<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhiraj Bhamare on Medium]]></title>
        <description><![CDATA[Stories by Dhiraj Bhamare on Medium]]></description>
        <link>https://medium.com/@dhiraj_db?source=rss-d6b85d33483f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZqMaU4v6GaA16C3qbAlxpQ.png</url>
            <title>Stories by Dhiraj Bhamare on Medium</title>
            <link>https://medium.com/@dhiraj_db?source=rss-d6b85d33483f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:52:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhiraj_db/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Migrating from Redshift Python UDFs to Lambda UDFs: What Changes, What Breaks, and What to Do About…]]></title>
            <link>https://medium.com/@dhiraj_db/migrating-from-redshift-python-udfs-to-lambda-udfs-what-changes-what-breaks-and-what-to-do-about-01db627d7656?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/01db627d7656</guid>
            <category><![CDATA[pyhton-udf]]></category>
            <category><![CDATA[lambda-udf]]></category>
            <category><![CDATA[lambda-function]]></category>
            <category><![CDATA[aws-redshift]]></category>
            <category><![CDATA[redshift]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Fri, 26 Jun 2026 12:25:43 GMT</pubDate>
            <atom:updated>2026-06-26T12:25:43.715Z</atom:updated>
            <content:encoded><![CDATA[<h3>Migrating from Redshift Python UDFs to Lambda UDFs: What Changes, What Breaks, and What to Do About It</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lys1d9QZ6tn3M70qwKK6wQ.png" /><figcaption>generated using Gemini</figcaption></figure><p>Amazon Redshift users who relied on Python UDFs are now being pushed toward Lambda UDFs as the long-term replacement. On paper, that sounds straightforward: move the same business logic into Lambda, expose it through an external function, and continue processing data. In practice, the migration can introduce a meaningful performance gap for high-volume workloads.</p><p>This post summarizes the technical reality behind that transition, what teams should expect when moving from Python UDFs to Lambda UDFs, where the main bottlenecks come from, and what the practical alternatives are when performance matters.</p><h3>The change: Python UDFs are reaching end of support</h3><p>Redshift Python UDFs have been a convenient way to run custom logic directly inside the Redshift execution engine. For many teams, they became the easiest path for string manipulation, custom parsing, enrichment logic, and other transformations that were awkward to express in SQL alone.</p><p>With Python UDFs reaching end of support, the supported replacement path is Lambda UDFs, implemented through Redshift external functions. AWS documents the Lambda UDF model here: <a href="https://docs.aws.amazon.com/redshift/latest/dg/udf-creating-a-lambda-sql-udf.html">Scalar Lambda UDFs</a> and <a href="https://docs.aws.amazon.com/redshift/latest/dg/user-defined-functions.html">User-defined functions in Amazon Redshift</a>.</p><p>That replacement is functionally valid, but it is not architecturally equivalent.</p><h3>Why the migration is not a like-for-like swap</h3><p>The key difference is execution model.</p><p>A Python UDF runs in-process inside the Redshift runtime. The data stays within the cluster execution path, and the function runs close to the data.</p><p>A Lambda UDF is remote. Redshift batches input rows, serializes them to JSON, sends the payload over the network to Lambda, waits for the function to execute, receives the response, deserializes it, and then continues query execution.</p><p>That means the migration is not simply “same code, new location.” It is a shift from in-engine execution to RPC-style execution.</p><p>For lightweight logic over large analytical workloads, that distinction matters a lot.</p><h3>What benchmark testing typically shows</h3><p>In migration testing, the function logic is often not the problem. Even a very simple text or string transformation can become noticeably slower after being moved from a Python UDF to a Lambda UDF.</p><p>The pattern tends to look like this:</p><pre>| Implementation                    | Typical outcome                                                     |<br>|-----------------------------------|---------------------------------------------------------------------|<br>| Python UDF                        | Fast baseline because execution remains inside Redshift             |<br>| Lambda UDF with low memory        | Significant slowdown, especially at scale                           |<br>| Lambda UDF with higher memory     | Material improvement, but usually still slower than Python UDF      |<br>| Lambda UDF with very high memory  | Diminishing returns once CPU is no longer the main bottleneck       |</pre><p>The two consistent lessons are straightforward.</p><p>First, the initial regression can be surprisingly large, especially for workloads that process large volumes of rows.</p><p>Second, increasing Lambda memory often helps a lot at first, but only up to a point. Once the CPU-side bottleneck is reduced, the remaining latency is usually structural.</p><h3>What causes the slowdown</h3><p>The root cause is usually not bad business logic. It is the overhead of the Lambda UDF model itself.</p><p>For large-scale analytical processing, the performance penalty comes from several layers:</p><ul><li>JSON serialization of each batch on the Redshift side</li><li>Network round-trip between Redshift and Lambda</li><li>Lambda invocation overhead</li><li>JSON parsing and response serialization inside Lambda</li><li>Deserialization of results back into Redshift</li><li>Repetition of that cycle across thousands of invocations</li></ul><p>In many real migrations, batching is already reasonably efficient. That means the problem is not necessarily undersized requests. The problem is that even with healthy batching, repeated remote invocations can accumulate enough overhead to dominate runtime.</p><h3>What was tested and ruled out</h3><p>All of the documented tuning levers available at the external function layer should be reviewed:</p><ul><li>STABLE volatility</li><li>MAX_BATCH_SIZE</li><li>MAX_BATCH_ROWS</li><li>RETRY_TIMEOUT</li><li>Input datatype sizing where applicable</li></ul><p>In practice, these settings can help ensure the function is configured correctly, but they do not fundamentally change the external execution model. If those parameters have already been tested and the function remains slow at scale, the issue is usually architectural rather than a missed tuning option.</p><p>AWS documentation and field experience both point to the same conclusion: there are only a small number of exposed controls for Lambda UDF behavior, and none of them eliminate the remote-call overhead.</p><h3>The mitigation that usually helps most</h3><p>The most effective immediate improvement is often increasing Lambda memory.</p><p>That recommendation matters because Lambda memory also increases available CPU. For simple but high-volume batch processing, more CPU can reduce time spent in parsing, execution, and serialization.</p><p>However, higher memory does not solve everything.</p><p>A common pattern is:</p><ul><li>Low memory makes the function clearly underpowered</li><li>Moving to a moderate memory tier delivers a strong improvement</li><li>Moving to very high memory delivers only modest additional gains</li></ul><p>That diminishing return is an important signal. It usually means the remaining bottleneck is no longer CPU. At that point, network and invocation overhead dominate.</p><p>In other words:</p><ul><li>Low Lambda memory can make the problem much worse</li><li>Higher Lambda memory can reduce the CPU component</li><li>Higher Lambda memory cannot remove the fundamental remote-execution overhead</li></ul><p>For many workloads, a mid-range memory setting becomes the best cost-performance compromise, but teams should validate that with their own benchmarks.</p><h3>The real constraint teams need to understand</h3><p>Lambda UDFs are not a drop-in performance replacement for Python UDFs when all of the following are true:</p><ul><li>The logic is invoked across large row volumes</li><li>The transformation is row-oriented and lightweight</li><li>The workload is latency-sensitive</li><li>The business requirement expects near in-engine performance</li></ul><p>That combination is exactly where Lambda UDFs are weakest.</p><p>The architecture is flexible and supported, but it is not optimized for high-volume, low-latency, per-row transformations that previously ran in-process inside Redshift.</p><p>This is the core migration constraint.</p><h3>Recommended alternatives</h3><p>When Lambda UDF overhead is unacceptable, the answer is usually not “find one more setting.” The better answer is to reduce or eliminate repeated external invocations.</p><p>The practical alternatives are:</p><h3>1. Rewrite the logic in native SQL where possible</h3><p>This is the best option when feasible. Native SQL avoids Lambda invocation overhead entirely and keeps execution inside Redshift.</p><p>Not every Python UDF has a clean SQL equivalent, but if the transformation can be expressed with built-in SQL functions, that should be the first choice.</p><h3>2. Precompute the transformation upstream</h3><p>If the transformation happens during ingestion or ETL, compute it before the data lands in its final Redshift table.</p><p>This is often the most effective pattern for transformations such as decoding, normalization, parsing, or enrichment that do not need to be recomputed at query time.</p><p>It turns repeated query-time overhead into a one-time pipeline cost.</p><h3>3. Use staging tables or materialized views</h3><p>If upstream precomputation is not possible, materialized views or staging tables can still reduce repeated Lambda UDF execution.</p><p>This does not remove the cost entirely, but it moves it to a controlled refresh step instead of paying the same penalty on every user-facing query.</p><h3>4. Redesign query patterns to minimize repeated UDF calls</h3><p>If the UDF must remain, use it as sparingly as possible.</p><p>Typical improvements include:</p><ul><li>Running the transformation once and storing the result</li><li>Avoiding repeated invocation across ad hoc reporting layers</li><li>Restricting the transformed row set earlier in the query plan</li><li>Separating heavy batch enrichment from interactive analytics paths</li></ul><h3>What teams should plan for before migrating</h3><p>If you are replacing Python UDFs with Lambda UDFs, plan the migration as a performance revalidation project, not just a syntax conversion project.</p><p>At minimum:</p><ul><li>Benchmark the old and new implementations on realistic peak data volumes</li><li>Increase Lambda memory early in testing so you do not misdiagnose CPU starvation as the only problem</li><li>Measure total invocation count, payload sizes, and runtime distribution</li><li>Confirm whether the workload is batch-oriented, interactive, or latency-sensitive</li><li>Decide whether the function should stay at query time or move upstream into ETL</li></ul><p>That framing changes the conversation from “How do we port this function?” to “What is the right execution model for this transformation now?”</p><h3>Final takeaway</h3><p>The end of Python UDF support in Redshift does not just require code migration. In some cases, it requires workload redesign.</p><p>Lambda UDFs are the supported alternative, but they introduce unavoidable remote-execution overhead that becomes visible at scale. Increasing Lambda memory can reduce part of the penalty, and for some batch workloads that may be enough. But when the remaining gap still matters, the durable solutions are architectural: native SQL, upstream precomputation, staging tables, materialized views, and less frequent invocation.</p><p>The most important thing teams can do is recognize the limitation early. If a Python UDF currently runs fast because it executes in-process close to the data, assume that a Lambda UDF version may behave differently until benchmarking proves otherwise.</p><h3>Sources</h3><ul><li><a href="https://docs.aws.amazon.com/redshift/latest/dg/udf-creating-a-lambda-sql-udf.html">AWS documentation: Scalar Lambda UDFs</a></li><li><a href="https://docs.aws.amazon.com/redshift/latest/dg/user-defined-functions.html">AWS documentation: User-defined functions in Amazon Redshift</a></li><li><a href="https://aws.amazon.com/blogs/big-data/amazon-redshift-python-user-defined-functions-will-reach-end-of-support-after-june-30-2026/">Amazon Redshift Python user-defined functions will reach end of support after June 30, 2026</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01db627d7656" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Aurora PostgreSQL Serverless in Seconds with Express Configuration]]></title>
            <link>https://medium.com/@dhiraj_db/create-aurora-postgresql-serverless-in-seconds-with-express-configuration-71e9f7617799?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/71e9f7617799</guid>
            <category><![CDATA[aurora-serverless]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aurora-postgresql]]></category>
            <category><![CDATA[aurora]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Fri, 29 May 2026 07:28:59 GMT</pubDate>
            <atom:updated>2026-05-29T07:28:59.699Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YMZ43dHU3q8_mKP5HXDJ9Q.png" /><figcaption>Generated from Gemini</figcaption></figure><p>Amazon Aurora PostgreSQL now offers a <strong>Create with express configuration</strong> experience that lets you provision a serverless database in just a few clicks. The goal is simple: reduce the time and configuration overhead needed to get a working Aurora PostgreSQL environment for development, testing, proof-of-concept work, and fast experimentation.</p><p>With this new flow, you can create an Aurora PostgreSQL Serverless database with preconfigured defaults and start querying it within seconds, instead of spending time on networking, authentication, and a long list of optional settings up front.</p><h3>What is express configuration?</h3><p>Express configuration is a streamlined Aurora PostgreSQL creation flow available via the AWS Management Console, AWS CLI, and API. It creates both the Aurora cluster and a serverless instance in a single operation, using sensible defaults that are designed to help you get started quickly.</p><p>In the console, the workflow is intentionally short:</p><ol><li>Open the Amazon RDS console.</li><li>Choose <strong>Create with express configuration</strong>.</li><li>Optionally adjust a small number of settings, such as the cluster identifier or capacity range.</li><li>Create the database.</li></ol><p>The same capability is available through the AWS CLI using the --with-express-configuration option:</p><pre>aws rds create-db-cluster \<br>  --db-cluster-identifier sample-express-cluster \<br>  --engine aurora-postgresql \<br>  --with-express-configuration</pre><p>This single command provisions a ready-to-use Aurora PostgreSQL Serverless cluster with the express configuration defaults.</p><h3>What gets configured automatically?</h3><p>When you use express configuration, AWS provisions more than just the database engine. It also sets up several foundational pieces that are normally required to connect and start using the database.</p><p>Express configuration automatically creates:</p><ul><li>An <strong>Aurora PostgreSQL-compatible cluster</strong>.</li><li>An <strong>Aurora Serverless instance</strong> in that cluster.</li><li>An <strong>internet access gateway</strong> for connectivity from common development environments.</li><li><strong>IAM database authentication</strong> for the administrator user by default.</li></ul><p>This is what makes the feature feel fast in practice. You are not required to decide on VPC design, subnets, security groups, VPN connectivity, or traditional password bootstrap steps before running your first queries.</p><h3>A key architectural point: no VPC association</h3><p>One of the most important aspects to understand is that the cluster is <strong>not</strong> created inside your default VPC or any user-selected VPC.</p><p>The design is explicit:</p><ul><li>Clusters created with express configuration are <strong>not associated with a VPC</strong>.</li><li><strong>DB subnet groups are not used</strong> in this mode.</li><li><strong>VPC security groups are not applicable</strong>.</li></ul><p>This explains why the express workflow never asks you to select a VPC. That is not a missing field in the console; it is part of the feature’s architecture.</p><p>If your requirement is to place the database into a specific VPC, subnet group, or private network path, you should use the full configuration flow instead of express configuration.</p><h3>How connectivity works</h3><p>Instead of placing the cluster in a customer VPC, Aurora express configuration uses a <strong>managed internet access gateway routing layer</strong> that is enabled by default.</p><p>This routing layer:</p><ul><li>Speaks the <strong>PostgreSQL wire protocol</strong>, so standard drivers and tools continue to work as expected.</li><li>Allows <strong>secure connections from tools running outside AWS</strong>, without requiring VPN or AWS Direct Connect.</li><li>Is <strong>distributed across multiple Availability Zones</strong> for high availability.</li><li>Is <strong>managed by Aurora</strong>, so there is no need to patch or operate it directly.</li></ul><p>This means you should not think of it as a “database in a public subnet with an internet gateway.” Instead, it is an Aurora-managed routing layer that sits outside your VPC, with its own connectivity model.</p><h3>IAM-only authentication</h3><p>Authentication is another major design decision in express configuration. Clusters created with this flow use <strong>IAM database authentication by default</strong>, and this cannot be disabled while the internet access gateway model is active.</p><p>In practical terms:</p><ul><li>IAM authentication is <strong>enabled automatically</strong>.</li><li>Traditional <strong>master password handling does not apply</strong> in this model.</li><li>The <strong>default master authentication type is IAM</strong> (iam-db-auth).</li><li>Your IAM principal must have rds-db:connect permissions to generate the token required for access.</li></ul><p>For tools that expect a username and password, AWS provides a <strong>Get token</strong> utility. In this pattern, the short-lived IAM authentication token is used as the password.</p><p>Because the token is valid for <strong>15 minutes</strong>, any tool that disconnects must obtain a new token before reconnecting.</p><h3>Querying the database quickly</h3><p>Once the database is created, the <strong>Connectivity &amp; security</strong> tab in the console exposes several connection options that are tailored to express configuration.</p><p>Common options include:</p><ul><li><strong>Code snippets</strong> for SDKs, APIs, and third-party tools.</li><li><strong>CloudShell</strong> for quick browser-based access with pre-populated commands.</li><li><strong>Endpoints + Get token</strong> for clients like pgAdmin that rely on user/password-style connections.</li></ul><p>AWS also dynamically generates language-specific examples, including Python, Node.js, JDBC, Go, PHP, psql, and TypeScript snippets. These examples are prewired with the correct endpoint and authentication model, which makes it easy to connect and run test queries with minimal setup.</p><h3>What you can change later</h3><p>Express configuration is opinionated at <strong>creation time</strong>, but it is not completely rigid afterward. Many settings can still be modified later through the standard Aurora “Modify” workflows.</p><p>Examples of settings you can change later include:</p><ul><li><strong>Auto minor version upgrade</strong> behavior.</li><li><strong>Storage configuration</strong>, such as Aurora Standard vs. I/O-Optimized.</li><li><strong>Parameter groups</strong> and related engine settings.</li><li><strong>Enhanced Monitoring</strong> and <strong>Performance Insights</strong>.</li><li><strong>Log exports</strong> and <strong>backup retention</strong>.</li><li>Adding <strong>reader instances</strong> for increased availability and read scaling.</li></ul><p>This makes express configuration a strong option for “start small, refine later” workflows, as long as your architectural requirements remain compatible with the express model.</p><h3>Important limitations to understand</h3><p>Because the feature is intentionally simplified, it comes with several limitations. Some are minor, but others are fundamental and can determine whether express configuration is appropriate for your workload.</p><h4>1. Aurora PostgreSQL only</h4><p>Express configuration currently supports <strong>Aurora PostgreSQL</strong>. Other engines, such as Aurora MySQL, are not included in this flow.</p><h4>2. No VPC placement</h4><p>You cannot:</p><ul><li>Associate an express configuration cluster with a <strong>VPC</strong>.</li><li>Choose <strong>DB subnet groups</strong>.</li><li>Apply <strong>VPC security groups</strong>.</li></ul><p>The cluster exists outside customer VPCs and uses the managed internet access gateway instead.</p><h4>3. Internet access gateway cannot be disabled</h4><p>Clusters created with express configuration have the internet access gateway <strong>enabled and fixed</strong>. You cannot turn it off or switch the cluster into a traditional VPC networking model later.</p><h4>4. IAM authentication only</h4><p>You cannot:</p><ul><li>Disable IAM authentication.</li><li>Use features tied to other authentication models, such as <strong>managed master credentials in Secrets Manager</strong> or <strong>Kerberos</strong>.</li></ul><p>The authentication model is IAM-only while using express configuration.</p><h4>5. No customer-managed KMS keys at creation</h4><p>Clusters are encrypted using service-owned keys (AWS-managed keys for RDS). At creation time, you <strong>cannot</strong> choose a custom <strong>AWS KMS key</strong> in this mode.</p><h4>6. No specific engine version selection at creation</h4><p>Express configuration uses the <strong>default Aurora PostgreSQL engine version</strong> for the region. You cannot select a specific engine version up front in this flow, although you can upgrade later using the standard modification procedures.</p><h4>7. Some Aurora and RDS features are not supported</h4><p>Because the cluster is not in a VPC, several features that require VPC integration are not supported, including:</p><ul><li>Aurora Global Database</li><li>Aurora Limitless Database</li><li>RDS Proxy</li><li>Aurora Zero-ETL integration</li><li>RDS Query Editor</li><li>Blue/Green Deployments</li><li>Database Activity Streams</li><li>Zero Downtime Patching</li><li>Babelfish</li></ul><p>If your workload depends on these features, the standard configuration path is the better choice.</p><h3>When express configuration makes sense</h3><p>Express configuration is a strong fit when <strong>speed and simplicity</strong> matter more than detailed control over infrastructure. Typical use cases include:</p><ul><li>Quick <strong>proofs of concept</strong> and experiments.</li><li><strong>MVPs</strong> and early-stage application prototypes.</li><li><strong>Developer experimentation</strong> with Aurora PostgreSQL features.</li><li>Rapidly <strong>testing connection logic</strong> and IAM authentication.</li><li>Learning Aurora PostgreSQL without building out full networking first.</li></ul><p>In these scenarios, the ability to provision and connect to a serverless database in seconds is often more valuable than VPC-level control or advanced integrations.</p><h3>When full configuration is the better choice</h3><p>The standard full configuration flow remains the better option when <strong>infrastructure control and enterprise requirements</strong> are the priority.</p><p>You should choose the full configuration flow if you need:</p><ul><li>A specific <strong>VPC</strong> and <strong>subnet groups</strong>.</li><li><strong>Private networking only</strong>, with no public internet exposure.</li><li><strong>Custom KMS keys</strong> for encryption.</li><li>Strict <strong>compliance or security controls</strong> tied to your network design.</li><li>Support for <strong>features that require VPC association</strong>, such as RDS Proxy, Aurora Global Database, or Zero-ETL integrations.</li></ul><p>A practical rule of thumb:</p><ul><li>Use <strong>express configuration</strong> when <strong>speed of setup</strong> is the primary goal.</li><li>Use <strong>full configuration</strong> when <strong>infrastructure and feature control</strong> are the priority.</li></ul><h3>In Summary</h3><p>Aurora PostgreSQL Express configuration is more than a UI shortcut. It represents a different onboarding model in which AWS preconfigures networking access and IAM-based authentication, so you can create and query a <strong>serverless database in seconds</strong>.</p><p>That simplicity is powerful, but it comes with trade-offs. The most important one is that the cluster is <strong>not deployed into your VPC</strong>, and several VPC-dependent features are unavailable in this mode.</p><p>If your goal is <strong>rapid onboarding, quick testing, or early-stage application development</strong>, express configuration is a compelling option. If your workload depends on <strong>private networking, custom infrastructure controls, or advanced Aurora features</strong>, the full configuration path remains the better fit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71e9f7617799" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Troubleshooting a Redshift DROP USER Failure Caused by Datashare Privileges]]></title>
            <link>https://medium.com/@dhiraj_db/troubleshooting-a-redshift-drop-user-failure-caused-by-datashare-privileges-ad3f786ca467?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad3f786ca467</guid>
            <category><![CDATA[amazon-redshift]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-redshift]]></category>
            <category><![CDATA[redshift]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Wed, 27 May 2026 10:09:25 GMT</pubDate>
            <atom:updated>2026-05-27T10:09:25.524Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*eNI-K7w7-osBvpdZNZrxHA.png" /><figcaption>Generated using Gemini</figcaption></figure><p>Amazon Redshift user cleanup is usually straightforward — until it is not. A particularly confusing scenario arises when a user appears to have no ownership of objects and no visible privileges, yet a DROP USER operation still fails with:</p><p>ERROR: user “&lt;user_name&gt;” cannot be dropped because the user has a privilege on some object</p><p>This article walks through a troubleshooting pattern for that situation and highlights a commonly overlooked root cause: datashare privileges.</p><h3>The Symptom</h3><p>Attempting to remove a user results in:</p><pre>DROP USER &quot;&lt;user_name&gt;&quot;;<br><br>Error:<br><br>ERROR: user &quot;&lt;user_name&gt;&quot; cannot be dropped because the user has a privilege on some object</pre><p>At face value, this suggests that the user still owns or has been granted access to some database object. However, standard checks may show no evidence of such dependencies.</p><h3>Initial Checks That Show Nothing</h3><p>A series of typical investigations may return empty results:</p><ul><li>Owned object queries show no objects tied to the user</li><li>Object-level ACL checks reveal no explicit grants</li><li>Schema-level privilege checks return no matches</li><li>Default ACL queries show no privileges associated with the user</li><li>SVV_RELATION_PRIVILEGES does not list the user</li><li>Searches for the user as a grantee in others’ default ACLs return nothing</li></ul><p>At this point, it is reasonable to assume something is being missed rather than concluding the system is inconsistent.</p><h3>The Missing Piece: Datashare Privileges</h3><p>The root cause in this scenario can be privileges on a Redshift datashare.</p><blockquote>Datashares are not covered by many of the standard catalog views used for privilege inspection. As a result, a user can retain access to a datashare without appearing in typical ACL or ownership queries.</blockquote><p>This hidden dependency is sufficient to block DROP USER.</p><h3>How to Identify the Issue</h3><p>To surface datashare-related privileges, query:</p><pre>SELECT *<br>FROM svv_datashares<br>WHERE share_acl LIKE &#39;%&lt;user_name&gt;%&#39;;</pre><p>This reveals whether the user is referenced in the datashare access control list.</p><p>For example, the output may show that the user still has permissions such as ALTER or SHARE on a datashare, even though no traditional object privileges exist.</p><h3>Resolving the Blocker</h3><p>To proceed with dropping the user, revoke datashare-level privileges:</p><pre>REVOKE ALTER ON DATASHARE &lt;datashare_name&gt; FROM &lt;user_name&gt;;<br>REVOKE SHARE ON DATASHARE &lt;datashare_name&gt; FROM &lt;user_name&gt;;</pre><p>A key nuance is that revoking only SHARE may not be sufficient. If the user still holds ALTER on the datashare, the DROP USER command will continue to fail.</p><p>Both privileges must be removed if present.</p><h3>Recommended Validation Before Changes</h3><p>Before modifying permissions, it is good practice to review the current datashare configuration:</p><p>Check datashare permissions:</p><pre>SELECT *<br>FROM svv_datashares<br>WHERE share_name = &#39;&lt;datashare_name&gt;&#39;;</pre><p>Check datashare consumers:</p><pre>SELECT share_name, consumer_account, consumer_namespace<br>FROM svv_datashare_consumers<br>WHERE share_name = &#39;&lt;datashare_name&gt;&#39;;</pre><p>This ensures that administrative access to the datashare is not unintentionally removed from all principals.</p><h3>Impact of Revoking Datashare Privileges</h3><p>Removing ALTER and SHARE privileges affects only administrative control, not data availability:</p><ul><li>The user can no longer modify the datashare</li><li>The user can no longer grant access to additional consumers</li><li>Existing consumers continue to access shared data without disruption</li><li>No data loss occurs as a result of these changes</li></ul><p>The primary consideration is ensuring that another authorized user or role retains management access.</p><h3>Practical Troubleshooting Flow</h3><p>When encountering a persistent DROP USER failure, expand the investigation beyond standard checks:</p><ul><li>Verify object ownership and standard privileges</li><li>Inspect default and relation-level privileges</li><li>Query SVV_DATASHARES for hidden dependencies</li><li>Revoke ALTER and SHARE privileges if present</li><li>Retry DROP USER</li></ul><h3>Final Thoughts</h3><p>This scenario highlights an important nuance in Redshift’s permission model: not all privileges are visible through commonly used system views.</p><p>Datashare permissions can quietly persist and block administrative operations, even when everything else appears clean. Including SVV_DATASHARES early in your troubleshooting workflow can significantly reduce time to resolution and prevent unnecessary confusion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad3f786ca467" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding AWS DMS “cdcPosition = null” During CDC Task Resume]]></title>
            <link>https://medium.com/@dhiraj_db/understanding-aws-dms-cdcposition-null-during-cdc-task-resume-bc4024129377?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc4024129377</guid>
            <category><![CDATA[dms]]></category>
            <category><![CDATA[aws-dms]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-dms-to-s3]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Wed, 27 May 2026 09:50:26 GMT</pubDate>
            <atom:updated>2026-05-27T09:50:26.707Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DdW0-8D4ycARnNkoEFJY7w.png" /><figcaption>Generated using Gemini</figcaption></figure><p>AWS Database Migration Service (DMS) logs can sometimes surface messages that appear more alarming than they actually are. One such example is the following log entry observed during CDC task resume:</p><p>“Execute Request Task ‘…’ running CDC with flags resume with cdcPosition null and stop_at null”</p><p>At first glance, this message suggests that DMS may not know where to resume replication from. Naturally, this raises concerns about skipped transactions, duplicate processing, or even data loss. However, this interpretation is not always correct.</p><p>This article explains what this log entry actually means, when it is harmless, and when it requires deeper investigation.</p><h3>Observed Behavior</h3><p>In a scenario involving MySQL as the source and Amazon S3 as the target, CDC tasks were restarted after a maintenance window. During resume, logs displayed:</p><ul><li>cdcPosition = null</li><li>stop_at = null</li></ul><p>Despite this, the tasks resumed successfully and continued processing changes from the expected point. Data generated during the downtime window was correctly replicated to the target after restart.</p><p>This confirmed that the replication process did not lose its position, even though the logs suggested otherwise.</p><h3>What “cdcPosition = null” Really Means</h3><p>The key insight is that this message can be a logging display issue rather than a reflection of the actual internal state.</p><p>In a graceful stop-and-resume scenario:</p><ul><li>DMS persists the CDC checkpoint internally when the task is stopped.</li><li>Upon resume, DMS retrieves and uses that checkpoint.</li><li>The log message may incorrectly display “null” instead of the real CDC position.</li></ul><p>In other words, the engine still knows where to resume from, even if the log does not show it.</p><h3>Why This Causes Confusion</h3><p>From an operator’s perspective, a null CDC position typically implies serious risk, such as:</p><ul><li>Restarting from an incorrect position</li><li>Missing source transactions</li><li>Duplicate event processing</li><li>Inability to locate the correct binlog position</li></ul><p>These are valid concerns, especially in production environments.</p><p>Additionally, if a task is interrupted abruptly (for example, during unplanned maintenance), DMS may only resume from the last successfully saved checkpoint. In such cases, a gap can exist between the last checkpoint and the interruption point.</p><p>This distinction is critical:</p><ul><li>Graceful stop + retained logs → Likely safe; log message may be cosmetic</li><li>Forced stop or missing logs → Potential risk; requires validation</li></ul><h3>When It Is Likely Harmless</h3><p>You can generally treat “cdcPosition = null” as non-critical if all of the following are true:</p><ul><li>The task was stopped gracefully and reached a STOPPED state</li><li>Source transaction logs were retained and remain accessible</li><li>The task resumed without errors</li><li>Target data reflects changes that occurred during the downtime</li><li>Validation checks show no discrepancies</li></ul><p>In this scenario, the log message is misleading, but replication integrity is intact.</p><h3>When to Investigate Further</h3><p>Further validation is required if the same log entry appears alongside:</p><ul><li>Missing or purged source logs (for example, MySQL binlogs)</li><li>Data gaps or inconsistencies in the target</li><li>Validation failures after restart</li><li>Abrupt stop/start events during maintenance or upgrades</li></ul><p>In these situations, the issue may not be cosmetic. It becomes essential to verify whether DMS resumed from the correct checkpoint and whether the required source logs were available.</p><h3>Best Practices for Reliable CDC Operations</h3><p>To reduce ambiguity and ensure safe CDC behavior, the following operational practices are recommended:</p><ol><li>Stop tasks gracefully before maintenance<br>Ensure tasks reach the STOPPED state to allow DMS to persist the latest checkpoint cleanly.</li><li>Capture the recovery checkpoint<br>Record the recovery checkpoint using AWS CLI before maintenance to provide a reference for validation.</li><li>Ensure sufficient source log retention<br>For MySQL, configure binlog retention to comfortably cover the entire maintenance window plus buffer time.</li><li>Verify source logs post-maintenance<br>Confirm that required logs still exist before resuming the task.</li><li>Validate using data, not just logs<br>Do not rely solely on log messages. Instead, verify replication integrity using:</li></ol><ul><li>Row count comparisons</li><li>Timestamp or sequence checks</li><li>DMS validation tasks</li><li>CloudWatch metrics such as CDCLatencySource and CDCLatencyTarget</li></ul><h3>Practical Takeaway</h3><p>The presence of “cdcPosition = null” in DMS logs during task resume does not automatically indicate a problem. In graceful restart scenarios, it can simply be a misleading log message while the service continues to use the correct internal checkpoint.</p><p>The appropriate response is not to assume failure, but to validate system behavior. If the task was stopped properly, source logs are intact, and data replication is consistent, the message is likely cosmetic.</p><h3>Final Thoughts</h3><p>For DMS operators, the key lesson is straightforward: do not rely on a single log line to assess replication health.</p><p>Instead, combine proper operational practices such as graceful shutdowns, checkpoint tracking, and log retention with post-resume validation. This approach ensures that CDC pipelines remain reliable, even when logs appear ambiguous.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc4024129377" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Cost‑Efficient Aurora Cluster: Provisioned Writer + Serverless v2 Reader Pattern]]></title>
            <link>https://medium.com/@dhiraj_db/building-a-cost-efficient-aurora-cluster-provisioned-writer-serverless-v2-reader-pattern-de5cd0e54bc7?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/de5cd0e54bc7</guid>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[aurora-serverless-v2]]></category>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[aws-cost-optimization]]></category>
            <category><![CDATA[aurora-postgresql]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 19:35:08 GMT</pubDate>
            <atom:updated>2026-03-19T19:35:08.012Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*H2393oU5UEeQd_Kgt0xsAg.png" /></figure><p>Balancing consistent performance with cost efficiency remains one of the toughest challenges in modern cloud database design. Many systems end up over-provisioned because infrastructure is built for peak load even if that peak only happens rarely.</p><p>A practical and increasingly popular solution in Amazon Aurora PostgreSQL is to <strong>combine a provisioned writer with an Aurora Serverless v2 reader</strong>. This hybrid setup anchors your database in stability while adding elasticity exactly where it delivers the most value.</p><h3>The Core Idea: Separate Stability from Elasticity</h3><p>Not all database workloads behave the same way — and treating them identically often leads to inefficiency.</p><ul><li><strong>Writes</strong> are steady, critical, and highly latency-sensitive.</li><li><strong>Reads</strong> are typically bursty, unpredictable, and highly scalable.</li></ul><p>By separating these concerns architecturally, you can:</p><ul><li>Guarantee predictable performance for transactional workloads.</li><li>Enable dynamic scaling for read-heavy operations.</li><li>Avoid paying for unused capacity during quiet periods.</li></ul><p>This model aligns infrastructure behavior with real workload patterns rather than theoretical peak demand.</p><h3>Why Keep the Writer Provisioned</h3><p>The writer instance is central to your system’s integrity — processing every insert, update, delete, and coordinating transactions and locks. Because performance fluctuations can cascade into user-facing issues, <strong>stability is non-negotiable</strong>.</p><p>A provisioned writer ensures:</p><ul><li>Consistent throughput under sustained load.</li><li>Predictable latency for writes and commits.</li><li>No dependency on scaling lag or warm-up time.</li></ul><p>Think of it as the foundation layer of your database — solid and stable, supporting everything built on top.</p><h3>Where Aurora Serverless v2 Shines</h3><p>Read workloads fluctuate constantly, driven by user activity, analytics queries, or scheduled background jobs. Aurora Serverless v2 was built for this variability.</p><p>Instead of sizing for the unknown, Serverless v2:</p><ul><li>Scales compute up automatically during high traffic.</li><li>Scales down when idle.</li><li>Adjusts in fine-grained increments (Aurora Capacity Units, or ACUs).</li></ul><p><strong>Common use cases include:</strong></p><ul><li>Reporting and analytics queries.</li><li>Dashboard or read-heavy workloads with sudden spikes.</li><li>Batch jobs or exploratory queries.</li></ul><p>This makes the Serverless reader a natural complement to a provisioned writer.</p><h3>Understanding Priority Tiers in Mixed Clusters</h3><p>A subtle but critical aspect of mixed configurations is the <strong>priority tier</strong> setting. It determines how the Serverless reader scales and impacts cost efficiency.</p><p><strong>Priority Tiers 0–1 (Coupled Behavior):</strong></p><ul><li>The Serverless reader scales in sync with the writer.</li><li>Aurora estimates a baseline capacity based on the writer instance.</li><li>This ensures faster failover but increases baseline cost.</li></ul><p><strong>Priority Tiers 2–15 (Independent Behavior):</strong></p><ul><li>The Serverless reader scales purely based on its workload.</li><li>It can scale down to the minimum ACU when idle.</li><li>This delivers maximum elasticity and cost efficiency.</li></ul><p>For most use cases, <strong>Tier 2–15</strong> is ideal, enabling independent scale-down during low demand.</p><h3>Component Behavior Overview</h3><pre>Component                   | Scaling    | Role<br>-----------------------------|------------|------------------------------------------------------------<br>Provisioned Writer           | Fixed      | Handles all transactional operations (writes, commits)<br>Provisioned Reader (optional)| Fixed      | Serves latency-sensitive read queries<br>Aurora Serverless v2 Reader  | Auto-scales| Ideal for variable or bursty read workloads; depends on tier</pre><h3>Recommended Configuration</h3><p>To achieve the best balance of stability and elasticity:</p><ol><li>Use a <strong>provisioned writer</strong> sized for your peak write workload.</li><li>Optionally, include a <strong>provisioned reader</strong> for consistently low-latency queries.</li><li>Configure your <strong>Serverless reader</strong> in <strong>priority tier 2–15</strong> for independent scaling.</li></ol><p>This strategy ensures:</p><ul><li>Stability for mission-critical operations.</li><li>Flexibility for unpredictable read demand.</li><li>Cost savings during off-peak periods.</li></ul><h3>Cost Optimization Benefits</h3><p>This hybrid pattern directly reduces waste from over-provisioning.</p><ul><li><strong>Traditional approach:</strong> Provision for peak → Pay for peak year-round.</li><li><strong>Mixed approach:</strong> Provision stability → Let everything else scale.</li></ul><p>The result:</p><ul><li>Lower costs during idle times.</li><li>Automatic spike handling.</li><li>Minimal manual tuning over time.</li></ul><p>For workloads with uneven read traffic, cost savings can be substantial.</p><h3>Traffic Routing Best Practices</h3><p>Route traffic based on workload criticality to maximize reliability and elasticity:</p><ul><li><strong>Writes →</strong> Provisioned writer.</li><li><strong>Critical reads →</strong> Provisioned reader (optional).</li><li><strong>Scalable or bursty reads →</strong> Serverless reader.</li></ul><p>This separation isolates real-time operations from scaling behavior, keeping critical paths always stable.</p><h3>When This Pattern Works Best</h3><p>This architectural model fits perfectly for:</p><ul><li>Applications with unpredictable or seasonal read traffic.</li><li>Analytics or reporting-heavy systems.</li><li>Platforms with time-of-day or event-driven usage spikes.</li><li>Teams focused on optimizing database spend without losing performance.</li></ul><h3>Final Thoughts</h3><p>Combining a <strong>provisioned writer</strong> with an <strong>Aurora Serverless v2 reader</strong> represents a balanced, forward-looking approach to database scaling. It embodies the shift from static provisioning to <strong>workload-aware resource allocation</strong>.</p><p>By letting the write layer stay stable while the read layer adapts dynamically, you create a database architecture that is resilient, efficient, and cost-effective — one that scales with your business instead of against it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de5cd0e54bc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Unexpected Aurora SnapshotExportToS3 Charges — Zero-ETL Integration]]></title>
            <link>https://medium.com/@dhiraj_db/understanding-unexpected-aurora-snapshotexporttos3-charges-zero-etl-integration-4810bed4e57b?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4810bed4e57b</guid>
            <category><![CDATA[aurora-mysql]]></category>
            <category><![CDATA[zero-etl-integrations]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[aurora-postgresql]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Thu, 12 Mar 2026 18:48:45 GMT</pubDate>
            <atom:updated>2026-03-12T18:48:45.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4C4Bir3TDJU9okXkhKB6tA.png" /><figcaption>Image generated using Gemini</figcaption></figure><p>Unexpected AWS charges can be difficult to trace, especially when the billing usage does not directly correspond to visible console operations.</p><p>One such scenario involves Aurora: SnapshotExportToS3 usage appearing in billing, even though no snapshot export tasks were manually executed.</p><p>This situation can occur when <strong>Zero-ETL integrations between Aurora and Redshift are modified</strong>, triggering internal synchronization processes that use snapshot export mechanisms behind the scenes.</p><p>This article walks through:</p><ul><li>How to identify the cost anomaly</li><li>How to investigate SnapshotExportToS3 usage</li><li>Why Zero-ETL integration modifications can generate these charges</li></ul><p>The goal is to provide a general investigation approach that others can use when diagnosing similar billing anomalies.</p><h3>Identifying the Cost Anomaly</h3><p>A common way to detect unusual AWS costs is through <strong>AWS Cost Anomaly Detection</strong>.</p><p>To identify anomalies related to RDS or Aurora:</p><ol><li>Navigate to the Cost Anomaly Detection console.</li><li>Locate the anomaly listed under Amazon Relational Database Service.</li><li>Click “View potential root cause(s)” to see a breakdown by account and usage type.</li><li>Click “View root cause” to examine the Cost ($) and Usage (Hrs) breakdown, along with the actual date the charge was incurred.<br> The date range can also be adjusted to analyze usage patterns over a longer period.</li></ol><p>When investigating anomalies related to Aurora, a usage type that may appear is:</p><p><strong>Aurora: SnapshotExportToS3</strong></p><p>This usually indicates that an Aurora snapshot was exported to Amazon S3. However, there are cases where <strong>this usage appears even when no export task was manually initiated</strong>.</p><h3>Investigating Aurora: SnapshotExportToS3 Usage</h3><p>When this usage type appears, several checks should be performed to determine the source.</p><h4>1. Check RDS Export Tasks</h4><p>Start by verifying whether any snapshot export tasks exist.</p><p>This can be done in the <strong>RDS console under Exports in Amazon S3</strong>, or via the AWS CLI:</p><pre>aws rds describe-export-tasks --region &lt;region&gt;</pre><p>If a snapshot export was triggered manually or through automation, it should appear in this list.</p><h4>2. Review CloudTrail Logs</h4><p>Next, review logs in AWS CloudTrail for export-related API calls.</p><p>Look for events such as:</p><ul><li>StartExportTask</li><li>CancelExportTask</li></ul><p>These events indicate when snapshot exports are initiated or cancelled.</p><p>If no such events exist, it suggests the export may not have been triggered through standard user or automation workflows.</p><h4>3. Check Backup Services or Automation</h4><p>Exports may also be initiated indirectly by automation or backup services.</p><p>Possible sources include:</p><ul><li>AWS Backup</li><li>AWS Systems Manager automation</li><li>Scheduled Lambda functions</li><li>Third-party backup or data pipeline tools</li><li>Cross-account automation</li></ul><p>If none of these mechanisms initiated the export, it is possible that the operation was triggered internally by another AWS service.</p><h3>Understanding Aurora Zero-ETL Integrations</h3><p>A common architecture for analytics pipelines uses <strong>Zero-ETL integration between Aurora and Redshift</strong>, enabling near-real-time replication of operational data into a data warehouse.</p><p>This integration typically involves:</p><ul><li>Amazon RDS/ Aurora as the source database</li><li>Amazon Redshift as the analytics destination</li></ul><p>According to the Aurora pricing documentation:</p><blockquote><em>While AWS does not charge additional fees for the Zero-ETL integration feature itself, customers are billed for the underlying resources used during the data synchronization process.</em></blockquote><p>These resources may include:</p><ul><li>compute resources</li><li>storage</li><li><strong>snapshot export costs used for the initial data load</strong></li></ul><p>This behavior becomes particularly relevant when integrations are modified.</p><h3>How Integration Modifications Can Trigger Snapshot Export Usage</h3><p>Zero-ETL integrations allow configuration changes such as:</p><ul><li>adding or modifying <strong>data filters</strong></li><li>adjusting table inclusion rules</li><li>modifying synchronization scope</li></ul><p>For example, a filter might exclude specific tables:</p><pre>&quot;dataFilter&quot;: &quot;exclude: &lt;Table_Name&gt;./_new/&quot;</pre><p>When such a filter is applied to an existing integration, the system must re-evaluate the integration configuration.</p><p>According to AWS documentation for modifying Zero-ETL integrations:</p><blockquote><em>If a data filter is added to an existing integration, Amazon RDS reevaluates the filter as if it always existed and removes data in the target warehouse that does not match the new criteria.</em></blockquote><p>To enforce the new filtering rules, AWS performs data re-synchronization, which may include:</p><ul><li>removing previously synchronized data</li><li>re-exporting the filtered dataset</li><li>reloading the correct data into the target warehouse</li></ul><p>This re-synchronization process may internally use the same snapshot export mechanism used during the initial integration setup.</p><p>As a result, <strong>SnapshotExportToS3 usage may appear in billing</strong>, even though no explicit export task was initiated.</p><h3>Why No Export Tasks May Appear</h3><p>A key point that often causes confusion is the absence of visible export tasks.</p><p>Manual snapshot exports normally produce:</p><ul><li>an export task visible in the console</li><li>a StartExportTask API event</li><li>export status tracking</li></ul><p>However, when the operation is triggered internally by a managed service workflow (such as Zero-ETL synchronization):</p><ul><li>the snapshot export may <strong>not create a visible export task</strong></li><li>the <strong>StartExportTask API event may not appear</strong></li><li>the export occurs <strong>as part of an internal service workflow</strong></li></ul><p>This can lead to <strong>SnapshotExportToS3 charges appearing without any obvious export activity</strong>.</p><h3>Key Takeaways</h3><p>Several important insights emerge from this behavior.</p><h4>Zero-ETL integrations themselves are not billed</h4><p>The integration feature does not carry a direct charge, but the <strong>resources used to process and synchronize data are billed</strong>.</p><h4>Configuration changes may trigger large internal operations</h4><p>Modifications such as data filters can trigger:</p><ul><li>dataset reevaluation</li><li>resynchronization</li><li>snapshot export operations</li></ul><h4>Internal service workflows may not appear in export task lists</h4><p>Some AWS-managed operations may not generate visible export tasks, even though they still produce <strong>billable usage</strong>.</p><h3>Best Practices When Using Zero-ETL Integrations</h3><p>To minimize unexpected costs:</p><ul><li>Plan <strong>data filter changes carefully</strong>, especially for large datasets.</li><li>Monitor <strong>AWS Cost Explorer</strong> after integration modifications.</li><li>Combine <strong>CloudTrail analysis with cost breakdowns</strong> when investigating anomalies.</li><li>Treat integration modifications similarly to <strong>initial data loads</strong> from a cost perspective.</li></ul><h3>Conclusion</h3><p>Zero-ETL integrations between Amazon Aurora and Amazon Redshift simplify analytics architectures by eliminating complex ETL pipelines.</p><p>However, certain configuration changes — such as modifying data filters — can trigger <strong>internal data re-synchronization workflows</strong>. These workflows may use <strong>snapshot export mechanisms</strong>, resulting in <strong>Aurora: SnapshotExportToS3 billing usage even when no explicit export task exists</strong>.</p><p>Understanding this behavior can help engineers and cloud administrators <strong>more effectively investigate cost anomalies and avoid unexpected billing surprises</strong>.</p><h3>References</h3><ol><li><strong>Amazon Aurora Pricing</strong><br> Explains that while Zero-ETL integration itself has no additional charge, customers are billed for the underlying resources used during data synchronization, including snapshot exports used for the initial data load.<br> <a href="https://aws.amazon.com/rds/aurora/pricing/">https://aws.amazon.com/rds/aurora/pricing/</a></li><li><strong>Modifying Zero-ETL Integrations in Amazon Aurora</strong><br>AWS documentation describes how configuration changes (such as adding data filters) trigger reevaluation of the integration and may remove or resynchronize data in the target warehouse.<br> <a href="https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/zero-etl.modifying.html">https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/zero-etl.modifying.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4810bed4e57b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When an Aurora PostgreSQL Major Upgrade Fails: A Lesson from a Hidden View]]></title>
            <link>https://medium.com/@dhiraj_db/when-an-aurora-postgresql-major-upgrade-fails-a-lesson-from-a-hidden-view-27cda5842bbe?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/27cda5842bbe</guid>
            <category><![CDATA[aws-database]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[aurora-postgresql]]></category>
            <category><![CDATA[aurora]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Wed, 21 Jan 2026 15:27:54 GMT</pubDate>
            <atom:updated>2026-01-21T15:27:54.265Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdGZbaejJ3p3LTOSdIldMw.png" /></figure><p>Recently, I attempted an in-place major version upgrade of an Amazon Aurora PostgreSQL cluster from 15.x to 17.x.<br>The upgrade initially looked healthy. Initial checks passed, the temporary PostgreSQL environment started successfully, and metadata initialization completed.</p><p>Then it failed.</p><p>This post documents <strong>what went wrong, why it happened, and how to prevent it </strong>especially if you’re planning PostgreSQL 15 → 16/17 upgrades on Aurora or self-managed clusters.</p><h3>The Error That Stopped Everything</h3><p>During the data migration phase, the upgrade failed with the following error:</p><pre>pg_restore: error: could not execute query: <br>ERROR: column reference &quot;query_id&quot; is ambiguous</pre><p>From the logs:</p><pre>pg_restore: creating VIEW &quot;public.pg_stat_activityenric&quot;<br>pg_restore: while PROCESSING TOC:<br>pg_restore: from TOC entry 996; 1259 7344676 VIEW pg_stat_activityenric postgres<br>LINE 33: &quot;s&quot;.&quot;query_id&quot;,</pre><p>At first glance, this looks like a generic pg_restore failure.<br> In reality, it was a <strong>PostgreSQL version-compatibility issue hidden within a user-defined view</strong>.</p><h3>What Actually Happened During the Upgrade</h3><p>Aurora major version upgrades internally rely on <strong>pg_upgrade</strong>, which works roughly like this:</p><ol><li>Start a temporary PostgreSQL environment using the target version</li><li>Migrate system catalogs and metadata</li><li>Restore user objects (tables, views, functions, etc.) using pg_restore</li><li>Validate and finalize the upgraded cluster</li></ol><p>The failure occurred in <strong>step 3</strong>, while restoring a <strong>custom view</strong>:</p><pre>pg_stat_activityenric</pre><h3>Root Cause: Changes in PostgreSQL 16 Statistics System</h3><p>PostgreSQL 16 introduced changes to the <strong>Cumulative Statistics System</strong>.</p><p>Specifically:</p><ul><li>The internal function pg_stat_get_activity() returns <strong>more columns</strong> in PostgreSQL 16+ than it did in PostgreSQL 15</li><li>The view pg_stat_activityenric was defined with an <strong>explicit column list</strong> tailored for PostgreSQL 15</li><li>After the upgrade, the expanded column set caused <strong>column name ambiguity</strong>, particularly for query_id</li></ul><p>As a result:</p><blockquote><em>pg_restore could not recreate the view in the PostgreSQL 16/17 environment, causing the entire major version upgrade to terminate.</em></blockquote><h3>Why the Failure Kept Reappearing</h3><p>Even after removing the view from one database, the upgrade continued to fail.</p><p>Why?</p><p><strong>Major version upgrades process each database in the cluster independently.</strong></p><p>Later logs showed:</p><pre>pg_restore: creating VIEW &quot;public_isolated.pg_stat_activityenric&quot;<br>pg_restore: from TOC entry 667; 1259 7631429 VIEW pg_stat_activityenric postgres<br>ERROR: column reference &quot;query_id&quot; is ambiguous</pre><p>This confirmed that the same view still existed in another database/schema, and that pg_upgrade was hitting the same incompatible definition again.</p><h3>How to Identify the Problem Across the Cluster</h3><p>To ensure the view is completely removed, it must be checked <strong>in every database</strong>:</p><pre>SELECT *<br>FROM pg_catalog.pg_views<br>WHERE viewname = &#39;pg_stat_activityenric&#39;;</pre><p>Even <strong>one remaining instance</strong> of the view anywhere in the cluster will cause the upgrade to fail.</p><h3>The Fix (and Why It’s Mandatory)</h3><p>To successfully complete the major version upgrade:</p><h3>Before the Upgrade (PostgreSQL 15.x)</h3><ol><li>Drop the incompatible view from every database</li><li>Verify its removal using the catalog query above</li><li>Optionally scale the instance class up (major upgrades are memory-intensive)</li></ol><h3>Retry the Upgrade</h3><ul><li>Upgrade from PostgreSQL <strong>15.15 → 16.11 or later</strong> (or 17.x)</li><li>The upgrade should proceed past the data migration phase successfully</li></ul><h3>After the Upgrade</h3><ul><li>Recreate the view using a <strong>PostgreSQL 16+ compatible definition</strong></li><li>Avoid explicit column lists tied to internal functions unless absolutely necessary</li></ul><h3>Final Thoughts</h3><p>This issue wasn’t caused by Aurora, pg_upgrade, or resource constraints it was a <strong>perfectly valid PostgreSQL 15 view that became invalid in PostgreSQL 16</strong>.</p><p>If you’re planning a PostgreSQL 15 → 16 or 17 upgrade, <strong>audit your custom monitoring views first</strong>. It can save hours of failed upgrade attempts and head-scratching log analysis.</p><h3>Reference:</h3><p>PostgreSQL 16 Monitoring Stats Documentation- <a href="https://www.postgresql.org/docs/16/monitoring-stats.html">https://www.postgresql.org/docs/16/monitoring-stats.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27cda5842bbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Have You Noticed the New “Upgrade Rollout Order” in the RDS Console?]]></title>
            <link>https://medium.com/@dhiraj_db/have-you-noticed-the-new-upgrade-rollout-order-in-the-rds-console-d6a3b0c96ac5?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6a3b0c96ac5</guid>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[aws-aurora]]></category>
            <category><![CDATA[rds]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Tue, 25 Nov 2025 16:22:38 GMT</pubDate>
            <atom:updated>2025-11-25T16:22:38.792Z</atom:updated>
            <content:encoded><![CDATA[<p>If you work with Amazon RDS or Aurora, you may have spotted a new option in the console called <strong>Upgrade rollout order</strong>. It showed up quietly, but it solves a real problem: keeping automatic minor version upgrades predictable across all your environments.</p><p>This feature was introduced on <strong>November 21, 2025</strong>, which makes it one of the newest additions to the RDS platform.</p><p><a href="https://aws.amazon.com/about-aws/whats-new/2025/11/aws-organizations-upgrade-rollout-policy-amazon-aurora-rds/">https://aws.amazon.com/about-aws/whats-new/2025/11/aws-organizations-upgrade-rollout-policy-amazon-aurora-rds/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJtjoFNx0FwExzMzcAB8Lg.png" /></figure><h3>What Is Upgrade Rollout Order?</h3><p>Upgrade rollout order is part of the <strong>AWS Organizations upgrade rollout policy</strong>. It controls how automatic minor version upgrades move through your environments. Instead of everything upgrading at once, AWS now lets you group your databases into three waves:</p><ul><li><strong>first</strong> — usually development or test</li><li><strong>second</strong> — often QA</li><li><strong>last</strong> — production<br> <em>(If nothing is configured, resources default to “second”.)</em></li></ul><p>This turns automatic upgrades into a phased rollout. Lower risk environments upgrade first. You watch how they behave. If everything looks good, the next wave follows. Production comes last.</p><h3>How the Rollout Works</h3><p>When a new minor version becomes eligible:</p><ol><li>Databases in the <strong>first</strong> group become eligible during their maintenance windows.</li><li>AWS waits through a <strong>bake time</strong> to let you observe how things perform.</li><li>The <strong>second</strong> group becomes eligible.</li><li>Another bake time follows.</li><li>Finally, the <strong>last</strong> group becomes eligible.</li></ol><p>This creates a natural safety buffer between phases, giving you time to spot issues before they reach production.</p><h3>Supported Services</h3><p>Upgrade rollout order works with:</p><ul><li>Amazon Aurora MySQL-Compatible</li><li>Amazon Aurora PostgreSQL-Compatible</li><li>Amazon RDS for MySQL</li><li>Amazon RDS for PostgreSQL</li><li>Amazon RDS for MariaDB</li><li>Amazon RDS for SQL Server</li><li>Amazon RDS for Oracle<br> <em>(Supports automatic minor upgrades for engine versions released after January 2026)</em></li><li>Amazon RDS for Db2</li></ul><p>This is broad coverage, especially helpful for teams running mixed engine types.</p><h3>Ways to Manage the Feature</h3><p>You can manage upgrade rollout policies through:</p><ul><li>AWS Management Console</li><li>AWS CLI</li><li>AWS SDKs</li><li>AWS CloudFormation</li><li>AWS CDK</li></ul><h3>How To Change the Upgrade Rollout Order for RDS Instances</h3><p>If you want to adjust which group a database belongs to, here’s the simple workflow.</p><h3>1. Create or Update an Upgrade Rollout Policy</h3><p>In AWS Organizations, you define a policy that sets the default order and any tag-based rules.</p><p>A typical setup uses an environment tag, for example:</p><pre>{<br>  &quot;upgrade_rollout&quot;: {<br>    &quot;default&quot;: {<br>      &quot;patch_order&quot;: { &quot;@@assign&quot;: &quot;last&quot; }<br>    },<br>    &quot;tags&quot;: {<br>      &quot;Environment&quot;: {<br>        &quot;tag_values&quot;: {<br>          &quot;Development&quot;: { &quot;patch_order&quot;: { &quot;@@assign&quot;: &quot;first&quot; } },<br>          &quot;QA&quot;:         { &quot;patch_order&quot;: { &quot;@@assign&quot;: &quot;second&quot; } },<br>          &quot;Production&quot;: { &quot;patch_order&quot;: { &quot;@@assign&quot;: &quot;last&quot; } }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>Attach the policy to the organization units or accounts that hold your RDS resources.</p><h3>2. Tag Your RDS Instances</h3><p>AWS decides the rollout order based on tags. If your policy uses Environment, tag each instance or cluster accordingly.</p><p><strong>Console:</strong></p><ol><li>Open the RDS console</li><li>Choose <strong>Databases</strong></li><li>Select your DB instance</li><li>Go to <strong>Manage tags</strong></li><li>Add or edit tags (for example, Environment=Development)</li></ol><p><strong>CLI:</strong></p><pre>aws rds add-tags-to-resource \<br>  --resource-name arn:aws:rds:&lt;region&gt;:&lt;account&gt;:db:&lt;instance&gt; \<br>  --tags Key=Environment,Value=Development</pre><h3>3. Wait for Propagation</h3><p>Tag changes and policy updates can take some time to propagate throughout the system. Once they do, AWS will apply the correct rollout order to each resource.</p><h3>4. Changing the Order Later</h3><p>If you need to move a database to another wave, simply update its tag to match the new group and ensure that your policy maps the tag value to the correct order.</p><h3>Why This Feature Matters</h3><p>Automatic upgrades are great in theory, but risky when everything updates at once. This rollout model brings order to the process. You get:</p><ul><li>A safer upgrade path</li><li>Clear sequencing across environments</li><li>Less manual coordination</li><li>A structure that matches how most teams already release software</li></ul><p>Upgrade rollout order is a small feature with big impact. It gives you control without complexity and brings predictability to something that used to feel unpredictable. If you manage multiple RDS or Aurora environments, it’s worth enabling and aligning with your tagging strategy.</p><p>Reference :</p><p><a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_upgrade_rollout.html">https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_upgrade_rollout.html</a></p><p><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/RDS.Maintenance.AMVU.UpgradeRollout.html">https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/RDS.Maintenance.AMVU.UpgradeRollout.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6a3b0c96ac5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Impact of Reducing idle_in_transaction_session_timeout on RDS/Aurora PostgreSQL when using AWS DMS]]></title>
            <link>https://medium.com/@dhiraj_db/impact-of-reducing-idle-in-transaction-session-timeout-on-rds-aurora-postgresql-when-using-aws-dms-ae6302fa896f?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae6302fa896f</guid>
            <category><![CDATA[aws-dms]]></category>
            <category><![CDATA[aws-aurora]]></category>
            <category><![CDATA[aws-rds]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Mon, 27 Oct 2025 21:05:19 GMT</pubDate>
            <atom:updated>2025-10-27T21:05:44.358Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Goal: </strong>Reduce idle_in_transaction_session_timeout from the default (24 hours) to a shorter value. For example, 1 hour or less.</p><p><strong>Why:</strong> Long “idle in transaction” sessions hold locks and block VACUUM from cleaning up dead tuples. Over time, that causes <strong>table bloat</strong> and <strong>performance issues</strong>. A shorter timeout automatically kills those sessions and frees locks, which is a solid operational safeguard.</p><p>AWS DMS uses transactions that can appear idle for a while. If you lower this timeout globally, PostgreSQL may terminate DMS sessions mid-task, which can disrupt replication, interrupt CDC, and lead to WAL buildup.<br>AWS explicitly warns not to end idle transactions when using DMS but doesn’t give a clear solution for mixed workloads.</p><p>So you end up stuck between:</p><ol><li>Wanting shorter timeouts for regular app users (for performance and hygiene)</li></ol><p>2. But needing longer (or no) timeouts for DMS to avoid replication failures</p><h3>Background</h3><p>idle_in_transaction_session_timeout terminates any session that stays idle within an open transaction for longer than the set time. That rollback releases locks and helps prevent bloat. Almost every PostgreSQL tuning guide recommends setting a reasonable timeout here.</p><p>The problem is that <strong>AWS DMS</strong> holds open transactions while waiting for data changes. These can look idle to PostgreSQL even though they’re part of CDC replication. Killing those sessions causes DMS task failures and WAL growth.</p><p>AWS documentation warns against using this parameter for PostgreSQL sources used by DMS, but doesn’t provide a direct, role-scoped fix. You either keep the global timeout long for everyone (which hurts DB health) or risk breaking DMS.</p><p>AWS’s own documentation says:</p><blockquote><em>“Some AWS DMS transactions are idle for some time before the DMS engine uses them again. By using the parameter idle_in_transaction_session_timeout in PostgreSQL versions 9.6 and higher, you can cause idle transactions to time out and fail. Don’t end idle transactions when you use AWS DMS.”</em></blockquote><p>In short:</p><ul><li>They acknowledge the risk.</li><li>They warn against using the parameter globally.</li><li>But they don’t offer a clean way to <strong>exclude</strong> the DMS user from the timeout.</li></ul><p>That’s the gap this workaround fills.</p><h3>Practical Workaround</h3><p>Keep the shorter timeout at the cluster level (for all regular users), but <strong>exempt the DMS user</strong> by setting the timeout to 0 (disabled) at the role level.</p><p>That way:</p><ul><li>Regular users benefit from automatic session cleanup.</li><li>DMS keeps its long-lived transactions safe.</li></ul><pre>-- Run as a superuser or equivalent<br>ALTER USER your_dms_user SET idle_in_transaction_session_timeout = 0;</pre><p>Setting this to 0 disables the timeout for that user. You could also set a very high value instead, but 0 is simpler and effective.</p><p><strong>Why it works:</strong><br>PostgreSQL allows per-role config overrides. RDS and Aurora honor these role-level settings in addition to parameter group values. You get fine-grained control without compromising global behavior.</p><h3>Implementation Steps</h3><p><strong>1. Prepare</strong></p><ul><li>Make sure DMS uses a dedicated user (don’t reuse app users).</li><li>If not, create one and update your DMS endpoint credentials.</li></ul><p><strong>2. Set cluster-level timeout</strong></p><ul><li>In your parameter group, set:</li></ul><pre>idle_in_transaction_session_timeout = 3600000  # 1 hour</pre><ul><li>Apply and test on a non-production cluster first.</li></ul><p><strong>3. Exempt the DMS user</strong></p><pre>ALTER USER your_dms_user SET idle_in_transaction_session_timeout = 0;</pre><p><strong>4. Verify</strong></p><pre>SELECT usename, useconfig<br>FROM pg_user<br>WHERE usename = &#39;your_dms_user&#39;;</pre><p><strong>5. Test</strong><br>Run your DMS tasks (full load and CDC). Watch for timeout or disconnection errors in the task logs.</p><p><strong>6. Monitor</strong></p><ul><li>Monitor WAL size and replication lag.</li><li>Track DMS task status.</li><li>Confirm bloat reduction and improved VACUUM activity for app users.</li></ul><h3>Conclusion &amp; Recommendation</h3><p>Use a <strong>shorter </strong><strong>idle_in_transaction_session_timeout globally</strong> for your application workloads — but <strong>exempt your DMS user</strong> with:</p><pre>ALTER USER your_dms_user SET idle_in_transaction_session_timeout = 0;</pre><p>This gives you :</p><ul><li>Better performance and less bloat for your main workload</li><li>Stable, uninterrupted CDC replication for DMS</li></ul><p>AWS documentation warns about the issue but doesn’t describe this exclusion method. The role-level approach is a <strong>PostgreSQL-supported</strong> and <strong>RDS/Aurora-compatible</strong> workaround that solves the conflict cleanly.</p><p>We’ve tested it successfully in staging and production with good results.</p><h3>References</h3><ul><li><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.RDSPostgreSQL">AWS DMS — Using PostgreSQL as a Source</a></li><li><a href="https://www.postgresql.org/docs/current/runtime-config-client.html">PostgreSQL — idle_in_transaction_session_timeout</a></li><li><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html">AWS RDS Parameter Groups Documentation</a></li><li><a href="https://repost.aws/articles/ARlpbyXTvhQQy1dwsxu6czUw/troubleshooting-ongoing-idle-in-transaction-alerts-in-amazon-aurora-postgresql">Troubleshooting Ongoing “Idle In Transaction” Alerts in Amazon Aurora PostgreSQL</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae6302fa896f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linked Servers on Amazon RDS for SQL Server: What’s New, What’s Supported, and What’s Not]]></title>
            <link>https://medium.com/@dhiraj_db/linked-servers-on-amazon-rds-for-sql-server-whats-new-what-s-supported-and-what-s-not-af3f5845aaf1?source=rss-d6b85d33483f------2</link>
            <guid isPermaLink="false">https://medium.com/p/af3f5845aaf1</guid>
            <category><![CDATA[linked-server]]></category>
            <category><![CDATA[rds]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[aws-rds]]></category>
            <dc:creator><![CDATA[Dhiraj Bhamare]]></dc:creator>
            <pubDate>Tue, 22 Jul 2025 16:04:07 GMT</pubDate>
            <atom:updated>2025-07-22T16:04:07.971Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6LGoQ17xusP6TmQ9Za_zeA.png" /></figure><p>Amazon recently announced the addition of support for AWS RDS SQL Server with linked servers using Oracle OLEDB Driver version 21.16. This enhancement expands cross-database connectivity for enterprises that rely on heterogeneous environments, but it also raises important questions about what’s possible and what isn’t when it comes to linked servers on RDS SQL Server.</p><p>In this blog post, I will explain the update, clarify what’s officially supported, explain the limitations around drivers and the shared responsibility model, and unpack why some configurations like linking RDS SQL Server to MySQL are still off the table.</p><h3>What’s New: Oracle OLEDB Driver 21.16 Support</h3><p>As of <a href="https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-rds-sql-server-linked-servers-oracle-oledb-driver-version-21-16/">July 2025</a>, <strong>Amazon RDS for SQL Server now supports linked servers using Oracle OLEDB Driver version 21.16</strong>. This is significant for customers who need to query Oracle databases directly from their RDS-hosted SQL Server instances.</p><p>This update enables linked server configurations to Oracle databases (versions 21c, 19c, and 18c), allowing SQL Server users to run distributed queries, execute stored procedures, or join data across platforms using familiar T-SQL commands.</p><h3>What About Teradata and MySQL?</h3><h3>Teradata Support</h3><p>According to <a href="https://aws.amazon.com/blogs/database/configure-linked-servers-on-amazon-rds-custom-for-sql-server/">AWS documentation and troubleshooting references</a>, Teradata is also supported for linked servers via <strong>pre-configured ODBC drivers</strong>.</p><h3>MySQL Is Not Supported</h3><p>Let’s be clear: <strong>linked server connectivity from RDS SQL Server to MySQL is currently not supported</strong> on standard RDS instances.</p><p>Here’s why:</p><ul><li>Linked servers rely on database-specific drivers to function. SQL Server includes its own driver by default, but <strong>drivers for other databases like MySQL are not pre-installed</strong>.</li><li>On a self-managed SQL Server (EC2 or on-premise), you could manually install the MySQL ODBC or OLEDB driver and configure the linked server.</li><li>However, <strong>RDS for SQL Server is a managed service</strong>. You don’t get RDP or administrative access to the underlying host OS, so you can’t install third-party drivers yourself.</li></ul><p>This limitation ties directly into the <a href="https://aws.amazon.com/compliance/shared-responsibility-model/">AWS Shared Responsibility Model</a>. AWS manages the infrastructure, while customers manage the data and configuration within the bounds of what AWS allows. Installing system-level software like custom drivers isn’t in scope for customer responsibility under standard RDS offerings.</p><h3>Workaround: RDS Custom for SQL Server</h3><p>If you need a linked server to a database not supported in standard RDS (like MySQL), <strong>RDS Custom for SQL Server</strong> is your alternative.</p><p>RDS Custom gives you more control over the host operating system. You can:</p><ul><li>RDP into the server</li><li>Install custom drivers</li><li>Configure the SQL Server instance just like in a traditional EC2-based deployment.</li></ul><p>That flexibility enables linked server configurations to non-standard sources like MySQL, but with it comes more management responsibility. You now own patching, driver compatibility, system health, and database workloads.</p><p>For a walkthrough on configuring linked servers in RDS Custom, check out <a href="https://aws.amazon.com/blogs/database/configure-linked-servers-on-amazon-rds-custom-for-sql-server/">this AWS blog post</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af3f5845aaf1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>