<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by dikshant mali on Medium]]></title>
        <description><![CDATA[Stories by dikshant mali on Medium]]></description>
        <link>https://medium.com/@dikshantmali.dev?source=rss-7ee3d91ab4df------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rBZgIEmXMpC7yvdhNaKrgA.png</url>
            <title>Stories by dikshant mali on Medium</title>
            <link>https://medium.com/@dikshantmali.dev?source=rss-7ee3d91ab4df------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 06:35:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dikshantmali.dev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[IOS ReactNative App CI-CD: Code Magic Practicle Guide]]></title>
            <link>https://medium.com/@dikshantmali.dev/ios-reactnative-app-ci-cd-code-magic-practicle-guide-d3b0f696fc8f?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3b0f696fc8f</guid>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[codemagic]]></category>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 15:36:20 GMT</pubDate>
            <atom:updated>2025-06-25T15:36:20.589Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*8fnalWmKFQISazjrH0e-Pg.png" /></figure><p>Code-magic is a mobile CI/CD platform that provides an environment to build mobile applications. The document below describes the process to deploy a React-native CI/CD solution on Code-Magic.</p><p><strong>Features:</strong></p><ul><li>Code-magic offers 500 build minutes for free for getting started, then charges at the standard rates.</li><li>The platform offers a <strong>pay-as-you-go </strong>model, which means we are only charged for the build time we use. The standard rates are given below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*JI_GmAlYIDuL7lhx" /></figure><p>For more detailed billing, we can refer to:<a href="https://codemagic.io/pricing/"> https://codemagic.io/pricing/</a></p><p><strong>How to use CodeMagic?</strong></p><ol><li>Add IOS Certificates and Provisioning Profile on the code-magic saas(software as a service) platform.</li></ol><p>2. Go to Teams → personal account → code signing identities.</p><p><strong>Add certificate</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lpKZ5JMB45WXWbx5" /></figure><p><strong>Add Provisioning Profile</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ssIRvYDXu-iXyCO7" /></figure><p>2. Create a Code Magic workflow for your ReactNative iOS application that includes code signing and build steps for a successful build. The typical workflow that we have built for the successful build of IOS is given below.</p><pre># add provisioning profile and certificate in teams setting of codemagic account.<br>workflows:<br> ios-workflow:<br>   name: iOS Workflow For Dev<br>   max_build_duration: 120<br>   environment:<br>     xcode: 13.3<br>     node: 14.17.3<br>     vars:<br>       BUNDLE_ID: &quot;&lt;bundle-id&gt;&quot;<br>       XCODE_WORKSPACE: &quot;workspace.xcworkspace&quot; # &lt;-- Name of your Xcode workspace<br>       XCODE_SCHEME: &quot;workspace&quot; # &lt;-- Name of your Xcode scheme<br>     ios_signing:<br>       provisioning_profiles:<br>         - Provisioningprofile-LP<br>       certificates:<br>         - Certificate-LP<br>   scripts:<br>     - name: Set up code signing settings on Xcode project<br>       script: xcode-project use-profiles<br>     - name: Install npm dependencies<br>       script: |<br>         npm install  --save react-native --force<br>     - name:  Install CocoaPods dependencies<br>       script: |<br>         cd ios &amp;&amp; pod install<br>         pod update<br>     - name: Build ipa for distribution<br>       script: |<br>         xcode-project build-ipa \<br>         --workspace &quot;ios/$XCODE_WORKSPACE&quot; \<br>         --scheme &quot;$XCODE_SCHEME&quot; --archive-flags=&quot;-destination &#39;generic/platform=iOS&#39;&quot;<br>   artifacts:<br>     - build/ios/ipa/*.ipa<br>     - /tmp/xcodebuild_logs/*.log<br>     - $HOME/Library/Developer/Xcode/DerivedData/**/Build/**/*.app<br>     - $HOME/Library/Developer/Xcode/DerivedData/**/Build/**/*.dSYM<br>   publishing:<br>     email:<br>       recipients:<br>         - dikshant@gmail.com</pre><p>3. Put the <strong>codemagic.yaml</strong> file with the above sample workflow in the root of the repository branch.</p><p>4. On code magic, add the repository containing <strong>codemagic.yaml</strong> at the root and build the required branch from the codemagic console.</p><p>5. The email address added in the publishing section will receive status on failure and successful builds.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3b0f696fc8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynatrace: Infrastructure And Kubernetes Monitoring]]></title>
            <link>https://medium.com/@dikshantmali.dev/dynatrace-infrastructure-and-kubernetes-monitoring-b0876f081dd3?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0876f081dd3</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[infrastructure-monitoring]]></category>
            <category><![CDATA[dynatrace]]></category>
            <category><![CDATA[kubernetes-monitoring]]></category>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 15:35:21 GMT</pubDate>
            <atom:updated>2025-06-25T15:35:21.284Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes makes deploying and scaling apps easier, but monitoring it can be complex. Dynatrace simplifies this by automatically discovering your Kubernetes environment and providing real-time insights into nodes, pods, services, and performance issues. With its AI-powered engine, it helps detect problems early and reduces noise. In this article, we’ll explore how Dynatrace makes Kubernetes infrastructure monitoring easier, smarter, and more efficient.</p><p>We will be monitoring Linux servers and minikube Kubernetes through Dynatrace.</p><h3>Step 1: Register and log in to the Dynatrace Saas portal</h3><p><a href="https://www.dynatrace.com/trial/">https://www.dynatrace.com/trial/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G6sV2DTho6waH0CX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*jtxj_GcljeCb8e97Unznrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VOnV9zzaSL66GFvG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cadTWy1aYI8XXo31" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eynp2PvJ58NOgFG6" /></figure><h3>Monitor Linux Server</h3><h3>Step 2: Follow a wizard for Linux monitoring.</h3><p>Prerequisite: A running Linux VM anywhere (on cloud or VirtualBox)</p><ul><li>Install the agent in Linux</li><li>Start the agent in Linux.</li><li>Review metrics from Linux on Dynatrace</li></ul><h3>Step 2a: Go to -&gt; infrastructure -&gt; Hosts</h3><h3>Step 2b: Click on deploy Dynatrace</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MRU4ZuiraKUYOHvp" /></figure><h3>Step 2c: Click on Start Installation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZPMD6Q7ma6O9DkSd" /></figure><h3>Step 2d: Choose Host as Linux(You can choose Windows or other OS platforms if you want )</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yrRNLYwOW643DZUk" /></figure><h3>Step 2e: Click on create token</h3><h3>Step 2f: Install the one agent</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*cKyvUqUJHnRWwmD6nOCkTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*fPG-nxAXqUpZEEOOSvJw-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v_7rZx1zManSijva" /></figure><p>Now, view the deployment by clicking Show deployment status.</p><p>We can now see that a VM host has been discovered on the Dynatrace dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wkwV7qRsSK7i5jCh" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qXisY7QYXO3OjQce" /></figure><p>We can now explore the CPU, memory, and disk usage report, etc., from the metrics received from the OneAgent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*513OiBLnAVuTKzg-" /></figure><p>Process monitoring</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XyJDrbeADUvZxmVu" /></figure><h3>Monitor Kubernetes(Minikube Running On Local Machine)</h3><h3>Step3a: Go to -&gt; infrastructure -&gt; Kubernetes</h3><p>Click on Connect automatically via Dynatrace Operator</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ySzZ0_Ul1pLwRpry" /></figure><p>Fill in the required details</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uRROs364jBQU_wrw" /></figure><p>Note: Kindly download the dynakube.yaml</p><p>Output of the above commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*y9KtNgkM7ZmLRAYiNc5mOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Jg_7OToFD1xN3MBS6jm36g.png" /></figure><p>Now apply the dynakube.yaml we have downloaded.</p><pre>$ kubectl apply -f dynakube.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*RWJSvK18BNZf3qIkpg-f9g.png" /></figure><p>Now, click on -&gt; Show deployment status in Dynatrace UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HgLvAyf0sWynZlww" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2wgX-445yoMqztsC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UIAwu45-pN32ySPG" /></figure><p>Many more metrics we can explore from the dashboard regarding nodes, pods, services, etc.</p><p>Done!!!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0876f081dd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demo: Migrating MySQL from RDS To CloudSQL]]></title>
            <link>https://medium.com/@dikshantmali.dev/demo-migrating-mysql-from-rds-to-cloudsql-387c74e02966?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/387c74e02966</guid>
            <category><![CDATA[mysql-migration]]></category>
            <category><![CDATA[google-cloud-sql]]></category>
            <category><![CDATA[cloud-dms]]></category>
            <category><![CDATA[aws-rds]]></category>
            <category><![CDATA[database-migration]]></category>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Tue, 24 Jun 2025 15:27:41 GMT</pubDate>
            <atom:updated>2025-06-24T18:17:03.045Z</atom:updated>
            <content:encoded><![CDATA[<p>Google Cloud Platform <a href="https://cloud.google.com/database-migration/docs/release-notes">Database Migration Service</a>(DMS) makes it easier for us to “lift and shift” (Rehost) our MySQL and PostgreSQL workloads into Cloud SQL for MySQL and Cloud SQL for PostgreSQL, respectively.</p><p><strong>Prerequisite:</strong></p><p>Existing AWS RDS for MySQL with Public accessibility in any availability zone of your choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*_kwhljapWX8V02DvPcAXAg.png" /></figure><h3>Step 1: Enable DMS Service api</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gSOOMiDGozVebv-P" /></figure><h3>Step 2: Create a connection profile</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*WfRRU2YdiZtJI8JLCl6_jg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z0xGPIrTFBtAd42R" /></figure><h3>Step 3: Now let&#39;s create a<a href="https://console.cloud.google.com/dbmigration/migrations/create"> </a>migration job using the Migration Job wizard with the Migration job type set to One-time.</h3><p><strong>Note:</strong></p><p><strong>One-time =&gt;</strong> A point-in-time snapshot of the source, migrated to your destination as a full dump and load.</p><p><strong>Continuous =&gt; </strong>Real-time synchronization of the source and destination, capturing ongoing changes via continuous replication of data (CDC)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gQLKyxHlRqsWm72C" /></figure><h3>Step 4: Define the source using the Connection Profile created in step 2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*cqvIgP5Q-ERQoWgO_lGpkQ.png" /></figure><h3>Step 5: Create a destination, make sure you use the same MySQL version that was provisioned in AWS RDS.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rXcRHhPJmlWk3AoT" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H9NfNf6GMx0nAnbl" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*QbVdq8YBF6idQAU2-3O9fQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_xRuYDhW2hPXk8XV" /></figure><p>Choose =&gt; Create a destination and continue.</p><h3>Step 6: You can go to the Cloud SQL page to view the destination Cloud SQL for the MySQL instance provisioned.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*KLAczVqZvrpm8zjjhUAXSA.png" /></figure><h3>Step 7: Define the connectivity method as an IP allowlist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*q_oGoJNeIfsdUP820DsIOg.png" /></figure><p>Now, copy the Destination outgoing IP address and go to the AWS Security Group page and whitelist it for the 3306 port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JKwRL2YWvZP3X0k4" /></figure><p>Click Save rules.</p><h3>Step 8: The final step is to test and create a Migration Job by clicking on the Create Job button.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jp3epEazYp-a_55o" /></figure><h3>Step 9: Now, click on the start button in the corresponding migration job.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2S6zQuGd5K03-AJq" /></figure><h3>Step 10: Verify by connecting to your cloud instance if the data is propagated.</h3><p>You can view the information either from the CLI or the GUI.</p><pre>$ mysql -h 34.132.52.249 -u root -p</pre><p>Enter the password on the terminal prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*QiLDk2jhi6H6_ez7_oZ38A.png" /></figure><p>Verify the database and other details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hPkf40D5fpMOC36E" /></figure><p><strong>Done!!!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=387c74e02966" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ AWS: How to Provide Cross-Account Access]]></title>
            <link>https://medium.com/@dikshantmali.dev/aws-how-to-provide-cross-account-access-7674805ac008?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/7674805ac008</guid>
            <category><![CDATA[aws-cross-account]]></category>
            <category><![CDATA[cross-account-access]]></category>
            <category><![CDATA[aws-iam-best-practices]]></category>
            <category><![CDATA[aws-iam-security]]></category>
            <category><![CDATA[aws-iam]]></category>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Tue, 24 Jun 2025 15:27:13 GMT</pubDate>
            <atom:updated>2025-06-24T15:27:13.974Z</atom:updated>
            <content:encoded><![CDATA[<p>In multi-account AWS environments, securely sharing resources across accounts is a common need, whether for centralized logging, DevOps automation, or team collaboration. AWS provides powerful mechanisms like <strong>IAM roles with trust policies</strong> to enable cross-account access without compromising security. In this article, we’ll explore how to set up cross-account access in AWS, including role assumptions, permission boundaries, and best practices to ensure secure and controlled interactions between AWS accounts.</p><p><strong>Prerequisite:</strong></p><ul><li>2 AWS accounts</li></ul><p><strong>Use Case:</strong> We will provide temp-user(account 1) access to another account(account 2).</p><p><strong>Step 1:</strong> Create a user in <strong>account 1 </strong>with suitable permissions to the same account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OgIUB4N2KwUBnv2T" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*URhNMmHT8VadigDM" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PXML63bvX83H2u2K" /></figure><p><strong>Step 2: </strong>Now, create an inline policy for this user so that it can assume a cross-account role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HvZXrkbDe4s65v3G" /></figure><p><strong>Step 2a: </strong>Now, create a role in another account(account 2) and add the ARN of that role in the resource tab. This will allow this user to assume the cross-account role.</p><p><strong>NOTE:</strong> The cross-account role must also trust this user entity(configure in the next step).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X3J784fEor9KqLmB" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ePTQteNHXMWITsf" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*i2_4y7rksWZGwdS1OgyGnA.png" /></figure><p>Now assign permission to this role(we are assigning S3 read access for testing).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*phRg9Rog72_25qP4" /></figure><p>Now, after giving a suitable name, create a role.</p><p><strong>Step 3: </strong>Now, add this role to the STS resource for account 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*d1ETuvkV0eJhygqtIG6F5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9xLZuZNSPZxZgkZy" /></figure><p>Now create the policy.</p><p><strong>Gr8!!!</strong></p><p><strong>Step 4: </strong>Now sign in through the user of account1(temp-user) and access the second account as shown below.</p><p>Click on<strong> the switch role.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*AmvPXMuDwjz5fu022Ers9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*ZdgmtStcdec669Z6WdjPfA.png" /></figure><p>Now, switch roles and explore other accounts.</p><p><strong>Additional feature: </strong>By default, this role in account 2 trusts all the users in account 1, as shown in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*PhTmMU_umvztPv0qrQ1UiA.png" /></figure><p>The principal:</p><p><em>“AWS”: “arn:aws:iam::&lt;account-id&gt;:root” </em>trusts all users to assume this role. To allow a specific user from account 1 to assume this role. We should add only the ARN of a specific user.</p><p>Like:</p><blockquote>“AWS”: “arn:aws:iam::&lt;account-id&gt;:user/temp-user” principal will only allow this user to assume the role.</blockquote><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7674805ac008" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Migrating from AWS S3 to Google Cloud Storage]]></title>
            <link>https://medium.com/@dikshantmali.dev/migrating-from-aws-s3-to-google-cloud-storage-b9f8e90b65f9?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9f8e90b65f9</guid>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Tue, 24 Jun 2025 15:26:57 GMT</pubDate>
            <atom:updated>2025-06-24T15:26:57.330Z</atom:updated>
            <content:encoded><![CDATA[<h3>🚀 Practical Guide: Migrating from AWS S3 to Google Cloud Storage</h3><p>As cloud adoption grows and multi-cloud strategies become more common, organizations often face the need to move data between cloud providers. One of the most frequent scenarios is migrating object storage from Amazon S3 to Google Cloud Storage (GCS). Whether driven by cost optimization, regional availability, compliance, or a shift in cloud vendor strategy, such a migration must be planned and executed carefully to ensure minimal disruption and data integrity.</p><p>In this article, I’ll walk you through the process of migrating data from AWS S3 to GCS, step by step. We’ll explore key considerations like storage classes, permissions, access control, data transfer tools, and automation using native and third-party solutions. Whether you’re a DevOps engineer, cloud architect, or developer, this guide will help you understand the nuances of each platform and perform a seamless, secure migration.</p><p>To migrate S3 data from AWS to GCP, we can use the GCP Transfer Service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2SgPDXe7ZilO5_uFbrKVOg.png" /></figure><h3>Step 1: Create an S3 bucket in AWS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dDe4-X_CW1-5QB4R" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9MwhKNdDUZqXh8WU" /></figure><h3>Step 2: Create a cloud storage bucket on GCP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u07MyqGIN-hjPGJX" /></figure><h3>Step 3: Upload some files to AWS S3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*co5VoLrYIeH0sBzZ" /></figure><h3>Step 4: Create a transfer job for the cloud storage bucket in GCP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ov-JqsrSaivmEoai" /></figure><h3>Step 5: Create an IAM user in AWS to provide S3 bucket permission</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CDWu_VYSfTPUUj_3" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O1PqYWplPTy6-8oq" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-uq8cm6frJyUVYO4" /></figure><h3>Step 6: Choose the source and enter the AWS bucket name</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gLMZQXCl8gAxkCR5" /></figure><h3>Step 7: Add access key and secret key in the transfer job creation in GCP</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jjUrZMtX9fQAkJM3" /></figure><h3>Step 8: Choose a destination bucket</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-z07vjRcFtc6-wwC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iwQAzsRZDElHQDhV" /></figure><p>We can modify this option as per our requirement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y1jcHI34bDZk1ISG" /></figure><p>We can change this option if we want to run the job at a particular time.</p><h3>Step 9: Create a job</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MMiyIjVbTnIv3Rel" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lMG1sBMTAJHkKTll" /></figure><p>Done !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9f8e90b65f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backup And Restore Of GKE PV-PVC In Different Clusters On GCP Using Velero]]></title>
            <link>https://medium.com/@dikshantmali.dev/backup-and-restore-of-gke-pv-pvc-in-different-clusters-on-gcp-using-velero-51024299a116?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/51024299a116</guid>
            <category><![CDATA[pvc]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[veelro]]></category>
            <category><![CDATA[pv]]></category>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 12:43:35 GMT</pubDate>
            <atom:updated>2025-06-23T17:36:35.678Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*9QP25L7KHHjuhn57" /></figure><h3><strong>Prerequisite</strong></h3><p>An existing GKE cluster consists of persistent volumes (PVs) and persistent volume claims (PVCs).</p><h3><strong>Tools used</strong></h3><p><strong>Velero:</strong> Velero is an open-source tool for safely backing up and restoring resources in a Kubernetes cluster, performing disaster recovery, and migrating resources and persistent volumes to another Kubernetes cluster.</p><p><strong>Note: </strong>Velero does not support the migration of persistent volumes across cloud providers.</p><p><strong>Restic:</strong> Velero has support for backing up and restoring Kubernetes volumes using a free open-source backup tool called <a href="https://github.com/restic/restic">restic</a>.</p><p><strong>Note:</strong> k8s hostPath volumes are not supported.</p><p><strong>Step 1: </strong>Create a cloud storage bucket to upload all our backups from the k8s cluster.</p><pre>$ gsutil mb gs://gke-cluster-backup-bucket/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LYcgy76KplYFVecT" /></figure><p><strong>Step 2:</strong> Create a service account</p><pre>$ gcloud iam service-accounts create velero --display-name &quot;Velero service account&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*JzNB_my2t84-vr8DAda9XA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*TSi-pVE40016pUBH1ISFsw.png" /></figure><p><strong>Step 3:</strong> Set the $SERVICE_ACCOUNT_EMAIL variable to match its email value.</p><pre>SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \<br>  --filter=&quot;displayName:Velero service account&quot; \<br>  --format &#39;value(email)&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*lGu4caWHBL4lf9NQfmaXyw.png" /></figure><p><strong>Step 4: </strong>Attach policies to give Velero the necessary permissions to function:</p><p><strong>Step 4a:</strong> Export necessary roles to attach to the service account.</p><pre>ROLE_PERMISSIONS=(<br>compute.disks.get<br>compute.disks.create<br>compute.disks.createSnapshot<br>compute.snapshots.get<br>compute.snapshots.create<br>compute.snapshots.useReadOnly<br>compute.snapshots.delete<br>compute.zones.get<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*AqmABNmcwCoQZyIk-xJVJg.png" /></figure><p><strong>Step 4b: </strong>Create an IAM role with necessary permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*0fSbtK_m0FuSHyrp1N0HUQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aQt1afIbw07ns39O" /></figure><p><strong>Step 5: </strong>Attach this role to the Velero service account created in step 2.</p><pre>$ gcloud projects add-iam-policy-binding gcp200credit \<br>    --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \<br>    --role projects/gcp200credit/roles/velero.server</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*n0jQF2PRCMNCGMS_Kf683g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*bAWuP3eRxnN5te5saE8Whw.png" /></figure><p><strong>Step 6: </strong>Add Velero service account to bucket permission principals to access the <em>gke-cluster-backup-bucket</em> bucket with objectAdmin permission.</p><pre>$ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://gke-cluster-backup-bucket</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*376vvBRs-0yQ4FjyHCLZWw.png" /></figure><p><strong>Step 7: </strong>Create a service account key, specifying an output file (credentials-velero) in your local directory.</p><pre>$ gcloud iam service-accounts keys create credentials-velero \<br>    --iam-account $SERVICE_ACCOUNT_EMAIL</pre><p><strong>NOTE: </strong>If you run Google Kubernetes Engine (GKE), make sure that your current IAM user is a cluster-admin. This role is required to create RBAC objects.</p><p><strong>Step 8 (optional): </strong>Dynamically provision PersistentVolumes</p><p><strong>Note: </strong>Most of the time, you don’t need to directly configure PersistentVolume objects or create Compute Engine persistent disks. Instead, you can create a PersistentVolumeClaim, and Kubernetes automatically provisions a persistent disk for you.</p><blockquote><em>Demo YAML for configuring PV-PVC on GKE.</em></blockquote><pre># pvc-pod-demo.yaml<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: pvc-demo<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 30Gi<br>  storageClassName: standard-rwo<br>---<br>kind: Pod<br>apiVersion: v1<br>metadata:<br>  name: pod-demo<br>spec:<br>  volumes:<br>    - name: pvc-demo-vol<br>      persistentVolumeClaim:<br>       claimName: pvc-demo<br>  containers:<br>    - name: pod-demo<br>      image: nginx<br>      resources:<br>        limits:<br>          cpu: 10m<br>          memory: 80Mi<br>        requests:<br>          cpu: 10m<br>          memory: 80Mi<br>      ports:<br>        - containerPort: 80<br>          name: &quot;http-server&quot;<br>      volumeMounts:<br>        - mountPath: &quot;/usr/share/nginx/html&quot;<br>          name: pvc-demo-vol</pre><p>When you create this PersistentVolumeClaim object with $ <em>kubectl apply -f pvc-pod-demo.yaml</em>, Kubernetes dynamically creates a corresponding PersistentVolume object.</p><pre>$ kubectl apply -f pvc-pod-demo.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*C-CtXb8V8txdMvFvCR83JA.png" /></figure><pre>$ kubectl get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*TveScJLeLKwx7WkHvmN55Q.png" /></figure><pre>$ kubectl get pv,pvc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*ysW94fHXDiLrWcUwIKR9HQ.png" /></figure><p><strong>Step 9 (optional): </strong>Make changes to the <em>/usr/share/nginx/html </em>in the pod.</p><pre>$ kubectl exec -it pod/pod-demo /bin/bash</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*3LIc_iFA6J04WaHpZOm1Ug.png" /></figure><p><strong>Step 10: </strong>Install and start Velero with restic.</p><pre>$ velero install \<br>    --provider gcp \<br>    --plugins velero/velero-plugin-for-gcp:v1.0.1 \<br>    --bucket gke-cluster-backup-bucket \<br>    --secret-file ./credentials-velero \<br>    --use-restic</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*h916f0yyoVrf51zrYI2BMQ.png" /></figure><pre>$ kubectl get pods -n velero</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*Ri6K1zVdpRrpBfBYBOHE2g.png" /></figure><p><strong>Step 11: </strong>Take a backup of the PV, PVC using restic for the pod.</p><p><strong>Step 11a:</strong> Annotate the pod with the volume name while taking backup.</p><p>Get persistent volume names.</p><pre>$ kubectl get pv</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*5lhPnDcGZj5eXAfbSWEklA.png" /></figure><p>We can check the PV is claimed by which PVC by describing the PV.</p><pre>$ kubectl describe pv &lt;pv-name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0zt1jlEPFRapVxtC" /></figure><p>Now, annotate the pod with the corresponding volume.</p><pre>$ kubectl annotate pod pod-demo backup.velero.io/backup-volumes=pvc-demo-vol</pre><p><strong>Note: </strong>The volume name can be found in the manifest used to deploy the application pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*Zf59KD5esoJ-j9si" /></figure><p>Verify:</p><pre>$ kubectl describe pod pod-demo | grep Annotations</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*h6Bc0je-0IdtfAkTPDVdYA.png" /></figure><p><strong>Step 12: </strong>Perform the Velero backup</p><pre>$ velero backup create &lt;backup-name&gt; -- include-namespaces default</pre><blockquote><strong>Note: — — </strong>default-volumes-to-restic option Use restic by default to backup all pod volumes.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*S2IFRLw7_J0Mksp9cpKEhw.png" /></figure><pre>$ velero backup describe gke-pv-pvc-backup</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*rLSubRNRFhy9HnXiwJqGRw.png" /></figure><p><strong>Step 13: </strong>Create a new GKE cluster to restore the backup.</p><p><strong>Step 13a:</strong> Authenticate to the GKE cluster</p><pre>gcloud container clusters get-credentials &lt;cluster-name&gt; --zone &lt;zone-id&gt; --project &lt;project-id&gt;</pre><p><strong>Step 14: </strong>Install Velero with restic in the newly created cluster.</p><pre>$ velero install \<br>    --provider gcp \<br>    --plugins velero/velero-plugin-for-gcp:v1.0.1 \<br>    --bucket gke-cluster-backup-bucket \<br>    --secret-file ./credentials-velero \<br>    --use-restic</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*_eaNrkKvcImZlOEXOSxRlQ.png" /></figure><p><strong>Step 15:</strong> Restore the backup through Velero.</p><p><strong>Step 15a:</strong> Get available backup information.</p><pre>$ velero backup get</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*LDYeLVbEptxCe9Wv0ll2Zw.png" /></figure><p><strong>Step 15b:</strong> Restore backup</p><pre>$ velero restore create &lt;restore-name&gt;  --from-backup gke-pv-pvc-backup</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*8pnX-ZHF8ijMmnqah8JaaQ.png" /></figure><p><strong>Step 16:</strong> Verify the restoration.</p><p><strong>Step 16a:</strong> Exec into the pod in the namespace and verify the content we added.</p><pre>$ kubectl exec -it pod/pod-demo /bin/bash</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*XtBUHp735Bnks_QYqbWHKQ.png" /></figure><p>Done!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51024299a116" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud DMS: Migrating Postgresql AWS RDS To GCP Cloudsql Through VPN Tunnel]]></title>
            <link>https://medium.com/@dikshantmali.dev/cloud-dms-migrating-postgresql-aws-rds-to-gcp-cloudsql-through-vpn-tunnel-983c9359c8a6?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/983c9359c8a6</guid>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[database-migration]]></category>
            <category><![CDATA[postgresql-database]]></category>
            <category><![CDATA[cloud-dms]]></category>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 12:43:11 GMT</pubDate>
            <atom:updated>2025-06-23T12:43:11.520Z</atom:updated>
            <content:encoded><![CDATA[<p>As organizations adopt multi-cloud strategies or shift their workloads to Google Cloud, migrating databases becomes a critical step. One common scenario is moving a PostgreSQL database hosted on <strong>AWS RDS</strong> to <strong>Google Cloud SQL</strong>. While this may sound complex, <strong>Google Cloud Database Migration Service (DMS)</strong> simplifies the process, especially when paired with a secure <strong>VPN tunnel</strong> for private, encrypted communication between AWS and GCP.</p><p>In this article, we’ll walk through a step-by-step guide to securely migrate your PostgreSQL database from AWS to GCP using Cloud DMS. This approach ensures minimal downtime, data integrity, and a smooth transition to managed services on GCP.</p><p>To migrate CloudSQL to RDS, refer <a href="https://medium.com/@dikshantmali.dev/aws-dms-migrating-postgresql-gcp-cloudsql-to-aws-rds-through-vpn-tunnel-ceb48b476089">here</a></p><h4>🔑 Key Highlights:</h4><ul><li>Use <strong>Cloud VPN</strong> to establish a secure tunnel between AWS and GCP.</li><li>Configure <strong>AWS RDS</strong> as the source and <strong>Cloud SQL</strong> as the target.</li><li>Use <strong>Cloud DMS</strong> for continuous or one-time migration.</li><li>Handle network routing, replication, user permissions, and connectivity settings.</li><li>Monitor the migration and cut over with minimal disruption.</li></ul><h3>AWS → GCP S2S VPN</h3><p><strong>Step 1:</strong> Create a VPC in AWS[10.2.0.0/16] with 2 public, private subnets, 1 NAT gateway, and 1 internet gateway. Also, make sure DNS hostname and DNS Resolutions are enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*BfaxZ5Lleg3u7oAZh_xZag.png" /></figure><p><strong>Step 2:</strong> Create a VPC in GCP with 1 subnet with non-overlapping CIDR[subnet CIDR → 10.0.0.0/16].</p><p><strong>Step 3:</strong> Launch an instance in both the VPCs. Make sure cloud NAT is enabled in GCP if you deploy a private VM, and the NAT gateway is enabled in AWS if you want to have internet connectivity.</p><p><strong>Step 4:</strong> Set up S2S VPN between both of these VPCs.</p><p><strong>Step 4a:</strong> Create a static IP on gcp cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rgEtcGNUfbENIiwC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pg1LFBUuHyx5otkP" /></figure><p><strong>Step 4b:</strong> Create a customer gateway on AWS with the static IP created in <em>step 4a</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*aCJE-2-M-bSBm6xgJNb-BA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*dvt8V-2kKt6IeR5yayDBhg.png" /></figure><p><strong>Step 4c: </strong>Create a virtual private gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HyxpyhRmV1OCk_Ll" /></figure><p><strong>Step 4d:</strong> Attach VPG to your VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*IZ_ISGPUDYfJBrNQmiJPpQ.png" /></figure><p><strong>Step 4e:</strong> Create an S2S VPN connection on AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jwp0f2tXQIPcI7km" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*vxjjkbTXeH_sp7DW" /></figure><p>In the static IP prefixes, enter the GCP VPC CIDR. 10.0.0.0/16</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gFx4X4h97ooVc-Qe" /></figure><p><strong>Step 4f:</strong> Download the configuration for aws s2s vpn connectivity details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J4dy5YirtQgMduk2" /></figure><p><strong>Step 4g:</strong> Go to → GCP and create a VPN with classic mode with the configuration you downloaded.</p><p><strong>NOTE:</strong> Add configuration for both the tunnels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yqKUH6KsXxD3hS_a" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*26RS512UGYRx8Iuk" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K1M4YFTaqdgoHSlA" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*x8TJXOh7AuR4j_k3nriePw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*bERetI7ZHxRy_Bd7P32zWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*6SSezBZSLLb2GrCNQ9LmiA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*MLrzmBcvfoNMaGazhlCx3Q.png" /></figure><p><strong>Step 4h:</strong> Verify the tunnel setup. It should show active on both sides.</p><p><strong>GCP</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*TAdgojx7AAVOFbQ7lUJMPg.png" /></figure><p><strong>AWS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*1I-cgfW2NGTNoPQ7QRZZGQ.png" /></figure><p><strong>Step 5:</strong> Add a route through VPG to GCP CIDR in aws route table for the subnet and enable route propagation.</p><p><strong>Step 5a:</strong> Add Routes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NEs6Dr9qeeewomLN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*JKjPoP4rZGFLvPGOosreNQ.png" /></figure><p>NOTE: Once you enable route propagation, it will automatically add routes. So not necessary to add routes.</p><p><strong>Step 5b:</strong> Allow firewalls on both sides.</p><p><strong>AWS</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MM7NeCJMvQrB7yXu" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/0*SKTizmfs8VsOGyxs" /></figure><p><strong>GCP</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*baNx6K-zMprx4m9B" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wal3FYoKBiUYZIT9" /></figure><p><strong>Step 5c: </strong>Check the ping from both instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*Y-CAMy9Q37tRGfg5" /></figure><p><strong>Step 6:</strong> Create and deploy a private RDS in AWS.</p><p><strong>NOTE: </strong>Do add some dummy databases and tables.</p><p><strong>Step 6a:</strong> Create a private database subnet group [choose only private subnets].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*jptmlGwtiBW-oLWxx6RTSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*OkfLA_9S7b31Tpwn5EsehA.png" /></figure><p><strong>Step 6b:</strong> Deploy RDS Postgresql in this private subnet group.</p><p><strong>Step 7:</strong> Configure Cloud DMS on GCP.</p><p><strong>Step 7a:</strong> Enable Cloud DMS API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AYFERK5H6JgJsevf" /></figure><p><strong>Step 7b:</strong> Create a connection profile in GCP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C2TM-BmlJA2Qvt6U" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*fgSJJHPPbqWdlTUTXLCxqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OdMwg00Eeny-wkZ6" /></figure><p><strong>Step 7c: </strong>Create a continuous managed migration job.</p><p><strong>Step 7ca:</strong> Fill in to get started</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PgD-ZHdyCXHXpIUB" /></figure><p><strong>Step 7cb:</strong> Define Source</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*E-5oYq0MCxGzjnx5QWOWfQ.png" /></figure><p><strong>Step 7cc: </strong>Define Destination</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SDBsa_IPhUvmHrwa" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KLHQmlsYU8zX6KQR" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BZNcFw4ItSvG03d5" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RufXKXRhXnNDTZjv" /></figure><p>Click on → ALLOCATE AND CONNECT</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W5eeGZOjyty4yjKo" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k8wTTOaFC0NjGRqR" /></figure><p><strong>Step 7cd:</strong> Define connectivity method</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8X9pfzzCn1bw53vQ" /></figure><p><strong>Step 7ce:</strong> Test the migration job; it will fail because we haven’t configured the firewall &amp; route yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yjBZKwGRjqYzjvTk" /></figure><p>Allow firewalls on both sides and GCP.</p><p><strong>On GCP</strong></p><p>Edit this VPC peering and check Import Custom Routes and Export Custom Routes in the VPC Peering connection details for the Cloud SQL, and click SAVE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*dAqpTszxdqCtArm7yA__3w.png" /></figure><p>Copy internal IP ranges of Google-managed Cloud SQL [10.182.16.0/20]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eIyiuRwjOv7zBklG" /></figure><p>Internal IP ranges for Cloud SQL: 10.182.16.0/20 (Need these IP ranges to allow in the AWS security group of RDS)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lrvh3OBeO1vjP-F3" /></figure><p>Enable custom routes for private connection if not enabled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dVt5yg8DC4K9bzZL" /></figure><p>Add a static route in aws s2s vpn for Cloud SQL internal IPv4 CIDR [10.182.16.0/20].</p><p><strong>Note:</strong> Make sure route propagation is enabled on the route table for subnets of RDS. Otherwise, it won’t be added automatically. If not added, then do add manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PYfNtazHzixyG_Yy" /></figure><p>The route detail for Cloud SQL internal CIDR should be like this in aws route table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*oKBbRgSbJiCkk8RiaXEf6A.png" /></figure><p><strong>Step 7cf:</strong> Test the migration job once again after making changes to the firewall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*rk5E6fXw9YMhBUfBz7wikw.png" /></figure><p>We are getting these errors, which we have to fix for the successful creation of migration jobs. Let’s fix them.</p><blockquote>Check the below section: <a href="https://cloud.google.com/database-migration/docs/postgres/configure-source-database#awsrds-postgres">https://cloud.google.com/database-migration/docs/postgres/configure-source-database#awsrds-postgres</a></blockquote><blockquote><a href="https://aws.amazon.com/blogs/database/part-2-upgrade-your-amazon-rds-for-postgresql-database-using-the-pglogical-extension/">https://aws.amazon.com/blogs/database/part-2-upgrade-your-amazon-rds-for-postgresql-database-using-the-pglogical-extension/</a></blockquote><ol><li><strong>To fix →</strong> The source database ‘wal_level’ configuration must be equal to ‘logical’. wal_level should be set to logical instead of replica.</li></ol><ul><li>You need to set rds.logical_replication parameter to 1. It will automatically change wal_level to logical. With RDS Postgresql or Aurora Postgresql, you can only work on logical replication, because we can find the parameter rds</li></ul><p><strong>2. To fix → </strong>Invalid RDS database logical replication. rds.logical_replication should be set to ‘on’, got: off</p><ul><li>Update the same in the AWS RDS parameter group.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KZ-bqZ6i7fygWn5d" /></figure><p><strong>3. To fix → </strong>The source database must have a pglogical extension installed. No PostgreSQL extension installed on databases: mydatabase, postgres</p><ul><li>Update below parameter below in the AWS RDS parameter group.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5Jz3CvJmmWem1ovr" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oZE8JluoxeinHU8n" /></figure><p>Now, in AWS RDS, for all databases that do not have extensions for pglogical, create using the create extension pglogical.</p><pre>CREATE EXTENSION pglogical;<br>select * from pg_extension;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fmHjCVOR6RAGmtXD" /></figure><p>Now, updated output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0CAf5x13XGCHZD5a" /></figure><p>Done!!</p><p>Now, the test job should succeed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J6JjabgCar5WiGpR" /></figure><p>The data should replicate now from AWS to GCP Cloud SQL. The cloud DMS creates a read replica for the instance with a managed primary. So, do not forget to promote your Cloud SQL read replica to make it the main DB.</p><p><strong>Note: </strong>Once you promote the read replica, the replication from AWS RDS to GCP Cloud SQL will stop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DwDJl1uE3u09-R1q" /></figure><p>Promote a read replica from the cloud DMS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FvL4jDVxS-SdYE6y" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9kaJ-nTW1-GS6UcW" /></figure><h3>Thanks!! 🙌</h3><p>Downtime doc: <a href="https://cloud.google.com/database-migration/docs/manage-jobs">https://cloud.google.com/database-migration/docs/manage-jobs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=983c9359c8a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EC2: AWS CodeDeploy with AutoScaling]]></title>
            <link>https://medium.com/@dikshantmali.dev/ec2-aws-codedeploy-with-autoscaling-f4eefff6fe59?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4eefff6fe59</guid>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 12:42:39 GMT</pubDate>
            <atom:updated>2025-06-23T17:37:51.870Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PCauzdxQeznZ4LOS" /></figure><p><strong>GITURL:</strong> <a href="https://github.com/dikshantmali/codedeploy-testing-python">https://github.com/dikshantmali/codedeploy-testing-python</a></p><blockquote><strong>appspec.yaml →</strong></blockquote><blockquote><strong>The application specification file (AppSpec file) is a YAML -formatted or JSON-formatted file used by CodeDeploy to manage a deployment. The AppSpec file for an EC2/On-Premises deployment must be named appspec.yml or appspec. yaml.</strong></blockquote><p>The <strong>appspec.yaml</strong> file has to be present in the root directory of your code that you want to deploy.</p><h3>Step 1: Create an AMI from the instance with the dependencies your code requires</h3><p>The Dependencies my AMI has are given below:</p><ul><li>Node and NPM</li><li>Python3</li><li>PM2</li><li>Flask</li><li>Nginx</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bqsMImwwO-YPQxrp" /></figure><p><em>The user data script we use to install Node.js and Python dependencies for our app is given below:</em></p><pre>#!/bin/bash<br>sudo apt update -y<br># Using Ubuntu<br>curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -<br>sudo apt-get install -y nodejs<br>npm install pm2 -g &amp;&amp; pm2 update<br>sudo apt install python3-pip</pre><p><strong>Note:</strong> The additional dependencies for Python we have installed manually after SSHing to the EC2 instance.</p><p><strong>Note:</strong> The AMI name for this demo is python-dependencies-img</p><h3>Step 2: Create Application Load Balancer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-jFFgRIXg_tiaMgw" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d8VhGjY38a-vcyW_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e76rvbertH-S2LmG" /></figure><h3>Step 2a: Create a Security Group For Application Load Balancer</h3><p><strong>Note:</strong> Only allow all traffic at port 80(listen port) of your load balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z0kCBcjjmkoV5MxK" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jGuOrtpuouFrHfbr" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*obZZ0vVX4-N3jAAN" /></figure><h3>Step 2b: Choose Security Group</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_Pjdps4CnoydRO_i" /></figure><h3>Step 2c: Create Target group</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rGXeAzljyfRhGQCN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K02-N58ecyYw9Y-i" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3i6owDvIuYYtbiAB" /></figure><p>Do not attach any instances and simply create a target group; also, add the port on which the load balancer should route traffic to instances available in this target group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NOSHnaLOc-kAmoDi" /></figure><p><strong>Note: </strong>Change the port in case your application is listening on another port.</p><h3>Step 2d: Choose the target group</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JtUfPaKDdieH3BSV" /></figure><p>Click Create -&gt; Load Balancer</p><h3>Step 3: Now, create a launch configuration for the Autoscaling group</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s32Qj_so3UxttBIV" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MVRcCRSlKbv1BlPp" /></figure><p><strong>Note:</strong> The ami chosen above has all the dependencies installed</p><h3>Step 3a: Create role for ec2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pByK7d6iGBzy5ZSZ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYePyYUVV6hpJqnc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t3GbOpIi_0GoT60a" /></figure><p>Click -&gt; Create Role and attach it to the instance profile on the launch configuration page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LNlYk-rTnKHtxYRG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pA5Dspff9-bE-3F5" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vjdBhZU-FacBH-8x" /></figure><p>The security group selected above allows only load balancers to access 5000 ports on EC2 instances launched with this launch configuration.</p><p><strong>Note:</strong> To achieve the above scenario, only allow access to 5000 ports to the security group attached to the load balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I_9c8M7PaR8WIybx" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dGHoLD2MlLVQnCYb" /></figure><h3>Step 3b: Add the codedeploy agent installation script to the user data of the launch configuration.</h3><p>The userdata script is given below:</p><pre>#!/bin/bash<br>## userdata script for codedeploy agent<br>sudo apt-get update -y<br>sudo apt install ruby-full -y<br>sudo apt-get install wget -y<br>cd /home/ubuntu<br>wget https://aws-codedeploy-us-east-1.s3.us-east-1.amazonaws.com/latest/install<br>chmod +x ./install<br>sudo ./install auto &gt; /tmp/logfile<br>sudo service codedeploy-agent start<br>sudo service codedeploy-agent status</pre><p>Click -&gt; Create launch configuration</p><h3>Step 4: Create a service role for CodeDeploy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GpFTl25-xlRa5AvS" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oQaZyQlBW1n0_MlZ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tPEiL4LKRJwzaNNu" /></figure><p>Click -&gt; create role</p><h3>Step 5: Create ASG</h3><p><strong>Note:</strong> Choose the launch configuration created in step 3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bgr1Tjkt-jUJj8Rv" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7kmhhiQZA2Q-M3H5" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6XAqZoEwgM0DE6Hs" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TGozNeL0dTwQshJm" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kzqgLXrVpqtSIHTS" /></figure><h3>Step 6: Configure code deploy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1c7i3jwVmGQi824T" /></figure><h3>Step 6a: Create application</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rRcWjrrmtxJfJDi-" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5niTvuZ5IF9wTYzA" /></figure><h3>Step 6b: Create a deployment group</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SIrb_lJVboTQacfE" /></figure><p><strong>Note: </strong>Choose the role created in step 4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*35_rThFaie9l_HsM1vHdxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Ha4bJfOrHkDKom1RUvWJAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_kXWwbU3PXH-12oC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QaGkaD_JsXq1DxPh" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p8stgSkQMBEsAPBE" /></figure><p>Click -&gt; create deployment group</p><h3>Step 6c: Create deployment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TQv4Q39zzsFPNGAH" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*xGpwxctteUEGMYB99mWsQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fn5yuChOOsBFUVBG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EaM8-6cw52Ce-g1X" /></figure><p>Click on create deployment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-9FRUnumC-1Gg55x" /></figure><p>Done!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4eefff6fe59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS DMS: Migrating Postgresql GCP Cloudsql To AWS RDS Through VPN Tunnel]]></title>
            <link>https://medium.com/@dikshantmali.dev/aws-dms-migrating-postgresql-gcp-cloudsql-to-aws-rds-through-vpn-tunnel-ceb48b476089?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceb48b476089</guid>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Sun, 22 Jun 2025 11:28:10 GMT</pubDate>
            <atom:updated>2025-06-22T11:28:10.046Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_FaC_pE6fjJHrSpE" /></figure><p><strong>NOTE: </strong>With the below guide, you will only be able to migrate the data of tables. It does not migrate schema, indexes, constraints, and sequences. These things you have to migrate manually, before and during the migration activity.</p><p>For migrating data alongside the above objects, use AWS DMS convert and migrate for homogeneous PostgreSQL migration via their serverless offering.</p><h3>Introduction</h3><p>In today’s fast-paced technological landscape, organizations often find themselves needing to adapt their infrastructure to meet evolving business requirements. One common scenario is the migration of databases from one cloud provider to another. This article aims to provide a comprehensive guide on migrating PostgreSQL Cloud SQL from Google Cloud Platform (GCP) to Amazon Web Services (AWS). The process involves multiple steps, including preparation, data transfer, and validation to ensure a smooth transition. By the end of this article, you will have a clear understanding of the migration process and the best practices to follow.</p><h3>Why Migrate from GCP to AWS?</h3><p>Before diving into the migration process, it’s essential to understand the reasons why an organization might choose to migrate its PostgreSQL database from GCP to AWS. Some of the common motivations include:</p><p>1. Cost Efficiency: AWS may offer more cost-effective pricing models tailored to your specific needs.</p><p>2. Performance: AWS might provide better performance due to its global infrastructure and advanced networking options.</p><p>3. Feature Set: AWS offers a rich set of features and services that might be more aligned with your business requirements.</p><p>4. Compliance and Security: AWS’s compliance certifications and security features may be more suitable for your industry standards.</p><h3>Assessment and Planning</h3><p>1. Inventory and Assessment: Conduct a thorough inventory of your existing PostgreSQL instances, including database sizes, configurations, and dependencies.</p><p>2. Downtime Tolerance: Determine the acceptable level of downtime during the migration process. This will influence your choice of migration strategy.</p><p>3. Compliance and Security: Ensure that the migration complies with all relevant legal and regulatory requirements. Assess the security implications of transferring data between cloud providers.</p><p><strong>The approach we are going to discuss in this article will help you migrate a private cloud SQL Postgresql instance with multiple databases from GCP to AWS RDS with change data capture using Postgres logical replication.</strong></p><h3>Important Points to Notice For Migration</h3><ul><li>We will be using AWS DMS to migrate the PostgreSQL cloud SQL databases from GCP to AWS</li><li>To allow migrating ongoing changes or to implement change data capture, we have to enable logical replication on source databases, i.e., on Cloud SQL instances.</li><li>To allow connectivity between both the private networks, we have to establish a VPN tunnel using GCP cloud VPN and AWS site-to-site VPN service.</li><li>The AWS DMS will allow us to fully load as well as sync tables for a specific Postgres database using a database migration task job.</li><li>We need to create a separate DMS task as well as separate endpoints in case we are planning to migrate more than 1 database from a single cloud SQL instance.</li><li>AWS DMS doesn’t allow you to migrate a database directly; rather, it migrates schemas from that database. If you do not have any schema within your database, you can always choose a public<strong> </strong>schema, which is the default.</li><li>Do not migrate pglogical, pg_catalog, and information_schema from the source to the destination. Each Postgres database handles this automatically.</li><li>Always create a separate user with limited privileges to connect source endpoints. The user you use to connect to the source will be able to create tables until you stop the migration and disconnect the endpoint.</li><li>The AWS DMS also replicates newly created tables to AWS RDS.</li><li>Source endpoint (Cloud SQL private instance will work as the source)</li><li>Destination endpoint (AWS RDS PostgreSQL instance)</li><li>Replication instance (The actual server where DMS tasks run to perform replication. Replication instance connects to both source and target and performs replication)</li><li>Database migration task (The actual job to migrate data with ongoing changes. Each task allows you to migrate schemas)</li><li>After replication, simply stop the DMS task and clean up DMS resources.</li></ul><p>Let’s begin with the implementation now.</p><p><strong>Step 1:</strong> Create a VPC in AWS[10.1.0.0/16] with 2 public and private subnets, 0 NAT gateway, and 1 internet gateway(will use this to download dependencies).</p><p>Amazon Virtual Private Cloud (VPC) is a service that allows you to provision a logically isolated section of the AWS cloud where you can launch AWS resources in a virtual network that you define. With VPC, you have complete control over your virtual networking environment, including selecting your IP address range, creating subnets, and configuring route tables and network gateways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*A2txe98mXf_9dhd1ZaVlbQ.png" /></figure><p><strong>Step 2:</strong> Create a VPC in GCP with 1 subnet with non-overlapping CIDR[subnet CIDR → 10.0.0.0/16] with no NAT gateway(Cloud NAT), only internet gateway(will use this to download dependencies).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*rkbu3gnli2Z0n56H" /></figure><p><strong>Step 3:</strong> Launch a sample instance in both the GCP and AWS networks with a public IP( using a public IP to utilize the internet gateway for any dependency installation like psql client and save cost on the NAT gateway).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*6I_k9yZb0Bk7TQBK" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*s3UawN94xYOui8Bk_qn68Q.png" /></figure><p><strong>Step 4:</strong> Set up S2S VPN between both of these VPCs.</p><p>Using a Site-to-Site VPN with Cloud VPN allows you to establish secure, private connectivity between Google Cloud Platform (GCP) and Web Services (AWS). This setup leverages IPsec VPN tunnels to encrypt data as it travels over the public internet, ensuring a secure communication channel between your cloud environments. By configuring a Site-to-Site VPN, you can seamlessly extend your network infrastructure across both cloud providers, enabling hybrid cloud architectures, improved disaster recovery capabilities, and consistent network policies. This approach provides a cost-effective and reliable way to integrate services and data hosted on GCP and AWS, facilitating a unified and secure multi-cloud strategy.</p><p><strong>Step 4a:</strong> Create a static IP on the GCP cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*tmpLZSawZfM0W1x9" /></figure><p><strong>Step 4b:</strong> Create a customer gateway on AWS with the static IP created in step 4a.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*MoVEQE6QpxMouxu34bLeGw.png" /></figure><p><strong>Step 4c: </strong>Create a virtual private gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*t-h9S73Ql7inhrWH" /></figure><p><strong>Step 4d:</strong> Attach VPG to your VPC.</p><p>Select VPG and attach it to your AWS VPC.</p><p><strong>Step 4e:</strong> Create a S2S VPN connection on AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*XafrTzM_NTo5t19YiQL4Kw.png" /></figure><p>In the static IP prefixes, enter the GCP VPC CIDR. 10.0.0.0/16 [Later will add cloud SQL service networking CIDR as well. Since in GCP, the database is created in Google Managed VPC, and access is allowed through VPC peering between Google Service Networking vpc and GCP VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Imbv8-PM_3H3QVwe" /></figure><p><strong>Step 4f:</strong> Download the configuration for AWS S2S VPN connectivity details.</p><p>Make sure IKEv2 is selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*z1O5ayh_UvBQG-RL" /></figure><p><strong>Step 4g:</strong> Go to → GCP and create a VPN with classic mode with the configuration you downloaded.</p><p><strong>NOTE:</strong> Add configuration for both tunnels from the file you downloaded. Add a pre-shared key and tunnel IP from the file you downloaded. Details are available in the configuration file.</p><p>In the file, we can find settings for both the tunnel, the pre-shared key is available there, and for IP, simply add a Virtual Private Gateway IP address from outside the IP address block available in the config.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*MNFRtIKwlmpISEFG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ZKLZ7cpfD_cHJ7r7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*hxFan8eJm3ckwDuZYUIG6w.png" /></figure><p>Fill in the IP of the AWS VPC in the remote CIDR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GJ3m4bmmRdWVv3YQ" /></figure><p>Now, similarly, add tunnel2; the screenshot for that is not shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*VmmBl3znZVNRiWki-0DB7Q.png" /></figure><p><strong>Step 4h:</strong> Verify the tunnel setup. It should show active on both sides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*a7Byz3w3sQtLhg3hStSKJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*wdXrvN5S_qtvcVLyt4A6gw.png" /></figure><p><strong>Step 5:</strong> Add a route through VPG to GCP VPC CIDR in the AWS route table for the subnet and enable route propagation.</p><p>Note: We can simply enable route propagation on the route table, and it will automatically add the static route we have added in the S2S vpn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*LYfvU8v0KNyBzh6zuUGpWQ.png" /></figure><p>Enabled it for the public subnet, where our sample instance resides. You can also enable it on other subnets from wherever we need connectivity to GCP [Later will do this in this article to allow the DMS replication instance to connect to GCP Cloud SQL DB].</p><p><strong>Step 6:</strong> Allow firewalls on both sides, for instance.</p><p>On the GCP side allows complete VPC CIDR to all instances in the target to allow all traffic from AWS to GCP VPC[You can restrict this as needed].</p><p>AWS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8CYN4QgfVhdI3J2H" /></figure><p>GCP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*y_1APhHJeilJD7OF" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1LoabTMgHs-CuzR5" /></figure><p><strong>Step 7: </strong>Try accessing nginx from both instances.</p><p>Prerequisite: Make sure that nginx is already installed by you on both sides.</p><p>From AWS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*oj9jZX6LJ2ZRMowp" /></figure><p>From GCP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Vvn3NO97N0cDAuHU" /></figure><p><strong>Step 8:</strong> Create and deploy a private Cloud SQL Postgres in GCP.</p><p><strong>NOTE: </strong>Do add some dummy databases and tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*OW7Xw0n_Q9mTOSDy" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*7547TSjS5BLJ3XP7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*bqChY5ss48Hkeuc7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*HQdpazmyy6pqLGIQ" /></figure><p>Enable private service connection for your VPC through pvt svc access. GCP will use the allocated CIDR to create Cloud SQL instances in Google Google-managed VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8HUjboPeRuzM8TnK" /></figure><p>Since GCP-managed VPC will only allow our GCP VPC CIDR to access Cloud SQL, please enable the import-export route for service networking peering to allow the AWS VPC to connect to this Cloud SQL instance. This will automatically import routes for any tunnel attached to this VPC [In our case, the GCP-AWS tunnel].</p><p>Go to VPC network peering, then choose one with GCP VPC &lt;-&gt; servicenetworking</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/0*-rEe_lGAwctlmFKR" /></figure><p>Now, add the IP CIDR of your Cloud SQL to GCP static routes for your S2S VPN connection. Since route propagation is enabled on the route table. This route will be added automatically to your subnet&#39;s route table where the setting is enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xdnpKeAa4d3O-cek" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*OAp4Jyh-GBZFLmNH9cZn_w.png" /></figure><p>This will create a route in the AWS VPC public subnet route table to send a connection to the cloud SQL CIDR via a VPN tunnel.</p><p><strong>Step 9:</strong> Create 1 sample database and insert some dummy tables.</p><pre>$ CREATE DATABASE franchdb;</pre><p>For database tables, I have used some free databases available on the internet; you can also choose any of your choice, or simply create some dummy tables with sample entries.</p><p><strong>Step 10:</strong> Prepare the GCP Cloud SQL database for replication</p><p>Create a user for replication to AWS RDS in GCP Cloud SQL.</p><pre>$ CREATE USER replication_user WITH REPLICATION IN ROLE cloudsqlsuperuser LOGIN PASSWORD &#39;secretpassword&#39;</pre><p>Now we have to configure a pglogical extension on our source cloudsql db to allow logical replication. This step requires a source database restart, so plan a short downtime in the appropriate timeframe for your services. Enable the following flags on your GCP Postgres cloud SQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*NHO-dArm3a-RnVh9" /></figure><p>Now, connect to this cloud SQL database with a Postgres user and run the command to create a pglogical extension in the frenchdb database. You can also create this extension to any other database if you plan to migrate that as well.</p><pre>$ CREATE EXTENSION pglogical;</pre><p>Run the following query to check the installation:</p><pre>$ SELECT * FROM pg_catalog.pg_extension where extname=&#39;pglogical&#39;;</pre><p>Grant privileges required for initial data copy and CDC to replication_user on frenchdb:</p><pre>$ grant usage on schema pglogical to replication_user;<br>$ grant select on all tables in schema pglogical to replication_user;<br>$ grant usage on schema public to replication_user;<br>$ grant select on all tables in schema public to replication_user;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*sJDpIOf5BKp6X3d1" /></figure><p>To allow the newly added table to be replicated, assign permission to replication_user.</p><pre>$ ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO replication_user;</pre><p>Also, once the migration starts, any user on the source database won’t be able to create a new user except replication_user, so grant that user the replication_user role, as shown below.</p><pre>GRANT replication_user TO postgres;</pre><p>Since AWS DMS allows you to migrate schema from a database rather than the DB directly, you can use the below command to get all available schemas for that DB. Make sure that for this demo, we will be only migrating the public schema from frenchdb. There is no need to migrate pg_catalog, information_schema, and pglogical schemas.</p><pre>$ SELECT schema_name FROM information_schema.schemata;</pre><p>The above command will show you 4 schemas in the<strong> frenchdb</strong> database</p><ul><li>pg_catalog</li><li>information_schema</li><li>public</li><li>pglogical</li></ul><p><strong>Step 11:</strong> Create a Private AWS RDS instance in your VPC</p><p>Create a subnet group with private subnets</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*55OJUw3JNUhANd5RdVw2LQ.png" /></figure><p>Create RDS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1d0kHDei02jq8d5Z" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*0Rp0_-dImSkz47lV" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xFTYIN8n5f0YIoi2" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ynetEplYdmogN0im" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*TvlMgi2pGw_W_ZIpwZKTfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*umu4J4dG6C_QLo1-" /></figure><p>Rest settings are the same and can be chosen as per requirement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*2gowapK29CiF-fRy" /></figure><p>Now, we will start setting up AWS DMS</p><p><strong>Step 12:</strong> Set up AWS DMS</p><p>AWS Database Migration Service (AWS DMS) is a fully managed service designed to streamline and expedite the migration of databases to Amazon Web Services (AWS). It supports a variety of database engines, including both commercial and open-source databases, enabling seamless migrations with minimal downtime.</p><p><strong>Step 12a:</strong> Create a subnet group for your replication instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*arGyQjSHwt9WNQ5u" /></figure><p>Choose a private subnet from the list of your VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*oY5y2rr7CPVUCMVz" /></figure><p>Create a replication instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*OtoV5P0XsG9P8j3z" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*rggSpvjzAqowzXH_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*zpYwsZqGixJghxZQVUbUsQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*kv0FaebTC0swYNDK" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*PChh_sRekwuo6HFY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*0pwwiaM7C38qqNJE" /></figure><p>Now, we have to configure the source and destination DB in the endpoints and have to test the connectivity.</p><p><strong>Step 12b:</strong> Configure Source endpoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ApO1h9mejSmcfmpT" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3KFjiBIsRZ05-XwX" /></figure><p>Now, download an SSL certificate from the GCP console and add it to AWS for an SSL-encrypted connection.</p><p>Connections -&gt; Security</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mBkBhaysr1YmTgVy" /></figure><p>Now, add it to the source endpoint in AWS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*iPo1vbYCAEtvORHs" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GgWJtQGCEAZalftc" /></figure><p>Now, here in the database name provides the database to migrate, which is frenchdb. You can not provide a list of databases, etc., you have to provide the name of a single database from your Cloud SQL instance that you are planning to migrate.</p><p>Now, here only, you will see an option to test your connection. Do test the connection and fix any errors observed.</p><p>For us, we have an error while connecting to the database because the private subnet where the replication instance is deployed does not have a route to the GCP Cloud SQL Instance via the tunnel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*x6ABxq5NtUqF6U4eB2QqKQ.png" /></figure><p>To fix this, add the route, simply enable route propagation for the private route table where the AWS DMS instance is going to get deployed. It will auto-add a static route from your S2S VPN connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*mKBiu_vlSabk_hUbvIlsyA.png" /></figure><p>Also, create a firewall rule on the GCP side to allow all connections from AWS VPC [10.1.0.0/16].</p><p>On GCP earlier, we created the rule to allow only port 80. But let’s allow either all traffic or the 5432 port for AWS VPC [For testing, I have allowed all ports].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_Sh4k5KgFHEdTSpD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*srz-6CbdV8TWxubd" /></figure><p>Now, as routes are available and firewall issues are fixed, let’s retry the connection test. And this time it was successful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*nklevPtTajRd8ha5" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*25UyWyiKV81NWMtL" /></figure><p><strong>Step 12c:</strong> Configure Target endpoint</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*j8ln9vN1EW7CJyY8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/0*JN4O3APsLFvDxat_" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*245qoz3SHRO-d5_j" /></figure><p>Here, also in the database name, provide the name frenchdb. Once the name is provided, you have to manually create this database on the destination. So, connect the destination RDS and simply create the database there.</p><p>$ create database frenchdb;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*JlgJe2phVn2mLCOa" /></figure><p>Allow your AWS VPC CIDR to access your RDS. [This will allow DMS to connect RDS] You can also only allow the DMS instance&#39;s private IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1xFIO0sSdRC6uu5S" /></figure><p>Test connection</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*379CWT8sX8vQ8_7h" /></figure><p>Now, we have replication instances ready, and source and destination connections are also ready. Let’s start migrating <em>frenchdb</em>.</p><p>Now, we have the source and destination ready, along with data replication instances. We have to create database migration tasks to full load as well as replicate ongoing changes.</p><p>One thing to note here is that the database you specify while making the endpoint, the replication instance establishes a connection to it. For our case, we have mentioned frenchdb, so it establishes an IDLE connection to frenchdb in the cloud SQL instance.</p><p>We can use the below command to check the connection. It should be in an idle state.</p><pre>$ SELECT pid, usename, application_name, client_addr, state, query FROM pg_stat_activity WHERE datname = &#39;frenchdb&#39;;</pre><p><strong>Step 12d:</strong> Create a DMS Task To Migrate the Public Schema</p><p>Let’s make a DMS task now to migrate cloud SQL to AWS.</p><p><strong>TaskName:</strong> migrate-french-db-aws-gcp</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*lORLPgPH94Bel5QQ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*UoAGZfQQhJL1ISBf" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3T1j-lN5s9C2nrJY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*dvuzXtIPhodYIyZD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Sy0N4KDHLMrLfWEx" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*9QVbuVpD98SE1lB_" /></figure><p>To work with pre-migration tasks, we have to create an S3 bucket in the same region as DMS and an AWS role that this service can use to access this bucket.</p><p>For now, unselect the option we will configure and use it later. Simply create a task with a manual apply.</p><p><strong>Step 12e:</strong> Set pre-migration assessment role policy and bucket. Very high-level steps are given below since it is easy to configure.</p><p><a href="https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.AssessmentReport.Prerequisites.html">https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.AssessmentReport.Prerequisites.html</a></p><p>Prerequisites: s3 bucket with name dms-poc-gcp-cloudsql-to-rds exists.</p><p>Create Policy</p><p>Name: POC-DMS-<strong>DMSPremigrationAssessmentS3Policy</strong></p><pre>{<br>  &quot;Version&quot;:&quot;2012-10-17&quot;,<br>  &quot;Statement&quot;:[<br>     {<br>        &quot;Effect&quot;:&quot;Allow&quot;,<br>        &quot;Action&quot;:[<br>           &quot;s3:PutObject&quot;,<br>           &quot;s3:DeleteObject&quot;,<br>           &quot;s3:GetObject&quot;,<br>           &quot;s3:PutObjectTagging&quot;<br>        ],<br>        &quot;Resource&quot;:[<br>           &quot;arn:aws:s3:::&lt;bucket-name&gt;/*&quot;<br>        ]<br>     },<br>     {<br>        &quot;Effect&quot;:&quot;Allow&quot;,<br>        &quot;Action&quot;:[<br>           &quot;s3:ListBucket&quot;,<br>           &quot;s3:GetBucketLocation&quot;<br>        ],<br>        &quot;Resource&quot;:[<br>           &quot;arn:aws:s3:::&lt;bucket-name&gt;&quot;<br>        ]<br>     }<br>  ]<br>}</pre><p>Create Role</p><p><strong>Name:</strong> POC-DMS-DMSPremigrationAssessmentS3Role</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ghdTIu49X37qW8wc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*UqWszdvemA1ZP9W_" /></figure><p>Now, you can go to the pre-migration assessment tab on the DMS task and configure it. The steps are straightforward to configure.</p><p>For us, we are getting only 1 warning below, which is easy to understand, and you can take action on it as you wish. For our DB config, the setting is good enough. Similarly, if you face any issue in a pre-migration task, then simply fix it and restart/resume the task.</p><p>1 Warning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*DKEWF0134SzGNBQb" /></figure><p><strong>Solution: </strong>Update this in case you have a long-running transaction, and ignore it.</p><p><strong>Step 12f:</strong> Start DMS Task</p><p>Now, simply start the task, and your database <em>frenchdb</em> will be synced to RDS.</p><p>For us, this has run successfully, and we have migrated all sample tables from my database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1A3IiMjfrakfMnx8" /></figure><p><strong>Step 13:</strong> Validate insert, update, and delete operations on frenchdb DB.</p><p>Insert operation verified.</p><pre>INSERT INTO &quot;Track&quot;<br>(&quot;TrackId&quot;, &quot;Name&quot;, &quot;AlbumId&quot;, &quot;MediaTypeId&quot;, &quot;GenreId&quot;, &quot;Composer&quot;, &quot;Milliseconds&quot;, &quot;Bytes&quot;, &quot;UnitPrice&quot;)<br>VALUES<br>(4000, &#39;New Track&#39;, 1, 1, 1, &#39;Composer Name&#39;, 300000, 5000000, 1.99);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*fpPHPt0qUvIK1cDb" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*CMjg_LSOeyQtfdkN" /></figure><p>Delete operation verified</p><p>Simply deleted 310 entries from my frenchdb source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*cOEQBDqmojJ4ikgV" /></figure><p>Update</p><pre>UPDATE &quot;Track&quot;<br>SET<br>&quot;Name&quot; = &#39;Updated Track Name&#39;,<br>&quot;Composer&quot; = &#39;Updated Composer Name&#39;,<br>&quot;UnitPrice&quot; = 2.99<br>WHERE<br>&quot;TrackId&quot; = 4000;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*tR7O1-vnDXhhyoXD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*buCOj2gmF6dVIhwp" /></figure><p>Final Table Stats</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/0*17cb_2MbJrT7hvar" /></figure><p>Once all is done to stop migration, you can stop the job once all data is synced, and can clean up all the resources.</p><p>Done! Thanks for following along 🙌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceb48b476089" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learned: Migrating Prod Redis from GCP Memorystore to AWS ElastiCache]]></title>
            <link>https://medium.com/@dikshantmali.dev/lessons-learned-migrating-prod-redis-from-gcp-memorystore-to-aws-elasticache-f6e292ac9c5d?source=rss-7ee3d91ab4df------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6e292ac9c5d</guid>
            <dc:creator><![CDATA[dikshant mali]]></dc:creator>
            <pubDate>Sun, 22 Jun 2025 11:00:51 GMT</pubDate>
            <atom:updated>2025-06-22T11:00:51.850Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*iSL_M-no8oOBvYNL" /></figure><p><strong>Introduction:</strong></p><p>Migrating a production Redis instance is never a trivial task, especially when it involves moving from one cloud provider to another. Recently, I undertook the challenge of migrating my Redis data from Google Cloud’s Memory Store to AWS ElastiCache. While the goal was clear — to leverage AWS’s robust infrastructure and integrations — what I encountered was a series of unexpected challenges that tested my planning and execution strategies.</p><p>One of the initial approaches I considered was using ElastiCache’s migration feature via its endpoint. This method appeared promising due to its potential to reduce downtime and automate much of the process. However, as I dug deeper, I encountered significant limitations that made this option less feasible for my production environment. These limitations included compatibility issues and less control over Redis configuration, since Cloud memory store is a managed solution from GCP.</p><p><strong>Key Challenges I Faced:</strong></p><ul><li><strong>Restricted Customization:</strong> Due to less control over the Redis configuration. I was not able to make any changes to the Redis configuration on the Google Cloud provider.</li><li><strong>Version Incompatibility:</strong> Ensuring that data remained consistent during the migration was a primary concern, as any discrepancies could lead to application downtime or corrupted data. What I faced was that migrating Redis version 7.2 from GCP to AWS Elasticache 7.0.1 is not feasible due to version conflicts.</li></ul><p>Given these challenges, I ultimately decided to abandon the migration via the endpoint method. Instead, I opted for a more traditional approach: exporting my Redis data as an RDB (Redis Database) snapshot from GCP Memorystore and then restoring an ElastiCache instance on AWS.</p><p>This method, while requiring more manual effort, provided me with greater control over the migration process and minimized the risk of data inconsistency. The RDB dump and restore method allowed me to verify the data at each step, ensuring that the migration was successful without any loss or corruption.</p><p>In this article, I’ll delve into the details of the challenges I encountered, why I chose to pivot my approach, and how I ultimately achieved a seamless migration with minimal downtime.</p><p><strong>Prerequisite:</strong> You need to have a VPN tunnel established between your AWS and GCP networks. You can follow the steps mentioned <a href="https://medium.com/@dikshantmali.dev/hands-on-demo-s2s-vpn-between-gcp-and-aws-network-acd55c69fb11">here</a>.</p><p><strong>Step 1:</strong> Create a Redis instance on GCP.</p><p>The Redis instance is going to be created in a GCP-managed VPC. We will be accessing it through Google private service access.</p><p>As you can see in my GCP network, I already have an IP allocated for some different purposes. Let’s allocate a new IP address with /29 netmask, which GCP will use to deploy our Redis instance in google Google-managed tenant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3OecF4VqMiSKXxfM" /></figure><p>Adding a new range</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*MUpYfl_1GjWDWI8t" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*j_pEe3ouiVuCvFti" /></figure><p>Now, let’s update VPC peering with servicenetworking to populate this new range in its route. So, we can use it to deploy our Redis there.</p><p>Note: Update project ID in the below command, and ranges as per your setup.</p><pre>gcloud services vpc-peerings update --service=servicenetworking.googleapis.com --ranges=vpn-gcp-vpc-ip-range,vpn-gcp-vpc-redis-ip --network=vpn-gcp-vpc --project=&lt;project-id&gt; --force</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*JMZwqAOKgIhswNm8" /></figure><p>Let’s make a Redis memory store in GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*c3gyag98SKJH5jem" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*UX19Isk7YcwlPj5R" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*vz2UkLCeFq1W4lVT" /></figure><p>You can choose security settings as per your requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jzQMCyn5XIyCknwk" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*5LYOTUuGdaPeqlAa" /></figure><p>Now, connect to this Redis and import some sample keys to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*gzI_WjBokjyfazME" /></figure><p>The AUTH token can be found in the memory store security section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*OqG-VYkL9yqiJvykd0P2aA.png" /></figure><p>Now, to migrate this to AWS, ensure the import-export custom route is enabled on the service networking VPC peering. This will automatically allow VPN tunnel routes to import and export for GCP GCP-managed VPC, in our case AWS VPC, so that AWS VPC can access Redis through this peering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*eEyFaY6KUgnPcpyNE7lYPw.png" /></figure><p>Add your Redis-allocated CIDR to your s2s VPN static route so that AWS VPC can route traffic for it. Since route propagation is enabled on route tables, an automatic entry is going to be created in the route table as soon as we add this static route to the S2S VPN on the AWS side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*PHrJP9zT2wdWGxywWBSaWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*Qc5aVMQFL_FDpNEeoqpTgw.png" /></figure><p>Also, ensure that the AWS instance attempting to access this Redis is allowed through the GCP firewall. Simply create a firewall rule on GCP to allow port 6379 from AWS VPC CIDR to the instances in the GCP network.</p><p>Cool 😎, all set….let’s go to AWS and see the data there.</p><p>First, create a private subnet group in your AWS Elasticache to launch a Redis instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*junj7nMI64klxZoEY47q4w.png" /></figure><p>Create a sample Redis instance on Elasticache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*MEXTZJ9P_YFQUuGD" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*VvRcrQalykZcsqaj" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jENZfXzyWHZCjUnh" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xZPIO97KmSHKnvvB" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8LKe66MdY_ay7kN8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*B3PR92tLEUMEbPbF" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*kxY3cRMaLphjHlwE" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Z9AK2xfJIblIUlXR" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*4aYj1jJGqK5cliFq" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ye2wJs_LsXCwZkIB" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*nFLiM49AOi3-IvmO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*qR5u9k805hqSz4HS" /></figure><p><strong>NOTE:</strong> Make sure your security group allows connection.</p><p>Connect to this Redis instance created above and check connectivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Z4QDaIbU3x-uMnnJ" /></figure><p>Now, let’s perform migration.</p><p>On AWS, we have<em> migrated Redis with an endpoint</em> feature available that can migrate data from the source endpoint to your Elasticache. The only thing is that it cleans all the data from your destination cache and loads new data keys to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*pR43f1qVEqkP5Il-" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*CRixXEVteovbtvjX" /></figure><p>As we can see, Migration is not supported for auto-failover disabled instances. So, let’s enable it to perform our migration.</p><p>As we have created a single-shard Redis with 1 primary node, let’s add one read replica for automatic failover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*NdKJupkzwV1ol6Z-" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*yxo-lVKR0UepCFwa" /></figure><p>Now, we have one shard with 1 primary and 1 replica node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*h0h6-8OG1PLkIzQJ" /></figure><p>Let’s enable automatic failover now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*6Z1b5913OFpTdCZv" /></figure><p>Let’s test migration via endpoint one more time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xClSgzn0klNJcuDz" /></figure><p>As we can see, this failed because AUTH is enabled on the source instance. Let’s disable it on the source and try again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*EAPPKUNI9ufZM7-0" /></figure><p>Disable AUTH on the GCP memory store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Rt9zGGp2zOTaJqlH" /></figure><p>It took 30 seconds to update the Redis on GCP. After disabling AUTH, we can observe that we can verify data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3WKqScZGL3nCDqcQ" /></figure><p>Let’s start the migration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*bsHWlVGpuWaHe9Dv" /></figure><p>Again, the migration status has failed for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*SrSbSMValSE9dCSn" /></figure><p>On GCP, we have the Redis version as 7.0, and on the AWS destination, we have 7.1.0. So, let’s try upgrading the GCP cache and see if it fixes the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*wqbrvyOhRphGWI-F" /></figure><p>7.2 is the only version available on the GCP memory store for upgrade.</p><p>It took around 12 minutes to upgrade the Redis instance. Although the upgrade operation is ongoing, we can access the instance, so we did not face any downtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*6nhT1URERfK4Jdsh" /></figure><p>The migration to Elasticache via endpoint is still not working, as per my findings, the option is only available for self-managed Redis OSS. It does not support managed solutions.</p><p>So, to confirm, let’s try migrating the Redis running on the EC2 instance to Elasticache.</p><p>Set up Redis on some EC2 systems and insert some keys into it.</p><pre>version: &#39;3.8&#39;<br><br>services:<br>  redis:<br>    image: redis:7.0.1<br>    container_name: redis-7.0.1<br>    ports:<br>      - &quot;6379:6379&quot;</pre><pre>$ docker compose up -d</pre><p>Let’s insert 50 keys into it</p><p># Connect to Redis and insert 50 keys with a prefix</p><pre>for i in {1..50}; do<br>redis-cli -h localhost -p 6379 SET &quot;prefix_key_$i&quot; &quot;value_$i&quot;<br>done</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*cYGFmjRIb52eSaHa" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*gSJSHlmBqNbdhX0v" /></figure><p>Now, let’s try migrating this to Elasticache.</p><p><strong>NOTE:</strong> Allow connection to the instance from Elasticache VPC. The instance is also available in the same VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3-RCIx8M-sF29Iud" /></figure><p>As we can see, the migration test is also passed, and the cluster is in a migrating condition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*bE8TGM-3JKvrFNeq" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*VQ5278LencJRY_FJ" /></figure><p>Let’s verify the data in Elasticache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*9FN3Rou2j5WHLT0n" /></figure><p>Now, to complete the migration first, we need to ensure all keys are synced. To verify the migration is completed, check that the Redis OSS master_link_status is up on the ElastiCache primary node(s) and set to 1.</p><p><a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Migration-Verify.html">https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Migration-Verify.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*U3Vl9Sd2d_CFyhVR" /></figure><p>Now, we can stop the migration and complete it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*T8b7t2rwyPlciiWI" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*zeAufKjM8M7yzgTR" /></figure><p>So, as per our findings, we ensure that migrating via the Elasticache endpoint is only possible for self-managed Redis as of the writing this document. Since the managed provider comes with their limitations.</p><p>Now, let’s try one more time after completing all the possibilities below and see if we can migrate the memory store. The document below will guide us through the prerequisites for migrating Redis OSS to Elasticache.</p><p><a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Migration-Prepare.html">https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Migration-Prepare.html</a></p><p>After the findings as per things mentioned in the above AWS document, we concluded that GCP memory store does not allow modification to any parameter of their Redis instance, so migrating via endpoint is not possible for GCP memory store. Due to compatibility issues not able to migrate to Elasticache via the endpoint service.</p><p>This means we can migrate Redis from GCP to AWS via .rdb dump, so let’s do that.</p><p>Export .rdb dump from your GCP and upload it to some S3 location in your AWS account in the same region as Elasticache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*MU1VPRMqb996LcKO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*oArkUTOALG8MTvN4" /></figure><p>Download and upload it to the S3 location. You may leverage a migration service on AWS to do so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*j3BU-xtkyO1r_zdL" /></figure><p>We can get help from the AWS document below.</p><p><a href="https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-seeding-redis.html">https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-seeding-redis.html</a></p><p><strong>Step 1:</strong> Allow Elasticache to read this .rdb file.</p><p>Enable ACL on the bucket</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*6vMfaJC9psI3bv3e" /></figure><p>Now, go to the permission tab in your .rdb file and add the grantee to Elasticache. With the below permission. Choose the canonical ID as per your setting. We are operating in the default region, so we have chosen that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1LpHDPRfva23k6q4" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*aGjvZ2VpJSC8i4BU" /></figure><p><strong>Step 2: </strong>Now, load this RDB to your Elasticache.</p><p>Go to -&gt; Elasticache console, then create a new cache with the option to restore from backup.</p><p>We can only create a new cache, but we can not import an RDB dump into the existing cache.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_ibrsU8xXgPRXdJY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*rp5vlrZsBqAIHnjt" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*q1cCJFDxH18l8Ti8" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*QkisCLGYCUCtoJoe" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*D9pzv2_VZRoXVeo7" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*uFXpDMMxIAZS3HKd1CQ39g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jV80KMRUkoY9RacC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*kSTeSZTE8mkxI553" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*QIlURxY6KYc2KLOB" /></figure><p>Then -&gt; create</p><p>While creating this, we are getting an error, it says the RDB file is not valid. This is due to the version compatibility issue on both Redis. Let’s create a new Redis on GCP with 7.0 as the version and try to load its dump. With 7.2, it is not working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*tMGX0wzhB4XmMCWR" /></figure><p><strong>Do the same thing:</strong> export the dump and upload it to S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*bolrnsRnl8BPnJN8" /></figure><p>Add a grantee for Elasticache to this dump and create a cluster as described above.</p><p>Cool, this time it worked</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*iUldADFV41cKSl8J" /></figure><p>Also, verify if keys are populated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*5ZUffcCl46evyAwH" /></figure><p>Done!</p><p>So, this is our learning while migrating from memory store to Elasticache. Do let me know if you were able to migrate the Cloud Memorystore to AWS Elasticache via the migrate to endpoint feature in the comments below.</p><p>Thanks for following along 🙌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6e292ac9c5d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>