<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pnpavan Pn on Medium]]></title>
        <description><![CDATA[Stories by Pnpavan Pn on Medium]]></description>
        <link>https://medium.com/@pnpavan.pn?source=rss-222c3fc16558------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*yUP-vawToAc8Ovdp</url>
            <title>Stories by Pnpavan Pn on Medium</title>
            <link>https://medium.com/@pnpavan.pn?source=rss-222c3fc16558------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 07:51:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pnpavan.pn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solving GCP MIG Graceful Scale-In: A Production-Ready Solution for Queue-Based Workloads]]></title>
            <link>https://medium.com/niveus-solutions/solving-gcp-mig-graceful-scale-in-a-production-ready-solution-for-queue-based-workloads-ff5a02326309?source=rss-222c3fc16558------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff5a02326309</guid>
            <category><![CDATA[graceful-shutdown]]></category>
            <category><![CDATA[infra-modernization]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Pnpavan Pn]]></dc:creator>
            <pubDate>Thu, 18 Sep 2025 08:57:35 GMT</pubDate>
            <atom:updated>2025-09-19T06:24:31.729Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hoirsKIS81zakAXo.png" /></figure><h3>What works by default in GCP Auto-Scaling</h3><p><strong>Google Cloud Platform’s auto-scaling is impressive — until it isn’t.</strong></p><p>Managed Instance Groups (MIGs) handle the heavy lifting for millions of applications worldwide. They’re the backbone of scalable cloud infrastructure, delivering enterprise-grade capabilities that just work:</p><ul><li><strong>Automatic scaling</strong> based on CPU, memory, or custom metrics</li><li><strong>Health checking</strong> and automatic instance replacement</li><li><strong>Rolling updates</strong> with zero-downtime deployments</li><li><strong>Load balancing</strong> integration with seamless traffic distribution</li><li><strong>Multi-zone deployment</strong> for high availability</li></ul><p>But there’s a significant gap in GCP’s auto-scaling capabilities: <strong>graceful scale-down operations</strong>.</p><p>When your MIG decides it’s time to reduce capacity, the process is brutally simple:</p><ol><li>Pick an instance to terminate</li><li>Send a termination signal</li><li>Wait for n(default) seconds</li><li>Force kill the instance</li></ol><p>No checking if the instance is processing critical work. No coordination between instances. No application-aware shutdown logic. Just an immediate termination.</p><h4><strong>The Challenge:</strong> <strong>When Auto-Scaling Meets Real-World Applications</strong></h4><p>Picture this scenario: You have a fleet of instances processing tasks from a Redis queue, running smoothly with Google Cloud Platform’s Managed Instance Groups (MIGs). The scale-up works beautifully — when your Redis queue grows, new instances spin up automatically. But then comes the scale-down challenge.</p><p>The problem isn’t just theoretical. When GCP decides to scale down your MIG, it simply terminates instances. No questions asked. No checking if they’re in the middle of processing critical tasks. No graceful shutdown. Just… gone.</p><p>For queue-based workloads running services like Celery workers, this creates a cascade of issues:</p><ul><li><strong>Data loss</strong> from incomplete task processing</li><li><strong>Service disruption</strong> from abrupt connection termination</li><li><strong>Queue inconsistency</strong> when tasks are partially processed</li></ul><h3>The Evolution: From CPU to Custom Metrics</h3><p>Our journey didn’t start with Redis metrics. Like most teams, we initially relied on CPU-based auto-scaling. But CPU usage doesn’t tell the full story for queue-based applications:</p><ul><li>High CPU might mean efficient processing, not overload.</li><li>Low CPU could indicate waiting for I/O, not idle capacity.</li><li>CPU doesn’t reflect queue depth or processing urgency.</li></ul><p>We needed something smarter — metrics that actually represented our application’s load state. That’s when we moved to <strong>custom Redis metrics</strong>, specifically monitoring queue depth to make intelligent scaling decisions.</p><p>But even with the right metrics, the graceful shutdown problem remained. That’s where our real innovation began.</p><p><strong>A Complete Overview </strong>before we deep dive into the solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*K5Mcj60Vsigw3iBSeto5Iw.png" /><figcaption>flow diagram</figcaption></figure><h3>The Solution: Redis Metric-Aware Graceful Scale-In</h3><p>We designed a system that combines three critical components:</p><h3>1. Intelligent Metric Monitoring</h3><p>Instead of relying on basic CPU metrics, our system continuously monitors custom Redis metrics from Google Cloud Monitoring:</p><pre>METRIC_TYPE=&quot;custom.googleapis.com/redis/test_autoscale_keys_int64&quot;<br>QUEUE_LOW=20  # Scale-in threshold<br>LOOKBACK_SEC=120  # Historical data window</pre><p>The system calculates average queue depth over a look-back period, filtering out recent data to account for ingestion delays. This provides a stable, representative view of actual workload.</p><h3>2. Distributed Coordination</h3><p>Here’s where it gets interesting. Multiple instances can’t all decide to scale-in simultaneously — that would cause chaos. We implemented a <strong>distributed locking mechanism</strong> using Google Cloud Storage’s atomic operations.</p><pre># Atomic lock creation - only one instance succeeds<br>curl -X POST &quot;https://storage.googleapis.com/upload/storage/v1/b/$BUCKET/o&quot; \<br>  --data &quot;holder=$INSTANCE_NAME time=$(date)&quot; \<br>  -H &quot;ifGenerationMatch: 0&quot;</pre><p>This ensures only one instance can initiate scale-in operations at a time, preventing race conditions and maintaining system stability.</p><h3>3. Two-Mode Operation Strategy</h3><p>We support both <strong>Drain Mode</strong> (recommended) and <strong>Standard Mode</strong>:</p><p><strong>Drain Mode</strong> (the game-changer):</p><ul><li>Remove instance from MIG immediately (abandon-instances)</li><li>Stop receiving new work while completing existing tasks</li><li>Release coordination lock early for parallel operations</li><li>Complete cleanup independently before self-termination</li></ul><p><strong>Standard Mode</strong>:</p><ul><li>Execute cleanup while remaining in MIG</li><li>Remove instance only after cleanup completion</li><li>Fully serialized operation</li></ul><h3>The Technical Architecture:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhycsrkZdEMMd6fC_joM6w.png" /><figcaption>basic architecture flow diagram</figcaption></figure><h3>The Complete Process Flow:</h3><ol><li><strong>Initialisation</strong>: Dependencies installed, scripts deployed, service configured</li><li><strong>Warmup Period</strong>: 90-second stabilisation to prevent premature decisions</li><li><strong>Continuous Monitoring</strong>: Redis metrics evaluated against thresholds</li><li><strong>Preflight Verification</strong>: Comprehensive system readiness checks</li><li><strong>Lock Acquisition</strong>: Atomic coordination to prevent conflicts</li><li><strong>Heartbeat Mechanism</strong>: Prevent deadlocks with periodic lock refresh</li><li><strong>MIG Validation</strong>: Ensure scale-in appropriateness and safety</li><li><strong>Graceful Operation</strong>: Execute chosen mode (Drain/Standard)</li><li><strong>Cleanup &amp; Termination</strong>: Complete processing and clean shutdown</li></ol><h3>Key Features That Make It Production-Ready:</h3><h4>Comprehensive Error Handling</h4><ul><li>Automatic stale lock detection and cleanup</li><li>API resilience with exponential backoff</li><li>Guaranteed autoscaler recovery even in failure scenarios</li><li>Service auto-restart with proper signal handling</li></ul><h4>Monitoring and Observability</h4><ul><li>Comprehensive logging in /var/log/graceful-scalein.log</li><li>Service status monitoring via systemctl</li><li>Lock state tracking in GCS</li><li>Key metrics for operational visibility</li></ul><h3>Real-World Benefits:</h3><p>After implementing this solution in production, we’ve seen:</p><ul><li><strong>Zero data loss</strong> during scale-in operations</li><li><strong>Improved queue consistency</strong> with proper task completion</li><li><strong>Reduced operational overhead</strong> from automated coordination</li><li><strong>Better resource utilisation</strong> with intelligent metric-based decisions</li></ul><h3>Beyond Redis: Universal Applicability</h3><p>While we developed this for Redis queue metrics, the architecture is universally applicable:</p><ul><li><strong>Message Queue Systems</strong>: RabbitMQ, Apache Kafka, Google Pub/Sub</li><li><strong>Database Connection Pools</strong>: Graceful connection draining</li><li><strong>File Processing Systems</strong>: Complete current file processing</li><li><strong>API Gateway Instances</strong>: Drain active connections</li><li><strong>Batch Processing Jobs</strong>: Complete current batch before termination</li></ul><p>The key is customising the metric monitoring and cleanup scripts for your specific use case.</p><h3>Conclusion: Graceful Scale-In as a Competitive Advantage</h3><p>In today’s cloud-native world, the ability to scale dynamically while maintaining data integrity isn’t just nice-to-have — it’s essential. Our Redis metric-aware graceful scale-in system transforms a fundamental limitation of GCP MIGs into a robust, production-ready solution.</p><p>The combination of intelligent metric monitoring, distributed coordination, and flexible cleanup handling creates a system that’s both powerful and adaptable. Whether you’re processing Redis queues, handling API requests, or managing batch jobs, the principles and architecture we’ve shared can be adapted to your specific needs.</p><p>The result? A scaling system that respects your application’s actual state, coordinates intelligently across instances, and ensures graceful operations even during the most dynamic workloads.</p><p><em>Have you faced similar challenges with auto-scaling and graceful shutdowns? What solutions have you implemented? Share your experiences in the comments below.<br></em>#GCP #AutoScaling #Redis #CloudArchitecture #DevOps #ProductionSystems #ManagedInstanceGroups</p><h3>Ready to Implement?</h3><p><strong>Get the Complete Solution:</strong></p><p><a href="https://docs.google.com/document/d/1j7DljOdmKtkWWxCWD6CiezLL7JPdnPF15Y6G6V16zH4/edit?pli=1&amp;tab=t.0"><strong>Download Complete Startup Script</strong></a></p><p>The Google Doc contains:</p><ul><li>Production-ready startup script (fully tested)</li><li>Configuration parameters explanation</li><li>Troubleshooting and monitoring</li><li>Custom cleanup script templates</li></ul><p><strong>Quick Start Checklist:</strong></p><ol><li>Create GCS bucket for distributed locking</li><li>Set up custom Redis metrics in Cloud Monitoring</li><li>Copy startup script to your MIG instance template</li><li>Configure the key parameters for your environment</li><li>Deploy and monitor!</li></ol><p>Happy implementing and testing! Would love to hear how this solution works in your environment or any adaptations you make for different use cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff5a02326309" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/solving-gcp-mig-graceful-scale-in-a-production-ready-solution-for-queue-based-workloads-ff5a02326309">Solving GCP MIG Graceful Scale-In: A Production-Ready Solution for Queue-Based Workloads</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End-to-End CI/CD Pipeline for Cloud Run using GitLab, Cloud Build, and Cloud Deploy]]></title>
            <link>https://medium.com/niveus-solutions/end-to-end-ci-cd-pipeline-for-cloud-run-using-gitlab-cloud-build-and-cloud-deploy-09c7f007bb38?source=rss-222c3fc16558------2</link>
            <guid isPermaLink="false">https://medium.com/p/09c7f007bb38</guid>
            <category><![CDATA[cloudrun]]></category>
            <category><![CDATA[infra-modernization]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[cloud-deploy]]></category>
            <category><![CDATA[cloud-build]]></category>
            <dc:creator><![CDATA[Pnpavan Pn]]></dc:creator>
            <pubDate>Mon, 03 Mar 2025 11:02:44 GMT</pubDate>
            <atom:updated>2025-03-24T16:01:25.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m9Zb3Dd5k_a9RR8Via5zLQ.png" /></figure><h3>Introduction</h3><p>A <strong>fully automated CI/CD pipeline</strong> is essential for <strong>fast, secure, and reliable deployments</strong> in cloud-native applications. This blog will walk you through setting up a <strong>GitLab-powered CI/CD pipeline</strong> that integrates with:</p><p>✅ <strong>Google Cloud Build</strong> for building and storing container images<br>✅ <strong>Artifact Registry</strong> for versioned and secure image management<br>✅ <strong>Cloud Deploy</strong> for automated releases and rollbacks<br>✅ <strong>Cloud Run</strong> for a scalable, serverless application runtime<br>✅ <strong>Skaffold</strong> for simplified deployment workflows<br>✅ <strong>IAM Policies</strong> to securely manage service access</p><p>By the end of this blog, you’ll have a solid understanding of how these components fit together. The next blog post will focus on <strong>implementing End-to-End CI/CD Pipeline with Automated Rollbacks using Cloud Deploy configurations in depth</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*ol_NaUyBZEEEmD8qyErv0w.jpeg" /><figcaption><strong>End-to-End CI/CD Flow: From Code Commit to Cloud Run Deployment</strong> 🚀</figcaption></figure><h3>Architecture Overview</h3><p>This CI/CD pipeline automates the process from <strong>code push to deployment</strong> while ensuring <strong>secure and version-controlled releases</strong>. Below is a step-by-step breakdown of the workflow.</p><h3>🔗 CI/CD Workflow Overview</h3><h3>1. GitLab Push Trigger &amp; Cloud Build Execution</h3><ul><li>A <strong>developer commits code</strong> to the GitLab repository, triggering a <strong>GitLab CI/CD pipeline</strong>.</li><li>The trigger executes <strong>Google Cloud Build</strong>, which follows the steps defined in cloudbuild.yaml.</li><li>Cloud Build spins up an isolated <strong>build-worker</strong> environment to process the code.</li></ul><h3>2. Fetching Secrets &amp; Preparing Build Environment</h3><ul><li>Cloud Build fetches sensitive values (like keys, database credentials) from <strong>Google Secret Manager</strong>.</li><li>The build-worker retrieves two crucial files:</li><li>.env → Holds environment-specific configuration</li><li>Dockerfile → Defines how the application is containerized</li></ul><h3>3. Containerization &amp; Pushing to Artifact Registry</h3><ul><li>The application is <strong>built into a Docker image</strong> using the Dockerfile.</li><li>The <strong>final container image</strong> is tagged with a unique version and pushed to <strong>Google Artifact Registry</strong> for secure storage.</li><li>This ensures that every build is <strong>reproducible, versioned, and rollback-friendly</strong>.</li></ul><h3>4. Deployment with Skaffold &amp; Cloud Deploy</h3><ul><li><strong>Skaffold</strong> automates the <strong>deployment workflow</strong>, ensuring that each release follows the same process.<br><em>📂 </em><strong>Skaffold </strong>would have <em>Deploy Configuration Files as below<br> </em>✅ <strong>pipeline.yaml</strong> (Delivery Pipeline)<br> ✅ <strong>target.yaml</strong> (Cloud Deploy Target)<br> ✅ <strong>prod-run.yaml</strong> (Cloud Run Service)</li><li><strong>Google Cloud Deploy</strong> uses the <strong>latest container image</strong> from <strong>Artifact Registry</strong> and initiates deployment to <strong>Cloud Run</strong>.</li><li>Cloud Deploy maintains a <strong>versioned history</strong> of releases, simplifying rollbacks if needed.</li></ul><h3>5. Health Checks &amp; Validation</h3><ul><li>After deployment, a <strong>health check</strong> is performed to verify the application’s status.</li><li>If the health check <strong>fails</strong>, the pipeline automatically triggers a <strong>rollback</strong> to the last successful deployment.</li><li>If successful, the deployment <strong>proceeds to production</strong>.</li></ul><h3>6. Secure Deployment with IAM &amp; Cloud Run</h3><ul><li><strong>Cloud IAM permissions</strong> enforce strict access control, ensuring that only authorized entities can deploy to Cloud Run.</li><li>The final <strong>redeployment step</strong> ensures the <strong>new version of the application</strong> is live and accessible.</li></ul><h3>Key Benefits of This CI/CD Pipeline</h3><p>✅ <strong>Automated Deployment</strong> — Reduces manual intervention with <strong>GitLab CI/CD triggers</strong>.<br>✅ <strong>Secure Image Storage</strong> — Uses <strong>Artifact Registry</strong> to ensure images are safely stored and versioned.<br>✅ <strong>Rollback Mechanism</strong> — If a deployment fails, it <strong>automatically reverts to the last successful release</strong>.<br>✅ <strong>Secret Management</strong> — Uses <strong>Google Secret Manager</strong> to handle API keys and credentials securely.<br>✅ <strong>Cloud-Native Efficiency</strong> — Optimized for <strong>Google Cloud Run</strong>, enabling <strong>serverless scalability</strong>.</p><h3>Next Steps for Implementation</h3><p>For the next author working on implementation, the following steps need to be completed:</p><p>1️⃣ <strong>Setup GitLab CI/CD with Cloud Build</strong></p><ul><li>Define the cloudbuild.yaml file for <strong>building and pushing</strong> Docker images.</li><li>Ensure Cloud Build has permissions to read from <strong>Secret Manager</strong>.</li></ul><p>2️⃣ <strong>Configure Cloud Deploy &amp; Skaffold</strong></p><ul><li>Define <strong>release pipelines</strong> using Cloud Deploy.</li><li>Automate deployments with Skaffold, ensuring <strong>zero-downtime rollouts</strong>.</li></ul><p>3️⃣ <strong>Implement Health Checks &amp; Rollbacks</strong></p><ul><li>Set up <strong>Cloud Run health checks</strong> to verify deployment success.</li><li>Configure <strong>automatic rollback mechanisms</strong> using <strong>Cloud Deploy</strong>.</li></ul><p>4️⃣ <strong>Enforce IAM Security</strong></p><ul><li>Ensure <strong>Cloud IAM permissions</strong> restrict deployment access.</li><li>Only authorized GitLab pipelines should trigger <strong>Cloud Run updates</strong>.</li></ul><h3>Final Thoughts</h3><p>This <strong>end-to-end CI/CD pipeline</strong> provides a <strong>fully automated, secure, and rollback-friendly</strong> solution for deploying applications to <strong>Google Cloud Run</strong>. By integrating <strong>GitLab, Cloud Build, Artifact Registry, Cloud Deploy, and Cloud Run</strong>, developers can focus on coding while the pipeline handles <strong>deployment reliability and security</strong>.</p><p>🔹 <strong>Stay tuned for the next blog by </strong><a href="https://medium.com/u/d03093f38f8b"><strong>Prajeshharikanth</strong></a><strong> : </strong><a href="https://medium.com/@prajeshharikanth05/implementing-cloud-run-deployments-a-ci-cd-pipeline-with-automated-rollbacks-using-cloud-deploy-8f5b569b404d"><strong>Implementing Cloud Run Deployments: A CI/CD Pipeline with Automated Rollbacks using Cloud Deploy</strong></a> , where we dive into the actual <strong>implementation of Cloud Build, Cloud Deploy, and Skaffold setup!</strong> 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09c7f007bb38" width="1" height="1" alt=""><hr><p><a href="https://medium.com/niveus-solutions/end-to-end-ci-cd-pipeline-for-cloud-run-using-gitlab-cloud-build-and-cloud-deploy-09c7f007bb38">End-to-End CI/CD Pipeline for Cloud Run using GitLab, Cloud Build, and Cloud Deploy</a> was originally published in <a href="https://medium.com/niveus-solutions">Niveus Solutions Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>