<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Ankercloud Engineering - Medium]]></title>
        <description><![CDATA[Cloud | Bigdata | AI - Medium]]></description>
        <link>https://medium.com/ankercloud-engineering?source=rss----374303d0a301---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Ankercloud Engineering - Medium</title>
            <link>https://medium.com/ankercloud-engineering?source=rss----374303d0a301---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:07:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/ankercloud-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Event-Driven Data Transfer: Syncing AWS S3 with Google Cloud Storage]]></title>
            <link>https://medium.com/ankercloud-engineering/event-driven-data-transfer-syncing-aws-s3-with-google-cloud-storage-d957d1a66f4a?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/d957d1a66f4a</guid>
            <category><![CDATA[live-sync]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[aws-s3-bucket]]></category>
            <category><![CDATA[storage-transfer-service]]></category>
            <dc:creator><![CDATA[Anand Manivannan]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 06:58:45 GMT</pubDate>
            <atom:updated>2025-09-29T06:43:54.436Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PIM23QKD3PNuDyO9WnsihA.png" /></figure><p>In today’s multi-cloud environments, moving data seamlessly across platforms is essential. One common requirement is to replicate or migrate files from AWS S3 into Google Cloud Storage (GCS). Instead of running scheduled batch jobs, we can make this process event-driven, so new objects in S3 are immediately transferred into GCS. This approach reduces lag, improves efficiency, and ensures real-time availability of data for applications running on Google Cloud.</p><p>In this blog, we’ll walk through how to configure AWS SQS, link it with S3 bucket notifications, and then set up a Google Cloud Storage Transfer Service job that consumes those events.</p><h3>Step 1: Create an SQS Queue in AWS</h3><p>We’ll first create an SQS queue in AWS that will receive event notifications from S3 whenever a new object is created.</p><p>Steps:</p><p>1. In the AWS Console, navigate to **SQS → Create Queue**.</p><p>2. Provide a **name** for your queue.</p><p>3. Configure access policies so that your S3 bucket can publish events to the queue.</p><p>Here’s a sample IAM policy JSON (replace placeholders with your values):</p><pre>{<br>&quot;Version&quot;: &quot;2012–10–17&quot;,<br>&quot;Id&quot;: &quot;example-ID&quot;,<br>&quot;Statement&quot;: [<br>{<br>&quot;Sid&quot;: &quot;example-statement-ID&quot;,<br>&quot;Effect&quot;: &quot;Allow&quot;,<br>&quot;Principal&quot;: {<br>&quot;Service&quot;: &quot;s3.amazonaws.com&quot;<br>},<br>&quot;Action&quot;: &quot;SQS:SendMessage&quot;,<br>&quot;Resource&quot;: &quot;RESOURCE&quot;,<br>&quot;Condition&quot;: {<br>&quot;StringEquals&quot;: {<br>&quot;aws:SourceAccount&quot;: &quot;AWS&quot;<br>},<br>&quot;ArnLike&quot;: {<br>&quot;aws:SourceArn&quot;: &quot;S3_BUCKET_ARN&quot;<br>}<br>}<br>}<br>]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6iJVm2l86nhD3tKnDaIww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eVBXDiXduhDiKtSZBtG-oQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nu0PCAe2s0RZkV8ephU1XQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zMRoj2DZ6HazVS2zSGqSIA.png" /></figure><p>The values of the placeholders in the preceding JSON use the following format:</p><ul><li>AWS is a numeric value representing your Amazon Web Services project. For example, “aws:SourceAccount”: “1234567890”.</li><li>RESOURCE is an Amazon Resource Number (ARN) that identifies this queue. For example, “Resource”: “arn:aws:sqs:us-west-2:01234567890:test”.</li><li>S3_BUCKET_ARN is an ARN that identifies the source bucket. For example, “aws:SourceArn”: “arn:aws:s3:::example-aws-bucket”. You can find a bucket’s ARN from the <strong>Properties</strong> tab of the bucket details page in the AWS console.</li></ul><p>JSON file Document for reference:</p><p><a href="https://cloud.google.com/storage-transfer/docs/event-driven-aws#create_an_sqs_queue">https://cloud.google.com/storage-transfer/docs/event-driven-aws#create_an_sqs_queue</a></p><h3>Step 2: Configure S3 Event Notifications</h3><p>Now, connect the S3 bucket to the SQS queue so that events (e.g., object creation) are pushed to SQS.</p><p>Steps:</p><p>1. Open your source S3 bucket in the AWS Console.</p><p>2. Navigate to **Properties → Event Notifications → Create Event Notification**.</p><p>3. Provide a **name** for the event (e.g., create).</p><p>4. Select **All object create events**.</p><p>5. For **Destination**, choose the SQS queue created in Step 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vF5wqtaG-YJiLWo_ktj-Og.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SSbVhU-NBJIWPh59vY3oYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5eodpU6-Ze_5YfonDisKLw.png" /></figure><h3>Step 3: Create a GCS Bucket</h3><p>In Google Cloud, we need a target bucket where the data will be synced.</p><p>Steps:</p><p>1. Go to **Cloud Storage → Buckets → Create**.</p><p>2. Provide details such as:</p><p>- Bucket Name</p><p>- Region</p><p>- Storage Class (e.g., Standard)</p><p>3. Click **Create**.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2L2q6rYVB2Ea689O9cQLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pS-Ck6qANVrGCQHkzqASzQ.png" /></figure><h3>Step 4: Create an Event-Driven Transfer Job in GCP</h3><p>Now we link AWS SQS and GCS using Google’s Storage Transfer Service.</p><p>Steps:</p><p>1. In the GCP Console, navigate to **Storage Transfer Service → Create Transfer Job**.</p><p>2. Select **Event-driven transfer**.</p><p>3. Enter the **Amazon SQS Queue ARN** from Step 1.</p><p>4. Choose the GCS bucket created in Step 3 as the **destination**.</p><p>5. Define scheduling preferences and click **Create**.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31IUbUU_c78NB9ZC5sj0bA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mht9kQnZ25zwISurtlVyTQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*Dr2XRjp9IkODNBEQQ81Aaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aoNFMM_BEc22lrzl2lqSHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v09wXZC5Ky4tq-2YYV4G-w.png" /></figure><h3>Validation</h3><p>Once the setup is complete, the Storage Transfer Service will automatically sync objects whenever they are added to the S3 bucket. You can validate by uploading a file to your S3 bucket and checking if it appears in the destination GCS bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCc-TGhoOWzANb-WGdhDWw.png" /></figure><h3>Conclusion</h3><p>By integrating AWS SQS with Google Cloud Storage Transfer Service, you can achieve a seamless, event-driven pipeline for cross-cloud data synchronisation. This setup is efficient, scalable, and ideal for real-time workloads that depend on up-to-date data replication between AWS and Google Cloud.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d957d1a66f4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/event-driven-data-transfer-syncing-aws-s3-with-google-cloud-storage-d957d1a66f4a">Event-Driven Data Transfer: Syncing AWS S3 with Google Cloud Storage</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing/Securing Internal and External Traffic with GKE Gateway API]]></title>
            <link>https://medium.com/ankercloud-engineering/designing-securing-internal-and-external-traffic-with-gke-gateway-api-c224b843a5c9?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/c224b843a5c9</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernates]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Bharath]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 06:58:40 GMT</pubDate>
            <atom:updated>2026-03-30T06:58:39.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqJcWZ73eMxrjcncZTuUOA.png" /></figure><h3>Introduction</h3><p>As Kubernetes workloads scale in enterprise environments, managing <strong>secure, flexible, and maintainable ingress traffic</strong> becomes increasingly important. Traditional Ingress controllers often fall short when dealing with:</p><ul><li>Separation of public and internal traffic</li><li>IP-based access control for corporate users</li><li>Centralized TLS management</li><li>Fine-grained HTTP routing and health checks</li></ul><p>Google Kubernetes Engine (GKE) <strong>Gateway API</strong> addresses these challenges by providing a <strong>modern, role-oriented traffic management model</strong> deeply integrated with Google Cloud Load Balancers and Cloud Armor.</p><p>This blog walks through a <strong>production-grade GKE Gateway API architecture</strong> supporting both <strong>external (corporate/public)</strong> and <strong>internal-only</strong> traffic, following enterprise security best practices.</p><h3>Architecture Overview</h3><h3>Traffic Flow</h3><ol><li><strong>Public / Corporate Traffic</strong></li></ol><pre>Internet / Corporate Network<br>↓<br>Public Gateway (public-gateway)<br>- External HTTP(S) Load Balancer<br>- Cloud Armor (IP Whitelisting)<br>↓<br>Backend Service (corp-backend-service)<br>↓<br>GKE Pods (Application Pods)</pre><p><strong>2. Internal Traffic</strong></p><pre>Internal Network / VPC<br>↓<br>Internal Gateway (internal-gateway) - Internal HTTP(S) Load Balancer<br>↓<br>Kubernetes Service<br>↓<br>GKE Pods</pre><p>This architecture enforces <strong>clear trust boundaries</strong>, ensuring only intended traffic reaches application workloads.</p><h3>1. Why GKE Gateway API?</h3><p>The Gateway API enables platform teams to:</p><ul><li>Expose services using <strong>Google-managed load balancers</strong></li><li>Strictly separate <strong>external and internal ingress</strong></li><li>Apply <strong>Cloud Armor policies</strong> at the backend level</li><li>Delegate routing ownership via HTTPRoute</li><li>Manage <strong>TLS, health checks, and routing declaratively</strong></li></ul><p>Compared to legacy Ingress, Gateway API offers <strong>better scalability, clearer ownership, and future-proof design</strong>.</p><h3>2. Enable Gateway API in GKE</h3><p>Enable the required API before deploying any Gateway resources:</p><pre>gcloud services enable gateway.googleapis.com project &lt;project-id&gt;</pre><p>This enables support for:</p><ul><li>Gateway</li><li>HTTPRoute</li><li>GCPBackendPolicy</li><li>HealthCheckPolicy</li></ul><h3>3. Gateway Deployment Strategy</h3><p>Two gateways are deployed in a shared infrastructure namespace.</p><h3>3.1 Public Gateway (Corporate Access)</h3><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: Gateway<br>metadata:<br>  name: public-gateway<br>  namespace: infrastructure<br>  annotations:<br>    networking.gke.io/certmap: gcp-certmap-prod<br>spec:</pre><p># Listener and LB configuration managed via Helm or Gateway addon</p><p><strong>Key Characteristics</strong></p><ul><li>External HTTP(S) Load Balancer</li><li>Cloud Armor supported</li><li>TLS managed via <strong>GCP CertMap</strong></li><li>Used for corporate or controlled public access</li></ul><h3>3.2 Internal Gateway</h3><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: Gateway<br>  metadata:<br>  name: internal-gateway<br>  namespace: infrastructure<br>spec:</pre><p># Internal LB configuration via Gateway addon</p><p><strong>Key Characteristics</strong></p><ul><li>Internal HTTP(S) Load Balancer</li><li>Accessible only within the VPC or Shared VPC</li><li>Cloud Armor <strong>not supported</strong></li><li>Ideal for internal services and east-west traffic</li></ul><h3>4. Backend Service Definition</h3><p>Each application exposes a Kubernetes Service that becomes a <strong>GCP backend service</strong>.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: corp-backend-service<br>  namespace: apps<br>spec:<br>  selector:<br>    app: sample-app<br>  ports:<br>    - port: 80<br>      targetPort: 8080</pre><p>This service is referenced by:</p><ul><li>HTTPRoute</li><li>GCPBackendPolicy</li><li>Health check policies</li></ul><h3>5. Securing External Access with Cloud Armor</h3><p>Cloud Armor is applied <strong>only to external gateways</strong> using GCPBackendPolicy.</p><h3>Cloud Armor Strategy</h3><ol><li>Deny all traffic by default</li><li>Allow only approved corporate IP ranges</li><li>Attach the policy to the backend service</li></ol><h3>GCPBackendPolicy Example</h3><pre>apiVersion: networking.gke.io/v1<br>kind: GCPBackendPolicy<br>metadata:<br>  name: corp-backend-ip-whitelist<br>  namespace: apps<br>spec:<br>  default:<br>    securityPolicy: corp-ip-whitelist-policy<br>  targetRef:<br>    kind: Service<br>    name: corp-backend-service</pre><p>Ensures only authorized networks can access the application.</p><h3>6. HTTP Routing with HTTPRoute</h3><p>HTTPRoute defines hostname- and path-based routing and is typically managed via Helm by application teams.</p><h3>Public Gateway Route Example</h3><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: HTTPRoute<br>metadata:<br>  name: app-public-corp-route<br>  namespace: apps<br>spec:<br>  parentRefs:<br>    - name: public-gateway<br>      namespace: infrastructure<br>  hostnames:<br>    - app.example.corp<br>  rules:<br>    - matches:<br>        - path:<br>            type: PathPrefix<br>            value: /<br>      backendRefs:<br>        - name: corp-backend-service<br>          port: 80</pre><p><strong>Benefits</strong></p><ul><li>Clean domain-based routing</li><li>No Gateway changes required for app updates</li><li>Supports multi-team ownership</li></ul><h3>7. Health Check Management</h3><p>Health checks are configured using:</p><ul><li>GCPBackendPolicy</li><li>HealthCheckPolicy via Helm</li></ul><p>This ensures:</p><ul><li>Traffic is routed only to healthy pods</li><li>Custom readiness endpoints (e.g., /healthz)</li><li>Independent health checks per service</li></ul><h3>8. Firewall Rules for Internal Load Balancers</h3><p>Internal Gateways require <strong>explicit firewall rules</strong>.</p><h3>Firewall Rule Configuration</h3><ul><li><strong>Name:</strong> proxy-lb-backend</li><li><strong>Network:</strong> Shared VPC (Host Project)</li><li><strong>Allowed Ports:</strong> 80, 3000, 8080</li><li><strong>Source Ranges:</strong> Approved internal networks</li></ul><h3>Example Source Subnet Ranges</h3><p>10.150.64.0/20, 10.150.128.0/18</p><p>This prevents unintended exposure of internal services.</p><h3>9. TLS / SSL Configuration in Gateway API</h3><h3>External Gateway — CertMap (Recommended)</h3><pre>metadata:<br>  annotations:<br>    networking.gke.io/certmap: gcp-certmap-prod</pre><p>✔ Centralized certificate management<br> ✔ GCP-managed lifecycle<br> ✔ External gateways only</p><h3>Internal Gateway — Listener-Level TLS</h3><pre>- name: app-gateway-https<br>  protocol: HTTPS<br>  port: 443<br>  tls:<br>    mode: Terminate<br>    options:<br>      networking.gke.io/cert-manager-certs: internal-app-cert<br>  allowedRoutes:<br>    namespaces:<br>      from: All</pre><p>✔ Supported for internal gateways<br> ✔ Also valid for external gateways<br> ✔ Ideal for private PKI and mTLS</p><h3>10. Deployment Flow</h3><ol><li>Enable Gateway API</li><li>Deploy public and internal gateways</li><li>Create backend services</li><li>Apply Cloud Armor policies (external only)</li><li>Deploy HTTPRoutes via Helm</li><li>Configure health checks</li><li>Add firewall rules for ILB</li><li>Validate TLS, routing, and access controls</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c224b843a5c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/designing-securing-internal-and-external-traffic-with-gke-gateway-api-c224b843a5c9">Designing/Securing Internal and External Traffic with GKE Gateway API</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Cloud NGFW: Why It’s Better Than Traditional Firewall Rules and Differences]]></title>
            <link>https://medium.com/ankercloud-engineering/google-cloud-ngfw-why-its-better-than-traditional-firewall-rules-and-differences-9f7293c375cc?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/9f7293c375cc</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Bharath]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 06:58:31 GMT</pubDate>
            <atom:updated>2026-03-30T06:58:29.807Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RFwcJsKLEA-055SqHj93A.png" /></figure><h3>Google Cloud NGFW: Why It’s Better Than Traditional Firewall Rules and Differences</h3><p>Security in Google Cloud has evolved far beyond basic VPC firewall rules. As workloads become more dynamic and distributed, <strong>Google Cloud Next-Generation Firewall (NGFW)</strong> provides a modern, scalable, and context-aware approach to network security.</p><p>In this blog, we’ll cover:</p><ul><li>What is Google Cloud NGFW</li><li>Why NGFW Is Better Than Traditional Firewall Rules</li><li>NGFW vs Traditional GCP Firewall Rules</li><li>Where to Use GCP NGFW</li><li>Create Secure Tags and Attach Them to Network &amp; VM (Blocking websites, regions, and ports using NGFW)</li><li>Why Secure Tags + NGFW Is Powerful</li></ul><h3>What is Google Cloud NGFW?</h3><p>Google Cloud NGFW (Next-Generation Firewall) is an advanced firewall service that integrates <strong>Layer 7 inspection, threat intelligence, and identity-based enforcement</strong> into GCP networking.</p><p>Unlike traditional firewall rules that rely only on <strong>IP addresses and ports</strong>, NGFW allows you to enforce security based on:</p><ul><li>Application awareness</li><li>Domain names (FQDN)</li><li>Geographic locations</li><li>Secure tags (identity-based targeting)</li><li>Advanced threat protection</li></ul><h3>Why NGFW Is Better Than Traditional Firewall Rules</h3><p>Traditional VPC firewall rules work well for simple allow/deny traffic control, but they struggle in modern cloud environments where:</p><ul><li>IPs change frequently</li><li>Workloads scale dynamically</li><li>Security must follow <strong>workloads, not networks</strong></li></ul><h3>Key Advantages of NGFW</h3><ul><li><strong>Identity-based security</strong> instead of IP-based rules</li><li><strong>Layer 7 (application-level) filtering</strong></li><li><strong>Domain and URL filtering</strong></li><li><strong>Geo-based traffic control</strong></li><li><strong>Centralized security policy management</strong></li><li><strong>Better auditability and compliance</strong></li></ul><p>In short, NGFW provides <strong>security that moves with your workload</strong>, not static infrastructure.</p><h3>NGFW vs Traditional GCP Firewall Rules</h3><p><strong>Feature</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dINsJOnw0UvrsWFyMh5NFw.png" /></figure><h3>Where to Use GCP NGFW</h3><p>NGFW is ideal when:</p><ul><li>You run <strong>internet-facing applications</strong></li><li>You want to <strong>block access to malicious or unwanted websites</strong></li><li>You need <strong>region-based traffic restrictions</strong></li><li>You operate <strong>multi-project or multi-VPC architectures</strong></li><li>You want <strong>zero-trust or identity-based security</strong></li><li>Compliance requires <strong>advanced logging and inspection</strong></li></ul><p>Typical use cases:</p><ul><li>Production workloads</li><li>Financial and healthcare applications</li><li>SaaS platforms</li><li>Enterprises with strict security policies</li></ul><h3>What Are Secure Tags?</h3><p><strong>Secure Tags</strong> are identity labels used by NGFW to target resources like:</p><ul><li>VMs</li><li>Subnets</li><li>Networks</li></ul><p>Unlike network tags, secure tags are:</p><ul><li><strong>IAM-controlled</strong></li><li><strong>More secure</strong></li><li><strong>Consistent across projects</strong></li></ul><p>They allow you to apply firewall policies based on <strong>what a resource is</strong>, not where it lives.</p><h3>Create Secure Tags and Attach Them to Network &amp; VM</h3><p>Secure Tags act as <strong>selectors</strong> in NGFW firewall policies.</p><p>Example:</p><p>Tag: env=prod</p><p>Policy:</p><p>Block SSH, restrict outbound internet, allow only approved domains</p><p>Any resource (VM or network) with env=prod automatically inherits the policy — <strong>no IP updates needed</strong>.</p><h3>Step 1: Enable Required APIs</h3><p>Before creating Secure Tags, ensure the following APIs are enabled:</p><ol><li>Go to <strong>GCP Console</strong></li><li>Navigate to <strong>APIs &amp; Services → Library</strong></li><li>Enable:</li></ol><ul><li><strong>Resource Manager API</strong></li><li><strong>Compute Engine API</strong></li><li><strong>Network Security API</strong></li></ul><h3>Step 2: Create a Secure Tag Key</h3><ol><li>Open <strong>GCP Console</strong></li><li>Go to <strong>IAM &amp; Admin → Tags</strong></li><li>Click <strong>Create Tag Key</strong></li><li>Fill in:</li></ol><ul><li><strong>Tag Key name</strong>: env</li><li><strong>Parent</strong>: Select your <strong>Project</strong> or <strong>Organization</strong></li></ul><p>Click <strong>Create</strong></p><p>The Tag Key represents the <strong>category</strong>, similar to a label name.</p><h3>Step 3: Create a Secure Tag Value</h3><ol><li>Under the newly created Tag Key (env)</li><li>Click <strong>Create Tag Value</strong></li><li>Enter:</li></ol><ul><li><strong>Tag Value</strong>: prod</li></ul><ol><li>Click <strong>Create</strong></li></ol><p>You can create multiple values under the same key:</p><ul><li>env=prod</li><li>env=staging</li><li>env=dev</li></ul><h3>Step 4: Attach Secure Tag to a VPC Network</h3><h3>Why Tag a Network?</h3><p>Tagging a VPC allows <strong>all workloads inside it</strong> to inherit NGFW policies automatically.</p><h3>Steps:</h3><ol><li>Go to <strong>VPC network → VPC networks</strong></li><li>Select your <strong>VPC Network</strong></li><li>Click <strong>Edit</strong></li><li>Scroll to <strong>Tags (Secure Tags section)</strong></li><li>Click <strong>Add Tag</strong></li><li>Select:</li></ol><ul><li><strong>Key</strong>: env</li><li><strong>Value</strong>: prod</li></ul><p>Save changes</p><p>The entire network is now tagged and ready for NGFW enforcement.</p><h3>Step 5: Attach Secure Tag to a VM Instance</h3><h3>When to Tag a VM?</h3><ul><li>Fine-grained control</li><li>Different security policies within the same VPC</li></ul><h3>Steps:</h3><ol><li>Go to <strong>Compute Engine → VM instances</strong></li><li>Select the VM</li><li>Click <strong>Edit</strong></li><li>Scroll to the <strong>Secure Tags</strong> section</li><li>Click <strong>Add Tag</strong></li><li>Choose:</li></ol><ul><li><strong>env = prod</strong></li></ul><p>Save</p><p>This VM can now be targeted directly by NGFW policies.</p><h3>Why Secure Tags + NGFW Is Powerful</h3><p>By combining <strong>Secure Tags</strong> and <strong>NGFW</strong>, you get:</p><ul><li>Firewall rules that follow workloads automatically</li><li>Zero dependency on IP addresses</li><li>Cleaner and reusable security policies</li><li>Better separation of security and infrastructure teams</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f7293c375cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/google-cloud-ngfw-why-its-better-than-traditional-firewall-rules-and-differences-9f7293c375cc">Google Cloud NGFW: Why It’s Better Than Traditional Firewall Rules and Differences</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure Your Kubernetes Secrets with Sealed Secrets: A Step-by-Step Guide]]></title>
            <link>https://medium.com/ankercloud-engineering/secure-your-kubernetes-secrets-with-sealed-secrets-a-step-by-step-guide-3c1b6d0179e3?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c1b6d0179e3</guid>
            <dc:creator><![CDATA[Hrishikesh Nangare]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 05:03:01 GMT</pubDate>
            <atom:updated>2025-09-17T05:02:22.305Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPCjZR1jqbmNo1w-CyhMgA.png" /></figure><h3>The Problem: Kubernetes Secrets Security Risk</h3><p>In Kubernetes-based workflows, especially those embracing GitOps, it’s common to define infrastructure and application configurations in version-controlled repositories. This is great for reproducibility and automation — <strong>until you hit the problem of secrets</strong>.</p><p>You might be tempted to store Kubernetes Secret manifests in your Git repository, but these are only base64-encoded — <strong>not encrypted</strong>. Anyone with access to the repo can easily decode sensitive credentials like passwords, API tokens, or TLS keys. That’s a serious security concern.</p><h3>The Solution: Sealed Secrets</h3><p><a href="https://github.com/bitnami-labs/sealed-secrets">Sealed Secrets</a> by Bitnami offers a solution that’s simple, secure, and Git-friendly.</p><p>It lets you encrypt your secrets into a <strong>SealedSecret</strong> — a custom Kubernetes resource that <strong>can be safely stored in Git</strong>, because only your Kubernetes cluster can decrypt it. The actual Kubernetes Secret is generated from the SealedSecret <strong>inside the cluster</strong> using a controller.</p><h3>Step-by-Step Setup: Sealed Secrets</h3><h4>Install the Sealed Secrets Controller in Your Cluster</h4><p>This controller will decrypt Sealed Secrets into Kubernetes Secrets.</p><blockquote>Using Helm<strong> (</strong>recommended<strong>)</strong></blockquote><pre>helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets<br>helm repo update<br><br>helm install sealed-secrets sealed-secrets/sealed-secrets \<br>  --namespace kube-system \<br>  --create-namespace</pre><blockquote>Alternatively, install using manifests</blockquote><pre>kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.4/controller.yaml</pre><blockquote>Install the kubeseal CLI</blockquote><p>This tool is used to encrypt your secrets locally before committing them to Git.</p><p><strong>For macOS (Homebrew)</strong></p><pre>brew install kubeseal</pre><p><strong>For Linux</strong></p><pre>wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.4/kubeseal-0.24.4-linux-amd64.tar.gz<br>tar -xvzf kubeseal-0.24.4-linux-amd64.tar.gz<br>sudo install kubeseal /usr/local/bin/</pre><h4>Create a Kubernetes Secret (Local or Temporary)</h4><p>Start with a normal Kubernetes Secret file. So first, create a Kubernetes secret file with encoded values</p><pre># secret.yaml<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: my-secret<br>  namespace: default<br>type: Opaque<br>data:<br>  username: bXl1c2Vy<br>  password: bXlwYXNzd29yZA==</pre><p>You can use echo -n &quot;myuser&quot; | base64 to encode values.</p><p><strong>Seal the Secret Using </strong>kubeseal</p><p>You encrypt the secret using the public key from the controller.<br>Now, using kubeseal and the above secret file, we create sealsecret <br>Run the following command to create sealsecret file</p><pre>kubeseal --format=yaml &lt; secret.yaml &gt; sealed-secret.yaml</pre><p>Now, sealed-secret.yaml is created and safe to store</p><h4>Apply the Sealed Secret to the Cluster</h4><p>Once the Sealed Secret is applied, the controller will decrypt it and create a standard Secret.</p><pre>kubectl apply -f sealed-secret.yaml</pre><p>Check if it created the Secret:</p><pre>kubectl get secrets</pre><pre>kubectl get sealsecret </pre><p>You will see the sealsecret was created, and sealsecret will create the kubernets secret.</p><h3>Best Practices for Using Sealed Secrets</h3><h4>✅ Store Only SealedSecrets in Git</h4><ul><li><strong>Do:</strong> Commit only SealedSecret YAMLs to Git.</li><li><strong>Don’t:</strong> Commit raw Secret manifests — even base64-encoded.</li></ul><h4>🔁 Rotate Sealing Keys Securely</h4><ul><li>Rotate the sealing key every 3–6 months.</li><li>Back up the controller’s private key securely:</li></ul><pre>kubectl get secret -n kube-system sealed-secrets-key -o yaml &gt; backup.yaml</pre><h4>📍 Namespace-Scoped vs Cluster-Wide Sealing</h4><ul><li><strong>Namespace-scoped (</strong><strong>--scope namespace-wide)</strong>: Default and more secure. Secret only decrypts in a specific namespace.</li><li><strong>Cluster-wide (</strong><strong>--scope cluster-wide)</strong>: More flexible but less secure — avoid unless necessary.</li></ul><h4>📁 Organize Secrets by Environment</h4><p>Structure your repo like this:</p><pre>environments/<br>├── dev/<br>│   └── sealed-secret-db.yaml<br>├── staging/<br>│   └── sealed-secret-db.yaml<br>└── prod/<br>    └── sealed-secret-db.yaml</pre><p>Avoid deploying dev secrets to prod by mistake.</p><h4>🧪 Validate Before Committing</h4><p>Use --validate to ensure the SealedSecret is still valid and matches the controller&#39;s current key:</p><pre>kubeseal --validate --sealed-secret-file sealed-secret.yaml</pre><h3>Conclusion</h3><p>Sealed Secrets offer a secure, GitOps-friendly way to manage Kubernetes secrets without compromising on automation or safety. By encrypting secrets with a cluster-scoped controller and storing only the encrypted versions in Git, you gain the benefits of:</p><ul><li><strong>Version control</strong> for secrets</li><li><strong>Reduced risk</strong> of data exposure</li><li><strong>Seamless integration</strong> with CI/CD workflows</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c1b6d0179e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/secure-your-kubernetes-secrets-with-sealed-secrets-a-step-by-step-guide-3c1b6d0179e3">Secure Your Kubernetes Secrets with Sealed Secrets: A Step-by-Step Guide</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating AWS ↔ GCP VPN Setup with BGP Using Bash]]></title>
            <link>https://medium.com/ankercloud-engineering/automating-aws-gcp-vpn-setup-with-bgp-using-bash-8443086dbaaa?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/8443086dbaaa</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Bharath]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 06:57:50 GMT</pubDate>
            <atom:updated>2025-12-11T13:31:14.706Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-IBS08dHBOT7YzxEZbI5Q.png" /></figure><p>Hybrid cloud is becoming the new normal. Many enterprises run workloads across <strong>AWS and GCP</strong>, requiring secure and highly available connectivity between the two clouds. Traditionally, setting up a <strong>site-to-site VPN with BGP</strong> between AWS and GCP is a lengthy manual process.</p><p>In this blog, I’ll walk you through a <strong>Bash automation script</strong> that provisions:</p><ul><li>GCP VPN Gateways (HA VPN)</li><li>AWS Virtual Private Gateways (VGW) &amp; Customer Gateways (CGW)</li><li>IPsec tunnels (4 total) with pre-shared keys</li><li>Cloud Router + BGP sessions</li><li>AWS route propagation</li></ul><p>This script transforms hours of manual configuration into a <strong>fully automated cross-cloud VPN setup</strong>.</p><h3>Prerequisites Before Execution</h3><p>Before running the script, ensure the host machine is properly set up with the following requirements:</p><p><strong>Install AWS CLI &amp; GCP CLI</strong></p><ul><li>AWS CLI Installation Guide</li><li>Google Cloud SDK (gcloud) Installation Guide</li></ul><p><strong>Verify installation:</strong></p><p><em>aws — version</em></p><p><em>gcloud — version</em></p><p><strong>Configure AWS Credentials<br></strong> Set up your <strong>AWS Access Key and Secret Key</strong> with the desired region:</p><p><em>aws configure</em></p><p><strong>Activate GCP Service Account<br></strong> Ensure you have a GCP <strong>service account JSON key</strong> with sufficient IAM permissions (Compute Admin, Network Admin).<br> Activate it using:</p><p><em>gcloud auth activate-service-account &lt;sa-mail-id&gt; — key-file=&lt;path-to-service-account-key.json&gt;</em></p><p><em>gcloud config set project &lt;your-project-id&gt;</em></p><p><strong>Private Service Access connect range should be allocated</strong></p><p><strong>Download the Script<br></strong> Clone or download the automation script from GitHub (for example aws-gcp-vpn-setup.sh) to your host machine.</p><p><em>git clone </em><a href="https://github.com/bharathprime/aws-gcp.git"><em>https://github.com/bharathprime/aws-gcp.git</em></a></p><p><em>cd aws-gcp/aws_gcp_ha_tunnel</em></p><p><strong>Update the GCP Project ID in script</strong></p><p>Update the gcp project id with real project id of your account in script 1st line &lt;project_id=”gcp-project-id”&gt;</p><p><strong>Update Preshared key:</strong></p><p>Update the preshared key with unique string in the variable “PSK1, PSK2, PSK3 and PSK4”</p><p><strong>Set Execute Permission<br></strong> Grant execution permission to the script:</p><p><em>chmod +x aws-gcp-vpn-setup.sh</em></p><p><strong>Run the Script<br></strong> Execute the script:</p><p><em>./aws-gcp-vpn-setup.sh</em></p><h3>Script Workflow</h3><p>Here’s the breakdown of what the script automates.</p><h3>1. Project, Region &amp; VPC Selection (GCP Side)</h3><ul><li>Lists all available GCP regions and prompts user to pick one.</li><li>Lists all GCP VPCs and allows user to select the target VPC.</li><li>Checks if <strong>Private Service Connect (PSC) peering</strong> exists for the chosen VPC, then enables custom route exchange.</li></ul><p>This ensures that hybrid traffic can flow through PSC ranges.</p><h3>2. AWS VPC Selection</h3><ul><li>Lists all available AWS VPCs with Name, ID, and CIDR.</li><li>Prompts user to select the correct AWS VPC for VPN attachment.</li></ul><h3>3. Creating VPN Gateways</h3><ul><li>Creates a <strong>GCP HA VPN Gateway</strong> with two interfaces.</li><li>Retrieves external IPs of the VPN interfaces.</li></ul><p>On AWS side:</p><ul><li>Creates two <strong>Customer Gateways (CGWs)</strong> pointing to GCP interfaces.</li><li>Creates an <strong>AWS Virtual Private Gateway (VGW)</strong> with ASN 65001 and attaches it to the AWS VPC.</li></ul><h3>4. Creating VPN Connections &amp; Tunnels</h3><ul><li>Establishes <strong>two VPN connections on AWS</strong> (one per CGW).</li><li>Each connection has <strong>two tunnels</strong>, giving <strong>4 tunnels total</strong> for redundancy.</li><li>Uses predefined <strong>PreSharedKeys (PSKs)</strong> for security.</li></ul><h3>5. Configuring GCP External VPN Gateway</h3><ul><li>Creates an <strong>external VPN gateway in GCP</strong> representing AWS.</li><li>Defines <strong>four external interfaces</strong> (one per AWS tunnel).</li><li>Creates <strong>four VPN tunnels</strong> in GCP with IKEv2 and BGP enabled.</li></ul><h3>6. Cloud Router &amp; BGP Peering</h3><ul><li>Creates a <strong>Cloud Router</strong> with ASN 65534 in GCP.</li><li>Adds interfaces for each VPN tunnel.</li><li>Establishes <strong>BGP sessions</strong> with AWS ASN 65001 for all four tunnels.</li><li>Verifies tunnel and BGP session status.</li></ul><p>Note: Update the BGP ASN value or cidr range from script as per the requirments</p><h3>7. AWS Route Propagation</h3><ul><li>Retrieves the VGW ID attached to AWS VPC.</li><li>Enables <strong>VGW route propagation</strong> for all route tables in that VPC.</li></ul><p>This ensures AWS instances automatically learn GCP subnets via BGP.</p><h3>Why BGP with HA VPN?</h3><p>Using BGP with multiple tunnels provides:</p><ul><li><strong>High Availability</strong> — If one tunnel goes down, traffic shifts seamlessly to others.</li><li><strong>Dynamic Routing</strong> — No need to manually update routes. Subnets are advertised automatically.</li><li><strong>Scalability</strong> — New prefixes are propagated without changes in configuration.</li></ul><h3>End-to-End Flow</h3><ol><li>User selects <strong>region &amp; VPCs</strong>.</li><li>Script provisions <strong>HA VPN on GCP</strong> + <strong>VGW on AWS</strong>.</li><li>IPsec tunnels + BGP sessions established.</li><li>Routes propagated dynamically on both sides for subnets and PSA(private service access)</li></ol><p>The end result: a <strong>secure, redundant, and automated AWS–GCP hybrid network</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8443086dbaaa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/automating-aws-gcp-vpn-setup-with-bgp-using-bash-8443086dbaaa">Automating AWS ↔ GCP VPN Setup with BGP Using Bash</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Kubernetes Deployments with Argo CD]]></title>
            <link>https://medium.com/ankercloud-engineering/automating-kubernetes-deployments-with-argo-cd-51319226e600?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/51319226e600</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Bharath]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 04:42:42 GMT</pubDate>
            <atom:updated>2025-09-09T04:42:42.576Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zX0VurPA0JgYqDOB" /></figure><p>Argo CD is a declarative, GitOps-based continuous delivery tool designed for Kubernetes. It allows you to manage and automate application deployment using Git as the single source of truth. Argo CD continuously monitors your Git repository and ensures the Kubernetes environment matches the desired state described in your manifest.</p><h3>Step 1: Create and Connect to a Kubernetes Cluster</h3><h4>Steps to Create and Connect</h4><p><strong>Create a Kubernetes Cluster<br></strong>If you’re using Google Kubernetes Engine (GKE), you can create a cluster using the following command:</p><blockquote>gcloud container clusters create &lt;cluster name&gt; — zone &lt;zone of cluster&gt;</blockquote><p>Replace &lt;cluster name&gt; with your desired cluster name and &lt;zone of cluster&gt; with your preferred zone.</p><p><strong>Connect to the Cluster<br></strong>Once the cluster is created, configure kubectl (the Kubernetes CLI) to interact with it:</p><blockquote><em>gcloud container clusters get-credentials argo-test — zone us-central1-c</em></blockquote><p>Verify the connection by listing the nodes in the cluster:<br><em>kubectl get nodes</em></p><h3>Step 2: Install Argo CD</h3><p>Installing Argo CD means deploying its server, UI, and supporting components as Kubernetes resources in a namespace.</p><h4>Steps to Install</h4><p><strong>Create a Namespace for Argo CD<br></strong>A namespace in Kubernetes is a logical partition to organize resources:</p><blockquote><em>kubectl create namespace argocd</em></blockquote><p><strong>Install Argo CD Components<br></strong>Use the official installation manifest to deploy all Argo CD components:</p><blockquote><em>kubectl apply -n argocd -f </em><a href="https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml"><em>https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</em></a></blockquote><p>This deploys key components like the API server, repository server, application controller, and web UI.</p><h3>Step 3: Expose Argo CD Publicly</h3><p>By default, the argocd-server service is configured as a ClusterIP, making it accessible only within the cluster. You need to expose it for external access.</p><h4>Options to Expose Argo CD</h4><p><strong>Option-1 LoadBalancer<br></strong>Change the service type to LoadBalancer to get an external IP address:</p><blockquote><em>kubectl patch svc argocd-server -n argocd -p ‘{“spec”: {“type”: “LoadBalancer”}}’</em></blockquote><p><strong>Ingress<br></strong>For advanced routing and SSL support, create an Ingress resource. This approach is recommended if you want to add HTTPS to your setup.</p><p><strong>Option-2 Port Forwarding<br></strong>If you only need temporary access:</p><blockquote><em>kubectl port-forward svc/argocd-server -n argocd 8080:80</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AGxf6ErR2_xuXxFn" /></figure><h3>Step 4: Access the Argo CD Dashboard</h3><p><strong>Retrieve the External IP<br></strong>After exposing the service as a LoadBalancer, get the external IP address:</p><blockquote><em>kubectl get svc argocd-server -n argocd</em></blockquote><p><strong>Login Credentials</strong></p><p><strong>Username</strong>: admin</p><p><strong>Password</strong>: Retrieve it from the secret:</p><blockquote><em>kubectl get secret argocd-initial-admin-secret -n argocd -o yaml</em></blockquote><p>Decode the base64 password:</p><blockquote><em>echo “&lt;base64_encoded_password&gt;” | base64 — decode</em></blockquote><p>Access the dashboard by navigating to https://&lt;external-ip&gt; in your browser.</p><h3>Step 5: Install the Argo CD CLI</h3><p>The Argo CD CLI enables you to interact with the Argo CD server programmatically for managing clusters, applications, and configurations.</p><h4>Steps to Install</h4><p><strong>Download the CLI</strong></p><blockquote><em>curl -sSL -o argocd-linux-amd64 </em><a href="https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64"><em>https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64</em></a></blockquote><p><strong>Install the CLI</strong></p><blockquote><em>sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd</em></blockquote><blockquote><em>rm argocd-linux-amd64</em></blockquote><p><strong>Verify Installation</strong></p><blockquote><em>argocd version</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*UMFhOZ4LFpgRhYzM" /></figure><h3>Step 6: Add a Kubernetes Cluster to Argo CD</h3><p>Argo CD requires access to the Kubernetes cluster where it will deploy applications.</p><h4>Steps to Add</h4><p><strong>Log in to Argo CD via CLI</strong></p><blockquote><em>argocd login &lt;argocd-server-url&gt;:&lt;port&gt; — username admin — password &lt;password&gt;</em></blockquote><p><strong>Get the Kubernetes Context</strong></p><blockquote><em>kubectl config get-contexts -o name</em></blockquote><p><strong>Add the Cluster</strong></p><blockquote><em>argocd cluster add &lt;context-name&gt;</em></blockquote><p>This command creates a service account (argocd-manager) with cluster-wide permissions to deploy applications.</p><p>To Verify the added cluster via cli use below command else navigate to the ui dashboard setting -&gt; cluster</p><blockquote><em>argocd cluster list</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MnzUU69efRSz812N" /></figure><h3>Step 7: Add a Git Repository</h3><p>The Git repository serves as the source of truth for application manifests.</p><h4>Steps to Add</h4><ol><li><strong>Navigate to Repositories<br></strong>Log in to the Argo CD dashboard, go to Settings -&gt; Repositories, and click Connect Repo.</li><li><strong>Enter Repository Details</strong></li></ol><ul><li>Choose a connection method (e.g., HTTPS or SSH).</li><li>Provide the repository URL and credentials.</li><li>Assign a project to organize repositories.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m2YQ4GjJt1xRJ6rK" /></figure><h3>Step 8: Create an Application in Argo CD</h3><p>An Argo CD application represents the Kubernetes resources defined in a Git repository.</p><h4>Steps to Create</h4><p><strong>Click New App<br></strong>Enter the application details:</p><ul><li><strong>Application Name</strong>: e.g., hello-world</li><li><strong>Project</strong>: Assign the application to a project.</li><li><strong>Source</strong>: Select the Git repository and specify the manifest file path.</li><li><strong>Destination</strong>: Select the cluster and namespace for deployment.</li></ul><ol><li><strong>Enable Auto-Sync policy<br></strong>Enable this option for automated synchronization between the Git repository and the Kubernetes cluster.</li><li><strong>Create the Application<br></strong>Click Create. Argo CD will deploy the application and monitor its state.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-2LIH65tCImOCJoA" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DXltsJjeklObHiti" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XR5Dj_L7adAFO0u1" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51319226e600" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/automating-kubernetes-deployments-with-argo-cd-51319226e600">Automating Kubernetes Deployments with Argo CD</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow on Argo CD — a Step‑by‑Step GitOps Guide (with git‑sync for DAGs)]]></title>
            <link>https://medium.com/ankercloud-engineering/airflow-on-argo-cd-a-step-by-step-gitops-guide-with-git-sync-for-dags-464bd0d7e2f5?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/464bd0d7e2f5</guid>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[gda]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Prachi Tiwari]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 04:42:02 GMT</pubDate>
            <atom:updated>2025-09-09T04:41:46.205Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Airflow on Argo CD — a Step‑by‑Step GitOps Guide (with git‑sync for DAGs)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7fxL_d7XLzH4HIkLpyvtSQ.png" /></figure><p>Running Apache Airflow on Kubernetes can feel like a constant juggling act. Manual deployments are error-prone, configs drift out of sync, and even small changes often require too much babysitting. That’s where GitOps — and specifically ArgoCD — comes to the rescue.</p><p>With ArgoCD, your Airflow deployment becomes fully automated: configs live in Git, updates roll out in seconds, and you always know your cluster state matches what’s in version control. Combine it with Helm and git-sync, and you’ve got a rock-solid setup for keeping DAGs fresh without lifting a finger.</p><p>So lets walk through</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*y7yE-6rwcEufj8RJ.png" /></figure><p><strong>Architecture at a glance</strong></p><p>Git → Argo CD → Airflow→DAG’s <br>1)You commit Helm values and overlay manifests.<br>2)Argo CD reconciles the cluster to Git.<br>3)Airflow runs with the CeleryExecutor, backed by Postgres + Redis.<br>4)A git‑sync sidecar inside Airflow components pulls DAGs on a schedule (period: 5s).</p><p><strong>Prerequisites</strong></p><p><strong>1)</strong>A Kubernetes cluster and Argo CD installed<br>2)Official Airflow HELM Chart<br>3)Access to a container registry <br>4)A DAGs repo (here we use <a href="https://github.com/airflow-dags.git">https://github.com/airflow-dags.git</a>)<br>5)kubectl, helm, and (optionally) ArgoCD CLI</p><p><strong>Repo layout</strong></p><pre>airflow-helm/<br>├─ Chart.yaml<br>├─ values.yaml <br>├─ custom-values.yaml # &lt;— full config used by Argo CD<br>├─ templates/ <br>└─ charts/postgresql/ </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3e8mDdkUEWIfTN-jB_HH0g.png" /></figure><p>Optionally, add a small deploy overlay for secrets/PVCs:-<br>Note:-Explicit PVCs for workers/triggerer logs.</p><pre>overlays/prod/<br>├─ kustomization.yaml<br>├─ git-creds.yaml # Secret for git‑sync <br>├─ webserver-secret-key.yaml <br>├─ pvc-worker.yaml <br>└─ pvc-triggerer.yaml </pre><h3>kustomization.yaml</h3><pre>apiVersion: kustomize.config.k8s.io/v1beta1<br>kind: Kustomization<br><br>resources:<br>  - pvc-triggerer.yaml<br>  - pvc-worker.yaml<br>  - git-creds.yaml<br>  - webserver-secret-key.yaml</pre><p>airflow-webserver-secret-key.yaml</p><pre>apiVersion: bitnami.com/v1alpha1<br>kind: SealedSecret<br>metadata:<br>  creationTimestamp: null<br>  name: airflow-webserver-secret-key<br>  namespace: airflow<br>spec:<br>  encryptedData:<br>    webserver-secret-key: PjdkjyuevdhjshnmlkshjnhsbjCWifyY74bzXYe7iFtmsiClsV8ZCVyeziYeV1qE/M/CccFzaDXCAX06crHk49RGuJQUMCiNtq1GJYj4rnxndNwhU8F33vc5bSEYrw+izs5jZyZ1guD5x+SWlsXh+wu8l2ULR9zzc/Y3kAkUqMvVPHVK1/1v8Q3Ybqlmd51PKpC35LMhpJ/bgsptbVt+d0C1RFzJ7HYZKl/p6Ras5Jn7EqCroWa5nI043FoGae0erJqogGnvYvEc7wG7CkfpZPu+wLOZ7NmxtAd7gNeDn7NbHXqWC0aJaN/C+ROcVAuyR5+1H/Ip2j90wzuL1UsV+1fAkp8iuzD7g+XuZoAy9i8Liz5svEjd11yOg9AE1cnjFjVmzQ1kHyTbuLvnmJ5DgKbHF5fsQvtjJ1c/HuN8CuMpSTG4SCzqIaRQYOZXxDwcb5ut4vCrGoMCtlegAU0d4nj4w14nY4ztOqBqQ/v8z8L4wIM/FTd9+cO21wLvL+AYPWoQkl38nfMfZKxV34EUv7MLeUZ/iEKVYqmgkXCwrFNqmD8moXe566nWfti5O25piWlLMyI58fzIsKXNLLC+7IwSWCAHdDM5hngZgYwIiKOs827ywd7ElcTU6wpdl8T2vLuth/4qgcaJUKb/g5Z9NLdCmgCSLyMiq9GaulKBwYFKTQb1Rtr+l17Q3lwV/Hk8aXFZzmtX+xgHe6KWSi771RI7C9X2MQMTW0Ikq5xW71uxALo=<br>  template:<br>    metadata:<br>      creationTimestamp: null<br>      name: airflow-webserver-secret-key<br>      namespace: airflow<br>    type: Opaque</pre><p>pvc-triggerer.yaml</p><pre><br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: airflow-pvc-triggerer<br>  namespace: airflow<br>  annotations:<br>    argocd.argoproj.io/sync-wave: &quot;-1&quot;<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 5Gi<br>  storageClassName: airflow</pre><p>Note:-We have to create similar file like triggerer for <strong>worker.yaml</strong></p><p>Now all the config setup is here in custom-values.yaml</p><pre>defaultAirflowRepository: apache/airflow<br>defaultAirflowTag: 2.9.2<br><br>postgresql:<br>  enabled: false<br><br>externalDatabase:<br>  type: postgresql<br>  host: airflow-postgres.example.com<br>  port: 5432<br>  database: airflow<br>  userSecretName: airflow-postgres-secret<br>  passwordSecretName: airflow-postgres-secret<br>  userSecretKey: username<br>  passwordSecretKey: password<br><br>redis:<br>  enabled: false<br><br>externalRedis:<br>  host: airflow-redis.example.com<br>  port: 6379<br>  passwordSecretName: airflow-redis-secret<br>  passwordSecretKey: password<br><br>data:<br>  metadataConnection:<br>    userSecretName: airflow-postgres-secret<br>    passSecretName: airflow-postgres-secret<br>    userSecretKey: username<br>    passSecretKey: password<br>    protocol: postgresql<br>    host: airflow-postgres.example.com<br>    port: 5432<br>    db: airflow<br>    sslmode: require<br>  brokerUrlSecretName: airflow-redis-secret<br>  brokerUrlSecretKey: brokerUrl<br><br>fernetKeySecretName: &quot;airflow-fernet-key&quot;<br>webserverSecretKeySecretName: &quot;airflow-webserver-secret-key&quot;<br><br>webserver:<br>  enabled: true<br>  defaultUser:<br>    enabled: true<br>    secretName: airflow-admin-user<br>    usernameKey: username<br>    passwordKey: password<br><br>ingress:<br>  enabled: true<br>  web:<br>    hosts:<br>      - airflow.example.com<br>    tls:<br>      - secretName: airflow-tls<br>        hosts:<br>          - airflow.example.com<br>  api:<br>    enabled: true<br>    hosts:<br>      - airflow-api.example.com<br>    tls:<br>      - secretName: airflow-api-tls<br>        hosts:<br>          - airflow-api.example.com<br><br>executor: CeleryExecutor<br><br>workers:<br>  replicas: 3<br>  resources:<br>    requests:<br>      cpu: 500m<br>      memory: 1Gi<br>    limits:<br>      cpu: 2<br>      memory: 4Gi<br>  persistence:<br>    enabled: true<br>    existingClaim: airflow-pvc-worker<br>  autoscaling:<br>    enabled: true<br>    minReplicas: 3<br>    maxReplicas: 10<br>    targetCPUUtilizationPercentage: 75<br><br>scheduler:<br>  resources:<br>    requests:<br>      cpu: 500m<br>      memory: 1Gi<br>    limits:<br>      cpu: 1<br>      memory: 2Gi<br><br>triggerer:<br>  replicas: 2<br>  resources:<br>    requests:<br>      cpu: 200m<br>      memory: 512Mi<br>    limits:<br>      cpu: 500m<br>      memory: 1Gi<br>  logs:<br>    persistence:<br>      enabled: true<br>      existingClaim: airflow-pvc-triggerer<br><br>dags:<br>  gitSync:<br>    enabled: true<br>    repo: https://github.com/org/airflow-dags.git<br>    branch: main<br>    ref: main<br>    credentialsSecret: git-credentials<br>    subPath: dags<br>    period: 60s<br><br>metrics:<br>  enabled: true<br>  statsd:<br>    enabled: true<br>  prometheusRule:<br>    enabled: true<br><br>createUserJob:<br>  useHelmHooks: false<br>migrateDatabaseJob:<br>  useHelmHooks: false<br>  jobAnnotations:<br>    &quot;argocd.argoproj.io/hook&quot;: Sync</pre><p>You can adjust this custom-values.yaml according to your requirements.<br>Now create all the secrets and apply this kustomization.yaml</p><p><strong>Create the Argo CD Application</strong></p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>name: airflow-helm<br>namespace: argocd<br>spec:<br>project: default<br>source:<br>repoURL: https://your.git/airflow-helm.git<br>targetRevision: master<br>path: .<br>helm:<br>valueFiles:<br>- custom-values.yaml<br>destination:<br>server: https://kubernetes.default.svc<br>namespace: airflow<br>syncPolicy:<br>automated: {} # optional; enable if you want continuous sync<br>syncOptions:<br>- CreateNamespace=true</pre><p>Now apply this</p><pre>kubectl apply -f argo-airflow-app.yaml</pre><p>now you can see that in your gitbash</p><pre>kubectl get pods -n airflow</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*0di8B-9KHutMKy8wpwxyQw.png" /></figure><p>Add a test DAG (git‑sync flow)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63FEtpTxlpD1AX49iUJPVw.png" /></figure><p>expose the airflow in port 8080 and get a UI of airflow</p><pre>$ kubectl port-forward svc/airflow-helm-api-server -n airflow 8080:8080<br>Forwarding from 127.0.0.1:8080 -&gt; 8080<br>Forwarding from [::1]:8080 -&gt; 8080<br>Handling connection for 8080<br>Handling connection for 8080<br>Handling connection for 8080</pre><p>Retrieve the initial admin password:-</p><pre>kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath=&quot;{.data.password}&quot; | base64 -d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vZZUsRYhKVgu8q-8dpOAIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCJ7dYmnCt5Zbw924RLrQQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSfnuWu_voTGUTN6AmqLbQ.png" /></figure><p><strong>Common issues which I faced during deployment:-</strong></p><p><strong>DAGs not visible</strong> → check the git‑sync sidecar logs; verify credentials + sub Path and that the branch/ref exists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p4zi_OH5wbmi0B3xHJqttg.png" /></figure><p><strong>Conclusion</strong>:-<br>Here we have seen that ArgoCD to deploy Apache Airflow on Kubernetes offers a scalable and maintainable solution for workflow orchestration. By adopting a GitOps tools, we can avoid a configuration drift, and system state is consistently aligned with version control tools . The integration of Helm ensures extensibility in cluster management, while the git-sync mechanism provides seamless DAG updates.</p><p><strong>🎯Visit our website </strong><a href="https://www.ankercloud.com/"><strong>https://www.ankercloud.com</strong></a></p><p><strong>Reference:-<br>1)</strong><a href="https://airflow.apache.org/docs/helm-chart/1.9.0/"><strong>https://airflow.apache.org/docs/helm-chart/1.9.0/</strong></a><strong><br>2)</strong><a href="https://medium.com/@isochoa95/setting-airflow-helm-chart-to-be-deployed-using-argocd-on-eks-3b4fe7d9d85b"><strong>https://medium.com/@isochoa95/setting-airflow-helm-chart-to-be-deployed-using-argocd-on-eks-3b4fe7d9d85b</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=464bd0d7e2f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/airflow-on-argo-cd-a-step-by-step-gitops-guide-with-git-sync-for-dags-464bd0d7e2f5">Airflow on Argo CD — a Step‑by‑Step GitOps Guide (with git‑sync for DAGs)</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Securely Access Private Google Cloud Buckets with Cloud CDN (Step-by-Step Guide)]]></title>
            <link>https://medium.com/ankercloud-engineering/how-to-securely-access-private-google-cloud-buckets-with-cloud-cdn-step-by-step-guide-ef8c16b882d4?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/ef8c16b882d4</guid>
            <dc:creator><![CDATA[Vaibhav Wanare]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 04:41:21 GMT</pubDate>
            <atom:updated>2025-09-09T04:41:18.778Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vvYuGL2QmqZ2Kq8MBtiNuA.png" /></figure><p>Have you ever wanted to serve content from a <strong>private Google Cloud Storage (GCS) bucket</strong> but still enjoy the <strong>speed and caching benefits of Cloud CDN</strong>?</p><p>Until recently, the only way to do this was by generating <strong>signed URLs</strong>, which can get complicated for large apps. But now, thanks to <strong>Cloud CDN’s private origin authentication</strong>, you can set it up with no changes on the client side.</p><p>In this post, we will see how to set up Cloud CDN to serve content securely from a private GCS bucket.</p><p><strong>What You’ll Learn</strong></p><ol><li>We will learn What Private Origin Authentication is and why it matters</li></ol><p>2. Also, How to create and secure a private GCS bucket</p><p>3. We will also see how to configure a service account and HMAC keys</p><p>4. We will learn about how to connect your bucket to a load balancer and cloud CDN</p><p>5. We will see how to troubleshoot common issues</p><p>By following the steps, at the end we will have a production-ready setup that serves private bucket content securely through Cloud CDN.</p><p><strong>Prerequisites</strong></p><p>We have to make sure we have enabled a Google Cloud project with billing permission</p><p>We should have permission to create the following resources:</p><p>1. Buckets (Cloud Storage) 2. Service Accounts (IAM) 3. Network Endpoint Groups (NEG) 4. Load Balancers (Cloud CDN) 5. Basic knowledge of GCP Console and gcloud CLI</p><p><strong>Step 1: How to Create a Private GCS Bucket</strong></p><p>1.Cloud Storage → Buckets → Create Bucket.<br>2.Enter a unique bucket name (e.g., test-bucket-ankercloud).<br>3.Given that the bucket names are globally unique, we may need to add a prefix/suffix.<br>4.Choose a bucket region or multi-region depending on our needs.<br>5.Access Control: Select Uniform access.<br>6.We have to enable public access prevention → That will make sure that no object in our bucket can be made public accidentally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n9_zSOn8wh-Wlj-zxWTlzA.png" /></figure><p>Example:</p><p>Bucket name: test-bucket-ankercloud</p><p>I have uploaded the file hello.png in the bucket.<br>File: hello.png</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fgCQe2AxIFrSlUifFzFkDA.png" /></figure><p>Why this matters: Without “public access prevention,” someone could accidentally set an object as public. This step guarantees your bucket stays private.</p><p><strong>Step 2: Create a Service Account</strong></p><p>The service account acts like a “robot identity” that Cloud CDN will use to fetch your bucket objects.</p><p>Go to IAM &amp; Admin → Service Accounts → Create Service Account.</p><p>Give it a clear name like cdn-access-sa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*fvTOz12tYyREwn4cPXtngA.png" /></figure><p>Assign role: Storage Admin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQl2zftYZzHSfYJhTH3m1Q.png" /></figure><p>Then click on done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*b6jWFZJqfOtFvMihT1C21w.png" /></figure><p>Now, this service account has the authority to read your bucket objects.</p><p>Go to the bucket you created in Step 1. Open the bucket’s <strong>Permissions</strong> tab, and under <strong>View by principals</strong>, locate the service account gcs-origin@gtgmigration.iam.gserviceaccount.com. You will see that it has been assigned the role <strong>Storage Admin</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*RDN95k0sACS8E5w2e33THA.png" /></figure><p><strong>Step 3: Generate HMAC Keys</strong></p><p>HMAC keys are like a username + password pair that Cloud CDN uses to sign requests to your private bucket.</p><ol><li>Go to Cloud Storage → Settings → Interoperability tab.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*kVNJrwdyb6ocPkAwUmneRA.png" /></figure><p>2. Under Access Keys for Service Accounts, click Create a key for another service account.</p><p>3. Pick the service account you just created (cdn-access-sa).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*-Bykg5aY5AB37vsMTXfiNA.png" /></figure><p>4. Copy the Access Key and Secret Key — you will never see them again after closing the dialog!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*IX6Kbsl4FJ2Au-eUiWnaAA.png" /></figure><p>Store them securely (e.g., in a password manager).</p><p><strong>Step 4: Create an Internet NEG (Network Endpoint Group)</strong></p><p>An NEG tells the load balancer where your backend (in this case, the GCS bucket) is.</p><ol><li>Go to Compute Engine → Network Endpoint Groups → Create NEG.</li></ol><p>2. Name it: gcs-neg.</p><p>3. Type: Internet NEG <br> <br>Select scope as global.</p><p>4. QDN: &lt;your-bucket-name&gt;.storage.googleapis.com</p><p>5. Eg: test-bucket-ankercloud.storage.googleapis.com</p><p>Port: 443 (HTTPS).</p><p>Note : Always use HTTPS when connecting to GCS—it ensures secure data transfer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*vbtsqfb8veymCYGBOBWHnA.png" /></figure><p><strong>Step 5: Create the Load Balancer</strong></p><p>This is where we glue everything together:</p><p>1.A frontend IP (the entry point for users)</p><p>2. A backend service (our NEG pointing to GCS)</p><p>3. Cloud CDN to cache and speed up delivery</p><p><strong>Steps:</strong></p><ol><li>Go to Network Services → Load Balancing → Create Load Balancer.</li></ol><p>2. Choose: Application Load Balancer (HTTP/HTTPS) → External → Global.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*92tj4w50AODJilWiJBNX-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*VHWZ8GGu0-EUVulxaySmOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*B229efe9X6rXW-TWMAROAw.png" /></figure><p>Click on configure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*AVUE1rMdsy0C2lr8sEkm2Q.png" /></figure><p>3. Configure Frontend:</p><p>Here, for configuring the frontend, I am using the HTTP protocol; you can also use HTTPS if you have SSL certificates.</p><p>Protocol: HTTP (or HTTPS if you have SSL certificates).</p><p>I am using Reserve a static IP so it doesn’t change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*XuzR-tQr6Gp26P-bhuMDFA.png" /></figure><p>4. Configure Backend:<br>Click on “create a backend service.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UnJ2ymtn1fXSNsvIPHyMWg.png" /></figure><p>Backend type: Internet NEG.</p><p>For the backend type choose “Internet network endpoint group” and select the internet NEG that we created in the previous step. Choose protocol as HTTPS. Enable Cloud CDN on this backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*ibFi5sLaR-NMNJR3gCvmsA.png" /></figure><p>4.2 Caching Behavior</p><p>By default, private buckets set:</p><pre>Cache-Control: private, max-age=0</pre><p>Meaning they aren’t cacheable.</p><p>Options:<br>1. Cache static content. It is recommended you can use that.</p><p>2. Use origin headers. → Respect bucket settings (you can override with gsutil setmeta)</p><p>3. Force cache all → Cache everything regardless of headers</p><p>You can also reduce the TTL as per your need:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*X1KmA6dL5h7X4pEdeFjmSw.png" /></figure><p>4.3. Click on Advanced Configuration and under Custom request headers, add a host header and set the value to the GCS hostname -</p><p>Here&#39;s what I have added in the header value:</p><p>&lt;your-bucket-name&gt;.storage.googleapis.com<br>test-bucket-ankercloud.storage.googleapis.com</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*PyCWkR69FF9hpAWoDLc8jA.png" /></figure><p>Click through the remaining steps in the load balancer configuration and click on create/done.</p><p><strong>6: Enable Private Origin Authentication</strong></p><p>Here’s the secret sauce . Without this step, your CDN would get a 403 Forbidden when trying to access private objects.</p><p>Unfortunately, this feature is <strong>only available via gcloud CLI</strong> right now.</p><p>6. 1. Export your backend service config:</p><pre><br>gcloud compute backend-services describe  test-cdn-bs --global</pre><p>If you have a different name for your backend service, remember to use that instead of “ test-cdn-bs.”</p><p>Copy over the output to a .yaml file. Here I am using cdn-private.yaml. You can instead run a single command as below -</p><pre><br>gcloud compute backend-services describe test-cdn-bs --global &gt; cdn-private.yaml</pre><p>6.2. Add the origin authentication section. For more details, refer <a href="https://cloud.google.com/cdn/docs/configure-private-origin-authentication">here</a>.</p><p>1. The code snippet we will be using :</p><pre>securitySettings:<br> awsV4Authentication:<br>  accessKeyId: [the access key from the HMAC Creation step ]<br>  accessKey: [the access key secret from the HMAC Creation step ]<br>  accessKeyVersion: Optional<br>  originRegion: Can be any string as GCS does not require this parameter for signatures</pre><p>Also note: The Cloud CDN private origin feature does not rely on Secret Manager; hence, the accessKeyVersion is optional — this is different from the private origin setup in Media CDN.</p><pre>securitySettings:<br> awsV4Authentication:<br>  accessKeyId: [the access key you got from the HMAC creation step ]<br>​​  accessKey: [the access key secret from the HMAC creation step <br>  originRegion: us-east-2</pre><p>You can add that by using the below lines directly. It will automatically add the lines accesskeyId and accesskey in the file:</p><pre>cat &gt;&gt; cdn-private.yaml &lt;&lt;&#39;YAML&#39;<br>securitySettings:<br>  awsV4Authentication:<br>    accessKeyId: &quot;[the access key from the HMAC Creation step]&quot; <br>    accessKey: &quot;[the access key from the HMAC Creation step]&quot;<br>    originRegion: &quot;us-east-2&quot;<br>YAML</pre><p>Here, give your accessKeyId and accessKey created from HMAC creation step.<br>Here in the origin region, you can give anything (like us-east-2 or asia-south-1, etc.).</p><p><strong>This appends</strong> a new securitySettings block to the end of cdn-private.yaml with accessKeyId and accessKey.</p><p>2. We have to remove the below lines from the YAML file.</p><pre>fingerprint: &lt;value&gt;<br>Id: &lt;value&gt;<br>usedBy:<br>- reference: &lt;value&gt;</pre><p>We can remove this by using the below line:<br><br>1. sed -i ‘/^fingerprint:/d’ cdn-private.yaml</p><p>This line removes fingerprint line</p><p>2. sed -i ‘/^id: /d’ cdn-private.yaml <br>It removes id line.</p><p>3. sed -i ‘/^usedBy:/,/^sessionAffinity:/d’ cdn-private.yaml<br>It removes # remove usedBy block and its references</p><p>3. Now update your backend service using the below command:</p><pre><br>gcloud compute backend-services import test-cdn-bs \<br>  --global \<br>  --source cdn-private.yaml</pre><p>Note: If you only change settings that are not related to the secret key, you don’t need to enter the secret accessKey again. Also, whenever you check your backend service configuration using gcloud describe or the REST API, the secret key will not be shown. This is done on purpose to keep your key safe.</p><p>Notes:</p><ol><li>originRegion can be any value (GCS doesn’t check it).</li></ol><p>2. The secret key won’t show up if you later run describe—that’s intentional for security.</p><p><strong>Step 7: Test the Setup</strong></p><ol><li>Get your load balancer’s IP:</li></ol><pre>gcloud compute addresses list</pre><p>2. Test with curl:</p><pre>curl -I http://&lt;LB_IP&gt;/hello.png<br></pre><p>✅ Expected Response:</p><pre>curl -I http://load-balancer-ip/hello.png<br>HTTP/1.1 200 OK<br>content-type: image/png<br>x-guploader-uploadid: ABgVH88rd4XdZQOSzmEebmlyHvKekYVrF9aXrbOcNIeTcfJctBAE4QHsp7CVUo00_3McPAAS2wL0TQ<br>expires: Tue, 26 Aug 2025 05:39:45 GMT<br>date: Tue, 26 Aug 2025 05:39:45 GMT<br>cache-control: private, max-age=0<br>last-modified: Mon, 25 Aug 2025 09:51:20 GMT<br>etag: &quot;c01d2e212048e3f02ba62fd8ecda4072&quot;<br>x-goog-generation: 1756115480241734<br>x-goog-metageneration: 1<br>x-goog-stored-content-encoding: identity<br>x-goog-stored-content-length: 6990<br>x-goog-hash: crc32c=SRig6Q==<br>x-goog-hash: md5=wB0uISBI4/Arpi/Y7NpAcg==<br>x-amz-checksum-crc32c: SRig6Q==<br>x-goog-storage-class: STANDARD<br>accept-ranges: bytes<br>Content-Length: 6990<br>server: UploadServer<br>alt-svc: h3=&quot;:443&quot;; ma=2592000,h3-29=&quot;:443&quot;; ma=2592000<br>via: 1.1 google</pre><p>If you see 200 OK, congratulations—you’re serving private content via Cloud CDN!</p><p>You can also see that in a browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSs4D25SV3DVtUMXSdpK_Q.png" /></figure><p><strong>Troubleshooting (Common Errors)<br>1. </strong>❌ 403 Forbidden</p><ol><li>Service account missing permissions → Re-check Step 2.</li></ol><p>2. Wrong HMAC key → Verify Step 3 &amp; Step 6.</p><p>2. ❌ 404 Not Found</p><p>Host header not set → In your backend, add a custom request header with:</p><pre>Host: &lt;bucket-name&gt;.storage.googleapis.com</pre><p>3. ❌ 400 Bad Request</p><p>Extra headers (e.g., cookies) in the signed request → Strip or overwrite them in the load balancer config.</p><p>4. ❌ Signature Does Not Match</p><p>URL encoding mismatch → Remember, GCS has stricter rules than S3. Encode spaces as %20, not +.</p><p><strong>Wrap</strong>-Up</p><p>With Cloud CDN + Private Origin Authentication, you now have:</p><p>1. Security—Only Cloud CDN can access your bucket</p><p>2. Performance—Faster content delivery with caching</p><p>3. Simplicity—No client-side signed URLs required</p><p>This setup is production-ready and works for serving images, static websites, or even large datasets privately and securely.</p><h3>Visual Flow :</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0zTYWHbG2lIKixsg6phHw.png" /></figure><p><strong>Conclusion:</strong></p><p>By enabling <strong>Cloud CDN private origin authentication</strong>, you can seamlessly serve files from a private Google Cloud Storage bucket without relying on signed URLs or complex client changes. This setup ensures your bucket remains <strong>secure</strong> while still benefiting from Cloud CDN’s <strong>global caching and performance gains</strong>. Using service accounts, HMAC keys, and a load balancer with Internet NEG, you create a robust and production-ready pipeline for secure content delivery. The result is a solution that is <strong>faster, safer, and simpler</strong>—giving you the confidence to serve private content at scale with minimal effort.</p><p><strong>Reference:</strong></p><p><a href="https://cloud.google.com/cdn/docs/configure-private-origin-authentication">https://cloud.google.com/cdn/docs/configure-private-origin-authentication</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef8c16b882d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/how-to-securely-access-private-google-cloud-buckets-with-cloud-cdn-step-by-step-guide-ef8c16b882d4">How to Securely Access Private Google Cloud Buckets with Cloud CDN (Step-by-Step Guide)</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tunneling Made Easy: Tunneling Between AWS and GCP with Site-to-Site VPN]]></title>
            <link>https://medium.com/ankercloud-engineering/tunneling-made-easy-tunneling-between-aws-and-gcp-with-site-to-site-vpn-59b281aa4698?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/59b281aa4698</guid>
            <dc:creator><![CDATA[Vaibhav Wanare]]></dc:creator>
            <pubDate>Thu, 21 Aug 2025 07:30:42 GMT</pubDate>
            <atom:updated>2025-08-21T08:35:43.315Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-OpcC_m_jcCl6hxED3-LQ.png" /></figure><p>We know that In today’s hybrid and multi-cloud world, secure connectivity is no longer optional but its essential. when you are migrating workload in AWS and Google cloud or migrating step by step, we’ll need way of our network to talk to each other securely. That’s where Site-to-site vpn tunneling comes in. In this blog, I’ll guide you through setting up VPN tunneling between AWS and GCP VPC. This will helps to enable private communication across cloud without exposing services to public internet.</p><p><strong>Prerequisites:</strong></p><p>Before we begin the tunneling process, make sure we are having the following permissions:<br>1. We should have access to AWS Console with permissions for VPC, VPN, and routing.<br>2. We should have access to GCP Console with permissions for VPC, VPN, and firewall.<br>3. We should select Non-overlapping VPC CIDR ranges (example: AWS 172.18.0.0/16, GCP 10.250.0.0/17.</p><p><strong>Step 1: Reserve a Static IP in GCP</strong></p><p><strong>1. </strong>To find use the search bar to find “IP”<br>2. Click on “Reserve External Static IP address”<br>Name it gcp-aws-tunneling-ip.<br>Region: same as your GCP VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*x7CVuG-0_0MLAmWoaiREgw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9tCrkRkwfl2xGDR8DhlcA.png" /></figure><p><strong>Step 2: Create a Customer Gateway in AWS</strong></p><p>In the AWS console → VPC → Customer Gateways.</p><ol><li>Create a customer gateway with the GCP Classic VPN gateway IP (<strong>34.93.53.157</strong>).</li><li>Provide the following details:</li></ol><ul><li><strong>Name</strong>: aws-gcp-migration-cg</li><li><strong>IP Address</strong>: <strong>34.93.53.157 (GCP</strong> reserved static public IP)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xR7X-X7guxsjkiDMlTjXbQ.png" /></figure><p><strong>Step 3: Creating a Virtual Private Gateway (VGW) in AWS</strong></p><p>Go to VPC → Virtual Private Gateways.</p><p>Name: aws-gcp-migration-vgw.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gjYPWMgBRISUODJzH2ovpA.png" /></figure><p><strong>Step 4: Attach Virtual Private Gateway to VPC</strong><br>Attach it to your AWS VPC (172.18.0.0/16).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCUtQIhicwS_Qum7CNbbcw.png" /></figure><p><strong>Step 5: Creating a Site-to-Site VPN Connection in AWS</strong></p><p>We have to Navigate to VPN Connections → Create VPN Connection.</p><p>1. We have to create a site-to-site VPN connection with the following details:<br>Name: aws-gcp-migration-vpn-tunneling<br>Virtual Private Gateway: aws-gcp-migration-vgw.<br>Customer Gateway: aws-gcp-migration-cg<br>Routing Options: <strong>Static</strong></p><p>2. We have Specify the network CIDRs:<br>Local IPv4 Network CIDR: 172.18.0.0/16 (AWS VPC IP)<br>Remote IPv4 Network CIDR: 10.250.0.0/17 (GCP VPC IP)<br>Note: In GCP, We have already created two subnets (10.250.32.0/20 and 10.250.60.0/20) . Therefore, we are using general CIDR 10.250.0.0/17 is to include the both subnets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6FVlAfAYP0Cth-9THhdI0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GqB0sfv7BTjql6f9AEn7_Q.png" /></figure><p><strong>Step 6: Download the Configuration File</strong></p><ol><li>Download the configuration file.</li><li>Copy the <strong>Pre-shared Key</strong> from the file, which will be needed for the GCP VPN tunnel setup.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*UKgv-GBroy6dpAvbGLAudA.png" /></figure><p>Pre-shared Key: iESoTnUtp.gISz6wzJbl4mkuCgVzZmAO</p><p>To find the Pre-shared Key in the downloaded configuration file, search for the term “Shared Secret”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RsRMO00sb_zDie7JMTOtg.png" /></figure><p><strong>Step 7: Tunnel Status</strong></p><p>The tunnel status is currently down because the tunnel setup on the target side has not yet been completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WP07RM6_jlSUuybejfPlug.png" /></figure><p><strong>Step 8: Configure Inbound Security Rules to allow traffic from GCP VPC</strong></p><p>Configure inbound security rules to permit all protocols and ports from the GCP Cloud VPN with the IP range <strong>10.250.0.0/17</strong>. To allow all traffic, I am using the generalized GCP CIDR range, 10.0.0.0/8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nkasgZzA2HOxBu11fDVqcA.png" /></figure><p><strong>Step 9.A: We have to Configure Route Table Associations</strong></p><p><strong>We are having two option for configuring route table associations.<br>1.Option A — Associate subnets with the main VPC route table:</strong></p><p>We have Navigate to our VPC settings.</p><p>We have to Go to the route table section.</p><p>We have to Edit subnet associations to include both subnets.</p><p>OR</p><p><strong>2. Option B — Update route propagation:</strong></p><p>We have to Navigate to our VPC settings.</p><p>We have Go to the route table section.</p><p>We have to Edit route propagation settings as needed click on eanble.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*e6q_ebixhTOpWnLduM5bHA.png" /></figure><p><strong>Step 9.B: Adding the GCP CIDR in the route table</strong></p><p>We need to add GCP CIDR in the VPC route table that is attached to the Virtual Machine or Database that we are migrating.</p><p>Go to VPC-&gt;Route table-&gt;Route-&gt; Edit routes-&gt;Add route<br>Here, we are adding GCP CIDR (10.250.0.0/17) and select virtual private Gateway that we have already created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQoDT7fiNpHN5UZcD5TVAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*LPNiYA_tivteLLP7awbuAg.png" /></figure><p><strong>Step 10: Configure the Customer Gateway Device (GCP VPN)</strong></p><p>We have to Configure the customer gateway device, which in this case is the GCP VPN.<br><strong>Step 1: Enable Required API in GCP Console</strong></p><p>We have Enable the <strong>Compute Engine API</strong> in the GCP console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*1t7KiEQLgbjD0F0vCT1VJg.png" /></figure><p>Service Networking API</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*7TV3Ba5XzBrY958ZyDIFXw.png" /></figure><p><strong>Step 11: Create a Classic VPN Gateway</strong></p><p>We have Create a Classic VPN gateway for our network setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*LVEWHjT8Z9NgGvyrlmz5Hg.png" /></figure><p>I selected Classic VPN Gateway because it’s suitable for a smaller network and proof of concept (PoC) purposes. However, if you have a complex network and require dynamic routing and IPv6 support, you should opt for High-Availability (HA) VPN.</p><p>We have to Configure the Classic VPN gateway with the below details:<br>-Name: gcp--vpn-gateway<br>- Network: gcp-migration-vpc<br>- Region: asia-south-1<br>- IP Address: Reserved public IP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*IrHbZ37aITMoHR2u6Atv-Q.png" /></figure><p><strong>Step 12: Setup VPN Tunnel</strong></p><p>We have to Configure the VPN tunnel with the following settings:<br>- Name: <em>gcp-aws-migration-vpn-tunnel</em><br>- Remote Peer IP Address: 3.6.56.199(this IP is obtained from the external address of tunnel 1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BndzhkkKJX_TgtthISW16Q.png" /></figure><p>- IKE Version: IKEv1<br>- IKE Pre-shared Key: iESoTnUtp.gISz6wzJbl4mkuCgVzZmAO<br>- Routing Options: Route-based<br>- Remote Network IP Ranges: <strong>172.18.0.0/16</strong><em>(This is AWS VPC CIDR)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*hKy3s_HXOV1cve5o-zXFFw.png" /></figure><p><strong>Step 13: Update GCP firewall rules to allow AWS traffic:<br></strong>We have to Navigate to VPC networks → Firewalls, and add a new firewall rule to allow traffic from the AWS VPC CIDR (172.18.0.0/16).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*EpU5FuuytRdLL9p7_VP-Nw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*bbdiQJlUyXq0IDRzAkr-9g.png" /></figure><p><strong>Step 14: Validation<br></strong>Ensure that the tunnel status on the AWS side is “up”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e8c-34OPqf8k3ehYOQMA3g.png" /></figure><p><strong>Validation:<br></strong>Verify that the GCP VPN tunnel status is “established”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*vzn1At5KIlSjHnA24qCWKA.png" /></figure><p><strong>Step 15: Updating Static routes for GCP CIDR</strong></p><p>We have to update Static routes for GCP CIDR ranges(<strong>10.250.0.0/17</strong>):</p><p><em>Here, I have addes GCP side VPC as “10.0.0.0/8” to be more generic and to take all value from public and private subnet you can also take </em><strong>10.250.0.0/17.</strong></p><p>We need to Navigate to Site-to-Site VPN connections → Static routes → Edit routes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOFD4lu6_AvcQoS8U7bylg.png" /></figure><p><strong>Step 16: Test</strong></p><p>ping 172.18.x.x # AWS EC2 private IP from GCP VM<br>Here, i am pinging AWS VM from GCP side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*ywRSw4yb8LX0HPrlU1uePA.png" /></figure><p><strong>Step 17: Troubleshooting</strong></p><p>If Host(host unreachable) in traceroute → check routing table.</p><p>If ping works but app doesn’t → check firewall rules.</p><p>If tunnels won’t establish → verify IKE version + pre-shared key match.</p><p><strong>Conclusion:<br></strong>In this blog We have configured a Site-to-Site VPN between AWS and GCP that establishes secure, private connectivity between the clouds, that ensures workloads between two clouds can be able to communicate with each other without the public internet . The set up we have done simplifies the hybrid and multi-cloud strategies, that support seamless migration between the clouds.. For production, HA VPN is recommended to ensure high availability and dynamic routing.</p><p><strong>Reference:<br></strong><a href="https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws">https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59b281aa4698" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/tunneling-made-easy-tunneling-between-aws-and-gcp-with-site-to-site-vpn-59b281aa4698">Tunneling Made Easy: Tunneling Between AWS and GCP with Site-to-Site VPN</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Google Cloud Monitoring: A Step-by-Step Guide]]></title>
            <link>https://medium.com/ankercloud-engineering/getting-started-with-google-cloud-monitoring-a-step-by-step-guide-908934fee783?source=rss----374303d0a301---4</link>
            <guid isPermaLink="false">https://medium.com/p/908934fee783</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[cloud-functions]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[cloud-monitoring]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <dc:creator><![CDATA[Sakshi Naik]]></dc:creator>
            <pubDate>Tue, 12 Aug 2025 07:22:10 GMT</pubDate>
            <atom:updated>2025-08-12T07:22:10.250Z</atom:updated>
            <content:encoded><![CDATA[<h4>Your First Step Toward Pro-Level Insights</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5_EcDafI42GRkPN8Z_fnw.png" /></figure><p>In today’s cloud driven world, <strong>visibility is everything</strong>. Whether you are running a simple web app or a complex microservices architecture, you need real-time insights into <strong>latency, uptime, error rates, and resource consumption</strong> to keep systems healthy and users happy.</p><p>Google Cloud’s <strong>Cloud Monitoring</strong> service provides a powerful, integrated way to monitor, visualize, and alert on metrics across all your GCP resources (and even hybrid or multicloud setups).</p><p>From VM instances to Kubernetes clusters, APIs, and serverless functions, Cloud Monitoring provides end-to-end observability without extra setup or tooling.</p><p>In this blog, I will guide you through <strong>setting up Cloud Monitoring step by step</strong>, using <strong>real example graphs</strong> to show how you can build dashboards, interpret key metrics, and configure alerts for proactive incident response.</p><blockquote>What is Google Cloud Monitoring?</blockquote><ul><li>It is part of<strong> Google Cloud’s Operations Suite</strong> (formerly Stackdriver).</li><li>It lets you <strong>collect, visualize, and alert</strong> on metrics, logs, and events from GCP resources.</li><li>Supports <strong>custom dashboards</strong> for real-time insights.</li><li>Can <strong>integrate with Cloud Logging</strong> for troubleshooting and alert triggers.</li></ul><p>Now let us understand the Concept of Log Based Metrics and Alerting in Cloud Monitoring:</p><p>We will further also<strong> </strong>see how to create custom metrics from logs, use them to build dashboards, and finally visualize key insights through interactive graphs.</p><blockquote>Log-Based Metrics</blockquote><p>Log-based metrics lets you <strong>create custom metrics from log data</strong> in Cloud Logging. They’re useful for tracking specific events (e.g., 500 errors or login failures) not covered by default metrics. These metrics can then be <strong>visualized on dashboards or used to trigger alerts</strong> in Cloud Monitoring.</p><blockquote>Alerting</blockquote><p>Alerting allows you to <strong>define conditions and thresholds</strong> on metrics (system or log-based). When triggered, it sends <strong>notifications (email, Slack etc.)</strong> to your team for quick action. Alerts can include incident details, charts, and links to logs for faster troubleshooting.</p><blockquote>Now that we understand what log-based metrics and alerting are, let’s put them into action.</blockquote><p>So what we will do is we will create two custom metrics that is one for error count and another for warning count and then we will configure alerts to notify us when these counts exceed a threshold.</p><p>These metrics will later provide the graphs in our dashboard, giving us a clear view of system.</p><p>Let’s get our hands dirty !!🚀</p><p>We’re going to create <strong>two custom metrics,</strong></p><ul><li>One that counts all <strong>Error logs (severity = ERROR)</strong></li><li>Another that counts all <strong>Warning logs (severity = WARNING)</strong></li></ul><p>☝️ You can use <strong>any GCP resource type</strong> here (VMs, GKE, BigQuery, etc.) but in this example we are focusing on logs from a <strong>Cloud Run Revision</strong>.</p><h3>📌Creating Log Based Metrics</h3><p><strong>Step 1: Open the Log-Based Metrics Page in Cloud Monitoring</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*-bBo0cs_vo24XScRskd2bQ.png" /></figure><ul><li>Now Click <strong>“Create Metric”</strong></li></ul><p><em>Lets make </em><strong><em>Error Count Metric</em></strong></p><ul><li>Fill out the below details,</li><li><strong>Name:</strong> error_count</li><li>Add <strong>Description</strong> (its optional)</li><li><strong>Type:</strong> Counter</li><li><strong>Filter:</strong></li></ul><pre>resource.type=&quot;cloud_run_revision&quot;<br>resource.labels.service_name=&quot;******-cf-staging-partitions&quot;<br>resource.labels.location=&quot;europe-west3&quot;<br>severity=&quot;ERROR&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KqH7lW9pIklFodOHxxQcyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0imxkQDyf_qzZpHwboteFQ.png" /></figure><ul><li>Click Create Metric✅</li></ul><p><em>Similarly Lets make Warning Count Metric</em></p><ul><li>Again, click <strong>“Create Metric”</strong></li><li>Fill out:</li><li><strong>Name:</strong> <em>warning_count</em></li><li><strong>Type:</strong> Counter</li><li><strong>Filter:</strong></li></ul><pre>resource.type=&quot;cloud_run_revision&quot;<br>resource.labels.service_name=&quot;******-cf-staging-partitions&quot;<br>resource.labels.location=&quot;europe-west3&quot;<br>severity=&quot;WARNING&quot;</pre><ul><li>Boom !! Smash that <strong>Create</strong> Metric button 💥</li></ul><h3>📌 Creating Alert Policy</h3><p><strong>Step 2: Create Alerts for These Metrics</strong></p><p><em>What good are metrics if nobody knows when stuff’s going wrong? Let’s make Cloud Monitoring talk to you.</em></p><p><strong>Steps to Create Alerts:</strong></p><ul><li>Go to <strong>Cloud Monitoring → Under Detect → Alerting → + Create Policy</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YBcniTrI6jfujl_dnfeEXQ.png" /></figure><ul><li>Click <strong>“Add Condition”</strong></li><li>Select <strong>“Metric”</strong> and search for your shiny new metric (e.g., logging/user/error_count)</li></ul><blockquote>NOTE : Newly created Log Based Metrics are present in Inactive metrics by default so select accordingly.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*qplHTJ132qyaiwiNHdpEBg.png" /><figcaption>Click Select a metric</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JHKZ3LupM57QHJIU0aFawQ.png" /><figcaption>Select this and Click Apply button</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AJrgedr_VuTDL5cvgcFXYg.png" /></figure><ul><li>Set the condition, like:</li><li><strong>Trigger if error count &gt; 5 within 1 minute ( Make this according to your requirement )</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u55obrIcnnjqiNBj7WUXOA.png" /></figure><ul><li>Add who to notify (email, Slack, PagerDuty, maybe your boss 👀)</li><li>Give your alert a name (something dramatic, like <em>🔥 SYSTEM MELTDOWN 🔥</em>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rwNtSvuoj5dThrwVM6ik_Q.png" /><figcaption>Click on Manage Notification you can see various options you want to get Notified</figcaption></figure><ul><li>Click <strong>Create Policy</strong></li></ul><p>Repeat the same steps for <em>warning_count</em> if you want double protection. 🛡️</p><p>And that’s it! You’re now officially monitoring your Cloud Run app like a boss. 👑</p><blockquote>Metrics✅</blockquote><blockquote>Alerts ✅</blockquote><blockquote>Peace of mind? Also 😂✅</blockquote><p>Next up !! we will build a dashboard to make those numbers look 🔥</p><h3>📌 Creating Dashboard</h3><p><strong>Step 3: Build a Dashboard to Visualize Your Metrics 🎨📊</strong></p><p>Time to turn those dry numbers into some eye catchy.</p><p>Cloud Monitoring dashboards let you see the heartbeat of your existing system or the project you are working on.<strong> </strong>Let’s plug in the metrics we just created and make them pop.</p><p><strong>To create your dashboard:</strong></p><ul><li>Head over to <strong>Cloud Monitoring → Dashboards</strong></li><li>Click <strong>“Create Dashboard”</strong></li><li>Give it a cool name of your wish like <strong><em>Log Insights — Cloud Run</em></strong>💡</li><li>Click <strong>“Add Chart”</strong></li></ul><p><strong>🔧 Configure your chart:</strong></p><ul><li><strong>Chart Type:</strong> Line or Stacked Bar (your call go wild !!🎨)</li><li><strong>Metric Type: </strong>Search for logging/user/error_count (or warning_count)</li></ul><p><em>(If you don’t see it, check under </em><strong><em>User defined metrics</em></strong><em> → </em><strong><em>Inactive metrics</em></strong><em> → new metrics hide here by default.)</em></p><ul><li><strong>Resource Type: </strong><em>cloud_run_revision</em></li><li><strong>Filter (Optional): </strong>Add <em>service_name </em>and <em>location</em> if you want to zoom in:</li></ul><pre>resource.labels.service_name = &quot;*****-cf-staging-partitions&quot;<br>resource.labels.location = &quot;europe-west3&quot;</pre><ul><li><strong>What to Choose Delta or Sum ? </strong>💡</li></ul><p>Here is where we make the chart <strong>actually readable</strong>:</p><ul><li><strong>Delta</strong> → Shows the <em>change</em> between data points. Perfect for spotting sudden spikes of errors (<em> I prefer using delta since it produces cleaner, more precise graphs </em>)</li><li><strong>Sum</strong> → Rolls up the total errors over each time window.<br> Perfect for cumulative tracking ( <em>Eg. How many errors in the last 1h? </em>)</li></ul><p>💡 <strong>My recommendation for real-time alerting</strong> → Use <strong>Delta</strong> with a <strong>1-minute alignment</strong>.<br> 💡 <strong>My recommendation for trend analysis</strong> → Use <strong>Sum</strong> with a <strong>5-minute alignment</strong>.</p><p>In the widget settings:</p><ul><li><strong>Alignment period</strong>: 5m (or 1m if you want high-resolution)</li><li><strong>Aligner</strong>: sum or delta based on your use case.</li></ul><p>Make It Pop</p><ul><li><strong>Widget title</strong>: <em>Error Count in Staging (Delta 5m)</em></li><li><strong>Legend</strong>: Turn it on so you can compare multiple services if you add them later.</li><li>Save Your Masterpiece</li><li>Hit <strong>Apply</strong> → then <strong>Save</strong> dashboard.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ok-CR_hogqvLzV5n0nLs2A.png" /><figcaption>Warning Widget</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OL3fHiViX-ABs0-Yt0g3tw.png" /><figcaption>The Red icon you see is the Alert Policy being applied</figcaption></figure><blockquote><strong>Now that we have seen how to create Log Based Metrics using the Counter type</strong>, let us also understand the <strong>Distribution</strong> type.</blockquote><p>But before that, here’s the quick difference between the two:</p><ul><li><strong>Counter → </strong>Counts how many log entries match your filter.<br>Example: <em>“How many ERROR logs in the last 5 minutes?” this is </em>good for event counts.</li><li><strong>Distribution</strong> → Tracks a numeric value from logs and shows statistics like min, max, average, and percentiles.<br>Example: <em>What’s the API latency in milliseconds? this is </em>good for performance trends.</li></ul><h3><strong>📌 Example: Distribution Metric for Response Payload Size</strong></h3><p>If your logs contain a <em>response_size_bytes </em>field and you want to track how large responses are over time:</p><ul><li>Go to <strong>Log-Based Metrics</strong> → <strong>Create Metric</strong></li><li><strong>Name</strong>:<strong> api_response_size_distribution</strong></li><li><strong>Type</strong>: <strong>Distribution</strong></li><li><strong>Filter</strong>:</li></ul><pre>resource.type=&quot;cloud_run_revision&quot;<br>resource.labels.service_name=&quot;my-api-service&quot;<br>jsonPayload.response_size_bytes &gt; 0</pre><ul><li><strong>Distribution field</strong>: <em>jsonPayload.response_size_bytes</em></li><li>Click <strong>Create Metric</strong> ✅</li></ul><p>📊 On your dashboard, you will now see <strong>min</strong>, <strong>max</strong>, <strong>average</strong>, and <strong>percentile</strong> response sizes that are super handy for spotting unusual spikes (e.g., an API suddenly returning huge payloads)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Ot5D0b0-XExCR3bHZscS-g.gif" /></figure><p>Now that we have seen Cloud Run Revision based, let us switch our gears to another awesome GCP service that BigQuery 🚀😎</p><h3>📌 Monitoring BigQuery Activity with Log-Based Metrics</h3><p>We have conquered Cloud Run metrics, so now it’s time to grab our <strong>data detective hat</strong> 🕵️‍♀️ and start tracking BigQuery 😎because let us all be honest, <strong>queries don’t run themselves ..😂</strong></p><p>Here is what we are Looking on 🔎now:</p><ul><li>📅 <strong>All queries</strong> that ran in the last 24 hours</li><li>📥 <strong>Tables</strong> that got fresh data loaded</li><li>🕵️ Who is running those queries</li></ul><p><strong>Step 1: Count Every Query Like a Boss 😎</strong></p><p><strong><em>Go to Cloud Monitoring → Configure → Log Based Metrics → Create Metrics</em></strong></p><ul><li><strong>Type:</strong> Counter</li><li><strong>Name:</strong> <em>bq_query_count</em></li><li><strong>Description:</strong> Counts all completed BigQuery queries</li><li><strong>Filter:</strong></li></ul><pre>resource.type=&quot;bigquery_resource&quot;<br>protoPayload.methodName=&quot;jobservice.jobcompleted&quot;<br>protoPayload.serviceData.jobCompletedEvent.job.jobConfiguration.query.query!=&quot;&quot;</pre><p><strong>Step 2: Catch All the Data Loads 🚚</strong></p><p>Bulk data drops? Heyyyy !! We are watching it all !! So this metric will track every time someone loads data into a BigQuery table.. Fun Right !!😁😉</p><ul><li><strong>Type:</strong> Counter</li><li><strong>Name:</strong> <em>bq_table_load_count</em></li><li><strong>Description:</strong> Counts all BigQuery table load jobs</li><li><strong>Filter:</strong></li></ul><pre>resource.type=&quot;bigquery_resource&quot;<br>protoPayload.methodName=&quot;jobservice.jobcompleted&quot;<br>protoPayload.serviceData.jobCompletedEvent.job.jobConfiguration.load.destinationTable!=&quot;&quot;</pre><p><strong>Step 3: Reveal the SQL Culprits 👤</strong></p><p>We can even see <strong>which user</strong> is behind each query because transparency is… fun in everything right ? if you know you know 😂😏</p><ul><li><strong>Type:</strong> Counter</li><li><strong>Name:</strong> <em>bq_query_by_user</em></li><li><strong>Filter:</strong></li></ul><pre>resource.type=&quot;bigquery_resource&quot;<br>protoPayload.authenticationInfo.principalEmail!=&quot;&quot;<br>protoPayload.methodName=&quot;jobservice.jobcompleted&quot;</pre><p><strong>Step 4: Make a Dashboard because it is Worth Showing Off 📊</strong></p><ol><li><strong>Chart 1</strong> → <em>Delta</em> of <strong><em>bq_query_count</em></strong> → See query spikes when your analysts are going wild !!!💥</li><li><strong>Chart 2</strong> → <em>Sum</em> of <strong><em>bq_table_load_count </em></strong>→ Track daily or weekly data load trends 📆</li><li><strong>Chart 3</strong> → <strong>The Hall of Query Fame</strong> 🏆 → Breakdown by principalEmail where we find out who’s been really hammering BigQuery 😉</li></ol><p>📌<strong>Full steps — Chart 1: Delta of </strong><em>bq_query_count </em><strong>(spike detector)</strong></p><ul><li>Console → <strong>Monitoring</strong> → <strong>Dashboards</strong>. Open an existing dashboard or <strong>Create Custom dashboard</strong>.</li><li>Click <strong>Add chart</strong>.</li></ul><p>In the chart editor:</p><ul><li><strong>Metric</strong>: search for<strong><em> logging/user/bq_query_count</em></strong> (or your project’s user metric namespace).</li><li><strong>Resource type</strong>: <em>bigquery_resource</em></li></ul><p>Aggregation settings:</p><ul><li><strong>Aligner</strong>: <strong>Delta</strong> (shows change between consecutive points).</li><li><strong>Alignment period</strong>: <strong>1 minute</strong> (for real-time spike detection) — choose 5m if you want slightly smoother lines.</li></ul><p>Visual &amp; display:</p><ul><li><strong>Chart type</strong>: Line chart.</li><li><strong>Title</strong>: <em>BigQuery: Query Activity (Delta — 1m)</em></li><li><strong>Legend</strong>: on/off as you prefer.</li><li>Click <strong>Apply</strong> → <strong>Save</strong> dashboard.</li></ul><p>Similarly, create the other two charts and add them to a dashboard and title the Dashboard with <strong>BigQuery Monitoring Dashboard</strong> for a complete view of your BigQuery activity.</p><blockquote>Conclusion:</blockquote><blockquote>With Google Cloud Monitoring, you are not just collecting data you are turning it into actionable insights. From custom log based metrics to real-time alerts and intuitive dashboards, you can spot issues before they impact users. In short, it is your all-in-one command center for keeping cloud systems healthy and performant.</blockquote><p>🎯😎 Visit our website <a href="https://www.ankercloud.com/">https://www.ankercloud.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=908934fee783" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ankercloud-engineering/getting-started-with-google-cloud-monitoring-a-step-by-step-guide-908934fee783">Getting Started with Google Cloud Monitoring: A Step-by-Step Guide</a> was originally published in <a href="https://medium.com/ankercloud-engineering">Ankercloud Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>