<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sergey Shcherbakov on Medium]]></title>
        <description><![CDATA[Stories by Sergey Shcherbakov on Medium]]></description>
        <link>https://medium.com/@sshcherbakov?source=rss-3df4bbb66206------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0Tga7Tnhm_u4CdvG</url>
            <title>Stories by Sergey Shcherbakov on Medium</title>
            <link>https://medium.com/@sshcherbakov?source=rss-3df4bbb66206------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:15:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sshcherbakov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Opinionated Take on Kubernetes Application Lifecycle Management in GKE on Google Cloud]]></title>
            <link>https://medium.com/google-cloud/opinionated-take-on-kubernetes-application-lifecycle-management-in-gke-on-google-cloud-4c1db70037fc?source=rss-3df4bbb66206------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c1db70037fc</guid>
            <category><![CDATA[config-sync]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[gke]]></category>
            <dc:creator><![CDATA[Sergey Shcherbakov]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 08:46:17 GMT</pubDate>
            <atom:updated>2026-11-18T08:46:17.343Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-IolboNQacKqdYnrIMBDfg.png" /></figure><p>I have been visiting many Google Cloud enterprise customers over the past several years, and I often noticed that infrastructure and software lifecycle management for a cloud environment — often shortened to “CI/CD” — was a source of confusion. Over time, I gathered notes and observations about approaches Google Cloud customers take to organize their platform operations and software development teams. Together, they can be consolidated into a coherent source control and CI/CD pipeline setup that could help you get started quickly and prepare for secure and scalable work in the future. Such a high-level approach, in the context of Kubernetes workloads in GKE clusters on Google Cloud, I would like to share in this article.</p><p>The approach below deliberately prefers approval gates with no direct human Kubernetes clusters access to the interactive “click-ops” target environment operation style. For experimental, sandbox and some development environments this may be too restrictive and undesirable.</p><h3>Important Terminology</h3><p>Before diving deep into technical and philosophical questions, it is important to disambiguate some terms that will be used further in this article.</p><p><em>Kubernetes Application</em> — for different people, “application” means different things. E.g., it may be an iPhone application or a set of documents you submit to the consulate for a visa. Often an application is thought of as a single versioned binary that is the result of an application software build. In the Kubernetes world, an application is usually a set of services (that is, more than one) working together to deliver the required business logic. That set of services may rely upon multiple distinct container image binaries. Those binaries might have been developed by distinct software development teams (I find the ability to scale teams in that way a major benefit of microservice architectures). Those teams work at their own pace and schedule and <em>release</em> application binaries independently of each other. Even if there is a single development team working on all related binaries, they are usually not constrained to update and release binaries all at once and under one version number.</p><p><em>Lifecycle</em> — lifecycle governs how code is produced, verified, delivered, and retired across business and operational boundaries: requirements, development, CI, QA, CD, operations, and observability. Whenever you can make code changes and tag the result with a unique version, that versioned source code undergoes a lifecycle different from the source code which implements other requirements, gets built by other pipelines, potentially by different people at different times. Lifecycle is the scope that deserves separation into a dedicated source control repository. Infrastructure Terraform code managed by a platform team has a distinct lifecycle from application service source code managed by a software development team. As will be explained in greater detail below, I suggest making the Kubernetes application workload definition undergo an independent lifecycle and potentially dedicating a separate source control repository for it.</p><p><em>Kubernetes Application Release</em> — taking into account the considerations above, a Kubernetes Application Release is a versioned (commit hash, tag) Kubernetes workload definition that contains unique, unambiguous version references to the immutable container binaries that comprise the Kubernetes application at runtime. Its versioning is independent of the versioning of dependent application (container) binaries and independent of the Google Cloud infrastructure code versioning: they have their own distinct lifecycles.</p><p>Regarding the latter item, you might wonder why not define a release as a versioned Helm chart that packages a Kubernetes workload definition into a single versioned artifact. In Helm terminology, a <a href="https://helm.sh/docs/intro/introduction#release">Release</a> is a <em>running</em>, named instance of a Helm chart <em>deployed</em> to a Kubernetes cluster. A Helm chart doesn’t include configuration values applied in particular target environments. <em>A Helm chart Release is a Helm chart joined with its configuration values at runtime.</em> Those values need to be captured somewhere (ideally in source control) to provide a repeatable environment setup. Even though the ConfigSync GitOps tool — discussed in more detail below — supports Helm charts and their “<a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>” approach to Kubernetes application definitions, ConfigSync encourages/prefers a “<a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself#WET">WET</a>” source control approach. That is, a place in source control where the Kubernetes workload is kept with all its variable configuration parameters resolved separately for each target environment and checked in unambiguously (at the expense of some manifest code repetition).</p><p>In our solution design, the <em>Kubernetes Application Release</em> definition above pins not only Kubernetes workload manifests definitions (YAML files) by version, but also includes configuration values populating each aspect of the Kubernetes workload for the particular target environment. That aligns with the <a href="https://docs.cloud.google.com/kubernetes-engine/config-sync/docs/concepts/gitops-best-practices#create-wet-repo">GitOps Best Practices and ConfigSync design preferences</a>. There are several consequences of this definition:</p><ul><li>The versions of <em>Kubernetes Application Release</em> for distinct target environments are different in a generic case. The hash code of the commit that ConfigSync resource (RepoSync, RootSync) currently points to in the GKE cluster is usually different for Dev, Test, Prod environments.</li><li>The <em>Kubernetes Application Release</em> consists of the reusable Kubernetes application manifest definitions as well as the actual configuration property values in those manifests. They are usually different between different environments. For example, we might want to configure the number of application replicas in Dev and Prod differently (unless we use auto-scaling). Even though the Deployment resource structure is same, the value for replicas field is different for Dev, Test and Prod clusters:</li></ul><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br> name: my-application<br> namespace: tenant-a<br>...<br>spec:<br> replicas: 3   # Infrastructure related environment specific config value<br>...</pre><ul><li>The Kubernetes infrastructure related configuration (replicas, resource requests/limits, environment variables, secrets, ports, etc.) is part of (and covered by) the Kubernetes Application Release version. In other words a Kubernetes Application Release version uniquely identifies all of the following: Application binary artifact versions; Kubernetes workload definition manifests; Configuration values for a particular target environment. That comprises the Kubernetes application at runtime and allows exact workload restoration after outage or in a new runtime environment.</li><li>In order to re-configure Kubernetes application in a particular target environment we must commit changes into the right place in the right source control repository (following established human approval process) and rely upon the GitOps mechanism to propagate that change to the target GKE cluster.</li></ul><h3>Main Principles</h3><p>A Continuous Integration¹ / Continuous Delivery²/Deployment³ (CI/CD) system, both structurally and from a technology stack perspective, is a matter of opinion, taste, and preferences. The same goal can be achieved in multiple different ways, using different concepts and technologies of which there are hundreds to select from.</p><p>That is why, when setting out to design one, it is important to set high-level requirements and principles that we are going to adhere to along the way. There is a high chance that otherwise it would end up with a convoluted, error-prone design.</p><p>A company’s organizational structure makes a big difference in design. If I am a single contributor on my free-time project, I won’t need the majority of the sophistication described below (although even in this case, some basic software development principles, such as the use of a source control management system, I would still be following). If we are building a CI/CD system for a large, regulated enterprise with the highest security posture and compliance expectations, we would probably want to add more requirements and features to our design.</p><p>To provide a more universal design option below, here is the list of main principles that we will try to adhere to:</p><ul><li>The use of a source control system — Every aspect of the target cloud environment, including application source code, cloud infrastructure code, Kubernetes workload definitions, and their configuration should be checked into a source control system before they can reach the target cloud environment. This may seem too harsh for agile software development: developers should be able to move with their experiments and tests as fast as possible, at least in a development environment. But many enterprise customers opt for source code as the only way of accessing target environments. This must also not be viewed as a “black &amp; white” decision; your strategy may foresee interactive “click-ops” access to development environments and strict source-control-based processes for higher environments and production.</li><li>Build once, deploy everywhere — This is the “immutable build artifacts” principle. We should not deploy to a production system a binary that we have never tested before (that was built specifically for the production environment).</li><li>Isolated Dev, Test, and Production environments — Regulated organizations commonly place strict restrictions on communication and data sharing between development, test, and production environments. The design can be much simpler if you don’t need strict security isolation between your environments.</li><li>Least privilege access principle — This principle raises the security posture of the organization and is applicable if your organization is concerned about security and sets a “high bar” for it. Every identity that executes actions in the process must possess a minimum set of permissions that are required for it to be able to execute its process role.</li><li>Promotion approval process with human-in-the-loop — Decisions to deploy a particular application version (“promote”) to higher environments (e.g., Prod) must be vetted by one or more human reviewers (decision makers) who are granted respective permissions for that.</li><li>Trunk-based software development — Please read more on the trunk-based development principle encouraged by Google <a href="https://dora.dev/capabilities/trunk-based-development/">here</a>. I would summarize it as “[long-lived] feature branches are evil.” Application code resides in a single main (trunk) branch, and software development occurs there and only there. In our CI/CD system design option, we utilize (fixed) branches not for application development, but for configuration storage.</li></ul><p>For the sake of universality, the examples below will also assume:</p><ul><li>Multi-tenant shared GKE cluster environments with multi-tenant cloud infrastructure resources — This design aspect can easily be simplified to single-tenant clusters and single application development environments. A tenant in a shared Kubernetes cluster gets its workloads deployed to its dedicated, isolated Kubernetes namespace and also receives its own dedicated set of Google Cloud resources.</li></ul><p>The latter is, strictly speaking, not required in the general case (of single-tenant clusters or a single application instance) but provides a comprehensive option for more advanced multi-tenant use cases, which can be simplified to the single-tenant case in a straightforward way.</p><h3>GitOps</h3><p>Alexis Richardson, the CEO and co-founder of Weaveworks, was the first to coin the term and introduce the principles of GitOps in August 2017 through a landmark <a href="https://medium.com/weaveworks/gitops-operations-by-pull-request-14e8b659b058">blog post</a> titled <em>“GitOps — Operations by Pull Request”</em> (co-authored with Google’s William Denniss). Since then, the GitOps approach has become popular, especially for Kubernetes workloads and runtimes.</p><p>GitLab <a href="https://about.gitlab.com/topics/gitops/">describes</a> GitOps as “<em>an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation</em>”.</p><p>I would highlight two distinctive properties of the GitOps approach in comparison to alternative ways of implementing CI/CD:</p><ul><li>Dependency direction from the runtime to the workload definition storage</li><li>Reconciliation loop</li></ul><p>Instead of a CD system pushing an application configuration update to the target runtime (a network connection <em>from</em> the CI/CD system <em>to</em> the target runtime system), a process continuously running in the target Kubernetes environment “pulls” configuration changes into its environment and ensures that the workload in the runtime environment matches its definition in the obtained configuration. The “reconciliation loop” that it implements essentially performs a continuous three-way merge of the desired state represented by the configuration source, the current workload definition in the runtime environment, and the “last applied state” (the “common ancestor” base).</p><p>This choice — the GitOps “pull” vs. the alternative imperative “push” approach — is an important one that we make when starting the CI/CD system design.</p><p>Things can be implemented, match requirements, and work fine in many different ways. In this article, we pick the GitOps approach, illustrated using the <a href="https://docs.cloud.google.com/kubernetes-engine/config-sync/docs/overview">ConfigSync</a> managed GitOps service included with GKE on Google Cloud.</p><p>ConfigSync is not a prerequisite; many other alternative technologies, such as <a href="https://argo-cd.readthedocs.io/en/stable/">Argo CD</a> and <a href="https://fluxcd.io/">Flux CD</a>, can support the design suggested below.</p><h3>Infrastructure-as-Code</h3><p>The target runtime environment for a software application is infrastructure and should itself be defined declaratively as code as well. In our context, this includes GKE clusters and accompanying Google Cloud organization resources.</p><p>An alternative to capturing infrastructure resource definitions in source control is “click-ops” — that is, interactive resource management using the GUI of the corresponding cloud provider (or executing shell commands manually on target servers directly). Such alternatives might work for learning, training, demo, or simple solo-development purposes, but they are never a good choice for a professional business organization.</p><p>The spectrum of technologies supporting infrastructure-as-code target runtime environment definitions ranges from “imperative” tools such as <a href="https://docs.ansible.com/">Ansible</a> or <a href="https://github.com/saltstack/salt-user-guide">Salt</a>, via more declarative tooling with imperative capabilities such as <a href="https://www.pulumi.com/">Pulumi</a>, to purely declarative resource definitions using HashiCorp’s <a href="https://developer.hashicorp.com/terraform">Terraform</a>, and to the declarative <a href="https://cloud.google.com/blog/topics/developers-practitioners/build-platform-krm-part-1-whats-platform">KRM</a> (Kubernetes Resource Model), which defines all infrastructure resources in the form of Kubernetes Custom Resource Definitions (CRDs).</p><p>For the opinionated design in this article, we pick the popular Terraform option.</p><p>One reason for that is that Google engineering is <a href="https://discuss.google.dev/t/introduction-to-terraform-on-google-cloud-solutions-benefits-resources-and-faqs/109065">committed to Terraform</a> and <a href="https://docs.cloud.google.com/docs/terraform/iac-overview#iac-usecases-tools">recommends it</a> for the general case. Another reason is our choice of the opinionated, Terraform-based <a href="https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/master/fast/README.md">Fabric FAST</a> framework for quick and structured deployment of Google Cloud organizations. This framework has been developed and is maintained by the Google Cloud Consulting Professional Services team and provides a reliable and battle-tested tool for both greenfield starters on Google Cloud as well as established enterprise organizations.</p><p>Strictly speaking, Google Cloud foundations are not part of the design. It is not too important in our context how exactly you have built the GKE Kubernetes runtime for your application. On the other hand, properly setting up a Google Cloud organization is very important for security posture as well as for establishing a secure baseline for our CI/CD system design. In particular, the passwordless Workload Identity Federation integration with external source control management and CI/CD tools, as well as individual service account identities for restricted access to the runtime environments, are provided by the foundational Infrastructure-as-Code layer.</p><h3>Source Control</h3><p>The need to keep both infrastructure and application code in a source control system is an obvious prerequisite. This software development practice should be followed in any case, even if you are a solo developer working on a temporary “throwaway” project.</p><p>Modern SaaS source control providers, such as <a href="https://about.gitlab.com/">GitLab</a> and <a href="https://github.com/">GitHub</a>, offer not only managed cloud-based source control as a service; they typically include CI/CD tooling functionality and offer enterprise versions that allow you to run CI builds and CD scripts on compute resources located on-premises or in your Google Cloud organization.</p><p>Both GitLab and GitHub integrate with Google Cloud in terms of passwordless authentication. That is, once properly set up, you don’t need to copy over, transfer, or manage any secret values to enable the source control system to authenticate securely to Google Cloud when your build or deployment script needs to access Google Cloud APIs. The Google Cloud mechanism that supports that integration is called <a href="https://docs.cloud.google.com/iam/docs/workload-identity-federation">Workload Identity Federation</a> (WIF).</p><p>If your security requirements include data residency restrictions and you would like to rely on Google Cloud managed services only, Google Cloud offers the <a href="https://docs.cloud.google.com/secure-source-manager/docs/overview">Secure Source Manager</a> (SSM) managed source control service. SSM also natively supports VPC Service Controls, which is an important building block of a high-security-posture Google Cloud organization.</p><p>Secure Source Manager doesn’t include CI/CD service functions such as GitHub Actions or GitLab CI/CD. When choosing SSM, we need to augment that choice with a Google Cloud managed service for CI (<a href="https://cloud.google.com/build">Cloud Build</a>) and (optionally, since we are opting for ConfigSync in this demo) CD (<a href="https://cloud.google.com/deploy">Cloud Deploy</a>).</p><p>No matter which source control and CI/CD tool you choose, the main idea is to <em>associate automated script execution with a dedicated identity</em> on the Google Cloud side and assign this identity a minimum set of [IAM] permissions to perform only those actions that we intend it to be able to execute.</p><h3>Workload Identity Federation</h3><p>External source control system integration using Workload Identity Federation provides a straightforward way to establish secure passwordless authentication to Google Cloud for GitHub, GitLab, and any other external service that supports it.</p><p>A WIF identity name on the Google Cloud side looks like:</p><pre>principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/SUBJECT_ATTRIBUTE_VALUE</pre><p>Or a more specific GitHub example:</p><pre>principal://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/github-pool/subject/repo:my-org/my-repo:ref:refs/heads/main</pre><p>Please note the branch name main, which is part of that identity. It can also be any Git reference or JWT claim value, e.g., a tag. (However, a tag is rather a static pointer and does not serve well our purpose of associating a Google Cloud identity with a dynamic entity on the source control system side.)</p><p>What happens when a CI/CD pipeline is running in an external system outside Google Cloud, such as GitHub Actions, is that it:</p><ol><li>exchanges its GitHub authentication OIDC JWT token for Google Cloud’s short-lived opaque federated “STS” access token (which can be “introspected” for claims using the dedicated Google Cloud <a href="https://sts.googleapis.com/v1/introspect">https://sts.googleapis.com/v1/introspect</a> API), and then</li><li>exchanges the federated access token for the OAuth access token of an impersonated Google Service Account, and then</li><li>passes the latter to every Google Cloud API call when needed.</li></ol><p>In step (1), Workload Identity Federation provides a way of <em>mapping</em> incoming GitHub OIDC JWT token claims to claims of a Google Cloud identity based on the token <em>claims</em> that the trusted GitHub system has placed in the token. You can use different pieces of information that the JWT token carries to map it to claims associated with the federated access token on the Google Cloud side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9mhCGybBUVO5PBYDrQjyhA.png" /><figcaption>Workload Identity Attributes Mapping</figcaption></figure><p>Please note the custom attribute.fast_subthat is constructed from parts of the incoming GitHub OIDC token claims and can later be used in conditional IAM binding policies to grant IAM permissions to matching identities only.</p><p>One such claim that is straightforward to think through and follow is the name of the Git branch on which the current CI/CD script is running. It is straightforward in that way to say “any build running on Git branch A is authenticated as Google Service Account A” and “any build running on Git branch B is authenticated as Google Service Account B”.</p><p>It is then possible to grant distinct IAM permissions on the Google Cloud side to those distinct service accounts and enforce the “least privilege access” principle, limiting what a script on a particular GitHub branch can do on the Google Cloud side.</p><p>The source control system (GitHub, GitLab, SSM), in turn, offers a mechanism based on OWNERS files that allows you to control interactive human user access permissions to distinct branches in the source control system. This mechanism allows us to build a guarding policy for the human-in-the-loop environment promotion approval process.</p><p>Eventually, only colleagues with granted source control permissions are allowed to perform commits to a particular branch, and an automated script that gets triggered has permissions to perform actions only against a single dedicated Google Cloud environment to which its Google Cloud identity has been granted access (IAM permissions).</p><h3>High Level Design</h3><p>Having the tools described above under our belt, let us construct a high-level design that addresses the requirements and principles we set at the beginning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*GRs-_Ln1qqR4XDihwAvnVg.png" /><figcaption>High level structure of the application binaries promotion mechanism</figcaption></figure><p>The main idea here is that one source control branch corresponds to one and only one Google Cloud identity (such as a service account) which, in turn, has permissions to invoke Google Cloud APIs for resources in one and only one target environment (Dev, Test, Prod).</p><p>The application binary artifact “promotion” between environments occurs via merging source code changes between branches in the source control system. That process is governed by the access permissions granted to users on the source control system side.</p><h3>Repository Structure</h3><p>Let’s take a look at the actual structure of source control repositories supporting the suggested design.</p><p>In the generic case, a Kubernetes application consists of multiple individual services running from their own container images. Each such container image binary is built once and only once in a CI system of choice (in our example, GitHub Actions) in a pipeline associated with its dedicated identity on the Google Cloud side. Every such binary artifact is a “candidate to be deployed to production,” provided that it passes all automated tests and human approval gates.</p><p>Things can certainly be done differently. For example, we may put all aspects of our system into a single source monorepo and develop sophisticated CI/CD scripts that know how to interpret such a monorepo folder structure.</p><p>In our opinionated design, we put code with different lifecycles and purpose into different source control repositories:</p><ul><li><em>Infrastructure-as-Code repository</em> — here we keep the foundational Terraform code that defines our Google Cloud organization or a subset of its resources that build up our runtime environment (e.g., GKE clusters).</li><li><em>Application Source Code repository</em> — contains the actual application software code and builds it into a versioned binary artifact (e.g., a container image). There can be multiple such artifacts and subsequently multiple such repositories, depending on how many individual components your software service consists of.</li><li><em>Kubernetes Platform repository</em> — this is a repository (optionally separate) containing GKE cluster-wide resource definitions: primarily the ConfigSync RootSync and tenants’ RepoSync objects, but also cluster-wide network policy resources, RBAC assignments, etc. Separating this repository from others is, in fact, optional. It can be colocated with the <em>Application Source Code</em> repository or the <em>Kubernetes Workload Configuration</em> repository if there is a single one for all possible tenants. We are separating it here to illustrate its distinct purpose, managing team, and lifecycle.</li><li><em>Kubernetes Workload Configuration repository</em> — this is a source for GitOps systems (such as ConfigSync) running in the target Kubernetes cluster. The repository stores the actual application Kubernetes workload manifests to be applied in the target clusters as well as their actual configuration values for the target environments.</li></ul><h3>Infrastructure-as-Code Repository</h3><p>In the Infrastructure-as-Code repository, we use the Fabric FAST framework for defining Google Cloud organization resources. Please take a look at the comprehensive description of this Terraform framework in its GitHub <a href="https://github.com/GoogleCloudPlatform/cloud-foundation-fabric">repository</a>. In short, the FAST framework separates the shared Terraform state of foundational, rarely changing Google Cloud resources from the Terraform state of lower-abstraction resources representing important aspects of a Google Cloud organization (networking, security, observability), as well as application-related resources such as GKE clusters and foundations of individual application tenants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SPal4vAp53Ud0eNbtdGtGA.png" /><figcaption>Detailed relationships between Google Cloud and source control resources</figcaption></figure><p>The infrastructure-as-code repository is not directly related to software CI/CD. Its code can be managed by a dedicated (Platform) team and undergoes its own independent lifecycle. It hosts the Terraform code for the target GKE clusters, but more importantly, it defines Workload Identity Federation resources that integrate the source control system of choice with Google Cloud and grant source control branch identities IAM permissions to access specific GKE clusters also defined in this repository.</p><p>A single main branch in the infrastructure-as-code repository is often sufficient. Its Terraform code is expected to match the Google Cloud organization resources for all environments (Dev, Test, Prod). Potential branches in this repository may host resource definitions for Dev, Test, and Prod Google Cloud <em>organizations</em>. Sometimes large companies maintain separate Google Cloud organizations as development and test environments for testing resource definitions that have org-wide impact (org policies, networking, VPC Service Controls, etc.).</p><p>The FAST framework <a href="https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/fast">documentation</a> provides more authoritative details about the source repository structure. Here, I would like to illustrate some parts of it that lay the foundation for the CI/CD system of our solution components. In particular:</p><ul><li>Workload Identity Federation (WIF) GitHub Enterprise provider resources: <a href="https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/master/fast/stages/0-org-setup/datasets/classic/projects/core/iac-0.yaml#L193">WIF Provider Pool</a> for CI/CD pipelines running terraform apply automatically for foundational FAST framework Terraform projects in the <em>Infrastructure-as-Code</em> repository; A similar WIF Provider Pool for CI/CD build and binary artifact promotion pipelines in the <em>Application Source Code</em> repository (the ones that publish container images to Google Artifact Registry repositories); Similar WIF Provider Pool definitions for the <em>Kubernetes Workload Configuration</em> repositories of every Google Cloud platform tenant. Each tenant gets its own folder in the Google Cloud organization for hosting dedicated managed cloud resources.</li></ul><p>This is how a workload identity pool configuration looks like using FAST framework:</p><pre>workload_identity_pools:<br> default:<br>   display_name: CI/CD pool for Organization IaC<br>   providers:<br>     github-default:<br>       display_name: GitHub Public<br>       attribute_condition: attribute.repository_owner==&quot;example-org&quot;<br>       identity_provider:<br>         oidc:<br>           template: github<br>     issuer_uri: https://token.actions.githubusercontent.com</pre><ul><li>Mappings between Git branches and WIF Google Cloud identities for Terraform CI/CD pipelines in the <em>Infrastructure-as-Code</em> repository; Application artifact promotion pipelines in the <em>Application Source Code</em> repository; Tenant Terraform IaC application pipelines in the <em>Kubernetes Workload Configuration</em> repositories that optionally also host tenant Terraform resource definitions adjacent to the tenant Kubernetes workload definitions for ConfigSync</li></ul><pre># Bind each environment&#39;s RW Service Account strictly to pushes/merges on its environment branch (deploy-DEV, deploy-UAT, deploy-PROD)<br>resource &quot;google_service_account_iam_member&quot; &quot;tenant_wif_apply&quot; {<br>   for_each           = local.tenant_environments<br>   service_account_id = module.factory.service_account_ids[each.value.apply_sa_key]<br>   role               = &quot;roles/iam.workloadIdentityUser&quot;<br>   member             = &quot;principalSet://iam.googleapis.com/${each.value.wif_pool}/attribute.fast_sub/repo:${each.value.repository}:ref:refs/heads/${each.value.branch}&quot;<br>}</pre><ul><li>Service accounts associated with those identities: Terraform IaC CI/CD Service Accounts; Application binary promotion pipeline Service Accounts; Tenant Environment Service Accounts</li></ul><pre>resource &quot;google_service_account&quot; &quot;promotion_cicd&quot; {<br>   for_each     = local.promotion_pipeline_stages<br>   project      = local.iac-product_core_project_id<br>   account_id   = each.value.account_id<br>   display_name = each.value.display_name<br>}</pre><ul><li>IAM permissions for all listed identities including CI/CD pipeline service accounts; GKE cluster node service accounts for accessing dedicated Google Artifact Registries only.</li></ul><h3>Application Source Code Repository</h3><p>The actual application business logic code is located in one or several <em>Application Source Code</em> repositories. It undergoes the usual software development lifecycle and practices, such as <a href="https://dora.dev/capabilities/trunk-based-development/">trunk-based</a> development. That is, all developer commits go to a single main branch.</p><p>A CI/CD system, often built into the source control system like GitHub Actions or GitLab CI/CD, gets triggered automatically on a merge to the main branch and performs compilation, automated testing, and eventually builds a uniquely versioned application binary artifact and deploys it to the binary artifact storage repository, such as Google Artifact Registry (GAR).</p><p>The Google Cloud Workload Identity associated with the main branch is granted IAM permissions to write to the build GAR repository in the Terraform code located in the <em>Infrastructure-as-Code</em> repository.</p><p>Each change to the application code in the main repository is built into a unique, immutable new binary version. Each such binary is a candidate to be promoted to run in the production environment (provided all tests and approvals succeed).</p><p>Theoretically, the entire system can live with a single binary artifact repository hosting all application binaries across all environments (Dev, Test, Prod). But that would mean that the Production environment would not be as thoroughly isolated from the Dev environment as it could be. Often there is a security requirement mandating strict isolation of production and non-production environments. In Google Cloud, that can even be enforced using the VPC <a href="https://cloud.google.com/security/vpc-service-controls?e=48754805">Service Controls</a> mechanism.</p><p>Hence, for such a generic security case, we dedicate separate GAR repositories for each environment and grant GKE clusters in those environments IAM permissions to read application container images only from their dedicated GAR binary artifact repository, and not from any other repository.</p><p>A GKE cluster in the Test and Prod environments would only be able to start a new application version if its container images are available in the respective GAR repository. There needs to be a process that <em>promotes</em> application binaries (container images) between environment-specific GAR repositories. This process is closely related to the application source code build. Our main branch build already pushes the container image to the build (or Dev) GAR repository. Let’s extend it to move the existing image from the Build/Dev artifact repository to the higher Test GAR repository and then, upon approval, to an even higher-level GAR repository. Certainly, to follow the least privilege access principle, each such promotion step needs to be executed with a Google Cloud Workload Identity that has the minimum permissions to perform that step only. And, as we have seen before, an identity associated with a source control branch is the most straightforward way to arrange that.</p><p>So, we add several long-lived <em>deployment</em> branches to the <em>Application Source Code</em> repository along with its main branch, e.g.:</p><ul><li>main</li><li>deploy-DEV</li><li>deploy-UAT</li><li>deploy-PROD</li></ul><p>To avoid unnecessary confusion, the deployment branches do not host the application code as the main branch does. Trunk-based development still occurs on the main branch only. The deployment branches hold a simple CI/CD binary artifact promotion script and a simple config file with the application container image version desired to be available in the respective target runtime environment.</p><pre># Main branch<br>application-repo/<br>|-- .github/<br>|   \-- workflows/<br>|       \-- 1-build-main.yaml # CI/CD script to build application code<br>|<br>|-- ...   # The actual application source code<br>|-- Dockerfile # Dockerfile defining the application container image<br>\-- release.env     # Container image name and tag to build and push to build GAR repo<br><br># Deployment branch<br>application-repo/<br>|-- .github/<br>|   \-- workflows/<br>|       \-- 2-promote-image.yaml # Script to copy image from lower to current env GAR repo<br>|<br>\-- release.env   # Container image name and tag to push to the target environment GAR repo</pre><p>The source control system users should now be separated into groups of people who have permissions to commit (merge PRs) to the main branch and a separate group (or groups) who may review and approve Pull Requests to the deployment branches, making the application code binary available to the Kubernetes workloads in the respective GKE cluster.</p><p>Both the main build branch and deployment branches contain release.env files in the root directory. This file contains the container image name and version (tag). In the main branch, those two values are evaluated by the GitHub Actions build script to package and push the application container image to the “build” Google Artifact Registry (GAR) repository. The “build” and “dev” artifact repositories can optionally be colocated. Having them separate allows artifact promotion scripts to remain consistent across environments and provides an additional level of isolation.</p><h3>Kubernetes Platform Repository</h3><p>Our demo solution supports multiple application tenants sharing the same GKE cluster in Dev, Test, and Prod environments. Those tenants have their own Kubernetes namespaces in those clusters. Each such tenant namespace in each Dev, Test, and Prod cluster must be initialized with its individual ConfigSync RepoSync resource placed into the tenant’s Kubernetes namespace. The overlays/ subfolders in the <em>Kubernetes Platform</em> repository are the targets of initial cluster-wide RootSync resources deployed once to each of the Dev, Test, and Prod clusters respectively. This repository is owned by the Kubernetes Platform team that operates the GKE clusters.</p><pre># Deployment branches<br>k8s-platform-repo/<br>|-- base<br>|   |-- kustomization.yaml  # Base tenant-a RepoSync resource definition<br>|   |-- tenant-a.yaml   # Base tenant-a RepoSync resource definition<br>|   \-- tenant-b.yaml   # Base tenant-b RepoSync resource definition<br>\-- overlays<br>    |-- dev<br>    |   \-- kustomization.yaml     # tenants RepoSync config overrides for Dev cluster<br>    |-- test<br>    |   \-- kustomization.yaml     # tenants RepoSync config overrides for Test cluster<br>    \-- prod<br>        \-- kustomization.yaml     # tenants RepoSync config overrides for Prod cluster</pre><p>The main application binary promotion process targeted by this design does not actually require any actions in the <em>Kubernetes Platform</em> repository. It is described in this article for completeness purposes: its resources are vital for setting up ConfigSync and application tenant configuration. This is done once. The repository code is updated only by the Kubernetes platform team for platform-wide purposes not related to the application lifecycle.</p><p>In order to bootstrap ConfigSync in our shared multi-tenant GKE clusters, RootSync resources (the primary, single cluster-wide resources) need to be distributed to the target GKE clusters. There are multiple ways to do that, including the kubectl command-line tool. One way to do that is to use the Google Cloud API or the ConfigSync Google Cloud console UI and deploy a ConfigSync <em>Package</em> into each of the target GKE clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B-k9h9SbK44oKqXh2PCvuA.png" /><figcaption>ConfigSync Packages</figcaption></figure><p>When you deploy a ConfigSync <em>Package</em>, you have to specify:</p><ul><li>Target clusters to deploy it into</li><li>Git or OCI image package type (we pick Git here)</li><li>Package name and scope (cluster or namespace)</li><li>Git repo URL</li><li>Git repo Revision — tag, commit, hash to check out (default: HEAD)</li><li>Git repo Path in the repository (default: /)</li><li>Git repo Branch — (default: main)</li></ul><p>The <em>Kubernetes Platform</em> repository is the place for those Git settings to be populated.</p><p>The Git URL is the URL of the <em>Kubernetes Platform</em> repository. The path in the repository should point to the GKE cluster’s environment Kustomize overlay subfolder, such as /overlays/dev in our example for the Dev cluster. And we specify the Dev deployment branch name (e.g., deploy-DEV) in the branch property value for the Dev cluster. Correspondingly, we populate those values for ConfigSync packages in the Test and Prod clusters.</p><p>Eventually, <em>Package</em> deployment creates a RootSync resource in the target GKE cluster, such as:</p><pre>apiVersion: configsync.gke.io/v1beta1<br>kind: RootSync<br>metadata:<br> annotations:<br>   applyset.kubernetes.io/tooling: configsync.gke.io/v1<br>   configsync.gke.io/requires-rendering: &quot;true&quot;<br> finalizers:<br> - configsync.gke.io/reconciler-manager<br> - configsync.gke.io/reconciler<br> name: gke-promotion-demo<br> namespace: config-management-system<br>spec:<br> git:<br>   auth: token<br>   branch: deploy-DEV<br>   dir: /overlays/dev<br>   period: 15s<br>   repo: https://github.com/demo/k8s-platform-repo.git<br>   revision: HEAD<br>   secretRef:<br>     name: git-creds<br> sourceFormat: unstructured<br> sourceType: git</pre><p>It could have been pushed to the target cluster with kubectl just as well.</p><p>The RootSync objects deployed in that way into a GKE cluster with ConfigSync installed pull Kubernetes resource definitions, in GitOps style, from the places in source control with which we have configured them: k8s-platform-repo/overlays/dev for the Dev cluster, containing value overrides for the k8s-platform-repo/base/tenant-X.yaml file containing tenant-namespace-specific RepoSync objects, as well as optionally other cluster- or namespace-wide resources that the Platform team would like to deploy:</p><pre># k8s-platform-repo/base/tenant-a.yaml<br># RepoSync in namespace &#39;tenant-a&#39;: watches Tenant A&#39;s folder in gke-promotion-config<br>apiVersion: configsync.gke.io/v1beta1<br>kind: RepoSync<br>metadata:<br> name: tenant-a<br> namespace: tenant-a<br>spec:<br> sourceFormat: unstructured<br> git:<br>   repo: https://github.com/demo/k8s-workload-repo.git<br>   branch: deploy-DEV<br>   dir: tenant-a/configsync/overlays/dev<br>   auth: token<br>   secretRef:<br>     name: tenant-a-git-creds</pre><p>This resource must be controlled by the Kubernetes Platform team to not allow tenant DevOps teams to modify their namespace GitOps resource source on their own.</p><p>The <em>Kubernetes Platform</em> repository consists of deployment branches only, e.g.:</p><ul><li>deploy-DEV</li><li>deploy-UAT</li><li>deploy-PROD</li></ul><p>They all contain similar file structures with overlay subfolders for each target environment in each branch. That seems redundant and is indeed optional. What this provides is source-control-level permissions to delegate change promotion for different environments to different people, and also make use of the PR merging mechanism to construct changes (for base resources) once and merge them in a straightforward way into higher branches (environments).</p><p>Now, once the RootSyncs have been deployed to target clusters, they create tenant namespaces and pull individual tenant RepoSyncs into those namespaces. The tenant RepoSyncs now point to the tenant-specific GitOps configuration located in the <em>Kubernetes Workload Configuration</em> repository, and ConfigSync controllers running in GKE clusters start pulling Kubernetes application workload definitions into the individual namespaces.</p><p>As mentioned before, for simplicity, the <em>Kubernetes Platform</em> repository contents can be colocated with other repositories if their lifecycle matches the Kubernetes platform resources and/or if no access separation is required.</p><h3>Kubernetes Workload Configuration Repository</h3><p>Last but not least, we come to the source repository that holds the actual hydrated (“WET”) Kubernetes application workload resource definitions (manifests) — optionally including infrastructure-as-code (Terraform) resource definitions of tenant-specific managed Google Cloud service resources — and is managed by the tenant’s application DevOps team.</p><p>If tenants sharing your multi-tenant GKE clusters are different, independent development teams, each of them will get their own <em>Kubernetes Workload Configuration</em> repository for managing their tenant resources. There will be multiple <em>Kubernetes Workload Configuration</em> repositories in that case. Tenant development team members will not have access to other tenant teams’ resource definitions. If your application tenants are being managed by the same single central team (which can happen!), then it can be a single <em>Kubernetes Workload Configuration</em> repository holding resource definitions for all tenants in respective subfolders. We will presume this latter case below.</p><pre># Deployment branches<br>k8s-workload-repo/<br>|-- .github/<br>|   \-- workflows/<br>|       \-- tenant-iac.yaml  # Terraform apply CI/CD pipeline<br>|-- tenant-a<br>|   |-- configsync<br>|   |   |-- base # K8s resources defining K8s application (services, deployments, etc.)<br>|   |   |   |-- kustomization.yaml <br>|   |   |   |-- application.yaml <br>|   |   |   \-- ...<br>|   |   \-- overlays # configuration value overrides for each target environment<br>|   |       |-- dev  # RepoSync source for tenant-a in Dev cluster<br>|   |       |   \-- kustomization.yaml <br>|   |       |-- test  # RepoSync source for tenant-a in Test cluster<br>|   |       |   \-- kustomization.yaml <br>|   |       \-- prod  # RepoSync source for tenant-a in Prod cluster<br>|   |           \-- kustomization.yaml <br>|   \-- terraform<br>|       |-- main.tf<br>|       \-- ...<br>|-- tenant-b<br>|   |-- configsync<br>|   |   |-- ...<br>|   \-- terraform<br>|       |-- main.tf<br>|       \-- ...<br>...</pre><p>Similarly to the <em>Kubernetes Platform</em> repository, the <em>Kubernetes Workload Configuration</em> repository contains only deployment branches, e.g.:</p><ul><li>deploy-DEV</li><li>deploy-UAT</li><li>deploy-PROD</li></ul><p>Each branch has the same redundant structure for the same reason: source-control-level access separation for configuration value promotion and check-ins across different target environments. ConfigSync in each target cluster is linked (via RepoSync) to a distinct branch in this repository. Only people with permissions to check in to deploy-PROD will be able to deploy new application versions to the Production cluster.</p><p>As shown in the previous section, the RepoSync resource in the tenant’s namespace points to:</p><ul><li>The tenant’s <em>Kubernetes Workload Configuration</em> repository — via Git URL</li><li>Its environment-related configuration — via branch name</li><li>Its tenant-specific Kubernetes workload definition — via a repository path pointing to the tenant’s Kustomization overlay folder (e.g., k8s-workload-repo/tenant-a/configsync/overlays/dev)</li></ul><p>The content of the k8s-workload-repo/tenant-a/configsync/base folder, merged with configuration values specific to the current GKE runtime environment defined and checked in explicitly in k8s-workload-repo/tenant-a/configsync/overlays/dev, fully defines the Kubernetes application workload. The Kustomization overlay that the tenant’s RepoSync points to contains environment- and tenant-specific configuration overrides for application container image names, versions, and other runtime configurations (such as the number of deployment replicas). For example:</p><pre># k8s-workload-repo/tenant-a/configsync/overlays/dev/kustomization.yaml<br>apiVersion: kustomize.config.k8s.io/v1beta1<br>kind: Kustomization<br>namespace: tenant-a<br>resources:<br> - ../../base<br>images:<br> - name: europe-west3-docker.pkg.dev/project-id/app-dev-gar/my-app<br>   newName: europe-west3-docker.pkg.dev/project-id/app-dev-gar/my-app<br>   newTag: v0.0.1<br>patches:<br> - target:<br>     kind: Deployment<br>     name: my-app<br>   patch: |-<br>    - op: replace<br>      path: /spec/template/spec/containers/0/image<br>      value: &quot;europe-west3-docker.pkg.dev/project-id/app-dev-gar/my-app:v0.0.2&quot;<br>    - op: replace<br>      path: /spec/template/spec/containers/0/env/2/value<br>      value: &quot;gkep0-dev-tenant-a-0&quot;<br>    - op: replace<br>      path: /spec/template/spec/containers/0/env/3/value<br>      value: &quot;development&quot;</pre><p>Following ConfigSync’s preferred paradigm, configuration values are checked into source control at the same source control location as the Kubernetes application workload manifest definitions.</p><p>The last piece of the puzzle is the infrastructure-as-code (Terraform) code for (optionally) tenant-specific Google Cloud resources. Your tenant application might want to talk to its own <a href="https://cloud.google.com/products/alloydb?e=48754805">AlloyDB</a> or <a href="https://cloud.google.com/memorystore?e=48754805">Memorystore</a> database instance. Or it might want to run its own <a href="https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/pipelines/components-introduction">Gemini Enterprise Agent Platform Pipelines</a>. Those are non-Kubernetes resources usually managed in Terraform.</p><p>One option would be defining those tenant-specific Google Cloud resources using <a href="https://docs.cloud.google.com/config-connector/docs/overview">Kubernetes Config Connector</a> (KCC) and the KRM resource definition model directly in the tenant’s Kubernetes workload manifests. Sometimes KCC’s cloud resource coverage lags behind its Terraform Google provider equivalent. Also, for those who find it more convenient to manage Google Cloud infrastructure resources in Terraform, we dedicate a place in the <em>Kubernetes Workload Configuration</em> repository per target environment (Dev, Test, Prod) and per tenant, alongside the Kubernetes application workload definition that works together with cloud resources defined in Terraform.</p><p>That is, the k8s-workload-repo/tenant-a/terraform subfolders colocated with the tenant’s K8s resource manifests in k8s-workload-repo/tenant-a/configsync.</p><p>As a result, both the tenant’s cloud infrastructure code and the Kubernetes application workload definition (<em>Kubernetes Application Release</em>) are versioned by the same source control repository commit hash and version tag (optionally).</p><p>The GitHub Actions workflow in k8s-workload-repo/.github/workflows/tenant-iac.yaml deploys changes to the tenant’s Terraform code using a dedicated service account identity assigned to this repository deployment branch, which has IAM permissions to modify the tenant’s Google Cloud resources only in the target tenant-specific Google Cloud projects.</p><h3>Promotion Process</h3><p>Once we have set up the Google Cloud organization by running all FAST stages in the <em>Infrastructure-as-Code</em> repository, we have successfully integrated all participating GitHub repositories with the Google Cloud organization.</p><p>The first step, then, is to check in application code changes to the main branch of the <em>Application Source Code</em> repository. The automated build script (GitHub Actions workflow) in that branch will use a dedicated Google Service Account associated with the main branch to access either a dedicated build Google Artifact Registry (GAR) repository or the GAR repository accessible by the Dev GKE cluster (via service accounts associated with the GKE cluster nodes).</p><p>Once the application container binary build has succeeded, its automated tests have passed, and the application container image binary is in the build GAR repository, we switch to the deployment branch of the <em>Application Source Code</em> repository (e.g., deploy-DEV or deploy-UAT), modify the application container image version in the configuration file, and check that change into the deployment branch. (To make our application available in the Dev cluster, we might use the latest image version and have a Kubernetes deployment that reloads the image on each manifest update).</p><p>The immutable application binary promotion script gets triggered in deploy-UAT. It runs with a dedicated Google Service Account identity that has only two permissions:</p><ul><li>Read images from the Dev GAR repository</li><li>Write images to the UAT GAR repository</li></ul><p>No other permissions to Google Cloud are required for the image promotion script. It can copy the container image (e.g., using Crane) with those IAM permissions alone to make the application container image available to any workloads in the UAT GKE cluster.</p><p>The same process repeats for higher environments, e.g., Production. The service account associated with the Production branch has permissions to read from the UAT GAR repository and put images into the Production GAR repository, making them available for Kubernetes workloads running in the Production GKE cluster. Only users with explicitly granted GitHub permissions to perform merges to the Production deployment branch will be able to do that.</p><p>But nothing has been deployed to the GKE clusters yet!</p><p>Once the application container image is made available in the GAR repository associated with the GKE cluster, we need to update the reference (version) to the application container image in the application Kubernetes workload manifest with hydrated config values in the <em>Kubernetes Workload Configuration</em> repository.</p><p>To do that, we open a deployment branch of the <em>Kubernetes Workload Configuration</em> repository, change the container image version in the hydrated application Kubernetes workload manifest, and (if our source control user has permission to do so) check in to that deployment branch.</p><p>On its polling cycle, the ConfigSync RepoSync controller will read the container image version change from the <em>Kubernetes Workload Configuration</em> repository and apply the updated workload manifest to the GKE cluster it is running in. The GKE cluster nodes that host application pods will pull the new version of the application container image from the GAR repository associated with the cluster.</p><p>If a commit to the <em>Kubernetes Workload Configuration</em> repository also contains changes to the tenant’s Terraform code (adjacent to the tenant’s Kubernetes workload definition), the GitHub CD script will apply those Terraform code changes to the tenant’s Google Cloud projects and will have IAM permissions to do so only in the given environment (Dev, UAT, Prod).</p><p>In this way, a Kubernetes application that also relies upon managed Google Cloud services (e.g., a Cloud SQL database) will be able to connect to its dedicated tenant instance and not to the instance of any other tenant.</p><p>A rollback to the previous application version is simply another check-in to the <em>Kubernetes Workload Configuration</em> repository reverting the last introduced changes (git revert).</p><p>For emergency access to protected non-development environments, one should resort to the <a href="https://docs.cloud.google.com/iam/docs/pam-overview">Privileged Access Manager</a> (PAM) service available in Google Cloud for such scenarios.</p><h3>Conclusion</h3><p>The presented solution is one of many technically possible. Its approach covers the following aspects of a cloud application design and decouples the lifecycles of:</p><ul><li>Cloud infrastructure resource definitions — the IaC code version in source control represents an application infrastructure release.</li><li>Application software code — programming language code that builds into an immutable container image that has unique, independent versioning and is a potential candidate to be deployed to production.</li><li>Kubernetes workload manifests and configuration — hydrated (WET) Kubernetes manifests of a Kubernetes application consisting of multiple application binaries. Its versioning is independent of the other types of related source code and uniquely identifies the Kubernetes Application Release to a particular target runtime environment, fixing the application binary release versions in the container image URI references.</li></ul><p>Source code identified across these three version “dimensions” uniquely identifies everything needed to repeatedly restore an equivalent cloud application deployment from scratch at any time.</p><p>A Kubernetes Application Release to the target environment is uniquely, repeatably, and immutably identified by a single commit hash (and optionally a version tag) in the <em>Kubernetes Workload Configuration</em> repository.</p><p>The solution also separates concerns in such a way that participating identities have the least privilege required to perform their work.</p><p>The process of application promotion between environments is guarded by human approvals implemented using the source control system’s permission system.</p><p>Please let me know what you think!</p><p>¹ <strong>Continuous integration</strong> (<strong>CI</strong>) is the practice of integrating <a href="https://en.wikipedia.org/wiki/Source_code">source code</a> changes frequently and ensuring that the integrated codebase is in a workable state.</p><p>² <strong>Continuous delivery</strong> (<strong>CD</strong>) is a <a href="https://en.wikipedia.org/wiki/Software_engineering">software engineering</a> approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time.</p><p>³ <strong>Continuous deployment</strong> (<strong>CD</strong>) is a <a href="https://en.wikipedia.org/wiki/Software_development_process">software engineering approach</a> in which software functionalities are delivered frequently and through automated <a href="https://en.wikipedia.org/wiki/Software_deployment">deployments</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c1db70037fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/opinionated-take-on-kubernetes-application-lifecycle-management-in-gke-on-google-cloud-4c1db70037fc">Opinionated Take on Kubernetes Application Lifecycle Management in GKE on Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simplified Google Cloud Certificate Authority Service CA Rotation for Kubernetes Workloads]]></title>
            <link>https://medium.com/google-cloud/simplified-google-cloud-certificate-authority-service-ca-rotation-for-kubernetes-workloads-8b0c98e0a17b?source=rss-3df4bbb66206------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b0c98e0a17b</guid>
            <category><![CDATA[ssl-certificate]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[tls-certificate]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Sergey Shcherbakov]]></dc:creator>
            <pubDate>Thu, 26 Mar 2026 09:41:32 GMT</pubDate>
            <atom:updated>2026-03-27T05:18:01.768Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0mo_679IHW9NH_o3Yzv4w.png" /></figure><h4>A new feature in the cert-manager plugin google-cas-issuer streamlines rotation of the organization root Certificate Authorities. This article explains when it can help and how to use it.</h4><p>If you are managing your workload TLS certificates using the ubiquitous <a href="https://cert-manager.io/">cert-manager</a> service for Kubernetes and the Google Cloud <a href="https://cloud.google.com/security/products/certificate-authority-service">Certificate Authority Service</a> (CAS), you have likely encountered the open-source project <a href="https://github.com/cert-manager/google-cas-issuer">google-cas-issuer</a>. This cert-manager plugin allows it to call CAS in Google Cloud to fetch certificates for Kubernetes workloads. This applies whether you are using Google GKE managed service, Google Distributed Cloud, or any arbitrary Kubernetes cluster with access to the Google Cloud API. While this is less critical for workloads managed by Google Cloud Service Mesh (which integrates with CAS and handles lifecycle management automatically), there are many other use cases where a TLS certificate is required within a Kubernetes cluster.</p><p>To obtain a TLS certificate from the Certificate Authority Service (CAS), you would typically install cert-manager in the cluster, define a Google Certificate Authority <em>Issuer</em> (or <em>ClusterIssuer</em> at the cluster level), and configure it with a reference to the CAS CA Pool resource in Google Cloud.</p><pre>apiVersion: cas-issuer.jetstack.io/v1beta1<br>kind: GoogleCASClusterIssuer<br>metadata:<br>  name: googlecasissuer<br>spec:<br>  project: my-project-id<br>  location: europe-west3<br>  caPoolId: server-pool</pre><p>Next, you must describe the required TLS certificate using the cert-manager API <em>Certificate</em> resource and reference a prepared Kubernetes Secret object that holds the keystore password. For example:</p><pre>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: gateway-certificate<br>  namespace: istio-system<br>spec:<br>  secretName: gateway-cert-tls<br>  commonName: crilb.hello.zone<br>  dnsNames:<br>    - crilb.hello.zone<br>  duration: 720h<br>  renewBefore: 8h<br>  issuerRef:<br>    group: cas-issuer.jetstack.io<br>    kind: GoogleCASClusterIssuer<br>    name: googlecasissuer<br>  keystores:    <br>    jks:<br>      create: true<br>      passwordSecretRef:<br>        name: gateway-cert-keystore-password<br>        key: password<br>    pkcs12:<br>      create: true<br>      passwordSecretRef:<br>        name: gateway-cert-keystore-password<br>        key: password<br>---<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: gateway-cert-keystore-password<br>  namespace: istio-system<br>data:<br>  password: Y2hhbmdlaXQ=</pre><p>Assuming everything is configured correctly (refer to the <a href="https://github.com/cert-manager/google-cas-issuer/blob/main/README.md">public documentation</a> for a detailed overview), you will receive a Kubernetes Secret object containing three key components.</p><pre>apiVersion: v1<br>kind: Secret<br>data:<br>  ca.crt: LS0tLS1C.....<br>  keystore.p12: MIIM4wIBAzCCD...<br>  tls.crt: LS0tLS1CRUdJ...<br>  tls.key: LS0tLS1CRUdJTiB...<br>  truststore.jks: /u3+7QAAAAIA...<br>  truststore.p12: MIIEHgIBAz...<br>metadata:<br>  annotations:<br>    cert-manager.io/alt-names: crilb.hello.zone<br>    cert-manager.io/certificate-name: gateway-certificate<br>    cert-manager.io/common-name: crilb.hello.zone<br>    cert-manager.io/ip-sans: &quot;&quot;<br>    cert-manager.io/issuer-group: cas-issuer.jetstack.io<br>    cert-manager.io/issuer-kind: GoogleCASClusterIssuer<br>    cert-manager.io/issuer-name: googlecasissuer<br>    cert-manager.io/uri-sans: &quot;&quot;<br>  labels:<br>    controller.cert-manager.io/fao: &quot;true&quot;<br>  name: gateway-cert-tls<br>  namespace: istio-system</pre><ol><li>tls.key — The private key of the TLS certificate, which the server uses to provide a TLS-encrypted endpoint.</li><li>tls.crt — The public certificate that the server process presents to clients to prove its authenticity.</li><li>ca.crt —The root CA certificate of the hierarchy from which the issuing CA originates.</li></ol><p>The ca.crt value requires a bit more explanation.</p><p>In many cases (and following recommended best practices), especially in large organizations, issuing certificate authorities (CAs) are organized into tree-like hierarchies. In these setups, the CAs issuing the actual TLS certificates (known as “leaf” certificates) are not the only entities that clients must trust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2DxqH4r5Slr9p3Rn6RqW2w.png" /><figcaption>Multi-level CA hierarchy example</figcaption></figure><p>There are several reasons for such a setup. One is the ability to independently replace the CAs in the lower levels of the hierarchy if they expire or are (hopefully not) compromised. This alone allows for shorter issuing CA validity times, which improves the overall organizational security posture and follows modern recommended security practices.</p><p>Clients do not need to be reconfigured each time the intermediate CAs are replaced. They can be configured to trust the single parent CA of the hierarchy and, therefore, transitively trust any certificate issued by any intermediate-level CA. The parent or <em>root</em> CAs in this hierarchy typically have longer validity (lifetime) configured.</p><p>So far, so good. But eventually (unless you have configured very long intervals against security best practices), the root CA will expire as well. Application clients must then be reconfigured with a new <em>trust bundle</em> that includes the new root CA certificate.</p><p>By default, cert-manager places this root CA certificate into the ca.crt field of the generated Kubernetes Secret. This is conveniently available to all workloads in the cluster — both application servers providing TLS endpoints and application clients establishing encrypted, trusted connections to those endpoints.</p><p>Relying on the ca.crt field for a trust bundle is tempting, but it presents a significant hurdle for root CA rotation in downtime-sensitive workloads. Since application clients trust a single root CA certificate, rotating it requires the simultaneous replacement of the server TLS certificate and the client trust bundle to maintain service availability. This is often very difficult, if not impossible, to achieve seamlessly.</p><p>The best practice is to configure application clients to fetch their trust bundle from a central, well-defined location, such as a Google Cloud Storage bucket (see the KubeTrust reference architecture explained in this IEEE CSCLOUD2025 <a href="https://ieeexplore.ieee.org/document/11261454">article</a>). In such a location, several valid root CA certificates can coexist during the rotation period. This allows the application client configuration to be updated to the new trust bundle version independently — the most critical factor — without incurring application downtime.</p><p>However, because cert-manager generates a convenient Secret accessible by application clients, following that best practice can feel like a burden. Consequently, applications often accumulate “technical debt” by relying on the ca.crt field (containing only the single root CA) as their trust bundle source.</p><p>Since root CAs must eventually be rotated, how can we minimize application downtime during that process?</p><p>Google Cloud Certificate Authority Service uses the concept of a CA Pool. A CA Pool is a container for multiple CAs. When you request a certificate from the CAS service, you target the CA Pool rather than a specific CAS CA resource.</p><p>This is highly convenient for rotation: when a CA in the pool nears expiration, a new CA with a longer validity period is created and placed in the pool (initially in a Disabled state). Once the new CA certificate is distributed to all application client trust bundles, the new CA can be activated in the pool to issue TLS certificates alongside the old CA. The old CA can then be disabled from issuing new certificates (while remaining trusted), and after application servers have updated their certificates, it can be fully deactivated and removed. This is the intended rotation workflow in CAS.</p><p>For application clients that rely solely on the cert-manager Secret’s ca.crt field, this rotation procedure would typically cause downtime.</p><p>However, starting with the v0.11.0 release, the google-cas-issuer plugin includes a new feature to simplify this for those clients.</p><p>By optionally specifying the caFetchMode: PoolCAs attribute, the ca.crt field in the resulting Kubernetes Secret will be populated with the root CAs of all CAs currently present in the specified CAS CA Pool:</p><pre>apiVersion: cas-issuer.jetstack.io/v1beta1<br>kind: GoogleCASClusterIssuer<br>metadata:<br>  name: googlecasissuer<br>spec:<br>  project: my-project-id<br>  location: europe-west3<br>  caPoolId: server-pool<br>  caFetchMode: PoolCAs</pre><p>If an application client now uses the ca.crt field as its trust bundle to verify application servers when establishing TLS connections, it will receive both the old, expiring root CA and the new, extended one. This allows the client to bridge the root CA rotation and avoid downtime by updating its configuration independently of the application server’s TLS update.</p><p>Here is what the resulting Kubernetes Secret might look like (the Base64-encoded ca.crt value has been decoded for illustration purposes):</p><pre>apiVersion: v1<br>kind: Secret<br>data:<br>  ca.crt: |-<br>    -----BEGIN CERTIFICATE-----<br>    MIID/TCCAuWgAwIBAgIUAOvVafxuYaRGgdctZqTWQ64ZJoowDQYJKoZIhvcNAQEL<br>    BQAwgYUxCzAJBgNVBAYTAkRFMRwwGgYDVQQIExNOb3JkcmhlaW4td2VzdGZhbGVu<br>    ...<br>    -----END CERTIFICATE-----<br>    -----BEGIN CERTIFICATE-----<br>    MIICbzCCAhagAwIBAgITfkwGizNFOiLKdSLvyTwKW7ekozAKBggqhkjOPQQDAjCB<br>    hTELMAkGA1UEBhMCR2UxHDAaBgNVBAgTE05vcmRyaGVpbi13ZXN0ZmFsZW4xEDAO<br>    ...<br>    -----END CERTIFICATE-----<br>  keystore.p12: MIIM4wIBAzCCD...<br>  tls.crt: LS0tLS1CRUdJ...<br>  tls.key: LS0tLS1CRUdJTiB...<br>  truststore.jks: /u3+7QAAAAIA...<br>  truststore.p12: MIIEHgIBAz...<br>metadata:<br>  annotations:<br>    cert-manager.io/alt-names: crilb.hello.zone<br>    cert-manager.io/certificate-name: gateway-certificate<br>    cert-manager.io/common-name: crilb.hello.zone<br>    cert-manager.io/ip-sans: &quot;&quot;<br>    cert-manager.io/issuer-group: cas-issuer.jetstack.io<br>    cert-manager.io/issuer-kind: GoogleCASClusterIssuer<br>    cert-manager.io/issuer-name: googlecasissuer<br>    cert-manager.io/uri-sans: &quot;&quot;<br>  labels:<br>    controller.cert-manager.io/fao: &quot;true&quot;<br>  name: gateway-cert-tls<br>  namespace: istio-system</pre><p>The great thing here is that all CAS CA Pool root CAs can also be requested in P12 and JKS formats within the same Secret. Cert-manager will generate all of them in the desired format, making them convenient for direct mounting to applications that support trust bundles in those formats.</p><p>Hopefully, this new google-cas-issuer feature will help save some gray hair during future root CA rotations.</p><p>Please let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b0c98e0a17b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/simplified-google-cloud-certificate-authority-service-ca-rotation-for-kubernetes-workloads-8b0c98e0a17b">Simplified Google Cloud Certificate Authority Service CA Rotation for Kubernetes Workloads</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-region HA in Google Cloud]]></title>
            <link>https://medium.com/google-cloud/multi-region-ha-in-google-cloud-823b9f706578?source=rss-3df4bbb66206------2</link>
            <guid isPermaLink="false">https://medium.com/p/823b9f706578</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Sergey Shcherbakov]]></dc:creator>
            <pubDate>Thu, 18 Apr 2024 14:48:48 GMT</pubDate>
            <atom:updated>2024-04-19T05:15:57.347Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-wMkHFcpMgOJtYixfAy_w.png" /></figure><p>Google Cloud is one of the remarkable cloud “hyperscalers”. Hyperscalers are designed for massive capacity. They possess immense data center networks spread globally, allowing them to handle the enormous computing demands of large enterprises and applications with vast user bases. With that, Hyperscalers can enable applications with unsurpassed capabilities in scalability, reliability and global reach.</p><p>In this article we will try to explore the levels of possible application availability in Google Cloud with a focus on private internal networks. We’ll also provide actual infrastructure configuration examples.</p><p>Let’s imagine a business critical web application or API that provides its important service to the, potentially internal, business customers or end users. Often the business needs require the application to minimize its downtime, make it accessible to the users and responsive most of the time. A common measure of success of such metric is the application service uptime metric often aiming for targets like “99.99%” (“four nines”) or even “99.999%” (“five nines”) which translate into very few minutes of allowed downtime per year.</p><p>The typical mechanisms that the application design can rely upon to improve application Availability (as measured by uptime) are</p><ul><li><strong>Redundancy</strong> — run application on multiple independent hardware instances</li><li><strong>Load Balancing </strong>— distribute incoming network traffic across multiple application instances running on multiple independent hardware instances</li><li><strong>Failover</strong> — mechanisms to automatically detect failures and switch operation to a working application instance seamlessly</li><li><strong>Monitoring &amp; Alerting</strong> — robust monitoring systems to detect problems quickly and preferably proactively notify the team responsible for addressing them</li><li><strong>Self-healing —</strong> ability of the application components restart themselves or re-provision failing resources with minimal manual intervention</li></ul><p>In this article we will concentrate on how Google Cloud can help with the first three means of improving cloud application availability: redundancy, load balancing, failover.</p><h3><strong>Redundancy</strong></h3><p>A single application instance or application running in a single failure domain cannot sustain underlying hardware failure and hence the application would not be available to the end users in case of an underlying hardware outage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mW4jv8zKfyr1rRv9xf-t7g.png" /><figcaption>Fig. 1: Single application instance on single GCE VM</figcaption></figure><p>If our business objectives require addressing only a single Google Compute Engine (GCE) VM outage we would need to apply <strong>Redundancy</strong> and <strong>Load Balancing</strong> in order to improve application availability and resilience to that failure scenario:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DpDuW6cKQnoVzTmeuk35Vw.png" /><figcaption>Fig. 2: Multiple application instances in single GCE Zone</figcaption></figure><p>This setup is addressing the single GCE VM or application instance outage failure scenario.</p><p>Google Cloud hardware is organized into <a href="https://cloud.google.com/compute/docs/regions-zones/zone-virtualization"><em>clusters</em></a>. A cluster represents a set of compute, network, and storage resources supported by building, power, and cooling infrastructure. Infrastructure components typically support a single cluster, ensuring that clusters share few dependencies. However, components with highly demonstrated reliability and downstream redundancy can be shared between clusters. For example, multiple clusters typically share a utility grid substation because substations are extremely reliable and clusters use redundant power systems.</p><p>A <a href="https://cloud.google.com/compute/docs/regions-zones"><em>zone</em></a> is a deployment area within a region and Compute Engine implements a layer of abstraction between zones and the physical clusters where the zones are hosted. Each zone is hosted in one or more clusters and you can check the <a href="https://cloud.google.com/compute/docs/regions-zones/zone-virtualization">Zone virtualization</a> article for more details about that mapping.</p><p>To simplify reasoning without sacrificing accuracy it would be fair to assume that a GCE zone is a deployment area within a geographic region mapped to one or more clusters that can fail together, e.g. because of the power supply outage.</p><p>GCE zone outage is <a href="https://status.cloud.google.com/summary">not an impossible scenario</a> and a highly reliable application on Google Cloud typically seeks to sustain its service during such unfortunate event by running application replicas in multiple zones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGaUrlTA6NCfc0RbYaClkQ.png" /><figcaption>Fig. 3: Multiple application instances in multiple GCE Zones</figcaption></figure><p>With high zone availability <a href="https://cloud.google.com/compute/sla">SLA levels</a> provided by the Google Cloud Compute engine, the application setup in Figure 3 should be sufficient for majority of business use cases even for very demanding customers requiring high application service SLA levels.</p><p>Unfortunately, a full region outage is also <a href="https://www.businessinsider.com/google-cloud-data-center-london-outage-hottest-day-record-uk-2022-7">not an impossible scenario</a>.</p><p>The power of cloud hyperscalers is especially in that they provide customers with significantly better tools to survive disasters similar to <a href="https://www.reuters.com/article/idUSKBN2B20NT/">this one</a>, for example, than other cloud providers. Amongst other things, that is what differentiates “Hyperscalers” from small-scale or localized cloud service providers. In Google Cloud an application can run its replicas not only on power independent hardware within one data center or geographic location (probably connected to the same power plant in the neighborhood) but also across geographic location and even across continents!</p><p>So we are coming to the next level of application redundancy that is possible with Google Cloud: multi-regional application deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jE7J83TUsHoguxvrK5OOMg.png" /><figcaption>Fig. 4: Multiple application instances in multiple GCE Regions with global external load balancing</figcaption></figure><p>With that, a business critical application can now have a strategy for the entire site (region) failure and promise uptime to its critical clients even in that unlikely case.</p><h3>Load Balancing</h3><p>There needs to be some magic happening in order to seamlessly direct clients from around the world to the application instance replicas running in multiple geographic locations. And not only that. Whenever a VM, GCE zone or even full region goes down that magic needs to seamlessly redirect application clients to the healthy locations in other surviving region.</p><p>What are the options for load balancing that Google Cloud provides?</p><p>On the picture in Figure 4 the load balancer is located in Google Cloud but outside of any particular region. That kind of a global service can be provided by the following <a href="https://cloud.google.com/load-balancing/docs/application-load-balancer">types</a> of Google Cloud Load balancers:</p><ul><li>Global External Application Load Balancer</li><li>Classic Application Load Balancer in Premium Tier</li><li>Global External proxy Network Load Balancer</li><li>Classic Proxy Network Load Balancer</li></ul><p>Load balancers of all of these listed types load balancing traffic coming from the clients on the internet to the workloads running on Google Cloud.</p><p>An enterprise organization on Google Cloud would keep VPC networks private and expose application workloads to the internal company clients, which are also often located across the world.</p><p><em>Internal</em> load balancers on Google Cloud restrict access to the application to the clients in internal networks only. Unlike global external, <em>internal</em> load balancers on Google Cloud currently rely on the regional infrastructure. Availability of the applications exposed by internal load balancers can hence be affected by a single cloud region outage.</p><p>That means that for the internal clients the multi-regional application deployment depicted in Figure 4 logically changes to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xauNw4H1lLTDmhzblFpk2g.png" /><figcaption>Fig. 5: Multiple application instances in multiple GCE Regions with internal load balancing</figcaption></figure><p>The choice of internal load balancers on Google Cloud is even bigger:</p><ul><li>Regional Internal Application Load Balancer</li><li>Cross-region Internal Application Load Balancer</li><li>Regional internal proxy Network Load Balancer</li><li>Cross-region internal proxy Network Load Balancer</li><li>Internal passthrough Network Load Balancer</li></ul><p>There is an open question with the regional internal load balancing though. How would application clients know and seamlessly failover to the healthy region in case of a full region outage (a high bar challenge we have set us up to)?</p><p>To address that challenge we can revert to a well known technique called <a href="https://en.wikipedia.org/wiki/Round-robin_DNS">DNS load balancing</a> (or round-robin DNS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*17RU8OR_tQTt7IOethwokQ.png" /><figcaption>Fig. 6: Internal DNS load balancing with regional application backends</figcaption></figure><p>Fully managed Google Cloud DNS service offers an important and convenient tool to setup such cross-regional client access, it is called <a href="https://cloud.google.com/dns/docs/policies-overview#geolocation-policy">Geolocation routing policies</a> and it “<em>lets you map traffic originating from source geographies (Google Cloud regions) to specific DNS targets. Use this policy to distribute incoming requests to different service instances based on the traffic’s origin. You can use this feature with the internet, with external traffic, or with traffic originating within Google Cloud and bound for internal passthrough Network Load Balancers. Cloud DNS uses the region where the queries enter Google Cloud as the source geography.</em>”</p><p>Using Cloud DNS Geolocation routing policies in the setup depicted in Figure 6 application clients will automatically receive IP address of the Internal Load Balancer nearest to their geographic location from the Cloud DNS server.</p><p>Please note, that the Google Cloud DNS is a fully managed <em>global</em> service offering impressive <a href="https://cloud.google.com/dns/sla">SLO targets</a>. DNS cache on the application client side helps sustaining the application service available in the rare case of possible Cloud DNS service outage.</p><p>In fact, many parts of the <a href="https://cloud.google.com/load-balancing/docs/l7-internal/setting-up-l7-cross-reg-internal">Cross-region Internal Application Load Balancers </a>are <em>global</em> Google Cloud resources as well. Here is a more detailed diagram borrowed from the public Google Cloud pages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PFjluXOPFbz_x475" /><figcaption>Fig. 7: Global resources of the internal cross-region load balancer</figcaption></figure><h3>Failover</h3><p>But what exactly happens to the client connections and overall application availability in case of an individual GCE VM, zone or region outage? How would a DNS service know that it needs to resolve application hostnames to a different IP address to direct clients to another (healthy) region?</p><p>The Cloud DNS service and its Geolocation routing policies in particular has yet another feature which completes the multi-regional application deployment puzzle. It is <a href="https://cloud.google.com/dns/docs/zones/manage-routing-policies#health-checks">Health Checks</a>.</p><p>For Internal Passthrough Network Load Balancers (L4), Cloud DNS checks the health information on the load balancer’s individual backend instances to determine if the load balancer is healthy or unhealthy. Cloud DNS applies a default 20% threshold, and if at least 20% of backend instances are healthy, the load balancer endpoint is considered healthy. DNS routing policies mark the endpoint as healthy or unhealthy based on this threshold, routing traffic accordingly.</p><p>For Internal Application Load Balancers and Cross-region Internal Application Load Balancers, Cloud DNS checks the overall health of the internal Application Load Balancer, and lets the internal Application Load Balancer itself check the health of its backend instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6FRNR7-8iegZykNfqZcSA.png" /><figcaption>Fig. 8: Internal DNS load balancing with cross-region application backends with health checks</figcaption></figure><p>Cloud DNS health checks are a crucial solution component for achieving maximum application uptime. Without Cloud DNS ability to test the current status of the internal load balancers and application backends it would not be able to reason about which IP address exactly should be returned to the clients on their DNS requests in case of a region outage. Hence the seamless client failover to the health application instances would not be possible.</p><p>Please note that in order to achieve best results, the Time-To-Live parameter of the application A-record in Cloud DNS needs to be set to a minimal value. It could even be zero, in which case applications would contact DNS for a current IP before every call to the application service. The choice of the DNS record TTL value is a tradeoff between the application availability requirements and DNS service load and client response latency.</p><p>Internal load balancers maintain their own application backends health checks (Cloud DNS health checks are using a different mechanism) and in case of the cross-region internal application load balancers a load balancer operating in a particular region can automatically failover and redirect client requests to the application replicas running in another healthy region.</p><p>This setup addresses the “partial” region outage scenario. That is when only application backend instances are not available (e.g. GCE VMs are down or there is a error in the application preventing it from accepting incoming connections) but other services in the affected region (such as networking and load balancing) continue working.</p><h3>Configuration with Managed Instance Groups</h3><p>Let’s combine all pieces of an HA solution discussed before into a single picture and see how the Google Cloud resources need to be configured together to achieve the desired effect.</p><p>GCE Managed Instance Groups based scenarios, discussed in this article, are also relevant to the managed Kubernetes service on Google Cloud, GKE, as well. Kubernetes node pools in GKE are implemented as GCE MIGs. Hence, Kubernetes workload deployed to GKE on Google Cloud can be made multi-regional by deploying the application service to several GKE clusters in distinct regions. The load balancer resources for such set up can be provisioned using</p><ul><li><a href="https://gateway-api.sigs.k8s.io/">Kubernetes Gateway API</a> and <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api#gateway_controller">GKE Gateway Controller</a> in GKE clusters</li><li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/multi-cluster-ingress">Multi Cluster Ingress</a> resources in GKE clusters</li><li>Terraform resources (outside of GKE cluster)</li></ul><p>In this example we will use Terraform to set up load balancers, a common tool for declarative cloud infrastructure definition and provisioning, but it is also possible to achieve the same setup using the other two approaches. You can find the full Terraform example in <a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing">this</a> GitHub project.</p><p>Our first example will be based on the regional <a href="https://cloud.google.com/load-balancing/docs/internal">Internal Network Passthrough Load Balancers</a> and Google Compute Engine (GCE) <a href="https://cloud.google.com/compute/docs/instance-groups#managed_instance_groups">Managed Instance Groups</a> (MIGs).</p><p>In the last section of this article we’ll discus pros and cons of different load balancer and application backends combinations.</p><p><a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing/mig.tf">First</a> we define the GCE MIGs in two Google Cloud regions:</p><pre>// modules/mig/mig.tf:<br>module &quot;gce-container&quot; {<br>  source = &quot;terraform-google-modules/container-vm/google&quot;<br>  container = {<br>    image = var.image<br>    env = [<br>      {<br>        name = &quot;NAME&quot;<br>        value = &quot;hello&quot;<br>      }<br>    ]<br>  }<br>}<br><br>data &quot;google_compute_default_service_account&quot; &quot;default&quot; {<br>}<br>module &quot;mig_template&quot; {<br>  source               = &quot;terraform-google-modules/vm/google//modules/instance_template&quot;<br>  version              = &quot;~&gt; 10.1&quot;<br>  network              = var.network_id<br>  subnetwork           = var.subnetwork_id<br>  name_prefix          = &quot;mig-l4rilb&quot;<br>  service_account      = {<br>    email  = data.google_compute_default_service_account.default.email<br>    scopes = [&quot;cloud-platform&quot;]<br>  }<br>  source_image_family  = &quot;cos-stable&quot;<br>  source_image_project = &quot;cos-cloud&quot;<br>  machine_type         = &quot;e2-small&quot;<br>  source_image         = reverse(split(&quot;/&quot;, module.gce-container.source_image))[0]<br>  metadata             = merge(var.additional_metadata, { &quot;gce-container-declaration&quot; = module.gce-container.metadata_value })<br>  tags = [<br>    &quot;container-vm-test-mig&quot;<br>  ]<br>  labels = {<br>    &quot;container-vm&quot; = module.gce-container.vm_container_label<br>  }<br>}<br><br>module &quot;mig&quot; {<br>  source             = &quot;terraform-google-modules/vm/google//modules/mig&quot;<br>  version            = &quot;~&gt; 10.1&quot;<br>  project_id         = var.project_id<br><br>  region             = var.location<br>  instance_template  = module.mig_template.self_link<br>  hostname           = &quot;${var.name}&quot;<br>  target_size        = &quot;1&quot;<br>  <br>  autoscaling_enabled = &quot;true&quot;<br>  min_replicas = &quot;1&quot;<br>  max_replicas = &quot;1&quot;<br>  named_ports = [{<br>    name = var.lb_proto<br>    port = var.lb_port<br>  }] <br><br>  health_check_name = &quot;${var.name}-http-healthcheck&quot;<br>  health_check = {<br>    type = &quot;http&quot;<br>    initial_delay_sec   = 10<br>    check_interval_sec  = 2<br>    healthy_threshold   = 1<br>    timeout_sec         = 1<br>    unhealthy_threshold = 1<br>    port                = 8080<br>    response            = &quot;&quot;<br>    proxy_header        = &quot;NONE&quot;<br>    request             = &quot;&quot;<br>    request_path        = &quot;/&quot;<br>    host                = &quot;&quot;<br>    enable_logging      = true<br>  }<br>}<br><br>// mig.tf<br>module &quot;mig-l4&quot; {<br>  for_each      = var.locations<br>  source        = &quot;./mig&quot;<br>  project_id    = var.project_id<br>  location      = each.key<br>  network_id    = data.google_compute_network.lb_network.id<br>  subnetwork_id = data.google_compute_subnetwork.lb_subnetwork[each.key].id<br>  name          = &quot;failover-l4-${each.key}&quot;<br>  image         = var.image<br>}</pre><p><a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing/l4-rilb-mig.tf">Then</a>, let’s define two Cross-regional Internal Network Passthrough Load Balancers (L4 ILBs), each in respective region:</p><pre>// modules/l4rilb/l4-rilb.tf<br>locals {<br>  named_ports = [{<br>    name = var.lb_proto<br>    port = var.lb_port<br>  }]<br>  health_check = {<br>    type                = var.lb_proto<br>    check_interval_sec  = 1<br>    healthy_threshold   = 4<br>    timeout_sec         = 1<br>    unhealthy_threshold = 5<br>    response            = &quot;&quot;<br>    proxy_header        = &quot;NONE&quot;<br>    port                = var.lb_port<br>    port_name           = &quot;health-check-port&quot;<br>    request             = &quot;&quot;<br>    request_path        = &quot;/&quot;<br>    host                = &quot;1.2.3.4&quot;<br>    enable_log          = false<br>  }<br>}<br><br>module &quot;l4rilb&quot; {<br>  source        = &quot;GoogleCloudPlatform/lb-internal/google&quot;<br>  project       = var.project_id<br>  region        = var.location<br>  name          = &quot;${var.lb_name}&quot;<br>  ports         = [local.named_ports[0].port]<br>  source_tags   = [&quot;allow-group1&quot;]<br>  target_tags   = [&quot;container-vm-test-mig&quot;]<br>  health_check  = local.health_check<br>  global_access = true<br><br>  backends = [<br>    {<br>      group       = var.mig_instance_group<br>      description = &quot;&quot;<br>      failover    = false<br>    },<br>  ]<br>}<br><br>// l4-rilb-mig.tf<br>module &quot;l4-rilb&quot; {<br>  for_each            = var.locations<br>  source              = &quot;./modules/l4rilb&quot;<br>  project_id          = var.project_id<br>  location            = each.key<br>  lb_name             = &quot;l4-rilb-${each.key}&quot;<br>  mig_instance_group  = module.mig-l4[each.key].instance_group<br>  image               = var.image<br>  network_id          = data.google_compute_network.lb_network.id<br>  subnetwork_id       = data.google_compute_subnetwork.lb_subnetwork[each.key].name<br><br>  depends_on = [ <br>    google_compute_subnetwork.proxy_subnetwork <br>  ]<br>}</pre><p>And now let’s also <a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing/dns-l4-rilb-mig.tf">add</a> the global Cloud DNS record set configuration:</p><pre>// dns-l4-rilb-mig.tf<br>resource &quot;google_dns_record_set&quot; &quot;a_l4_rilb_mig_hello&quot; {<br>  name         = &quot;l4-rilb-mig.${google_dns_managed_zone.hello_zone.dns_name}&quot;<br>  managed_zone = google_dns_managed_zone.hello_zone.name<br>  type         = &quot;A&quot;<br>  ttl          = 1<br><br>  routing_policy {<br>    dynamic &quot;geo&quot; {<br>      for_each  = var.locations<br>      content {<br>        location  = geo.key<br>        health_checked_targets {<br>          internal_load_balancers {<br>              ip_address         = module.l4-rilb[geo.key].lb_ip_address<br>              ip_protocol        = &quot;tcp&quot;<br>              load_balancer_type = &quot;regionalL4ilb&quot;<br>              network_url        = data.google_compute_network.lb_network.id<br>              port               = &quot;8080&quot;<br>              region             = geo.key<br>              project            = var.project_id<br>            }<br>        }<br>      }<br>    }<br>  }  <br>}</pre><p>After we apply the Terraform configuration to the target Google Cloud project:</p><pre>terraform init<br>terraform plan<br>terraform apply</pre><p>we get all solution infrastructure components including a test application running in the GCE VMs in two distinct Google Cloud regions needed to perform end-to-end testing.</p><p>Let’s see how the clients can now access our application.</p><p>For testing of continuous request flow we can use the <a href="https://fortio.org/">Fortio</a> tool, which is a common tool for testing service mesh application performance. We will run it from a GCE VM attached to the same VPC where the load balancers are installed:</p><pre>gcloud compute ssh jumpbox<br><br>docker run fortio/fortio load --https-insecure -t 1m -qps 1 http://l4mig.hello.zone:8080</pre><p>The results after a minute of execution should look similar to the following:</p><pre>IP addresses distribution:<br>10.156.0.11:8080: 1<br>Code 200 : 258 (100.0 %)<br>Response Header Sizes : count 258 avg 390 +/- 0 min 390 max 390 sum 100620<br>Response Body/Total Sizes : count 258 avg 7759.624 +/- 1.497 min 7758 max 7763 sum 2001983<br>All done 258 calls (plus 4 warmup) 233.180 ms avg, 17.1 qps</pre><p>Please note the IP address of the L4 internal regional load balancer in the nearest region that is getting all of the calls.</p><p>In the second console window SSH into the VM in the GCE MIG group in the nearest region</p><pre>export MIG_VM=$(gcloud compute instances list --format=&quot;value[](name)&quot; --filter=&quot;name~l4-europe-west3&quot;)<br>export MIG_VM_ZONE=$(gcloud compute instances list --format=&quot;value[](zone)&quot; --filter=&quot;name=${MIG_VM}&quot;)<br><br>gcloud compute ssh --zone $MIG_VM_ZONE $MIG_VM --tunnel-through-iap --project $PROJECT_ID<br><br>docker ps</pre><p>Now let’s run the load test in the first console window again.</p><p>While the test is running switch to the second console window and execute</p><pre>docker stop ${CONTAINER}</pre><p>Switch to the first console window and notice the failover happening. The output at the end of the execution should look like following</p><pre>IP addresses distribution:<br>10.156.0.11:8080: 16<br>10.199.0.48:8080: 4<br>Code -1 : 12 (10.0 %)<br>Code 200 : 108 (90.0 %)<br>Response Header Sizes : count 258 avg 390 +/- 0 min 390 max 390 sum 100620<br>Response Body/Total Sizes : count 258 avg 7759.624 +/- 1.497 min 7758 max 7763 sum 2001983<br>All done 120 calls (plus 4 warmup) 83.180 ms avg, 2.0 qps</pre><p>Please note that the service VM in the Managed Instance has been automatically restarted. This functionality is provided by the Google Compute Engine Managed Instance groups and implements the forth component of application high availability posture, as you remember from the beginning of the article, it is <strong><em>self-healing</em></strong>.</p><h3>Cloud Run Backends</h3><p>Let’s consider a second scenario and assume that our cloud-native application is implemented as a <a href="https://cloud.google.com/run">Cloud Run</a> service.</p><p>The Cloud Run based scenarios, discussed in this article, are relevant for the <a href="https://cloud.google.com/functions/docs/concepts/version-comparison#new-in-2nd-gen">Cloud Functions</a> (2nd generation) application backends as well. Cloud Functions can be configured as load balancer backends similarly to the Cloud Run instances using the same Serverless Network Endpoint Groups resources.</p><p>The overall multi-region application deployment changes slightly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBWHPTGg6H1Tne0vW6Ip0w.png" /><figcaption>Fig. 9: Internal DNS load balancing with cross-region application backends in Cloud Run</figcaption></figure><p>We <a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing/modules/cr/cr2.tf">start</a> with defining two regional Cloud Run instances allowing invocations from unauthenticated clients:</p><pre>// modules/cr/cr2.tf<br>resource &quot;google_cloud_run_v2_service&quot; &quot;cr_service&quot; {<br>  project       = var.project_id<br>  name          = &quot;cr2-service&quot;  <br>  location      = var.location<br>  launch_stage  = &quot;GA&quot;<br><br>  ingress = &quot;INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER&quot;<br>  custom_audiences = [ &quot;cr-service&quot; ]<br><br>  template {<br>    containers {<br>      image = &quot;gcr.io/cloudrun/hello&quot; # public image for your service<br>    }<br>  }<br>  traffic {<br>    percent         = 100<br>    type = &quot;TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST&quot;<br>  }<br>}<br><br>resource &quot;google_compute_region_network_endpoint_group&quot; &quot;cloudrun_v2_sneg&quot; {<br>  name                  = &quot;cloudrun-sneg&quot;<br>  network_endpoint_type = &quot;SERVERLESS&quot;<br>  region                = var.location<br>  cloud_run {<br>    service = google_cloud_run_v2_service.cr_service.name<br>  }<br>}<br><br>resource &quot;google_cloud_run_v2_service_iam_member&quot; &quot;public-access&quot; {<br>  name     = google_cloud_run_v2_service.cr_service.name<br>  location = google_cloud_run_v2_service.cr_service.location<br>  project  = google_cloud_run_v2_service.cr_service.project<br>  role     = &quot;roles/run.invoker&quot;<br>  member   = &quot;allUsers&quot;<br>}<br><br>// cr.tf<br>module &quot;cr-service&quot; {<br>  for_each    = var.locations<br>  source      = &quot;./modules/cr&quot;<br>  project_id  = var.project_id<br>  location    = each.key<br>  image       = var.image<br>}</pre><p>And <a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing/l7-crilb-cr.tf">then</a> define global Internal Cross-Region Application Load Balancer resources:</p><pre>// modules/l7crilb/l7-crilb.tf<br>resource &quot;google_compute_global_forwarding_rule&quot; &quot;forwarding_rule&quot; {<br>  for_each              = var.subnetwork_ids<br>  project               = var.project_id<br>  <br>  name                  = &quot;${var.lb_name}-${each.key}&quot;<br><br>  ip_protocol           = &quot;TCP&quot;<br>  load_balancing_scheme = &quot;INTERNAL_MANAGED&quot;<br>  port_range            = var.lb_port<br>  target                = google_compute_target_https_proxy.https_proxy.self_link<br>  network               = var.network_id<br>  subnetwork            = each.value<br>}<br><br>resource &quot;google_compute_target_https_proxy&quot; &quot;https_proxy&quot; {<br>  project  = var.project_id<br><br>  name    = &quot;${var.lb_name}&quot;<br>  url_map = google_compute_url_map.url_map.self_link<br><br>  certificate_manager_certificates = [<br>    var.certificate_id<br>  ]<br>  lifecycle {<br>    ignore_changes = [<br>      certificate_manager_certificates<br>    ]<br>  }<br>}<br><br>resource &quot;google_compute_url_map&quot; &quot;url_map&quot; {<br>  project  = var.project_id<br><br>  name            = &quot;${var.lb_name}&quot;<br>  default_service = google_compute_backend_service.backend_service.self_link<br>}<br><br>resource &quot;google_compute_backend_service&quot; &quot;backend_service&quot; {<br>  project  = var.project_id<br><br>  load_balancing_scheme = &quot;INTERNAL_MANAGED&quot;<br>  session_affinity = &quot;NONE&quot;<br>  <br>  dynamic &quot;backend&quot; {<br>    for_each          = var.backend_group_ids<br>    content {<br>      group           = backend.value<br>      balancing_mode  = var.balancing_mode<br>      capacity_scaler = 1.0      <br>    }<br>  }<br><br>  name        = &quot;${var.lb_name}&quot;<br>  protocol    = var.backend_protocol<br>  timeout_sec = 30<br><br>  // &quot;A backend service cannot have a healthcheck with Serverless network endpoint group backends&quot;<br>  health_checks = var.is_sneg ? null : [google_compute_health_check.health_check.self_link]<br><br>  outlier_detection {<br>    base_ejection_time {<br>      nanos     = 0<br>      seconds   = 1<br>    }<br>    consecutive_errors = 3<br>    enforcing_consecutive_errors = 100<br>    interval {<br>      nanos     = 0<br>      seconds   = 1<br>    }<br>    max_ejection_percent = 50<br>  }<br><br>}<br><br>resource &quot;google_compute_health_check&quot; &quot;health_check&quot; {<br>  project    = var.project_id<br><br>  name   = &quot;${var.lb_name}&quot;<br>  http_health_check {<br>    port_specification = &quot;USE_SERVING_PORT&quot;<br>  }<br>}<br><br>// l7-crilb-cr.tf<br>module &quot;l7-crilb-cr&quot; {<br>  source            = &quot;./modules/l7crilb&quot;<br>  project_id        = var.project_id<br>  lb_name           = &quot;l7-crilb-cr&quot;<br><br>  network_id        = data.google_compute_network.lb_network.name<br>  subnetwork_ids    = { for k, v in data.google_compute_subnetwork.lb_subnetwork : k =&gt; v.id }<br>  certificate_id    = google_certificate_manager_certificate.ccm-cert.id<br>  backend_group_ids = [ for k, v in module.cr-service : v.sneg_id ]<br>  is_sneg           = true<br>}</pre><p>Please note that all load balancer related resources in this case are global (not regional).</p><p>In this demo case we need to <a href="https://github.com/sshcherbakov/professional-services/blob/cloud-dns-load-balancing/examples/cloud-dns-load-balancing/dns-l7-crilb-cr.tf">define</a> <a href="https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/cloud-dns-load-balancing/dns-l7-crilb-cr.tf">define</a> the Cloud DNS resources as well:</p><pre>// dns-l7-crilb-cr.tf<br>resource &quot;google_dns_record_set&quot; &quot;a_l7_crilb_cr_hello&quot; {<br>  name         = &quot;l7-crilb-cr.${google_dns_managed_zone.hello_zone.dns_name}&quot;<br>  managed_zone = google_dns_managed_zone.hello_zone.name<br>  type         = &quot;A&quot;<br>  ttl          = 1<br><br>  routing_policy {<br>    dynamic &quot;geo&quot; {<br>      for_each  = var.locations<br>      content {<br>        location  = geo.key<br>        health_checked_targets {<br>          internal_load_balancers {<br>              ip_address         = module.l7-crilb-cr.lb_ip_address[geo.key]<br>              ip_protocol        = &quot;tcp&quot;<br>              load_balancer_type = &quot;globalL7ilb&quot;<br>              network_url        = data.google_compute_network.lb_network.id<br>              port               = &quot;443&quot;<br>              project            = var.project_id<br>            }<br>        }<br>      }<br>    }<br>  }  <br>}</pre><p>For each region where the Cloud Run instance with our application is running we need to create a dedicated Cloud DNS routing policy.</p><p>Let’s now apply the Terraform to the target Google Cloud project and see how the clients can access our Cloud Run application.</p><p>Similarly to the Network Passthrough load balancer case described in the previous section, we’ll call our application endpoint exposed by the Cloud Run via the configured FQDN hostname:</p><pre>gcloud compute ssh jumpbox</pre><pre>docker run fortio/fortio load --https-insecure \<br>    -t 5m -qps 1 <a href="https://l7-crilb-cr.hello.zone/">https://l7-crilb-cr.hello.zone</a></pre><p>The results after a minute of execution should look similar to the following:</p><pre>IP addresses distribution:<br>10.156.0.55:443: 4<br>Code 200 : 8 (100.0 %)<br>Response Header Sizes : count 8 avg 216 +/- 0 min 216 max 216 sum 1728<br>Response Body/Total Sizes : count 8 avg 226 +/- 0 min 226 max 226 sum 1808<br>All done 8 calls (plus 4 warmup) 17.066 ms avg, 1.4 qps</pre><p>With our Fortio tool setup of one call per second, all calls have reached their destination.</p><p>The IP address that shows up in the output is the IP of the L7 internal cross-regional load balancer in the nearest region that is receiving all of our calls at the moment.</p><p>To simulate Cloud Run backend service outage, while running Fortio tool started in the previous step, in the second console window we can delete the backend resource in the nearest region from the load balancer backend service definition, e.g.:</p><pre>gcloud compute backend-services remove-backend l7-crilb-cr \<br>   --network-endpoint-group=cloudrun-sneg \<br>   --network-endpoint-group-region=europe-west3 \<br>   --global</pre><p>We can also check, to which regions the load balancer sends the traffic using:</p><pre>gcloud compute backend-services list --filter=&quot;name:l7-crilb-cr&quot;<br><br>NAME         BACKENDS                                         PROTOCOL<br>l7-crilb-cr  us-central1/networkEndpointGroups/cloudrun-sneg  HTTPS</pre><p>There is only one backend left running in the remote region. Yet, the Fortio results in the first console session show no hiccup or interruption:</p><pre>IP addresses distribution:<br>10.156.0.55:443: 4<br>Code 200 : 300 (100.0 %)<br>Response Header Sizes : count 300 avg 216.33333 +/- 1.106 min 216 max 220 sum 64900<br>Response Body/Total Sizes : count 300 avg 226.33333 +/- 1.106 min 226 max 230 sum 67900<br>All done 300 calls (plus 4 warmup) 193.048 ms avg, 1.0 qps</pre><p>What we have seen so far was the failover at the Internal Cross-Regional Application load balancer backend side. That is, the client application (Fortio) was still accessing the load balancer IP address in the nearest europe-west3 region. That can also be verified by running host l7-crilb-cr.hello.zone which will return the internal load balancer IP address from the subnetwork in the europe-west3 region.</p><p>What would happen in case of a full local region outage?</p><p>The first use case discussed above (Network Passthrough Load Balancer with MIG backends) illustrates that case. The Cloud DNS L4 health checks for the Network Passthrough load balancer test connection all way through to the actual application process running in GCE VMs (it is not possible to configure this type of load balancer with <a href="https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts">Serverless Network Endpoint Groups</a> backends for Cloud Run instances) and flips the IP address for the application service host name to the load balancer IP address in another region automatically.</p><p>Unfortunately, the <a href="https://cloud.google.com/dns/docs/zones/manage-routing-policies#health-checks">Cloud DNS health checks</a> for application (L7) load balancers cannot detect the outage of the application backend service with that fidelity level yet. Regional and Cross-regional Application load balancers are built on Envoy proxies internally and Envoy proxy based load balancers are only health checking the state and availability of the Envoy proxy instances, not the application backends themselves.</p><p>If an application running in Cloud Run experiences malfunction (e.g. as a result of internal program error) and returns 500 response codes, the Cloud DNS won’t detect that and won’t take action switching the load balancer IPs for the application hostname. That situation would be detected by the <a href="https://cloud.google.com/load-balancing/docs/https/setting-up-global-traffic-mgmt#configure_outlier_detection">Outlier Detection</a> feature of the Internal Cross-regional Application Load Balancer and the load balancer will redirect traffic to the healthy backend by looking at the rate of successful calls towards each backend.</p><p>A missing load balancer backend is not considered as an outage by the Cloud DNS health checks though. When a Cross-regional Application Load Balancer backend resources are not properly configured, load balancer has none of them, or has malfunctioning backends, the Cloud DNS won’t take action and won’t flip load balancer IPs automatically. The Cloud DNS health checks only check the availability of the internal Google Cloud infrastructure (Envoy proxies) supporting the application (L7) load balancers.</p><p>Yet, in case a full Google Cloud region outage the load balancer infrastructure would not be available fully and the Cloud DNS health check could detect that and act as expected.</p><h3>Options Choice</h3><p>When designing a highly available application service distributed across multiple regions on Google Cloud we need to consider the currently existing constraints in the Google Cloud services and pick a combination that would support the application requirements.</p><p>Here are the constraints and trade-offs that you should consider when picking the Google Cloud load balancer type for your distributed application.</p><p><strong>1. External vs Internal Load Balancers</strong></p><p>The Global External Application Load Balancer offers the tools for building geographically distributed application service with best availability guarantees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-l8le_X_Hupbj5M4CIqaGw.png" /><figcaption>Fig. 10: External application load balancer with backends in Cloud Run</figcaption></figure><p>Instead of relying upon the DNS load balancing trick, it provides application endpoint availability to the clients via global anycast Virtual IP addresses and smart global network infrastructure that cleverly routes client traffic to the infrastructure and services in the healthy regions.</p><p>The Google Cloud <em>internal</em> load balancers infrastructure is available via regional IP addresses instead and hence require additional mechanism, Cloud DNS load balancing suggested in this article, to address the full regional outage scenario.</p><p><strong>2. Managed Instance Groups vs Cloud Run Backends</strong></p><p>The Network Passthrough (L4) Load Balancers cannot be configured with Cloud Run backends. They don’t support Serverless Network Endpoint Groups (NEGs) required for Cloud Run and Cloud Function based backends at the moment.</p><p>Hence, if you are building a multi-regional application service that should only be available in the <em>internal</em> company VPC network and would like to address majority of possible outage scenarios (full region outage, individual regional Google Cloud service outage, application service malfunction), then Network Passthrough (L4) Load Balancer with GCE Managed Instance Groups is the only option for the application backends. Please remember, that the GCE MIGs is the mechanism supporting GKE node pools as well. Hence, the GCE MIG backends option is also applicable to the Kubernetes workloads running in GKE.</p><p>An important consideration for the Cloud Run backends in multiple regions is authentication.</p><p>In order to seamlessly continue service for the authenticated Cloud Run clients the Cloud Run instances in different regions must be configured with <a href="https://cloud.google.com/run/docs/configuring/custom-audiences">Custom Audiences</a>. In that way, the access token that client passes along with an authenticated call can be validated and accepted by the Cloud Run backends in all regions. Please note that the Custom Audiences is a feature available in Cloud Run instances of the 2nd generation. Cloud Run instances of the 1st generation can be used in the suggested multi-regional setup only when the application service does not need to authenticate clients.</p><p><strong>3. Network Passthrough (L4) vs Application (L7) Load Balancers</strong></p><p>Selecting the load balancer type also depends on the functionality that a load balancer can provide. In case of a Passthrough (L4) load balancer the application would need to implement the following tasks by itself (to name a few):</p><ul><li>terminate TLS connections</li><li>authenticate incoming calls</li><li>implement request routing</li></ul><p>The Application (L7) load balancers can help with that but their internal versions address less failure scenarios, in comparison to the Network Passthrough (L4) load balancer based solution, because of the current feature level of the Cloud DNS health check mechanism. For example, Cloud DNS would not flip an application service IP address in case if the application is experiencing internal malfunction (e.g. returning 50x error codes) or the load balancer backend is unavailable or missing altogether.</p><p>This is not a problem with External Application (L7) load balancers, since there is no need in Cloud DNS load balancing solution for exposing application in a high available way in multiple regions.</p><p>These mentioned “partial” or individual regional service infrastructure outage scenarios are handled by the Internal Cross-Regional Application Load Balancers on their backend side though. In addition, optional <a href="https://cloud.google.com/load-balancing/docs/https/setting-up-global-traffic-mgmt#configure_outlier_detection">Outlier Detection</a> load balancer configuration can help detecting application level malfunctions at the cost of wasting certain percentage of actual client requests in the case of outage.</p><h3>Conclusion</h3><p>Google Cloud goes beyond usual redundant deployments and offers architects and developers tools for building highly available application services across multiple geographic locations also for internal security restricted corporate use cases.</p><p>The choice of the particular combination of Google Cloud resources for improving multi-regional application availability depends on the individual applications requirements and features currently supported in the Google Cloud services such as network load balancers and Cloud DNS.</p><p>Enterprise security features in Google Cloud services get special attention and differentiate Google Cloud from other cloud hyperscalers. Please check one of my previous articles on <a href="https://medium.com/google-cloud/application-secrets-encryption-in-kubernetes-and-anthos-products-ae5de5905224">Application Secrets Encryption in Google Cloud Kubernetes products</a> for an example of possible with Google Cloud products.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=823b9f706578" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/multi-region-ha-in-google-cloud-823b9f706578">Multi-region HA in Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Application Secrets Encryption in Kubernetes and Anthos Products]]></title>
            <link>https://medium.com/google-cloud/application-secrets-encryption-in-kubernetes-and-anthos-products-ae5de5905224?source=rss-3df4bbb66206------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae5de5905224</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[anthos]]></category>
            <category><![CDATA[kubernetes-security]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Sergey Shcherbakov]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 15:16:34 GMT</pubDate>
            <atom:updated>2023-07-10T17:51:22.161Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xU3ljBVIOhjXKiX17qI7Q.jpeg" /></figure><p>Here is Yet Another Summary of the tips and tricks for keeping application secret values encrypted at rest in Kubernetes applications. This time covering Google Anthos product flavors as well.</p><p>The following small table of contents should lead you directly to the section of interest:</p><ul><li><a href="#4f61">Default Encryption</a></li><li><a href="#8b7a">Open Source Tools</a> ··········· <a href="#5c02">External Secrets</a> | <a href="#fd7a">B3rg1a$</a></li><li><a href="#d8e8">Google Secret Manager</a> ···· <a href="#b071">Direct API Call</a> | <a href="#5217">CSI Driver</a> | <a href="#4229">Init-Container</a></li><li><a href="#745f">KMS Plugin</a> ····················· <a href="#35a0">HSM-based</a> | <a href="#a021">for Cloud KMS</a> | <a href="#4f4c">KMS + GSM</a></li></ul><h3>Definitions and Introduction</h3><p>Application Secret is a piece of sensitive information, such as passwords, API keys, or other credentials, that an application needs to access resources or services. The secrets need to be stored securely.</p><p>Let’s define an application secret as anything about application that its developer wants to keep secret. For example, passwords, private TLS certificate keys, credentials to access other services and anything else that only the application process must be able to read at runtime.</p><h4>Secrets in Kubernetes Applications</h4><p>In Kubernetes, a central distributed database in the control plane, known as Etcd, actively stores all Kubernetes resource definitions and metadata. This storage includes the content of Kubernetes Secrets, ConfigMaps, and virtually all Kubernetes resources defined in the cluster.</p><p>Kubernetes applications typically access the application secrets directly through Kubernetes Secret resources. These get either mounted into the application pod file system or duplicated into an environment variable.</p><h4>Expectations</h4><p>What would an organization with high security requirements expectations from application secrets in Kubernetes?</p><ul><li><em>Resistance to physical access to medium — </em>the data needs to be stored encrypted at rest so that nobody getting physical access to the storage device could retrieve the information</li><li><em>Encryption with custom keys </em>— it should be possible to provide and manage your own generated encryption keys</li><li><em>Encryption enforcement</em> — it would be good if secrets encryption could be enforced so that application developers accessing the cluster would not be able to switch off protection</li><li><em>Seamless encryption — e</em>ncryption should not incur additional burden and requirements for the applications running on the cluster though</li><li><em>Multi-tenant access segregation</em> — it would be great if one application would not be able to access secrets of another application running on the same cluster</li><li><em>Convenient access to secret values</em> — when needed, authorized administrators should be able to conveniently access and inspect contents of the secrets</li><li><em>Key rotation</em> — last but often not least, standard security recommendations prescribe regular rotation (that is replacement) of the encryption keys. Automated rotation also helps to ensure that in case of a credential leak, processes are in place to rotate the credential without risking production downtime</li></ul><h3>Default Encryption</h3><p>How does Kubernetes, and Anthos products in particular, encrypt secrets?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q1QIT1qLDzgyYEWV5Ox1pw.png" /><figcaption>Default data encryption in Anthos products</figcaption></figure><p>In GKE in Google Cloud everything persisted is encrypted automatically and by default by Google infrastructure. I recommend reviewing the Google <a href="https://cloud.google.com/docs/security/encryption/default-encryption">data-at-rest encryption whitepaper</a> for the details of that promise.</p><p>Anthos is container platform for running modern apps anywhere consistently at scale.</p><p>An Anthos on bare metal cluster is configured by default to encrypt Etcd database contents using the <strong>standard Kubernetes encryption</strong>.</p><p>In Anthos on VMware there are several other options available.</p><p>First, VMware vSphere can be configured to encrypt virtual machines with <strong>vSphere VM encryption</strong> altogether.</p><p>“<a href="https://cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/always-on-secrets-encryption#:~:text=Always%2Don%20secrets%20encryption%20works,the%20key%20currently%20in%20use."><strong>Always-on secrets encryption</strong></a>” automatically generates keys used to encrypt application secrets before they are persisted in the Etcd database. The encryption keys in this case are stored on the Admin cluster data disk in case of the Anthos control plane version 1, and the user cluster control plane node in case of control plane version 2.</p><p>There is an option of encrypting Etcd database using “<a href="https://cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/hsm-secrets-encryption"><strong>HSM-based secrets encryption</strong></a>” on-prem hardware HSM appliance. There is support for one of the HSM vendors which is in public preview at the moment.</p><p>In the next sections we will go through the other options that can help matching higher security expectations.</p><p>What if we would like to</p><ul><li>Use our own generated encryption keys and be able to revoke access to the secrets whenever we’d like</li><li>Enforce teams to encrypt their application secrets while not adding them more headache</li><li>Allow application team admins accessing their secrets, but only their secrets, and not those of a neighboring application?</li></ul><h3>Open Source Tools</h3><p>Here are a couple of Open Source Tools that can help with those questions.</p><h4>External Secrets</h4><p>There is a rather well known open source project <a href="https://github.com/external-secrets/external-secrets">External Secrets Operator</a>.</p><p>It is a Kubernetes Operator that can synchronize secret values from different sources including Google Cloud Secret Manager service with Kubernetes Secret objects. They can then be consumed by applications as usual.</p><p>This approach provides some benefits but the secret values eventually land in the Etcd database of the cluster as well. Hence leaving several questions open ended.</p><p>External Secrets is an open source tool that is not covered by Google support or SLA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bbWcjQLEtnw26WAB" /><figcaption>External Secrets Operator</figcaption></figure><p><strong><em>Pros:</em></strong></p><ul><li>Kubernetes native widely used tool</li><li>Supports Workload Identity</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Secrets are replicated to Kubernetes Secrets and land in the cluster Etcd database</li><li>Doesn’t provide fine-grained Audit Access Logs</li><li>Open source driver and controller that needs to be installed and maintained</li><li>No Google Cloud support and no SLA</li></ul><h4>B3rg1a$</h4><p>Another well known open source tool that can help keeping application secrets secret is <a href="https://github.com/GoogleCloudPlatform/berglas"><strong>Berglas</strong></a>. It is written by a <a href="https://github.com/sethvargo">Google engineer</a> and utilizes Google Cloud services such as Cloud KMS, Cloud Storage and optionally Google Secret Manager to actually store application secrets content.</p><p>In this case secrets are stored securely in Google Cloud and do not leave a footprint in the Kubernetes cluster.</p><p>Cloud KMS service stores encryption keys and provides rich set of features such as Bring Your Own custom keys and automatic key rotation.</p><p>An interesting feature of Berglas is that it allows organizing application secrets into “folders”, allowing setting up fine grained access permissions.</p><p>On the negative side: no Google support and SLA and no enforcement for application teams to use it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*UWpG9Bsg_SDROcif.png" /><figcaption><a href="https://insights.project-a.com/a-painless-way-to-manage-secrets-in-google-kubernetes-engine/">https://insights.project-a.com/a-painless-way-to-manage-secrets-in-google-kubernetes-engine/</a></figcaption></figure><p><strong><em>Pros:</em></strong></p><ul><li>Integrates with Google services (Google Cloud Storage, Google Secret Manager)</li><li>Integrates with GKE and Anthos</li><li>Advanced IAM management with “secret folders”</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Advanced cluster setup</li><li>No Google support and no SLA</li></ul><h3>Google Secret Manager</h3><p><a href="https://cloud.google.com/secret-manager">Google Secret Manager</a> is a secure and convenient storage system for API keys, passwords, certificates, and other sensitive data. Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud.</p><p>It is a convenient storage for all kinds of application credentials with access via Google API, gcloud, with Terraform support and rich UI interface.</p><p>It allows setting up fine grained access permissions to the secrets and it is also fully integrated into the Google Cloud ecosystem including “Cloud Logging and Monitoring” and customer managed key encryption by Cloud KMS.</p><p>There are several ways to utilize Secret Manager for Kubernetes secrets encryption.</p><h4>Direct API Call</h4><p>The first and straightforward way to use Secret Manager is by accessing its Google API from the Kubernetes application directly.</p><p>The Google Cloud SDK library is a convenient means to implement Secret Manager access in the application.</p><p>With this approach</p><ul><li>Secrets remain encrypted in the Google Cloud at all times and only show up in the application process memory when they are needed</li><li>Fine grained access permissions and audit logs are provided by the Secret Manager</li></ul><p>On the negative side, is the need to perform custom application modification to make use of this approach.</p><p>It doesn’t cover overall Kubernetes cluster data encryption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W81C81so8_Ls29bQ" /><figcaption>Direct API Call to Google Secret Manager</figcaption></figure><p><strong><em>Pros:</em></strong></p><ul><li>Secrets remain in the pod memory. Calls to the GSM API are HTTPS-protected</li><li>Supports Workload Identity for fine-grained access and audit logs</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Hard to adopt for existing applications</li><li>More code to maintain</li></ul><h4>Kubernetes Secret Store CSI Driver</h4><p>There is a <a href="https://github.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp"><strong>Kubernetes CSI driver for Google Secret Manager</strong></a>.</p><p>Using this Open Source mechanism, Google Secret Manager (GSM) secrets can be mounted into the application pod’s file system directly.</p><p>That is a more portable approach that does not require application modification.</p><p>You can still have secret access isolation and fine grained access and audit logs for your secrets.</p><p>No Google support and no SLA for this open source component. But the project is actively monitored by the Google engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I2uDs3B8JsnGKdSM" /><figcaption>Kubernetes Secret Store CSI Driver</figcaption></figure><p>GSM CSI driver can also synchronize the Secret Manager secret values in the Google Cloud with the Kubernetes Secret objects inside the Kubernetes cluster. This is similar to External Secrets Operator discussed above, and relies upon the “<a href="https://secrets-store-csi-driver.sigs.k8s.io/topics/sync-as-kubernetes-secret.html">Sync as Kubernetes Secret</a>” Kubernetes feature. It is also possible to update the value of the secret mounted by CSI driver when there is a corresponding update in Google Secret Manager. That capability relies upon “<a href="https://secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html">Secret Auto Rotation</a>” Kubernetes API feature which is in Alpha yet.</p><p><strong><em>Pros:</em></strong></p><ul><li>Kubernetes native way of fetching secrets</li><li>Abstracting secret store from the app makes applications more portable</li><li>Separation of duties possible</li><li>Workload Identity for fine grained access and audit logs</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Open source driver and controller that needs to be installed and maintained</li><li>No Google support and no SLA</li><li>Increased severity of vulnerabilities like risk for directory traversal attacks</li></ul><h4>Init-Container</h4><p>Another trick to fetch Secret Manager secrets into the Kubernetes application pods, is to fetch them, during pod start-up, in an init-container via a call to the Secret Manager API.</p><p>This can be done for example by scripting a standard gcloud call from a Cloud SDK container image and saving the secret value to the pod’s ephemeral file system. Please check <a href="https://github.com/boredabdel/gke-secret-manager/tree/main/hello-secret-init-container">this project</a> for an implementation example.</p><p>This approach is rather straightforward, and enjoys other benefits that Secret Manager offers.</p><p>The drawback is that</p><ul><li>secret values stay visible in the pods file system (directory traversal attack)</li><li>another container needs to be maintained</li><li>there may be difficulties in the service mesh environments as well</li><li>additional mechanism is needed to ensure that the application gets updated secret values at runtime</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aokCKbqaZkceSsrL" /><figcaption>Init-Container</figcaption></figure><p><strong><em>Pros:</em></strong></p><ul><li>Straightforward implementation</li><li>Supports Workload identity</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Needs to be added into each team manifest</li><li>Another container to scan and secure</li><li>Secrets can be stored in the file system</li><li>Secrets are cached and it’s hard to control their lifetime</li><li>Could be incompatible with Anthos Service Mesh</li></ul><h3>Kubernetes KMS Plugin</h3><p>Let’s now check the standard mechanism that Kubernetes provides for customizing encryption of the Etcd database content.</p><p>Kubernetes allows <strong>customization</strong> of the etcd data encryption.</p><p>KMS encryption provider uses an <strong>envelope encryption</strong> scheme to encrypt data in etcd. The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption. The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS. The KMS provider uses gRPC to communicate with a specific <strong>KMS plugin</strong>. The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s) as the Kubernetes control plane, is responsible for all communication with the remote KMS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*olKpUAaA48x5XyZ2" /><figcaption>Kubernetes KMS Plugin</figcaption></figure><p>Kubernetes KMS encryption provider uses so-called envelope encryption mechanism to encrypt data in Etcd.</p><p>That is, the case when there are two keys participating in the actual data encryption. One is the so-called key encryption key or (KEK) is always stored remotely at a remote KMS provider and never leaves it. Another one (DEK) is used for the actual data encryption and is kept encrypted by the remote key when stored locally together with the data that it encrypts.</p><p>Kubernetes KMS plugin is an additional control plane component that kube-apiserver contacts to decrypt the DEK.</p><p>Whenever the kube-apiserver stores secrets, or any other resources, to the Etcd, it first encrypts them with the data encryption key DEK.</p><p>And to decrypt the DEK, kube-apisever reaches out to the KMS plugin.</p><p>KMS plugin can now call any remote KMS provider to perform decryption or encryption .</p><p>The plugin is typically a container image that can either</p><ul><li>be started as a pod on the control plane node</li><li>or even be added as a sidecar to the kube-apiserver</li></ul><h4>HSM-based Secrets Encryption in Anthos on VMware</h4><p>Anthos on VMware clusters can be configured with <a href="https://cloud.google.com/anthos/clusters/docs/on-prem/latest/how-to/hsm-secrets-encryption"><strong>HSM-based secrets encryption</strong></a>. (This feature is yet in preview)</p><p>In that case, cluster Etcd data is getting effectively encrypted by the keys stored in an on-prem hardware HSM appliance.</p><p>The HSM provider provides this functionality via the (Kubernetes KMS plugin mechanism) as well.</p><p>Significant advantage of using the KMS plugin is that the data in Etcd is always encrypted and there is no way to bypass that without reconfiguring the entire cluster.</p><p>HSM keys also provide a high degree of security and compliance and remain in full customer control.</p><p>On the downside is to mention that an HSM appliance is an expensive system to maintain.</p><p>Also, when encrypted with a KMS plugin, Etcd data of all applications are encrypted with the same key.</p><p><strong><em>Pros:</em></strong></p><ul><li>Mandatory encryption of all cluster secrets at rest</li><li>Customer Managed hardware-backed encryption keys</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Requires on-prem HSM hardware appliance</li><li>Not possible to separate tenant access permissions</li><li>Only available in Anthos on VMware</li></ul><h4><strong>Kubernetes KMS Plugin for Cloud KMS</strong></h4><p>Google <a href="https://cloud.google.com/security-key-management"><strong>Cloud KMS</strong></a><strong> </strong>service can be encryption backend for Kubernetes Etcd data.</p><p>There is even a <a href="https://github.com/GoogleCloudPlatform/k8s-cloudkms-plugin"><strong>KMS plugin for Google Cloud KMS service</strong></a>.</p><p>It is an open source component that can be used in on-prem Anthos clusters to encrypt Etcd data on the control plane nodes with the key encryption keys stored in the Cloud.</p><p>We get the same advantages of mandatory secrets encryption and full customer control over the actual encryption keys.</p><p>And the same disadvantage of missing isolation by application.</p><p>With a reliable network connection to the Google Cloud such a combination can provide similar or sometimes even better reliability guarantees than on-prem HSM systems.</p><p>It should also be mentioned that this open source component is not covered by Google support and SLA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5OWmxWk7P0AJ9sXN" /><figcaption>Kubernetes KMS Plugin for Cloud KMS</figcaption></figure><p><strong><em>Pros:</em></strong></p><ul><li>Mandatory encryption of all cluster secrets at rest</li><li>Customer Managed Encryption Keys</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Not possible to separate tenant access permissions</li><li>Advanced cluster setup</li><li>No Google support and no SLA</li></ul><p>Here is an example of configuration that turns up Cloud KMS encryption in (an on-prem cluster’s Etcd database).</p><pre># Copyright 2023 Google LLC.<br># SPDX-License-Identifier: Apache-2.0<br><br>sudo mkdir -p /etc/kubernetes/user_patches<br>sudo bash<br>cat &lt;&lt;EOF &gt; /etc/kubernetes/user_patches/kube-apiserveruser500+strategic.yaml<br>apiVersion: v1<br>kind: Pod<br>spec:<br>  containers:<br>  - name: kube-apiserver<br>    volumeMounts:<br>    - mountPath: /var/run/kmsplugin<br>      name: socket-volume<br>  volumes:<br>  - hostPath:<br>      path: /var/kms-plugin<br>      type: DirectoryOrCreate<br>    name: socket-volume<br>EOF<br>exit</pre><pre># Copyright 2023 Google LLC.<br># SPDX-License-Identifier: Apache-2.0<br><br>sudo vi /etc/kubernetes/pki/encryption_config.yaml<br><br>apiVersion: apiserver.config.k8s.io/v1<br>kind: EncryptionConfiguration<br>resources:<br>  - resources:<br>    - secrets<br>    - configmaps<br>    providers:<br>    - kms:<br>        name: cloudKmsPlugin<br>        endpoint: unix:///var/run/kmsplugin/socket.sock<br>        cachesize: 100<br>    - aescbc:<br>        keys:<br>        - name: &quot;etcd_key&quot;<br>          secret: &quot;07kZOy4Y5cm6/EYm4aFayNvMDvIWvXRrThO58t/Gs+E=&quot;<br>    - identity: {}</pre><p>We basically need to</p><ul><li>Deploy plugin to the control plane nodes</li><li>Let the kube-apiserver know about the plugin</li><li>And turn in on in the Kubernetes encryption configuration</li></ul><h4>Kubernetes KMS Plugin + Secret Manager</h4><p>Application secrets encryption is not a single-choice decision.</p><p>KMS plugin and other (application secret encryption mechanisms) can be combined together.</p><p>Together they can help reaching our original goals:</p><ul><li>Mandatory encryption of all secrets in the cluster.</li><li>Fine grained access control and isolation of secrets of different applications</li><li>All that while keeping full customer control over encryption keys that protect the data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*Oak3PKiGyaf1_jfx" /><figcaption>Kubernetes KMS Plugin + Secret Manager</figcaption></figure><p><strong><em>Pros</em>:</strong></p><ul><li>Mandatory encryption of all cluster secrets at rest</li><li>Customer Managed Encryption Keys</li><li>Workload Identity for fine-grained access and audit logs</li></ul><p><strong><em>Cons</em>:</strong></p><ul><li>Advanced cluster setup</li></ul><h3>Summary</h3><p>Let’s now summarize our overview.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q_PxXBvX0QJWA8F_wxLLeQ.png" /><figcaption>Application Secrets Encryption Options Summary</figcaption></figure><p>We went through the (default application secrets encryption mechanisms) available in GKE and Anthos Kubernetes clusters out of the box.</p><p>Covered two common (open source tools) that help keeping application secrets outside of the Kubernetes clusters.</p><p>Discussed three ways of accessing application secrets in Google Secret Manager directly from the applications.</p><p>We have also explored the Kubernetes KMS provider and plugin mechanism for the cluster data encryption in the Kubernetes control plane.</p><p>And now to the takeaways..</p><p><strong>Google Anthos</strong> products can support security requirements of advanced customers</p><p><strong>Google Cloud</strong> offers services that help customers <strong><em>harden</em></strong> Kubernetes <strong><em>security</em></strong> and simplify reaching <strong><em>compliance</em></strong> goals</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae5de5905224" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/application-secrets-encryption-in-kubernetes-and-anthos-products-ae5de5905224">Application Secrets Encryption in Kubernetes and Anthos Products</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ingress in Google Kubernetes Products]]></title>
            <link>https://medium.com/google-cloud/ingress-in-google-kubernetes-products-f22ded21f4ed?source=rss-3df4bbb66206------2</link>
            <guid isPermaLink="false">https://medium.com/p/f22ded21f4ed</guid>
            <category><![CDATA[anthos]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[anthos-service-mesh]]></category>
            <dc:creator><![CDATA[Sergey Shcherbakov]]></dc:creator>
            <pubDate>Fri, 16 Dec 2022 08:27:31 GMT</pubDate>
            <atom:updated>2024-01-18T16:29:48.437Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*qs4OUGmey9hdVPhDBMCZKw.png" /></figure><p>Here is my attempt to summarise and disambiguate terms often used in technical discussions around arranging network ingress traffic into [single] Kubernetes clusters running in Google Cloud (<a href="https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview">GKE</a>) or on-premise (<a href="https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/concepts/about-bare-metal">Anthos on Bare Metal</a>, <a href="https://cloud.google.com/anthos/clusters/docs/on-prem/latest/overview">Anthos on VMware</a>).</p><p>The following small table of contents should lead you directly to the section of interest.</p><ul><li><a href="#bb7b">Options Breakdown</a></li><li><a href="#48d0">Service</a> ················ <a href="#73aa">GKE</a> | <a href="#e92f">Anthos</a> | <a href="#40c3">TLS</a> | <a href="#5020">Example</a></li><li><a href="#ac05">Ingress</a> ················ <a href="#d3f7">GKE</a> | <a href="#08eb">Anthos</a> | <a href="#9fdb">TLS</a> | <a href="#e81c">Example</a></li><li><a href="#cf3b">Gateway</a> ·············· <a href="#134a">GKE</a> | ··········· | <a href="#f3f4">TLS</a> | <a href="#bc1f">Example</a></li><li><a href="#e263">Istio Gateway</a> ······· <a href="#c3e2">GKE</a> | <a href="#9fe0">Anthos</a> | <a href="#2d5e">TLS</a> | <a href="#63a5">Example</a></li><li><a href="#5a14">NEG Controllers</a> ·· <a href="#4177">GKE</a> | ··········· | <a href="#e70b">TLS</a> | <a href="#137d">Example</a></li></ul><p>How many times did you need to connect a Kubernetes cluster to the outer world, make it accessible from public internet or within your organisation while keeping it secure and maintaining control over the setup?</p><p>Very often when such discussions commence there is no clear picture of the options available. The <strong><em>ambiguity</em></strong> of the component names and <strong><em>overloaded terms</em></strong> contribute to the confusion to large extent.</p><p><em>“We will set up an Ingress”</em> — wait, did you mean a particular implementation? In which system is that going to happen? Is that all that is going to be needed?</p><p><em>“TLS will be terminated at the Gateway”</em> — which kind of Gateway are you referring to? Are you sure that this will match your mutual TLS requirements?</p><p><em>“Who is going to own the network ingress in the team?” — </em>what am I referring to here?</p><p><em>“Do we need to take care about load balancer in front?” — </em>there are even more options about that!</p><p>Often times I’ve been in such discussions.</p><p>One way to navigate the rather ambiguous terminology and landscape is to list all possible variants and structurise them. Once all possible options are listed and structured it becomes possible to land on the same page and reduce confusion.</p><p>I will try to provide a succinct reference to the terms with their multiple meanings, short infrastructure design pattern diagrams and example configuration snippets.</p><h3>Options Breakdown</h3><p>It is convenient to start the network traffic ingress options list by classifying them by the product flavour.</p><p>Available options and short terms meaning depend on whether we are in the context of Google Cloud (GKE service) or Anthos clusters outside of Google Cloud. Whether the <a href="https://cloud.google.com/service-mesh/docs/overview">Anthos Service Mesh</a> (ASM) is available and/or required or not.</p><p>Please note that not all of the options provide an opportunity for setting up full fledged mutual TLS authentication. Such options are marked with a yellow question mark below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9UEtRTR4ZcMx-LxoEmOGOg.png" /></figure><p>This is the network flow summary for the options in the table (double arrows mean possible mutual TLS setup, single arrows are one-way TLS connections):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*GJnw4_1SgdcWGCYPAJ0AIA.png" /></figure><p>We will now look into each of them in a greater detail.</p><h3>Service</h3><p><a href="https://kubernetes.io/docs/concepts/services-networking/service/">Kubernetes Service</a> resource provides an option of defining the Kubernetes Service of type LoadBalancer. It is responsibility of the system external to the Kubernetes cluster to configure infrastructure resources outside of the Kubernetes cluster by looking at the Kubernetes Service definition.</p><p>Typically in this scenario the application service endpoints are exposed to the incoming (L4) traffic directly. Hence the TLS is terminated by the application itself.</p><p>This option can be useful in case if the application already has its own special component that can accept TLS connections and route them to the other application components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-ajcU50KY7627Y3g" /></figure><h3>Service — GKE</h3><p>In GKE, the LoadBalancer Service type automatically manages Google Cloud L4 TCP load balancer resources. They will be created by the Google Cloud automatically.</p><h3>Service — Anthos</h3><p>In Anthos on Bare Metal, the LoadBalancer Service can automatically manage the bundled MetalLB load balancer, which is part of the Anthos installation and is represented by the software components running on the cluster master nodes.</p><h3>Service — TLS</h3><p>The TLS certificates need to be configured in the application services themselves, e.g. using <a href="https://cert-manager.io/docs/">cert-manager</a> and <a href="https://github.com/jetstack/google-cas-issuer">Google CAS Issuer</a> issued certificates. These are eventually stored in the Kubernetes Secrets that get typically mounted into the application pods.</p><h3>Service — Example</h3><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: my-service<br>spec:<br>  selector:<br>    app.kubernetes.io/name: MyApp<br>  ports:<br>    - protocol: TCP<br>      port: 443<br>      targetPort: 9376<br>  clusterIP: 10.0.171.239<br>  type: LoadBalancer<br>status:<br>  loadBalancer:<br>    ingress:<br>    - ip: 192.0.2.127</pre><h3>Ingress</h3><p><a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Kubernetes Ingress</a> is an API object that manages external access to the services in a cluster, typically HTTP. Ingress may provide load balancing, SSL termination and name-based virtual hosting.</p><p>An <a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/">Ingress Controller</a> is responsible for fulfilling the Ingress.</p><p>The limitations of the Kubernetes Ingress resources include:</p><ul><li>Not possible to configure mutual TLS connection between the ingress proxy and the application backend services (corresponding settings are missing in the Kubernetes resource definition)</li><li>Single Ingress resource that incorporates both load balancer as well as routing configuration doesn’t allow separation of concerns and flexible access configuration for multi-tenant environments (all concerns are in responsibility of a single team)</li><li>In case of the Istio Ingress controller (which is present by default in ABM installations) it is also not possible to use TLS and encrypt network traffic between the reverse proxy and the application service</li></ul><p>Because of its limitations the use of Kubernetes Ingresses is being gradually discouraged in favor of the Kubernetes Gateway resources described in the next chapter.</p><h3>Ingress — GKE</h3><p>In GKE, the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress">GKE Ingress controller</a> automatically manages Google Cloud L7 External and L7 Internal Load Balancer resources.</p><p>The incoming L7 TLS connections terminate at the GCE L7 External Load Balancer or L7 Regional Internal Load Balancer. The GCE load balancers forward incoming traffic to the Kubernetes cluster nodes.</p><p><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#container-native_load_balancing">Container Native load balancing</a> can be configured on the target Kubernetes Service objects to make GCE load balancer forward traffic directly to the pods instead of the node ports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uMwhIuV8iKQlOyRS" /></figure><h3>Ingress — Anthos</h3><p>In ABM, the default out-of-the box installation of the <a href="https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/">Istio Ingress controller</a> provides an option of automatically managing the Istio Ingress proxy processes as Kubernetes workloads. Unfortunately, this out-of-the-box option has all Kubernetes Ingress limitations listed above.</p><p>In order to configure the downstream connection between the Ingress proxy process and application backend service with mutual TLS an Istio Ingress implementation that supports such implementation can optionally be used. One of such implementations is the <a href="https://docs.nginx.com/nginx-ingress-controller/">Nginx Ingress controller</a>, which can be configured with TLS downstream connection encryption and TLS client certificates via custom Kubernetes resource annotations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2AtifTxwdHQXtAPdlfHyw.png" /></figure><h3>Ingress — TLS</h3><p>TLS configuration such as ingress proxy server and client certificates are configured in the Kubernetes Ingress resource definition either directly or via additional custom Kubernetes resource annotations.</p><p>The Kubernetes Ingress can either use the TLS certificates and their private keys from the referenced Kubernetes Secret resources of type tls. The Kubernetes Secrets with TLS certificates and keys in that case can be managed by the <a href="https://cert-manager.io/docs/">cert-manager</a> and <a href="https://github.com/jetstack/google-cas-issuer">Google CAS Issuer</a>.</p><p>Alternatively, it is <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-multi-ssl#pre-shared-certs">possible</a> to specify the GCE SSL Certificate resource name in the ingress.gcp.kubernetes.io/pre-shared-cert Kubernetes Ingress annotation. The load balancer will then be configured with the referenced preconfigured SSL certificate.</p><h3>Ingress — Example</h3><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: ingress-myservicea<br>  annotations:<br>    cert-manager.io/issuer: &quot;letsencrypt-staging&quot;<br>spec:<br>  tls:<br>    - hosts:<br>        - ingress-demo.example.com<br>      secretName: ingress-demo-tls<br>  rules:<br>  - host: myservicea.foo.org<br>    http:<br>      paths:<br>      - path: /<br>        pathType: Prefix<br>        backend:<br>          service:<br>            name: myservicea<br>            port:<br>              number: 80<br>  ingressClassName: nginx</pre><h3>Gateway</h3><p><a href="https://gateway-api.sigs.k8s.io/">Kubernetes Gateway</a> is a collection of resources that model service networking in Kubernetes (GatewayClass, Gateway, HTTPRoute, TCPRoute, Service, etc.).</p><p>Kubernetes Gateway is a recent alternative to the Kubernetes Ingresses that addresses its drawbacks and limitations and is encouraged to be used instead of the Kubernetes Ingress going forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rx-LET3fcKs9t57O" /></figure><h3>Gateway — GKE</h3><p>In GKE, the Kubernetes Gateway is implemented by the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api">GKE Gateway controller</a> (this feature is in public preview at the moment). Via its Kubernetes resource configuration, it is possible to automatically manage:</p><ul><li>gke-l7-rilb<em> </em>Regional internal HTTP(S) load balancers built on <a href="https://cloud.google.com/load-balancing/docs/l7-internal">Internal HTTP(S) Load Balancing</a></li><li>gke-l7-gxlb<em> </em>Global external HTTP(S) load balancers built on <a href="https://cloud.google.com/load-balancing/docs/https">Global external HTTP(S) load balancer (classic)</a></li><li>gke-l7-rilb-mc<em> </em>Multi-cluster regional load balancers built on <a href="https://cloud.google.com/load-balancing/docs/l7-internal">Internal HTTP(S) Load Balancing</a></li><li>gke-l7-gxlb-mc<em> </em>Multi-cluster global load balancers built on <a href="https://cloud.google.com/load-balancing/docs/https">Global external HTTP(S) load balancer (classic)</a></li><li>gke-td<em> </em>Multi-cluster <a href="https://cloud.google.com/traffic-director/docs/gke-gateway-overview">Traffic Director service mesh</a></li></ul><p>The Anthos Service Mesh offers a preview of the <a href="https://cloud.google.com/service-mesh/docs/managed/service-mesh-cloud-gateway">ASM Cloud Gateway Controller</a>. In addition to the Kubernetes Gateway types listed previously, it introduces</p><ul><li>asm-l7-xlb<em> </em>for Global external HTTP(S) load balancers built on <a href="https://cloud.google.com/load-balancing/docs/https">Global external HTTP(S) load balancer (classic)</a></li></ul><p>The ASM Cloud Gateway provides unified way to configure Anthos Service Mesh ingress gateway with <a href="https://cloud.google.com/load-balancing">Cloud Load Balancing</a> simultaneously through <a href="https://gateway-api.sigs.k8s.io/">Kubernetes Gateway API</a>. When this ASM Gateway API object is deployed, the following things will happen:</p><ul><li>An external HTTP(S) Load Balancer is deployed and configured. It may take a few minutes for it to come up but when it does, the Gateway will indicate the IP address and will be annotated with the names of the Compute Engine load balancer resources that were created.</li><li>An Anthos Service Mesh ingress gateway Deployment is created in the istio-ingress namespace. This creates the Envoy proxy instances that will receive traffic from the Cloud Load Balancer.</li><li>The Cloud Load Balancer will encrypt and route all traffic to the Anthos Service Mesh ingress gateway.</li></ul><h3>Gateway — TLS</h3><p>The TLS certificates and keys for the Kubernetes Gateway managed load balancers can be <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-security#gateway-management">provided</a> using following methods</p><ul><li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-security#kubernetes-secrets">Kubernetes Secrets</a> — Can be supplied and managed by cert-manager and Google CAS Issuer</li><li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-security#ssl-certificates">SSL certificates</a> — Can be pre-provisioned (e.g. using Terraform) separately from the GKE cluster</li><li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-security#certificate-manager">Certificate Manager</a></li></ul><h3>Gateway — Example</h3><pre>kind: Gateway<br>apiVersion: gateway.networking.k8s.io/v1beta1<br>metadata:<br>  name: my-gateway<br>spec:<br>  gatewayClassName: gke-l7-rilb<br>  listeners:<br>  - name: https<br>    protocol: https<br>    port: 443<br>    tls:<br>      mode: Terminate<br>      options:<br>        networking.gke.io/pre-shared-certs: store-example-com<br>---<br>apiVersion: gateway.networking.k8s.io/v1beta1<br>kind: HTTPRoute<br>metadata:<br>  name: httproute-example<br>spec:<br>  parentRefs:<br>  - name: my-gateway<br>  hostnames:<br>  - my.example.com<br>  rules:<br>  - matches:<br>    - path:<br>        type: PathPrefix<br>        value: /bar<br>    backendRefs:<br>    - name: my-service1<br>      port: 8080</pre><p>Please note the separate Kubernetes resources “Gateway” and “HTTPRoute” that define different aspects of the Kubernetes Gateway. This separation allows Separation of Concerns and Segregation of Duties. Different teams with different access permissions can now be tasked with managing external and internal aspects of ingress network traffic.</p><h3>Istio Gateway</h3><p>When <a href="https://cloud.google.com/service-mesh/docs/overview">Anthos Service Mesh</a> (ASM) is installed and available in the Kubernetes cluster, it provides its own way of managing and organising the network traffic ingress into the service mesh.</p><p>ASM brings in the Istio Gateway Kubernetes resource and component. It is different from the Kubernetes Gateway resource object described in the previous section.</p><p>Anthos Service Mesh gives you the option to deploy and manage <a href="https://cloud.google.com/service-mesh/docs/gateways">Istio gateways</a> as part of your service mesh. A gateway describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections. Gateways are Envoy proxies that provide you with fine-grained control over traffic entering and leaving the mesh. Gateways are primarily used to manage ingress traffic, but you can also configure gateways to manage other types of traffic.</p><p>There are multiple ways of exposing Istio Gateways to the external clients. All options for ASM meshes running in the GKE clusters are well described in this <a href="https://cloud.google.com/architecture/exposing-service-mesh-apps-through-gke-ingress">public article</a>.</p><h3>Istio Gateway — GKE</h3><p>The basic way to expose the ASM mesh running in GKE cluster to the external clients is using the Kubernetes Service objects as depicted on the following diagram. Creating a Kubernetes Service resource object of LoadBalancer type on GKE creates corresponding Google Cloud L4 load balancers automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqXcYbwLEAKNYMc23oI4uw.png" /></figure><p>Please note that the Anthos Service Mesh can integrate with the <a href="https://cloud.google.com/certificate-authority-service">Certificate Authority Service</a> as a source for the mesh workload certificates.</p><h3>Istio Gateway — Anthos</h3><p>A Kubernetes Service resource object is needed to expose the ASM mesh running in an Anthos on Bare Metal (ABM) cluster too. The type of the load balancer automatically controlled by the ABM components depends on the <a href="https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/installing/load-balance">cluster installation configuration</a>:</p><ul><li>Bundled L2 load balancing</li><li>Bundled L3 load balancing with BGP</li><li>Manual load balancing</li></ul><p>The following diagram illustrates the Kubernetes component required to expose ASM mesh running in an ABM cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PXxurGWfPIShrTUJRxRWA.png" /></figure><h3>Istio Gateway — TLS</h3><p>The ASM installations on both GKE and ABM can be configured with TLS certificates provisioned by the Google <a href="https://cloud.google.com/certificate-authority-service/docs">Certificate Authority Service</a> (CAS).</p><p>The ASM can be configured with the CAS resource names during the installation process and manages the workload TLS certificates lifetime automatically at runtime fetching them from the supplied CAS service pool or instance.</p><h3>Example</h3><pre>apiVersion: networking.istio.io/v1beta1<br>kind: Gateway<br>metadata:<br>  name: frontend-gateway<br>  namespace: frontend<br>spec:<br>  selector:<br>    istio: ingressgateway<br>  servers:<br>  - port:<br>      number: 443<br>      name: https<br>      protocol: HTTPS<br>    hosts:<br>    - ext-host.example.com<br>    tls:<br>      mode: SIMPLE<br>      credentialName: ext-host-cert  # Kubernetes Secret name<br>---<br>apiVersion: networking.istio.io/v1beta1<br>kind: VirtualService<br>metadata:<br>  name: frontend-ingress<br>  namespace: frontend<br>spec:<br>  hosts:<br>  - &quot;*&quot;<br>  gateways:<br>  - frontend-gateway<br>  http:<br>  - route:<br>    - destination:<br>        host: frontend<br>        port:<br>          number: 80</pre><p>Please note the similar level of conserns separation as we have seen in the <a href="#bc1f">Kubernetes Gateway</a> section.</p><h3>GKE NEG and AutoNEG Controllers</h3><h3>NEG Controllers — GKE</h3><p>GKE provides a <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg">NEG controller</a> to manage the membership of GCE_VM_IP_PORT Network Endpoint Groups (NEGs). You can add the NEGs it creates as backends to the backend services of load balancers that you configure outside of the GKE API.</p><p>The NEG Controller with <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg#use_cases_of_standalone_negs">Standalone NEGs</a> covers the case when load balancer components automatically created by the Kubernetes Ingress do not serve your use case.</p><p>Additional <a href="https://github.com/GoogleCloudPlatform/gke-autoneg-controller">Autoneg Controller</a> provides simple custom integration between GKE and Google Cloud Load Balancing (both external and internal). It is a GKE-specific Kubernetes controller which works in conjunction with the GKE Network Endpoint Group (NEG) controller to manage integration between your Kubernetes service endpoints and GCLB backend services.</p><p>GKE users may wish to register NEG backends from multiple clusters into the same backend service, or may wish to orchestrate advanced deployment strategies in a custom or centralised fashion, or offer the same service via protected public endpoint and more lax internal endpoint. Autoneg controller can enable those use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fNwNDCdk1OJeI3lg" /></figure><p><strong><em>The benefit</em></strong> of using NEG controllers with Kubernetes Service resources is ability to further segregate the duties of teams who are responsible for the network traffic ingress into the Cloud including Load Balancers and the teams who have access and responsible for the Kubernetes internal resources and workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jC1IZK60UVV5_VO5LBLlow.png" /><figcaption>Components of the NEG Controller based network ingress</figcaption></figure><p>It becomes possible to set up network ingress by provisioning Cloud resources using the Infrastructure as Code (IaC) system of choice (e.g. Terraform) and Kubernetes resources via declarative Kubernetes API.</p><h3>NEG Controllers — TLS</h3><p>In this scenario the TLS connections are handled and terminated by the Google Cloud Load Balancers. With NEG Controller, it is possible to reuse the load balancer components and connect them to the Kubernetes Services as defined in the Kubernetes cluster resources.</p><p>The TLS certificates and private keys can be managed in this case outside of the Kubernetes cluster following existing Terraform IaC way or Google Cloud API based automation. The Kubernetes resources in this case would not lead to the load balancer components modification in a way that could damage the Terraform state.</p><h3>NEG Controllers — Example</h3><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: my-service<br>  annotations:<br>    cloud.google.com/neg: &#39;{&quot;exposed_ports&quot;: {&quot;80&quot;:{},&quot;443&quot;:{}}}&#39;<br>    controller.autoneg.dev/neg: &#39;{&quot;backend_services&quot;:{&quot;80&quot;:[{&quot;name&quot;:&quot;http-be&quot;,&quot;max_rate_per_endpoint&quot;:100}],&quot;443&quot;:[{&quot;name&quot;:&quot;https-be&quot;,&quot;max_connections_per_endpoint&quot;:1000}]}}<br>    # For L7 ILB (regional) backends <br>    # controller.autoneg.dev/neg: &#39;{&quot;backend_services&quot;:{&quot;80&quot;:[{&quot;name&quot;:&quot;http-be&quot;,&quot;region&quot;:&quot;europe-west4&quot;,&quot;max_rate_per_endpoint&quot;:100}],&quot;443&quot;:[{&quot;name&quot;:&quot;https-be&quot;,&quot;region&quot;:&quot;europe-west4&quot;,&quot;max_connections_per_endpoint&quot;:1000}]}}<br>spec:<br>  selector:<br>    app.kubernetes.io/name: MyApp<br>  ports:<br>    - protocol: TCP<br>      port: 443<br>      targetPort: 9376<br>  clusterIP: 10.0.171.239<br>  type: LoadBalancer<br>status:<br>  loadBalancer:<br>    ingress:<br>    - ip: 192.0.2.127</pre><h3>Conclusion</h3><p>Did you notice a mistake? Have an interesting or new solution for arranging network ingress into Kubernetes clusters? Will be glad to learn and amend the list. Please comment and PM me!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f22ded21f4ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/ingress-in-google-kubernetes-products-f22ded21f4ed">Ingress in Google Kubernetes Products</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>