<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ameer Abbas on Medium]]></title>
        <description><![CDATA[Stories by Ameer Abbas on Medium]]></description>
        <link>https://medium.com/@ameericus?source=rss-91845b0cfdde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*LWxGEvIHYd6cFobZQSbu4g.jpeg</url>
            <title>Stories by Ameer Abbas on Medium</title>
            <link>https://medium.com/@ameericus?source=rss-91845b0cfdde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:23:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ameericus/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Extending your Istio service mesh across GKE clusters and Compute Engine instances]]></title>
            <link>https://medium.com/google-cloud/extending-your-istio-service-mesh-across-gke-clusters-and-compute-engine-instances-4e8a69224f2b?source=rss-91845b0cfdde------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e8a69224f2b</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Ameer Abbas]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 17:03:33 GMT</pubDate>
            <atom:updated>2020-08-19T18:46:01.733Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial shows how to deploy a multi-tier microservices application that spans a Kubernetes cluster and a Compute Engine instance by using a <a href="https://istio.io/latest/docs/ops/deployment/deployment-models/#single-mesh">single Istio service mesh</a> over the entire application. This tutorial is intended for Kubernetes operators who have a basic knowledge of Kubernetes concepts. Knowledge of Istio is not required.</p><blockquote>I want to give a shout out to Megan O’Keefe (<a href="http://twitter.com/askmeegs">@askmeegs</a>) for her awesome work with Istio, parts of which I shamelessly plagiarized for this tutorial. You can find many Istio related samples maintained by Megan at <a href="https://github.com/GoogleCloudPlatform/istio-samples">https://github.com/GoogleCloudPlatform/istio-samples</a></blockquote><p><a href="https://istio.io/docs/examples/mesh-expansion/single-network/">Istio</a> lets you extend a service mesh that is running inside a Kubernetes cluster to services that run on virtual machine (VM) instances outside of the Kubernetes cluster.</p><p><strong>Note</strong>: Istio is an open source tool and not an official Google product.</p><p><a href="https://istio.io/">Istio</a> is an open source implementation of a <a href="https://istio.io/docs/concepts/what-is-istio/">service mesh</a> that lets you discover, dynamically route to, and more securely connect to Services that run on Kubernetes clusters. Istio also provides a policy-driven framework for routing, load balancing, throttling, telemetry, circuit breaking, authenticating, and authorizing service calls in the mesh with few to no changes to your application code.</p><p>When you install Istio in a Kubernetes cluster, the Istio control plane uses the Kubernetes <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/">service registry</a> to automatically discover and create a service mesh of interconnected Services (or microservices) that are running inside the local cluster. Istio uses <a href="https://www.envoyproxy.io/">Envoy</a> sidecar proxies running inside each Pod to manage Pod-to-Pod traffic routing and security and to provide observability for all Services and workloads that run inside the cluster. You can also deploy Envoy to VMs to extend the service mesh beyond Kubernetes clusters.</p><p>Services that are running in one Kubernetes cluster might need to talk to services running in VMs that are outside the cluster. For example, microservices that run inside a Kubernetes cluster might need access to a legacy monolithic service running in a VM or a database cluster running in VMs. Istio lets you <a href="https://istio.io/docs/setup/kubernetes/additional-setup/mesh-expansion/">create a service mesh</a> beyond a single Kubernetes cluster to include external services that run in VMs, outside of Kubernetes.</p><p>Istio provides two main configuration options for mesh expansion deployments:</p><ul><li><strong>Single network.</strong> Both the Kubernetes cluster and VMs are on the same network. Pods in the Kubernetes cluster and VMs can directly access each other by using their IP address.</li><li><strong>Multiple networks.</strong> The Kubernetes cluster and VMs are on separate networks and cannot access each other by using direct IP connectivity. In this scenario, Pods and VMs use a <a href="https://istio.io/docs/reference/config/networking/v1alpha3/gateway/">gateway</a> (Istio-managed Envoy edge proxy) to access each other.</li></ul><p>In this tutorial, you deploy Istio on a GKE cluster and expand the service mesh to a service running on a Compute Engine instance in a single Virtual Private Cloud (VPC) network. The Istio control plane cluster has direct IP connectivity to the VM through an IP address. For this tutorial, you use a 10-tier sample microservices app called <a href="https://github.com/GoogleCloudPlatform/microservices-demo">Online Boutique</a> split across a GKE cluster and a Compute Engine instance. You build the following architecture inside a Google Cloud project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YMDus-EqqYxbup4Pf8-e7A.png" /><figcaption>Tutorial Architecture</figcaption></figure><p>All Online Boutique Services except productcatalogservice run inside a GKE cluster. The Service productcatalogservice runs on a Compute Engine VM instance. Using Istio mesh expansion, you add productcatalogservice to the service mesh, which contains the rest of the Online Boutique services.</p><h3>Objectives</h3><ul><li>Create a GKE cluster called west.</li><li>Install Istio with mesh expansion enabled on the west cluster.</li><li>Create a Compute Engine instance called istio-gce in the same VPC as the west cluster.</li><li>Install the Online Boutique service productcatalogservice on the istio-gce instance.</li><li>Register productcatalogservice to the Istio service mesh in the west cluster.</li><li>Install the remaining Online Boutique app microservices on the west cluster.</li><li>Observe the expanded service mesh.</li></ul><h3>Costs</h3><p>This tutorial uses the following billable components of Google Cloud:</p><ul><li><a href="https://cloud.google.com/kubernetes-engine/pricing">GKE</a></li><li><a href="https://cloud.google.com/compute/all-pricing">Compute Engine</a></li><li><a href="https://cloud.google.com/vpc/network-pricing">Google Cloud networking</a></li><li><a href="https://cloud.google.com/compute/all-pricing#lb">Cloud Load Balancing</a></li></ul><p>Use the <a href="https://cloud.google.com/products/calculator">Pricing Calculator</a> to generate a cost estimate based on your projected usage.</p><h3>Before you begin</h3><ol><li>Select or create a Google Cloud project.</li></ol><p><a href="https://console.cloud.google.com/cloud-resource-manager">GO TO THE MANAGE RESOURCES PAGE</a></p><p>2. Enable billing for your project.</p><p><a href="https://support.google.com/cloud/answer/6293499#enable-billing">ENABLE BILLING</a></p><p>3. Enable the Kubernetes Engine and Source Repositories API.</p><p><a href="https://console.cloud.google.com/flows/enableapi?apiid=container,sourcerepo.googleapis.com">ENABLE APIS</a></p><p>When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See <a href="https://docs.google.com/document/d/1vaelwoytZYoZ5WvyzwdksRTysig0bncoThU9I-T0In0/edit#heading=h.mlrdlgcohh7k">Cleaning up</a> for more detail.</p><h3>Preparing your environment</h3><p>You run all of the terminal commands in this tutorial from Cloud Shell.</p><ol><li>Open Cloud Shell:</li></ol><p><a href="https://console.cloud.google.com/?cloudshell=true">OPEN CLOUD SHELL</a></p><p>2. Download the required files for this tutorial by cloning the Git repository:</p><pre>cd $HOME<br>git clone <a href="https://github.com/GoogleCloudPlatform/istio-multicluster-gke.git">https://github.com/GoogleCloudPlatform/istio-multicluster-gke.git</a></pre><p>3. Make the repository folder your $WORKDIR, from which you do all the tasks related to this tutorial:</p><pre>cd $HOME/istio-multicluster-gke<br>WORKDIR=$(pwd)</pre><p>You can delete the folder when you’re finished with the tutorial.</p><p>4. Install <a href="https://github.com/ahmetb/kubectx">kubectx and kubens</a>:</p><pre>git clone <a href="https://github.com/ahmetb/kubectx">https://github.com/ahmetb/kubectx</a> $WORKDIR/kubectx<br>export PATH=$PATH:$WORKDIR/kubectx</pre><p>These tools let you quickly switch contexts and namespaces, which makes it easier to work with multiple Kubernetes clusters.</p><h3>Creating the GKE cluster</h3><p>In this section, you create a GKE cluster in the default VPC with <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips">IP Alias</a> enabled. With <a href="https://cloud.google.com/vpc/docs/alias-ip">Alias IP addresses</a>, GKE clusters can allocate IP addresses from a CIDR block known to Google Cloud. This setup results in Pod IPs being routable within the VPC, giving Pods direct IP connectivity to Compute Engine instances in the same VPC.</p><ol><li>In Cloud Shell, create a GKE cluster that is called west in the us-west2 region:</li></ol><pre>gcloud container clusters create west \<br>--zone us-west2-a --username &quot;admin&quot; \<br>--machine-type &quot;n1-standard-2&quot; \<br>--cluster-version 1.14 \<br>--image-type &quot;COS&quot; --disk-size &quot;100&quot; \<br>--num-nodes &quot;5&quot; --network &quot;default&quot; \<br>--enable-cloud-logging --enable-cloud-monitoring \<br>--enable-ip-alias</pre><p>2. Connect to the west GKE cluster to generate an entry in the <a href="https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/">kubeconfig</a> file:</p><pre>export PROJECT_ID=$(gcloud info --format=&#39;value(config.project)&#39;)<br>gcloud container clusters get-credentials west --zone us-west2-a --project ${PROJECT_ID}</pre><p>You use the kubeconfig file to create authentication to GKE clusters by creating a user and context for every cluster.</p><p>3. For convenience, use kubectx to rename the context name:</p><pre>kubectx west=gke_${PROJECT_ID}_us-west2-a_west</pre><p>4. Give yourself (your Google user) the cluster-admin role for the cluster:</p><pre>kubectl create clusterrolebinding user-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account) --context west</pre><p>This role lets you perform administrative tasks on these clusters.</p><h3>Installing Istio</h3><p>In this section, you install and configure Istio with mesh expansion enabled on the west cluster.</p><h3>Download and install Istio</h3><ol><li>In Cloud Shell, download Istio:</li></ol><pre>cd ${WORKDIR}<br>export ISTIO_VERSION=1.6.5<br>curl -L <a href="https://istio.io/downloadIstio">https://istio.io/downloadIstio</a> | ISTIO_VERSION=${ISTIO_VERSION} sh -</pre><p>2. Create the istio-system namespace in the west cluster:</p><pre>kubectl --context west create namespace istio-system</pre><p>3. Using the sample certificates provided with the Istio release, create a Kubernetes secret:</p><pre>kubectl --context west create secret generic cacerts -n istio-system \<br>--from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/ca-cert.pem \<br>--from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/ca-key.pem \<br>--from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/root-cert.pem \<br>--from-file=${WORKDIR}/istio-${ISTIO_VERSION}/samples/certs/cert-chain.pem</pre><p>The Istio control plane uses this secret to sign workload certificates.</p><p>4. Install Istio:</p><pre>${WORKDIR}/istio-${ISTIO_VERSION}/bin/istioctl install \<br>--set profile=demo \<br>--set values.global.meshExpansion.enabled=true </pre><p>In this tutorial, you deploy Istio using the demo profile. You can deploy Istio by using any profile or even a custom profile as long as you enable the meshExpansion parameter.</p><p><strong>Note</strong>: Istio takes 2–3 minutes to install.</p><p>5. Ensure that all Istio deployments are running:</p><pre>kubectl --context west get pods -n istio-system</pre><p>The output is similar to the following:</p><pre><strong>OUTPUT (DO NOT COPY)</strong><br>NAME READY STATUS RESTARTS AGE</pre><pre>grafana-858f9bcbcd-jnjbf 1/1 Running 0 70s<br>istio-egressgateway-5548786875–4zwcv 1/1 Running 0 72s<br>istio-ingressgateway-dfcdf4d97–28rmn 1/1 Running 0 72s<br>istio-tracing-7cf5f46848-jbhsk 1/1 Running 0 70s<br>istiod-6ff4d846d7-pmtrx 1/1 Running 0 85s<br>...</pre><h3>Preparing the Compute Engine instance for Istio</h3><ol><li>Define a namespace that the Compute Engine instance joins:</li></ol><pre>export SERVICE_NAMESPACE=&quot;default&quot;</pre><p>2. Store the istio-ingressgateway external load balancer IP address in a file:</p><pre>export ISTIOD_IP=$(kubectl --context west get -n istio-system service istio-ingressgateway -o jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;)<br>echo $ISTIOD_IP &gt; istiod.txt</pre><p>When mesh expansion is enabled, Istio exposes the istiod (the Istio control plane) Service through istio-ingressgateway. The Compute Engine instance uses this IP address to access istiod.</p><p>3. Get the ClusterIP address range from the west cluster:</p><pre>ISTIO_SERVICE_CIDR=$(gcloud container clusters describe west --zone us-west2-a --project ${PROJECT_ID} --format &quot;value(servicesIpv4Cidr)&quot;)</pre><p>4. Generate a cluster.env configuration file that you deploy in the Compute Engine instance. This file contains the Kubernetes ClusterIP address ranges to intercept and redirect using Envoy.</p><pre>echo -e &quot;ISTIO_SERVICE_CIDR=$ISTIO_SERVICE_CIDR\n&quot; &gt; cluster.env<br>echo &quot;ISTIO_INBOUND_PORTS=3550&quot; &gt;&gt; cluster.env</pre><p>The service running on the Compute Engine instance uses the ISTIO_INBOUND_PORTS ports. In this tutorial, you deploy productcatalogservice to the Compute Engine instance. The Service productcatalogservice runs on port 3550.</p><p>5. Generate the certificates that Compute Engine uses:</p><pre>go get istio.io/istio/security/tools/generate_cert<br>go run istio.io/istio/security/tools/generate_cert \<br>-client -host spiffee://cluster.local/vm/vmname --out-priv key.pem --out-cert cert-chain.pem  --mode self-signed<br>kubectl --context west -n istio-system get cm istio-ca-root-cert -o jsonpath=&#39;{.data.root-cert\.pem}&#39; &gt; root-cert.pem</pre><p>To use mesh expansion, you must provision the Compute Engine instance with certificates that are signed by the same root CA as the rest of the mesh.</p><h3>Install and configure the Compute Engine instance</h3><p>In this section, you install a Compute Engine instance where one of the Online Boutique app microservices runs.</p><ol><li>In Cloud Shell, create a Compute Engine instance named istio-gce. Use a tag with the value of istio-gce. This tag is used in the next step for creating firewall rules.</li></ol><pre>GCE_INSTANCE_NAME=&quot;istio-gce&quot;<br>gcloud compute --project=$PROJECT_ID instances create $GCE_INSTANCE_NAME --zone=us-west2-a \<br>--machine-type=n1-standard-2 --subnet=default --network-tier=PREMIUM --maintenance-policy=MIGRATE \<br>--image-family=ubuntu-1604-lts --image-project=ubuntu-os-cloud --boot-disk-size=10GB \<br>--boot-disk-type=pd-standard --boot-disk-device-name=$GCE_INSTANCE_NAME --tags=&quot;istio-gce&quot;</pre><p>2. Get the GKE Pod IP CIDR range:</p><pre>export GKE_POD_CIDR=$(gcloud container clusters describe west --zone us-west2-a --format=json | jq -r &#39;.clusterIpv4Cidr&#39;)</pre><p>3. Create a firewall rule that allows the GKE Pod IP CIDR range to communicate with the istio-gce instance (using the istio-gce tag assigned in the previous step) on TCP port 3550:</p><pre>gcloud compute firewall-rules create k8s-to-istio-gce \<br>--description=&quot;Allow k8s pods CIDR to istio-gce instance&quot; \<br>--source-ranges=$GKE_POD_CIDR \<br>--target-tags=&quot;istio-gce&quot; \<br>--action=ALLOW \<br>--rules=tcp:3550</pre><p>The Service productcatalogservice runs in the istio-gce instance. This Service listens on TCP port 3550.</p><p>4. Generate an SSH key so that you can connect to the Compute Engine instance using SSH, and add the SSH private key to the SSH authentication agent:</p><pre>USER=$(gcloud config get-value account)<br>ssh-keygen -t rsa -N &#39;&#39; -b 4096 -C &quot;$USER&quot; \<br>-f $HOME/.ssh/google_compute_engine<br>eval &quot;$(ssh-agent -s)&quot;<br>ssh-add ~/.ssh/google_compute_engine</pre><p>5. Create the gce-script.sh file which deploys the Envoy proxy and the productcatalogservice to the istio-gce instance.</p><pre>export GWIP=$ISTIOD_IP<br>envsubst &lt; ./istio-mesh-expansion-gce/gce-script-tmpl.sh &gt; ./istio-mesh-expansion-gce/gce-script.sh</pre><p>6. Using secure copy protocol (SCP), copy the certificates, the istiod.txt file, and and the gce-script.sh shell script to the Compute Engine instance:</p><pre>gcloud compute scp --project=${PROJECT_ID} --zone=us-west2-a {key.pem,cert-chain.pem,cluster.env,root-cert.pem,istiod.txt,./istio-mesh-expansion-gce/gce-script.sh} ${GCE_INSTANCE_NAME}:~</pre><p>7. Run the gce-script.sh script in the Compute Engine instance:</p><pre>gcloud compute --project ${PROJECT_ID} ssh --zone us-west2-a ${GCE_INSTANCE_NAME} --command=&quot;chmod +x gce-script.sh; ./gce-script.sh&quot;</pre><p>8. The gce-script.sh script prepares the istio-gce instance to be part of the Istio service mesh. The script performs the following:</p><ul><li>Configures the etc/hosts file with the Istiod IP address to connect back to the Istio control plane in the west cluster.</li><li>Configures the Envoy agent on the Compute Engine instance.</li><li>Configures the certificates for mTLS communications.</li></ul><p>Installs Docker, then deploys the Service productcatalogservice Docker image on the istio-gce instance.</p><h3>Deploying Online Boutique</h3><p><a href="https://github.com/GoogleCloudPlatform/microservices-demo#service-architecture">Online Boutique</a> consists of 10 microservices that are written in different programming languages. Earlier in this tutorial, you installed productcatalogservice in the istio-gce instance. In this section, you install the remaining nine microservices in the west cluster in the default namespace.</p><ol><li>Label the default namespace for automatic Istio <a href="https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/#automatic-sidecar-injection">sidecar proxy injection</a>.</li></ol><pre>kubectl --context west label namespace default istio-injection=enabled</pre><p>This step ensures that all Pods created in the default namespace have the Envoy sidecar container deployed.</p><p>2. Install the Online Boutique app microservices in the west cluster:</p><pre>kubectl --context west -n default apply -f <a href="https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/release/kubernetes-manifests.yaml">https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/release/kubernetes-manifests.yaml</a><br>kubectl --context west -n default apply -f <a href="https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/release/istio-manifests.yaml">https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/release/istio-manifests.yaml</a></pre><p>3. Delete the productcatalogservice Service and Deployment from the west cluster:</p><pre>kubectl --context west -n default delete svc productcatalogservice<br>kubectl --context west -n default delete deployment productcatalogservice</pre><p>4. Wait a few moments, and then ensure that all workloads are up and running:</p><pre>kubectl --context west get pods</pre><p>The output is similar to the following:</p><pre><strong>OUTPUT (DO NOT COPY)</strong><br>NAME READY STATUS RESTARTS AGE<br>adservice-86674bf94d-vlkwn 2/2 Running 0 109s<br>cartservice-9cf968485-srbsq 2/2 Running 2 110s<br>checkoutservice-74df4f44c8-d7s9b 2/2 Running 0 111s<br>currencyservice-6444b89474-dwl75 2/2 Running 0 109s<br>...</pre><p>This output shows that the productcatalogservice Pod is not running inside the west cluster.</p><h3>Add the Compute Engine instance to the service mesh</h3><ol><li>Get the IP address of the Compute Engine instance:</li></ol><pre>export GCE_IP=$(gcloud --format=&quot;value(networkInterfaces[0].networkIP)&quot; compute instances describe ${GCE_INSTANCE_NAME} --zone us-west2-a)</pre><p>2. Add the Compute Engine instance to the service mesh:</p><pre>${WORKDIR}/istio-${ISTIO_VERSION}/bin/istioctl experimental add-to-mesh external-service productcatalogservice ${GCE_IP} grpc:3550 -n default</pre><p>3. This command creates a Kubernetes Service object as well as a ServiceEntry object for productcatalogservice that points to the IP address of the Compute Engine instance.</p><p>4. Restart the Istio Service on the Compute Engine instance:</p><pre>gcloud compute --project $PROJECT_ID ssh --zone us-west2-a ${GCE_INSTANCE_NAME} --command=&quot;sudo systemctl stop istio; sudo systemctl start istio;&quot;</pre><h3>Access the Online Boutique application</h3><ol><li>Get the Istio ingress gateway external IP address for the west cluster:</li></ol><pre>kubectl --context west get -n istio-system service istio-ingressgateway -o json | jq -r &#39;.status.loadBalancer.ingress[0].ip&#39;</pre><p>The output is similar to the following:</p><pre>EXTERNAL_IP</pre><p>2. Copy the Istio ingress gateway IP address in a web browser. The Online Boutique app main page is displayed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fI0cKDg_sX8evZyd" /><figcaption>Online Boutique App Frontend</figcaption></figure><p>To confirm that the app is fully functional across the west cluster and the istio-gce instance, browse the products, add products to your cart, and proceed with a checkout. The Service productcatalogservice is running in a dedicated Compute Engine instance while the remaining microservices are running on the GKE cluster.</p><h3>Monitoring the service mesh</h3><p>You can use <a href="https://www.kiali.io/">Kiali</a> to monitor and visualize the service mesh. Kiali is a service mesh observability tool that’s installed as part of the Istio installation.</p><ol><li>In Cloud Shell, expose the Kiali service on the west cluster:</li></ol><pre>${WORKDIR}/istio-${ISTIO_VERSION}/bin/istioctl dashboard kiali &amp;</pre><p>The output is similar to the following:</p><p>Failed to open browser; open <a href="http://localhost:35755/kiali">http://localhost:35755/kiali</a> in your browser.</p><p>2. Open the Kiali web interface by navigating to the HTTP link in the preceding output.</p><p>3. At the Kiali login prompt, log in with the username admin and password admin.</p><p>4. From the menu, select <strong>Graph</strong>.</p><p>5. From the <strong>Select a namespace</strong> drop-down list, select <strong>default</strong>.</p><p>6. From the <strong>Graph</strong> drop-down list, select <strong>Service graph</strong>.</p><p>7. (Optional) To see loadgenerator generating traffic to your app, from the <strong>Display</strong> drop-down list, select <strong>Traffic Animation</strong>.</p><p>The following diagram shows a single Istio service mesh for Services that are spread across a GKE cluster and a Compute Engine instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3w4S-DwwunlqzQS2" /><figcaption>Kiali Service Topology</figcaption></figure><p>This diagram shows the following:</p><ul><li>The microservices running in the west cluster are described by their names only and are denoted by a triangle symbol.</li><li>The services running in the istio-gce instance are denoted by a domain symbol (a 5 sided polygon).</li></ul><p>The Istio service mesh is extended to include applications running on VMs. The services are securely communicating between the GKE cluster and Compute Engine instances using the Istio ingress gateway with mTLS. You can use the steps in this tutorial to add services that run on Compute Engine instances.</p><h3>Cleaning up</h3><p>To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:</p><h3>Delete the project</h3><p>The easiest way to eliminate billing is to delete the project you created for the tutorial.</p><p>To delete the project:</p><ol><li>In the Cloud Platform Console, go to the Projects page.</li></ol><p><a href="https://console.cloud.google.com/iam-admin/projects">GO TO THE PROJECTS PAGE</a></p><p>2. In the project list, select the project you want to delete and click <strong>Delete</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*27vsBqi_71-0yHWy" /></figure><p>In the dialog, type the project ID, and then click <strong>Shut down</strong> to delete the project.</p><p>Thank you for going through this tutorial. If you have any comments or feedback about this tutorial, please feel free to leave a note.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e8a69224f2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/extending-your-istio-service-mesh-across-gke-clusters-and-compute-engine-instances-4e8a69224f2b">Extending your Istio service mesh across GKE clusters and Compute Engine instances</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Engine (GKE) multi-cluster life cycle management series]]></title>
            <link>https://medium.com/google-cloud/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-ee0f583d9b10?source=rss-91845b0cfdde------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee0f583d9b10</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Ameer Abbas]]></dc:creator>
            <pubDate>Wed, 22 Apr 2020 22:25:59 GMT</pubDate>
            <atom:updated>2020-04-28T16:23:58.400Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part V: GKE multi-cluster upgrades using Ingress for Anthos step-by-step tutorial</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2nN1m2qz6tLu_xk4fsKy0g.png" /><figcaption>GKE Lifecycle Management using Ingress for Anthos</figcaption></figure><p>This tutorial describes how to upgrade a multi cluster GKE environment using the rolling upgrade strategy (discussed in the previous blog) and the multi cluster ingress feature provided by <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-for-anthos">Ingress for Anthos</a>. This tutorial is the culmination of the concepts covered in this blog series.</p><p>GCP recommends using the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-upgrades#upgrading_automatically">GKE Autoupgrade</a> feature to upgrade your GKE clusters. Autoupgrade is a no-ops way of getting your clusters (master and nodes) automatically updated on a release schedule determined by GCP. This requires no intervention from the operator. However, in some cases, operators may want more control over how and when they upgrade clusters. This tutorial walks through a method of upgrading multiple clusters where your applications run on all clusters. It then utilizes the Ingress for Anthos (multicluster Ingress) feature to drain one cluster at a time before upgrading.</p><h3>Prerequisites</h3><ol><li>This tutorial requires multiple clusters (two or more) with the same applications (namespaces, deployments, services etc) running on all clusters.</li><li>This tutorial assumes autoupgrade feature has been turned off for the clusters to be upgraded.</li><li>This tutorial uses Ingress for Anthos. Ingress for Anthos requires:</li></ol><ul><li>GKE static versions 1.14.10-gke.17+ or GKE clusters in the Rapid or Regular <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels">release channels</a>.</li><li>Clusters in VPC-Native (Alias IP) mode. For more information, see <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips">Creating a VPC-native cluster</a>.</li><li>Have HTTP load balancing enabled (enabled by default).</li><li>gcloud — version must be 281 or higher. GKE cluster registration steps depend on this version or higher.</li></ul><h3>Topology</h3><p>This tutorial uses the following topology. There are a total of three clusters. Two clusters (blue and green) act as identical clusters with the same application deployed. One cluster (ingress-config) acts as the control plane cluster which configures Ingress for Anthos. In this tutorial, you deploy a sample app called the Hipster Shop app on two application clusters (blue and green clusters). <a href="https://github.com/GoogleCloudPlatform/microservices-demo">Hipster Shop</a> app is a sample microservices application comprising ten microservices simulating an e-commerce application. It consists of a web frontend (that clients can access) and a number of backend Services for example shopping cart, product catalog and recommendation services simulating a cloud-native e-retailer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2nN1m2qz6tLu_xk4fsKy0g.png" /><figcaption>GKE Lifecycle Management using Ingress for Anthos</figcaption></figure><h3>Objectives</h3><ul><li>Create three GKE clusters.</li><li>Register the clusters to GKE Hub.</li><li>Configure one GKE cluster (ingress-config) as the central config cluster. This cluster acts as the control plane for multi cluster ingress. It creates and manages the resources for Ingress for Anthos.</li><li>Deploy a sample application to the other GKE clusters. Use the <a href="https://github.com/GoogleCloudPlatform/microservices-demo">Hipster Shop</a> app as the sample application.</li><li>Configure Ingress for Anthos to send client traffic to the Hipster Shop running on both application clusters.</li><li>Set up a load generator to Hipster Shop app and configure monitoring with Cloud Monitoring.</li><li>Remove (drain) one application cluster from multi cluster ingress.</li><li>Upgrade the drained cluster.</li><li>Spill traffic back to the newly upgraded cluster using Ingress for Anthos.</li></ul><h3>Costs</h3><p>This tutorial uses the following billable components of Google Cloud Platform:</p><ul><li>GKE</li><li>Networking</li><li>Load Balancing</li><li>Cloud Monitoring</li></ul><p>Use the <a href="https://cloud.google.com/products/calculator">Pricing Calculator</a> to generate a cost estimate based on your projected usage.</p><h3>Before you begin</h3><ul><li>Select or create a GCP project.</li></ul><pre><a href="https://console.cloud.google.com/cloud-resource-manager">GO TO THE MANAGE RESOURCES PAGE</a></pre><ul><li>Enable billing for your project.</li></ul><pre><a href="https://support.google.com/cloud/answer/6293499#enable-billing">ENABLE BILLING</a></pre><ul><li>You run all the terminal commands in this tutorial from Cloud Shell. Open Cloud Shell:</li></ul><pre><a href="https://console.cloud.google.com/?cloudshell=true">OPEN CLOUD SHELL</a></pre><ul><li>Set your default project.</li></ul><pre>export PROJECT=$(gcloud info --format=&#39;value(config.project)&#39;)<br>gcloud config set project ${PROJECT}</pre><ul><li>Enable the GKE, GKE Hub and multiclusteringress APIs in the project.</li></ul><pre>gcloud services enable container.googleapis.com gkehub.googleapis.com multiclusteringress.googleapis.com</pre><p>When you finish this tutorial, you can avoid continued billing by deleting the resources you created. See <a href="https://docs.google.com/document/d/1vaelwoytZYoZ5WvyzwdksRTysig0bncoThU9I-T0In0/edit#heading=h.mlrdlgcohh7k">Cleaning up</a> for more detail.</p><h3>Set up the environment</h3><p>You run all the terminal commands in this tutorial from Cloud Shell.</p><ul><li>Open Cloud Shell:</li></ul><pre><a href="https://console.cloud.google.com/?cloudshell=true">OPEN CLOUD SHELL</a></pre><ul><li>Clone the repo to get the files for this tutorial. Create a WORKDIR. This way you can delete the WORKDIR at the end of this tutorial.</li></ul><pre>cd ${HOME}<br>git clone <a href="https://github.com/ameer00/gke-multicluster-upgrades.git">https://github.com/ameer00/gke-multicluster-upgrades.git</a></pre><pre>cd gke-multicluster-upgrades<br>export WORKDIR=`pwd`</pre><h3>Create and register GKE clusters to GKE Hub</h3><h4>Create GKE clusters</h4><ul><li>Create three GKE clusters.</li></ul><pre>gcloud container clusters create ingress-config --zone us-west1-a --num-nodes=3 --enable-ip-alias --async</pre><pre>gcloud container clusters create blue --zone us-west1-b --num-nodes=4 --enable-ip-alias --async</pre><pre>gcloud container clusters create green --zone us-west1-c --num-nodes=4 --enable-ip-alias</pre><ul><li>Wait a few minutes until all clusters are successfully created. Ensure that the clusters are running:</li></ul><pre>gcloud container clusters list</pre><pre><em>Output (do not copy)</em></pre><pre><em>NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS</em></pre><pre><em>ingress-config us-west1-a 1.14.10-gke.24 35.203.165.186 n1-standard-1 1.14.10-gke.24 3 </em><strong><em>RUNNING</em></strong></pre><pre><em>blue us-west1-b 1.14.10-gke.24 34.82.76.141 n1-standard-1 1.14.10-gke.24 4 </em><strong><em>RUNNING</em></strong></pre><pre><em>green us-west1-c 1.14.10-gke.17 35.197.46.200 n1-standard-1 1.14.10-gke.17 4 </em><strong><em>RUNNING</em></strong></pre><blockquote>Master/node version and IP addresses may be different.</blockquote><ul><li>Create a new <a href="https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/">kubeconfig</a> file and connect to all clusters to generate entries in the kubeconfig file:</li></ul><pre>touch gke-upgrade-kubeconfig<br>export KUBECONFIG=gke-upgrade-kubeconfig</pre><pre>gcloud container clusters get-credentials ingress-config --zone us-west1-a --project ${PROJECT}</pre><pre>gcloud container clusters get-credentials blue --zone us-west1-b --project ${PROJECT}<br>gcloud container clusters get-credentials green --zone us-west1-c --project ${PROJECT}</pre><p>You use the kubeconfig file to create authentication to clusters by creating a user and context for each cluster. After you create the kubeconfig file, you can quickly switch context between clusters.</p><ul><li>Verify that you have three clusters in the kubeconfig file:</li></ul><pre>kubectl config view -ojson | jq -r &#39;.clusters[].name&#39;</pre><pre><em>Output (do not copy)<br>gke_gke-multicluster-upgrades_us-west1-a_</em><strong><em>ingress-config</em></strong></pre><pre><em>gke_gke-multicluster-upgrades_us-west1-b_</em><strong><em>blue</em></strong></pre><pre><em>gke_gke-multicluster-upgrades_us-west1-c_</em><strong><em>green</em></strong></pre><ul><li>Get the context for the three clusters for use later.</li></ul><pre>export INGRESS_CONFIG_CLUSTER=$(kubectl config view -ojson | jq -r &#39;.clusters[].name&#39; | grep ingress-config)<br>export BLUE_CLUSTER=$(kubectl config view -ojson | jq -r &#39;.clusters[].name&#39; | grep blue)<br>export GREEN_CLUSTER=$(kubectl config view -ojson | jq -r &#39;.clusters[].name&#39; | grep green)<br>echo -e &quot;${INGRESS_CONFIG_CLUSTER}\n${BLUE_CLUSTER}\n${GREEN_CLUSTER}&quot;</pre><pre><em>Output (do not copy)<br>gke_gke-multicluster-upgrades_us-west1-a_ingress-config</em></pre><pre><em>gke_gke-multicluster-upgrades_us-west1-b_blue</em></pre><pre><em>gke_gke-multicluster-upgrades_us-west1-c_green</em></pre><h4>Register GKE clusters to GKE Hub</h4><p>GKE Hub enables you to operate your Kubernetes clusters in hybrid environments. GKE Hub also enables registered clusters to use advanced GKE features for example Ingress for Anthos. You need a GCP Service Account (SA) to register a GKE cluster to GKE Hub. Give IAM permission to the GCP SA to be able to register to GKE Hub.</p><ul><li>Create a GCP service account and download its credentials to use for registering the cluster.</li></ul><pre>gcloud iam service-accounts create ingress-svc-acct</pre><pre>gcloud projects add-iam-policy-binding ${PROJECT} \</pre><pre>--member=&quot;serviceAccount:ingress-svc-acct@${PROJECT}.iam.gserviceaccount.com&quot; \</pre><pre>--role=&quot;roles/gkehub.connect&quot;<br></pre><pre>gcloud iam service-accounts keys \</pre><pre>create ${WORKDIR}/ingress-svc-acct.json \</pre><pre>--iam-account=ingress-svc-acct@${PROJECT}.iam.gserviceaccount.com</pre><ul><li>Get the URIs from the GKE clusters.</li></ul><pre>export INGRESS_CONFIG_URI=$(gcloud container clusters list --uri | grep ingress-config)<br>export BLUE_URI=$(gcloud container clusters list --uri | grep blue)<br>export GREEN_URI=$(gcloud container clusters list --uri | grep green)<br>echo -e &quot;${INGRESS_CONFIG_URI}\n${BLUE_URI}\n${GREEN_URI}&quot;</pre><pre><em>Output (do not copy)<br>https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-a/clusters/ingress-config</em></pre><pre><em>https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-b/clusters/blue</em></pre><pre><em>https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-c/clusters/green</em></pre><ul><li>Register the three clusters to the GKE Hub.</li></ul><pre>gcloud container hub memberships register ingress-config \</pre><pre> --project=${PROJECT} \</pre><pre> --gke-uri=${INGRESS_CONFIG_URI} \</pre><pre> --service-account-key-file=${WORKDIR}/ingress-svc-acct.json</pre><pre>gcloud container hub memberships register blue \</pre><pre> --project=${PROJECT} \</pre><pre> --gke-uri=${BLUE_URI} \</pre><pre> --service-account-key-file=${WORKDIR}/ingress-svc-acct.json</pre><pre>gcloud container hub memberships register green \</pre><pre> --project=${PROJECT} \</pre><pre> --gke-uri=${GREEN_URI} \</pre><pre> --service-account-key-file=${WORKDIR}/ingress-svc-acct.json</pre><ul><li>Verify that the clusters are registered.</li></ul><pre>gcloud container hub memberships list</pre><pre><em>Output (do not copy)<br>NAME EXTERNAL_ID</em></pre><pre><em>blue d40521d9–693f-11ea-a26c-42010a8a0010</em></pre><pre><em>green d3027ecd-693f-11ea-ad5f-42010a8a00a9</em></pre><pre><em>ingress-config bb778338–693f-11ea-a053–42010a8a016a</em></pre><ul><li>Configure the ingress-config cluster as the config cluster for Ingress for Anthos. You do this by enabling multiclusteringress feature through the GKE Hub.</li></ul><pre>gcloud alpha container hub features multiclusteringress enable \</pre><pre> --config-membership=projects/${PROJECT}/locations/global/memberships/ingress-config</pre><p>Running the command above adds the MulticlusterIngress and MulticlusterService CRDs to the config cluster.</p><ul><li>The previous command takes a few minutes to complete. Run the following command to verify that the ingress-cluster has been successfully configured for Ingress for Anthos. Wait a few moments until you see code: OK for the ingress-config cluster (as shown below).</li></ul><pre>watch gcloud alpha container hub features multiclusteringress describe</pre><pre><em>Wait until the output is as shown below (do not copy)<br>createTime: ‘2020–03–18T18:13:46.530713607Z’</em></pre><pre><em>featureState:</em></pre><pre><em>details:</em></pre><pre><em>code: OK</em></pre><pre><em>description: Multicluster Ingress requires Anthos license enablement. Unlicensed</em></pre><pre><em>usage is unrestricted for the MCI Beta API. Note that licensing will be enforced</em></pre><pre><em>for use of the Generally Available MCI API.</em></pre><pre><em>detailsByMembership:</em></pre><pre><em>projects/960583074711/locations/global/memberships/blue:</em></pre><pre><em>code: OK</em></pre><pre><em>projects/960583074711/locations/global/memberships/green:</em></pre><pre><em>code: OK</em></pre><pre><strong><em>projects/960583074711/locations/global/memberships/ingress-config:</em></strong></pre><pre><strong><em>code: OK</em></strong></pre><pre><em>lifecycleState: ENABLED</em></pre><pre><em>multiclusteringressFeatureSpec:</em></pre><pre><em>configMembership: projects/gke-multicluster-upgrades/locations/global/memberships/ingress-config</em></pre><pre><em>name: projects/gke-multicluster-upgrades/locations/global/features/multiclusteringress</em></pre><blockquote>While lifecycleState is marked as enabled, this feature is not ready to use until your config cluster has an OK status. If you do not see code: OK after a few minutes, go to the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-for-anthos#troubleshooting">troubleshooting section</a>. When the feature is not ready you should see the following output.</blockquote><pre><em>featureState:</em></pre><pre><em>details:</em></pre><pre><em>code: OK</em></pre><pre><em>description: Multicluster Ingress requires Anthos license enablement. Unlicensed</em></pre><pre><em>usage is unrestricted for the MCI Beta API. Note that licensing will be enforced</em></pre><pre><em>for use of the Generally Available MCI API.</em></pre><pre><em>lifecycleState: ENABLED</em></pre><pre><em>multiclusteringressFeatureSpec:</em></pre><pre><em>configMembership: projects/qwiklabs-gcp-6dbb2ce9d43a8ef1/locations/global/memberships/ingress-config</em></pre><pre><em>name: projects/qwiklabs-gcp-6dbb2ce9d43a8ef1/locations/global/features/multiclusteringress</em></pre><pre><em>updateTime: ‘2020–03–23T15:55:36.087585371Z’</em></pre><p>Press <strong>CTRL-C</strong> or <strong>CMD-C</strong> on a Macbook to exit the watch command.</p><h3>Deploy Hipster Shop app to blue and green clusters</h3><ul><li>Deploy the Hipster Shop app to both blue and green clusters.</li></ul><pre>kubectl --context ${BLUE_CLUSTER} apply -f ${WORKDIR}/hipster-shop<br>kubectl --context ${GREEN_CLUSTER} apply -f ${WORKDIR}/hipster-shop</pre><ul><li>Wait a few minutes and ensure all Pods are Running in the blue and green clusters.</li></ul><pre>kubectl --context ${BLUE_CLUSTER} get pods<br>kubectl --context ${GREEN_CLUSTER} get pods</pre><pre><em>Output (do not copy)<br>NAME READY STATUS RESTARTS AGE</em></pre><pre><em>adservice-86f5dfb995-nlm5w 1/1 Running 0 10m</em></pre><pre><em>cartservice-76cf9686b6-rxf7b 1/1 Running 0 10m</em></pre><pre><em>checkoutservice-7766b946f5-qszvc 1/1 Running 0 10m</em></pre><pre><em>currencyservice-76975c7847-vmwn7 1/1 Running 0 10m</em></pre><pre><em>emailservice-c55cd96f-74rxs 1/1 Running 0 10m</em></pre><pre><em>frontend-f4b7cd95-lk4k8 1/1 Running 0 10m</em></pre><pre><em>loadgenerator-6784bc5f77-bkx4c 1/1 Running 0 10m</em></pre><pre><em>paymentservice-696f796545–8sjp5 1/1 Running 0 10m</em></pre><pre><em>productcatalogservice-7975f8588c-blrbq 1/1 Running 0 10m</em></pre><pre><em>recommendationservice-6d44459d79-xxb8w 1/1 Running 0 10m</em></pre><pre><em>redis-cart-6448dcbdcc-8qcb4 1/1 Running 0 10m</em></pre><pre><em>shippingservice-767f84b88c-8f26h 1/1 Running 0 10m</em></pre><h3>Configure multi cluster Ingress</h3><p>In this section, you create a multicluster Ingress that sends traffic to the Hipster Shop frontend to both blue and green clusters. A <a href="https://cloud.google.com/load-balancing/docs/load-balancing-overview">Google Cloud Load Balancer</a> (GCLB) is created which uses the frontend Services in both blue and green clusters as backends. To create the GCLB, you need two resources: a MultiClusterIngress and one or more MultiClusterServices. MultiClusterIngress and MultiClusterService objects are multi-cluster analogs to the existing Kubernetes Ingress and Service resources used in the single cluster context.</p><ul><li>Deploy the MulticlusterIngress resource to the ingress-config cluster.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mci.yaml</pre><pre><em>Output (do not copy)<br>multiclusteringress.networking.gke.io/frontend-multicluster-ingress created</em></pre><ul><li>Deploy the MulticlusterService resource to the ingress-config cluster.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mcs-blue-green.yaml</pre><pre><em>Output (do not copy)<br>multiclusterservice.networking.gke.io/frontend-multicluster-svc created</em></pre><ul><li>Inspect the MulticlusterIngress resource.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusteringress -o yaml</pre><pre><em>Output excerpt (do not copy)<br>spec:</em></pre><pre><em>template:</em></pre><pre><em>spec:</em></pre><pre><em>backend:</em></pre><pre><strong><em>serviceName: frontend-multicluster-svc</em></strong></pre><pre><em>servicePort: 80</em></pre><p>MulticlusterIngress resource looks very much like the Kubernetes Ingress resource except that the serviceName spec points to a MulticlusterService.</p><ul><li>Inspect the MulticlusterService resource.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusterservice -o yaml</pre><pre><em>Output excerpt (do not copy)<br>spec:</em></pre><pre><strong><em>clusters:</em></strong></pre><pre><strong><em>- link: us-west1-b/blue</em></strong></pre><pre><strong><em>- link: us-west1-c/green</em></strong></pre><pre><em>template:</em></pre><pre><em>spec:</em></pre><pre><em>ports:</em></pre><pre><em>- name: web</em></pre><pre><em>port: 80</em></pre><pre><em>protocol: TCP</em></pre><pre><em>targetPort: 8080</em></pre><pre><em>selector:</em></pre><pre><em>app: frontend</em></pre><p>MulticlusterService looks like a Kubernetes Service resource, except it has a clusters spec. The clusters value is the list of registered clusters where the MulticlusterService is created.</p><ul><li>The MulticlusterIngress creates a GCLB in the project with a backend service pointing to the MulticlusterService. Wait until you see a public VIP for the GCLB. Run the following command until you see the GCLB VIP in the status.</li></ul><pre>watch kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusteringress -o jsonpath=”{.items[].status.VIP}” </pre><pre><em>Output (do not copy)<br>EXTERNAL IP ADDRESS</em></pre><blockquote>This step takes a few minutes. Do not proceed until you see a VIP assigned in the MulticlusterIngress status. Until the GCLB VIP is created, you will see null as the output for the command above.</blockquote><ul><li>Press <strong>CTRL-C</strong> or <strong>CMD-C</strong> on a Macbook to exit the watch command.</li><li>Access the Hipster Shop app by navigating to the IP address on the GCLB VIP from the previous step.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*2Stj_aM8WlmvFM_l" /><figcaption>Hipster Shop frontend</figcaption></figure><blockquote>If you see a 404 error message, wait a bit longer and refresh the page until you see the Hipster shop frontend.</blockquote><p>Browse through the app and make a few purchases. At this point the Hipster Shop app is deployed to both blue and green clusters and the MulticlusterIngress is set up to send traffic to both clusters.</p><h3>Set up load generator</h3><p>In this section, you set up a loadgenerator that generates client traffic to the GCLB VIP. Initially, this sends traffic to both blue and green clusters since the MulticlusterService is set up to send traffic to both clusters. Later, you configure the MulticlusterService to send traffic to a single cluster.</p><ul><li>Get the GCLB VIP.</li></ul><pre>export GCLB_VIP=$(kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusteringress -o json | jq -r ‘.items[].status.VIP’)<br>echo ${GCLB_VIP}</pre><pre><em>Output (do not copy)<br>EXTERNAL IP ADDRES</em></pre><ul><li>Configure the loadgenerator to send client traffic to the GCLB VIP.</li></ul><pre>sed -i &#39;s/GCLB_VIP/&#39;${GCLB_VIP}&#39;/g&#39; ${WORKDIR}/load-generator/loadgenerator.yaml</pre><ul><li>Deploy the loadgenerator in the ingress-config cluster.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f ${WORKDIR}/load-generator</pre><ul><li>Verify loadgenerator Pod is Running in the ingress-config cluster.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} get pods</pre><pre><em>Output (do not copy)<br>NAME READY STATUS RESTARTS AGE</em></pre><pre><em>loadgenerator-5498cbcb86-hqscp 1/1 </em><strong><em>Running</em></strong><em> 0 53s</em></pre><pre><em>loadgenerator-5498cbcb86-m2z2z 1/1 </em><strong><em>Running</em></strong><em> 0 53s</em></pre><pre><em>loadgenerator-5498cbcb86-p56qb 1/1 </em><strong><em>Running</em></strong><em> 0 53s</em></pre><h3>Monitor with Cloud Monitoring</h3><p>In this section you monitor the traffic to the Hipster Shop app using Cloud Monitoring and Cloud Console.</p><h4>Monitoring GCLB metrics with Cloud Console</h4><p>In the previous section, you set up a loadgenerator deployment that simulates Hipster Shop client traffic by accessing it through the GCLB VIP. You can monitor GCLB metrics through the Cloud Console.</p><ul><li>Get the name of the forwarding rule for the frontend-multicluster-ingress.</li></ul><pre>export INGRESS_LB_RULE=$(gcloud compute forwarding-rules list | grep frontend-multicluster-ingress | awk &#39;{print $4}&#39;)</pre><pre>echo ${INGRESS_LB_RULE}</pre><pre><em>Output (do not copy)<br>mci-h8zu63-default-frontend-multicluster-ingress</em></pre><ul><li>Navigate to the link from the output of the command below.</li></ul><pre>echo &quot;https://console.cloud.google.com/net-services/loadbalancing/details/http/${INGRESS_LB_RULE}?project=${PROJECT}&amp;tab=monitoring&amp;duration=PT1H&quot;</pre><pre><em>Output (do not copy)<br>https://console.cloud.google.com/net-services/loadbalancing/details/http/mci-h8zu63-default-frontend-multicluster-ingress?project=gke-multicluster-upgrades&amp;tab=monitoring&amp;duration=PT1H</em></pre><p>You can also get to this page via the Cloud Console by navigating to <strong>Network services &gt; Load balancing</strong> page. Click on the forwarding rule that starts with ‘mci’ and then click on the <strong>Monitoring</strong> tab.</p><ul><li>In the <strong>Load balancer details Monitoring</strong> page, select the backend beginning with <strong>mci</strong> from the <strong>Backend</strong> dropdown menu.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/0*rDL3PaZ-5XHYjF7I" /><figcaption>Google Cloud Load Balancer Monitoring</figcaption></figure><ul><li>You can see that the traffic to the Hipster Shop app is going to both blue and green clusters (noted by the two zones the clusters are in). You can also see the timeline metrics chart showing traffic going to both backends.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mkDrXdndKvRKvBXE" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SCmi_tK1XnYli5T-" /><figcaption>Google Cloud Load Balancer Monitoring</figcaption></figure><blockquote>It takes a few minutes for load balancer metrics to populate. Refresh the page after a few minutes if needed.</blockquote><p>You can see the NEGs (under the Backend label starting with k8s1-) for the two frontend MulticlusterServices running in both the blue and green cluster.</p><h4>Monitoring with Cloud Monitoring</h4><p>You can monitor the metrics of your applications via Cloud Monitoring.</p><ul><li>Navigate to the link from the output of the command below to access Cloud Monitoring metric explorer.</li></ul><pre>echo &quot;https://console.cloud.google.com/monitoring/metrics-explorer?project=${PROJECT}&amp;pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22backend_scope%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D&quot;</pre><p>You can also access the same page via the Cloud Console. Click on <strong>Monitoring &gt; Metrics explorer</strong>. In the <strong>Find resource type and metrics</strong> search bar, type and select `https_lb_rule`. For <strong>Select a metric</strong>, search for and select `Request count`. In the <strong>Group by</strong> search bar, search for and select `backend_scope`.</p><ul><li>When accessing the link for the first time, Cloud Monitoring sets up a new workspace for you.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/0*NuHHbT1y9YVKZxk3" /></figure><ul><li>After the workspace is created, the chart shows the GCLB Request Count metrics grouped by backends.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c7TjbGh_k1UypRj-" /><figcaption>Cloud Monitoring Metrics Explorer</figcaption></figure><p>You see traffic going to the two backends (blue and green clusters). The backends are represented by the zones for the two frontend MulticlusterServices running in both clusters.</p><blockquote>You may see a third backend (named INVALID_BACKEND) for a short period of time. This is a UI anomaly in Cloud Monitoring and can be ignored.</blockquote><h3>Drain and upgrade the green cluster</h3><p>In this section, you “drain” the green cluster. Draining a cluster means you remove it from the load balancing pool. After you drain the green cluster, all client traffic destined for the Hipster Shop app goes to the blue cluster. You can monitor this process as described in the previous section. Once the cluster is drained, you can upgrade the drained cluster. After the upgrade, you can put it back in the load balancing pool. You follow the same steps to upgrade the other cluster (not shown in this tutorial).</p><p>To drain the green cluster, you update the MulticlusterService resource in the ingress-cluster and remove the green cluster from the clusters spec.</p><h4>Drain the green cluster</h4><ul><li>Update the MulticlusterService resource in the ingress-config cluster.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mcs-blue.yaml</pre><ul><li>Verify that you only have the blue cluster in the clusters spec.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusterservice -o json | jq &#39;.items[].spec.clusters&#39;</pre><pre><em>Output (do not copy)<br>{</em></pre><pre><em>   “link”: “us-west1-b/blue”</em></pre><pre><em>}</em></pre><p>You see only the blue cluster in the clusters spec. This means only the blue cluster is in the GCLB load balancing pool.</p><h4>Monitor Ingress metrics</h4><ul><li>Navigate to the Cloud Monitoring metrics chart page or by clicking the output of the following command.</li></ul><pre>echo &quot;https://console.cloud.google.com/monitoring/metrics-explorer?project=${PROJECT}&amp;pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22backend_scope%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D&quot;</pre><ul><li>The output shows that only the blue cluster is receiving traffic.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/0*BcL4BfJvXY4yVn6y" /><figcaption>Cloud Monitoring</figcaption></figure><blockquote>It may take a few minutes for Cloud Monitoring metrics to refresh.</blockquote><ul><li>You also see similar metrics from the GCLB metrics in the Cloud Console. Navigate to the GCLB Monitoring metrics page in Cloud Console or click on the output of the following command.</li></ul><pre>echo &quot;https://console.cloud.google.com/net-services/loadbalancing/details/http/${INGRESS_LB_RULE}?project=${PROJECT}&amp;tab=monitoring&amp;duration=PT1H&quot;</pre><ul><li>The chart also shows only the blue cluster is receiving Hipster Shop traffic.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N6ei9Wbt9u9WcOcW" /><figcaption>Google Cloud Load Balancer Monitoring</figcaption></figure><blockquote>It may take a few minutes for load balancer metrics to refresh.</blockquote><h4>Upgrade the green cluster</h4><p>Now that the green cluster is no longer receiving any client traffic, you can upgrade the cluster (master and nodes). Get the current cluster version for the master and the nodes. You can then find the available versions (in the zone). Finally, you can upgrade the cluster to the newer version.</p><ul><li>Get the current version of the clusters.</li></ul><pre>gcloud container clusters list</pre><pre><em>Output (do not copy)<br>NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS</em></pre><pre><em>ingress-config us-west1-a 1.14.10-gke.24 35.203.165.186 n1-standard-1 1.14.10-gke.24 3 RUNNING</em></pre><pre><em>blue us-west1-b 1.14.10-gke.24 34.82.76.141 n1-standard-1 1.14.10-gke.24 4 RUNNING</em></pre><pre><strong><em>green us-west1-c 1.14.10-gke.24 35.197.46.200 n1-standard-1 1.14.10-gke.24 4 RUNNING</em></strong></pre><blockquote>Your cluster versions may be different depending on the time you go through this tutorial.</blockquote><ul><li>Get the list of available master versions in the zone.</li></ul><pre>gcloud container get-server-config --zone us-west1-c --format=json | jq &#39;.validMasterVersions&#39;</pre><pre>Output (do not copy)<br>[</pre><pre>“1.15.9-gke.22”,</pre><pre>“1.15.9-gke.12”,</pre><pre>“1.15.9-gke.9”,</pre><pre>“1.15.9-gke.8”,</pre><pre>“1.15.8-gke.3”,</pre><pre>“1.14.10-gke.27”,</pre><pre>“1.14.10-gke.24”,</pre><pre>“1.14.10-gke.22”,</pre><pre>“1.14.10-gke.21”,</pre><pre>“1.14.10-gke.17”,</pre><pre>“1.13.12-gke.30”</pre><pre>]</pre><ul><li>Get a list of available node versions in the zone.</li></ul><pre>gcloud container get-server-config --zone us-west1-c --format=json | jq &#39;.validNodeVersions[0:20]&#39;</pre><pre>Output (do not copy)<br>[</pre><pre>“1.15.9-gke.22”,</pre><pre>“1.15.9-gke.12”,</pre><pre>“1.15.9-gke.9”,</pre><pre>“1.15.9-gke.8”,</pre><pre>“1.15.8-gke.3”,</pre><pre>“1.15.8-gke.2”,</pre><pre>“1.15.7-gke.23”,</pre><pre>“1.15.7-gke.2”,</pre><pre>“1.15.4-gke.22”,</pre><pre>“1.14.10-gke.27”,</pre><pre>“1.14.10-gke.24”,</pre><pre>“1.14.10-gke.22”,</pre><pre>“1.14.10-gke.21”,</pre><pre>“1.14.10-gke.17”,</pre><pre>“1.14.10-gke.0”,</pre><pre>“1.14.9-gke.23”,</pre><pre>“1.14.9-gke.2”,</pre><pre>“1.14.9-gke.0”,</pre><pre>“1.14.8-gke.33”,</pre><pre>“1.14.8-gke.21”</pre><pre>]</pre><ul><li>Pick the desired master and node version (higher than the current master and node version) for the green cluster.</li></ul><pre>export UPGRADE_VERSION=&quot;1.15.9-gke.26&quot;</pre><blockquote>Your cluster versions may be different depending on the time you go through this tutorial. In order to determine which versions to pick, refer to the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster#upgrading_the_cluster">upgrading clusters and node pools documentation</a>. Generally, it is best practice to pick the same version for the master and the nodes. Pick one of the available versions in your list for both master and node.</blockquote><ul><li>Upgrade the master for the green cluster.</li></ul><pre>gcloud container clusters upgrade green \<br> --zone us-west1-c --master --cluster-version ${UPGRADE_VERSION}</pre><ul><li>Type “<strong>Y</strong>” to confirm the upgrade.</li></ul><pre><em>Do you want to continue (Y/n)?</em> <strong>Y</strong></pre><pre><em>Upgrading green…⠧</em></pre><blockquote>This process takes a few minutes to complete. Please wait until the master upgrade is complete before proceeding. Once the update is complete, you should see the following message.</blockquote><blockquote>Updated [https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-c/clusters/green].</blockquote><ul><li>Once the master upgrade is completed, you can upgrade the nodes in the green cluster by running the following command.</li></ul><pre>gcloud container clusters upgrade green \</pre><pre> --zone=us-west1-c --node-pool=default-pool \</pre><pre> --cluster-version ${UPGRADE_VERSION}</pre><ul><li>Type “<strong>Y</strong>” to confirm the upgrade.</li></ul><pre><em>Do you want to continue (Y/n)?</em> <strong>Y</strong></pre><pre><em>Upgrading green… Done with 0 out of 4 nodes (0.0%): 1 being processed…⠶</em></pre><blockquote>This process takes a few minutes to complete. Please wait until the node upgrade is complete before proceeding. After the upgrade is complete, you should see the following message.</blockquote><blockquote>Upgrading green… Done with 4 out of 4 nodes (100.0%): 4 succeeded…done.</blockquote><blockquote>Updated [https://container.googleapis.com/v1/projects/gke-multicluster-upgrades/zones/us-west1-c/clusters/green].</blockquote><ul><li>Verify that the green cluster (master and nodes) are upgraded.</li></ul><pre>gcloud container clusters list</pre><pre><em>Output (do not copy)<br>NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS</em></pre><pre><em>ingress-config us-west1-a 1.14.10-gke.24 35.203.165.186 n1-standard-1 1.14.10-gke.24 3 RUNNING</em></pre><pre><em>blue us-west1-b 1.14.10-gke.24 34.82.76.141 n1-standard-1 1.14.10-gke.24 4 RUNNING</em></pre><pre><strong><em>green us-west1-c 1.15.8-gke.3 35.197.46.200 n1-standard-1 1.15.8-gke.3 4 RUNNING</em></strong></pre><blockquote>Your cluster versions may be different depending on the time you go through this tutorial.</blockquote><h3>Add green cluster to the load balancing pool</h3><p>In this section, you add the green cluster back in the load balancing pool.</p><ul><li>Verify that all Hipster Shop app deployments are running on the green cluster prior to adding it back to the load balancing pool.</li></ul><pre>kubectl --context ${GREEN_CLUSTER} get pods</pre><pre>Output (do not copy)<br>NAME READY STATUS RESTARTS AGE</pre><pre>adservice-86f5dfb995–2b25h 1/1 Running 0 16m</pre><pre>cartservice-76cf9686b6-ws7b7 1/1 Running 1 13m</pre><pre>checkoutservice-7766b946f5–6fhjh 1/1 Running 0 9m50s</pre><pre>currencyservice-76975c7847-rf8r7 1/1 Running 0 13m</pre><pre>emailservice-c55cd96f-pht8h 1/1 Running 0 13m</pre><pre>frontend-f4b7cd95-wxdsh 1/1 Running 0 13m</pre><pre>loadgenerator-6784bc5f77–6b4cd 1/1 Running 6 6m34s</pre><pre>paymentservice-696f796545–9wrl7 1/1 Running 0 9m49s</pre><pre>productcatalogservice-7975f8588c-kbm5k 1/1 Running 0 6m33s</pre><pre>recommendationservice-6d44459d79-km8vm 1/1 Running 0 9m49s</pre><pre>redis-cart-6448dcbdcc-sjg69 1/1 Running 0 13m</pre><pre>shippingservice-767f84b88c-gh9m4 1/1 Running 0 9m49s</pre><ul><li>Update the MutliclusterService to add the green cluster back in the load balancing pool.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} apply -f mcs-blue-green.yaml</pre><ul><li>Verify that you have both blue and green clusters in the clusters spec.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} get multiclusterservice -o json | jq &#39;.items[].spec.clusters&#39;</pre><pre><em>Output (do not copy)<br>{</em></pre><pre><em>   “link”: “us-west1-b/blue”</em></pre><pre><em>},</em></pre><pre><em>{</em></pre><pre><em>   “link”: “us-west1-c/green”</em></pre><pre><em>}</em></pre><p>You see both blue and green clusters in the clusters spec.</p><h4>Monitor Ingress metrics</h4><ul><li>Navigate to the Cloud Monitoring metrics chart page or by clicking the output of the following command.</li></ul><pre>echo &quot;https://console.cloud.google.com/monitoring/metrics-explorer?project=${PROJECT}&amp;pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22loadbalancing.googleapis.com%2Fhttps%2Frequest_count%5C%22%20resource.type%3D%5C%22https_lb_rule%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22unitOverride%22:%221%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22backend_scope%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D&quot;</pre><ul><li>The output shows that both blue and green clusters are receiving traffic.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6PtdMPPeONmAV1XB" /><figcaption>Cloud Monitoring</figcaption></figure><blockquote>It may take a few minutes for Monitoring metrics to refresh.</blockquote><ul><li>You also see similar metrics from the GCLB metrics in the Cloud Console. Navigate to the GCLB Monitoring metrics page in Cloud Console or click on the output of the following command.</li></ul><pre>echo &quot;https://console.cloud.google.com/net-services/loadbalancing/details/http/${INGRESS_LB_RULE}?project=${PROJECT}&amp;tab=monitoring&amp;duration=PT1H&quot;</pre><p>The chart also shows both blue and green clusters are receiving Hipster Shop traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WzhpIOgyAwEdW7pF" /><figcaption>Google Cloud Load Balancer Monitoring</figcaption></figure><blockquote>It may take a few minutes for load balancer metrics to refresh.</blockquote><blockquote>Congratulations. You successfully upgraded a GKE cluster in a multi cluster architecture using Ingress for Anthos. You can upgrade the blue cluster using the same process.</blockquote><h3>Cleaning up</h3><p>To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:</p><h4>Delete clusters</h4><ul><li>Unregister and delete blue and green clusters.</li></ul><pre>gcloud container hub memberships unregister green \</pre><pre> --project=${PROJECT} \</pre><pre> --gke-uri=${GREEN_URI}</pre><pre>gcloud container clusters delete green --zone us-west1-c --quiet</pre><pre>gcloud container hub memberships unregister blue \</pre><pre> --project=${PROJECT} \</pre><pre> --gke-uri=${BLUE_URI}</pre><pre>gcloud container clusters delete blue --zone us-west1-b --quiet</pre><ul><li>Delete the MuticlusterIngress from the ingress-config cluster. This deletes the GCLB resources from the project. Unregister and delete the ingress-config cluster.</li></ul><pre>kubectl --context ${INGRESS_CONFIG_CLUSTER} delete -f mci.yaml<br></pre><pre>gcloud container hub memberships unregister ingress-config \</pre><pre> --project=${PROJECT} \</pre><pre> --gke-uri=${INGRESS_CONFIG_URI}</pre><pre>gcloud container clusters delete ingress-config --zone us-west1-a --quiet</pre><ul><li>Verify all clusters are deleted.</li></ul><pre>gcloud container clusters list</pre><pre><em>Output (do not copy)<br>&lt;null&gt;</em></pre><ul><li>Unset the kubeconfig file. This resets to the default kubeconfig file.</li></ul><pre>unset KUBECONFIG</pre><ul><li>Remove the WORKDIR folder.</li></ul><pre>cd ${HOME}<br>rm -rf ${WORKDIR}</pre><h4>Delete the project</h4><p>The easiest way to eliminate billing is to delete the project you created for the tutorial.</p><p>To delete the project:</p><ul><li>In the Cloud Platform Console, go to the Projects page.</li></ul><pre><a href="https://console.cloud.google.com/iam-admin/projects">GO TO THE PROJECTS PAGE</a></pre><ul><li>In the project list, select the project you want to delete and click <strong>Delete</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*5LNtYYIuATvutSBm" /><figcaption>Google Cloud Console</figcaption></figure><ul><li>In the dialog, type the project ID, and then click <strong>Shut down</strong> to delete the project.</li></ul><h3>What’s next</h3><ul><li>Learn more about <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-for-anthos">Ingress for Anthos</a></li><li>Learn how to deploy Ingress for Anthos — <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-for-anthos">Deploying Ingress for Anthos across clusters</a></li><li>Try out other Google Cloud Platform features for yourself. Have a look at our <a href="https://cloud.google.com/docs/tutorials">tutorials</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee0f583d9b10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-ee0f583d9b10">Kubernetes Engine (GKE) multi-cluster life cycle management series</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Engine (GKE) multi-cluster life cycle management series]]></title>
            <link>https://medium.com/google-cloud/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-43ed97142c47?source=rss-91845b0cfdde------2</link>
            <guid isPermaLink="false">https://medium.com/p/43ed97142c47</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ameer Abbas]]></dc:creator>
            <pubDate>Wed, 22 Apr 2020 22:22:15 GMT</pubDate>
            <atom:updated>2020-04-24T23:09:06.798Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part IV: GKE multi-cluster lifecycle management</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZDweVMWRCFztrS-zwXuPw.png" /><figcaption>Distributed Service Foo on GKE</figcaption></figure><p>In this blog, I discuss common GKE life cycle strategies as well as planning and design considerations when picking the right one for you.</p><blockquote>I’ll assume you already know the reasons for <strong>multi-cluster architectures</strong> (<a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-17ff892b85c8">part I</a>), what are <strong>Distributed Services</strong> (<a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-b0d39080ac6b">part II</a>) and what constitutes a <strong>GKE upgrade</strong> (<a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-a73185392345">part III</a>).</blockquote><h3>Planning and design considerations</h3><p>GKE multi-cluster architecture plays a part in selecting a desirable cluster lifecycle management strategy. Before discussing these strategies, it is important to discuss certain design decisions that may affect or be affected by the cluster lifecycle management strategy.</p><p><strong>Type of clusters</strong> — If using the GKE auto upgrade as a cluster lifecycle management strategy, the type of cluster may matter. For example, regional clusters offer multiple masters where masters are auto upgraded one at a time vs zonal clusters which offer a single master. If you’re not using GKE auto upgrades and believe that all Kubernetes clusters are to be treated as infrastructure components (and disposable) then it may not matter what type of cluster you choose when deciding on a cluster lifecycle management strategy. The strategies discussed in the next section can be applied to any type of cluster.</p><p><strong>Cluster placement and footprint</strong> — There are a few factors that need to be considered when deciding on the cluster placement and footprint.</p><ol><li>Zones and regions clusters are required to be in</li><li>Number and size of clusters needed</li></ol><p>The first question is usually easy to answer as the zones/regions are dictated by your business and in which regions you are serving your users out of. The answer to the second question typically falls in two categories each with pros and cons.</p><ol><li><strong>Small number of large clusters</strong> — You may choose to utilize the redundancy and resiliency provided by regional clusters and place one (or two) large sized regional clusters per region. The benefit of this approach is low operational overhead of managing multiple clusters. The downside is it may affect a large number of services at once i.e. large blast radius.</li><li><strong>Large number of small clusters</strong> — Another strategy is to create a large number of small sized clusters. This reduces the cluster blast radius as your services are split across many clusters. This approach also works well for short lived ephemeral clusters (for example clusters running a batch workload). The downside of this approach is higher operational overhead as there are more clusters to upgrade. There may also be additional cost associated with a higher number of masters. This can be offset by automation, predictable schedule and strategy and careful coordination between teams and services affected.</li></ol><p>The guide does not recommend one approach over the other, they are simply there as options. In some cases, both design patterns can be chosen for different categories of services. The strategies discussed below work with either design choice. A guide for deciding on a strategy can be found <a href="https://cloud.google.com/solutions/scope-and-size-kubernetes-engine-clusters">here</a>.</p><p><strong>Capacity planning</strong> — When planning for capacity, it is important to take into account the chosen cluster lifecycle strategy. Capacity planning must take into account normal service load and maintenance events. There are two types of outages:</p><ol><li>Planned events like cluster upgrades</li><li>Unplanned events like cluster outages for example bad config pushes, bad rollouts etc.</li></ol><p>When capacity planning, you must take into account any total or partial outages. If you design for planned maintenance events only then all distributed services must have one additional cluster than required so that one cluster may be taken out of rotation at a time for upgrades without degrading the service. This approach is also referred to as “<a href="https://landing.google.com/sre/sre-book/chapters/software-engineering-in-sre/">N+1 capacity planning</a>”. If you design for planned and unplanned maintenance events then all distributed services must have two (or more) additional clusters than required to serve intended capacity — one for the planned event and one for an unplanned event in case it occurs during the planned maintenance window. This approach is also referred to as “<a href="https://landing.google.com/sre/sre-book/chapters/software-engineering-in-sre/">N+2 capacity planning</a>”.</p><p>In multi-cluster architectures, the terms <strong>draining</strong> and <strong>spilling</strong> are often used. These terms refer to the process of removing (or draining) traffic from a cluster and redirecting (or spilling) traffic onto other clusters during upgrades and maintenance events. This is accomplished using networking solutions like multi-cluster ingress or other load balancing methods. Careful use of draining and spilling are at the heart of some cluster lifecycle management strategies which are discussed in the following section. During capacity planning, these must be taken into account. For example: “if a single cluster is drained, do other clusters have enough capacity to handle the additional spilled traffic?” Other considerations include sufficient capacity in the desired zone or region or is there a need to send traffic to a different region (if using a single regional cluster per region for example).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOWGTbepRNuq04C7tvdEew.png" /><figcaption>Draining and Spilling</figcaption></figure><p><strong>Clusters and distributed services</strong> — Services based cluster design dictates that cluster architecture (number, size and location) is determined by services that are required to run on the clusters. Therefore the placement of your clusters are dictated by where the distributed services are needed. There are a number of considerations that are taken into account when deciding the placement of distributed services. Some of these are:</p><ol><li><strong>Location requirement</strong> — where/which regions does the service need to be served out of</li><li><strong>Criticality</strong> — how critical is the availability of a service to the business</li><li><strong>SLO</strong> — service level objectives for the service (typically based on criticality)</li><li><strong>Resilience — </strong>How resilient does the service need to be? Does it need to withstand cluster, zonal or even regional failures.</li></ol><p>When planning for cluster upgrades, you must consider the number of services a single cluster affects when it is drained and must account for spilling each of these services to other appropriate clusters. Clusters can be single tenant or multi-tenant. Single tenant clusters only serve a single service or a product represented by a set of services. They do not share the cluster with other services or products. Multi tenant clusters can run many services and products, typically partitioned into namespaces. When planning for cluster upgrades, you must consider the number of services a single cluster affects when it is drained and must account for spilling each of these services to other appropriate clusters.</p><p><strong>Impact to teams</strong> — A cluster event not only affects services but may also impact teams. For instance, the DevOps team might need to redirect or halt their CI/CD pipelines during a cluster upgrade. Likewise, support teams might get alerted for planned outages and need to be notified. It is therefore important to realize and plan for the fact that cluster events impact more than just services. AutomationCareful coordination and tooling must be in place to help ease the impact to multiple teamssurprises between operations and support teams. A cluster or a cluster fleet upgrade should be considered as routine and “uneventful” with all teams well informed.</p><p><strong>Timing, scheduling and coordination</strong> — Kubernetes releases a new minor version quarterly and maintains the last three releases. Timing and scheduling cluster upgrades must be planned carefully. Collectively, there must be an agreement on when these upgrades take place. There are a few considerations in this regard:</p><ol><li>How often do you upgrade? Do you upgrade every quarter or perhaps every half (two quarters)? Or a different timeline.</li><li>When to upgrade? Do you upgrade at the beginning of the quarter when business slows down or during other business downtime dictated by your specific industry.</li><li>When not to upgrade? Do you have clear planning around when not to upgrade. For example, Black Friday, Cyber Monday or during high profile conferences and other industry specific events.</li></ol><p>It is important to have a strategy in place. It is equally important that it is clearly communicated with the service owners as well as the operations and support teams. There should be no surprises. Everyone knows and expects when and how the clusters are upgraded. This requires clear coordination with all of the teams involved. A single service has multiple teams that interact with it. Typically they can be grouped into two categories. The service developer is the persona that is responsible for creating and coding the business logic into a service. And the service operator that is responsible for safely, and reliably running the service. The operators can be composed of multiple teams like policy/security admin, networking admin, support teams etc. They must all be kept in the loop during cluster upgrades so they can take proper actions during this time. One option is to plan for this the same way as an outage incident. You have an incident commander, a chat room, a postmortem (even if no users were impacted).</p><p>With these design and planning considerations, let’s discuss common GKE multi-cluster lifecycle management strategies.</p><h3>GKE Cluster lifecycle Strategies</h3><p>The section discusses three main cluster lifecycle management strategies often used in GKE multi-cluster architecture. It is important to note that one size does not fit all and you may end up choosing multiple strategies for various categories of services and needs of the business.</p><h3>Rolling Upgrades</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tHmO-z7HKAk8FWZ6yE9iaA.png" /><figcaption>Rolling Upgrade Strategy</figcaption></figure><p>This is the simplest and the most cost effective strategy. You start with <strong>N</strong> number of clusters running the old_ver (or current production version). You then drain <strong>m</strong> clusters at a time, where <strong>m</strong> is less than <strong>N</strong>. You then delete and recreate new clusters with the new desired version, or upgrade the drained clusters. The decision between deleting and upgrading new clusters depends upon the size of the clusters as well as your belief in immutable infrastructure. Immutable infrastructure dictates that instead of constantly upgrading a cluster which may produce undesirable results over time, you create new clusters instead, avoiding any unforeseen configuration drift. This is quite easily accomplished if using GKE as you can create a GKE cluster with a single command or an API call. New cluster strategy requires that you have the entire cluster configuration (cluster manifests) stored outside of the cluster, typically in Git. You can then use the same configuration template on the newly created cluster. If this is a new cluster, ensure that your CI/CD pipelines are pointing to the correct cluster. After the cluster is properly configured, you can push traffic back onto the cluster slowly while monitoring services’ SLOs.</p><p>The process is repeated for all clusters. Depending upon your capacity planning you may end up upgrading multiple clusters at a time without violating service SLOs.</p><p>This strategy is great if you value simplicity and cost over resiliency. During this strategy, you never exceed the GKE fleet’s required capacity for all distributed services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QC91YCAKy77a3ppRes7ieg.png" /><figcaption>Rolling Upgrade Strategy Timeline</figcaption></figure><h3>Blue/Green</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g9xyczG4yAQkTcOtuFewXA.png" /><figcaption>Blue/Green Strategy</figcaption></figure><p>This is a simple strategy, it provides some added resiliency but a bit more costly compared to the previous one. This strategy is very similar to the previous one. The only difference is instead of draining existing clusters first, you create <strong>m</strong> new clusters with the desired version first, where <strong>m</strong> is less or equal to <strong>N</strong>. You add the new clusters to the CI/CD pipelines and then slowly spill traffic over while monitoring the service SLOs. When the new clusters are fully taking traffic, you drain and delete clusters with the older version. This is akin to a blue/green upgrade strategy typically used for services. Creating multiple new clusters at a time will increase the overall cost but gives you the benefit of speeding up the fleet upgrade time. The added cost is only for the duration of the upgrade, during which additional clusters are used. The benefit of creating new clusters first is easy rollbacks in case of a failure. The new cluster can also be tested before sending production traffic to it. As these clusters co-exist with their old version counterparts for a small period of time, the additional cost should be minimal.</p><p>This strategy is great if you value simplicity and resiliency over cost. During the strategy, additional cluster(s) are added first hence exceeding the GKE fleet’s required capacity for the duration of the upgrades.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8HkZ5UofhQ-b2JszqzYfUA.png" /><figcaption>Blue/Green Strategy Timeline</figcaption></figure><h3>Canary Clusters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5Qabc_K0U3fOrZKGfJu9g.png" /><figcaption>Canary Clusters Strategy</figcaption></figure><p>This is the most resilient and complex strategy out of the three. This strategy completely abstracts cluster lifecycle management from services lifecycle management, thereby offering the lowest risk and highest resilience for your services. In the previous two strategies, you maintain your entire GKE fleet on a single version. In this strategy, you maintain two or perhaps three fleets of GKE clusters running different versions. Instead of upgrading the clusters, you migrate services from one fleet of clusters to the other over time. When the oldest GKE fleet is naturally drained (meaning all of the services have been migrated to the next versioned GKE fleet), you delete the fleet. This strategy requires you to maintain a minimum of two GKE fleets. One for the current production and one for the next production candidate version. You can also maintain more than two GKE fleets. This gives you more flexibility but your cost and operational overhead also goes up. It is important to note that this is not the same having clusters in different environments for example dev, stage and prod. Non production environments are great for testing the Kubernetes features and/or services with non production traffic. This strategy dictates that you maintain multiple GKE fleet versions in the production environment. This is similar to canary release strategies often used by services. With canary service deployments, the service owner can always pinpoint issues to a particular version of the service, with canary clusters, they must also take into account GKE fleet versions their services are running on. A single distributed service version could potentially run on multiple GKE fleet versions. The migration of a service can happen gradually so you can see the effects of the service on the new fleet prior to sending all of the traffic for the service to the new versioned clusters (as in the previous two strategies).</p><p>This strategy is great if you value resilience over everything else.</p><p>The decision tree below may be useful for determining which strategy is best for you based on the service and business needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJ5B6skNkddOgXlILp0_sA.png" /><figcaption>GKE Lifecycle Management Decision Matrix</figcaption></figure><blockquote>Up next… <a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-ee0f583d9b10">Part V Hands-on Lab: GKE multi-cluster rolling upgrades using Ingress for Anthos step-by-step tutorial</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43ed97142c47" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-43ed97142c47">Kubernetes Engine (GKE) multi-cluster life cycle management series</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GKE multi-cluster life cycle management series]]></title>
            <link>https://medium.com/google-cloud/gke-multi-cluster-life-cycle-management-series-a73185392345?source=rss-91845b0cfdde------2</link>
            <guid isPermaLink="false">https://medium.com/p/a73185392345</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Ameer Abbas]]></dc:creator>
            <pubDate>Tue, 21 Apr 2020 23:06:20 GMT</pubDate>
            <atom:updated>2020-04-23T23:38:13.968Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Engine (GKE) multi-cluster life cycle management series</h3><h4>Part III: GKE Cluster lifecycle management</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEX3Z66w9K0ZaHSTkcbCOg.png" /><figcaption>Distributed Service Foo on GKE</figcaption></figure><p>In the previous two parts (<a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-17ff892b85c8">part I</a> and <a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-b0d39080ac6b">part II</a>), I discussed the need for multi-cluster architectures and defined the concept of a Distributed Service, which is a Kubernetes Service running on multiple clusters. In this part, let’s dive into what we mean by GKE cluster lifecycle management. Cluster lifecycle management can be defined as strategies and planning required to maintain a healthy and updated fleet of Kubernetes clusters. Specifically, this refers to keeping the entire fleet of Kubernetes clusters updated without violating service SLOs. With proper strategies and planning in place, cluster lifecycle management should be routine, expected and “uneventful”.</p><p>Even though this blog series specifically focuses on <a href="https://cloud.google.com/kubernetes-engine">Kubernetes Engine</a> (GKE) lifecycle management, the concepts covered may be applied to other distributions of Kubernetes.</p><h3>GKE versioning and upgrades</h3><p>Before discussing strategies and planning for cluster lifecycle management, it is important to understand what constitutes a cluster upgrade.</p><p>A cluster is composed of two pieces: master(s) and nodes. A Kubernetes cluster upgrade requires all nodes and master(s) to be upgraded to the desired version. Kubernetes follow a <a href="https://semver.org/">semantic versioning</a> schema. Kubernetes versions are expressed as <strong>X.Y.Z</strong>. Where <strong>X</strong> is the major version, <strong>Y</strong> is the minor version and <strong>Z</strong> is the patch version. Minor releases occur approximately every three months (quarterly) and the Kubernetes project maintains release branches for the most recent three minor releases. This means a nine month old Kubernetes minor release is no longer maintained. In addition to not being maintained, lagging too far behind Kubernetes releases might also result in API changes when it is eventually upgraded to the latest versions. This may end up being more challenging and risky than keeping up to date with the Kubernetes versions. Needless to say, Kubernetes upgrades must be planned at a regular cadence. We suggest quarterly or every two quarters is a good time frame for planned GKE upgrades.</p><p>GKE clusters support running Kubernetes versions from any supported minor release. At least two, if not three, minor versions are available at any given time.</p><p>GKE offers three types of clusters.</p><ol><li>Single zone clusters — Single master and all node-pools in a single zone in a single region</li><li>Multi zonal clusters — Single master in one zone and node-pools in multiple zones in a single region</li><li>Regional clusters — Multiple masters and node-pools in multiple zones in a single region</li></ol><p>GKE is a managed service and offers auto-upgrades for both masters and nodes.</p><h3>GKE auto-upgrades</h3><p>GKE auto upgrades is a popular and often used cluster lifecycle strategy. GKE auto upgrades provide a no-ops way to keep your GKE clusters updated to supported versions. GKE auto upgrades upgrade master and nodes separately.</p><p><strong>Master auto upgrades</strong> — By default, GKE masters are automatically upgraded. Single zone and multi-zonal clusters have a single control plane (master). During master upgrade, workloads continue to run, however you cannot deploy new workloads, modify existing workloads, or make other changes to the cluster’s configuration until the upgrade is complete.</p><p>Regional clusters have multiple replicas of the control plane, and only one replica is upgraded at a time. During the upgrade, the cluster remains highly available, and each control plane replica is unavailable only while it is being upgraded.</p><p><strong>Node upgrades</strong> — Node pools are upgraded one at a time. Within a node pool, nodes are upgraded one at a time, in an undefined order. You can change the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-upgrades#surge">number of nodes upgraded</a> at a time. This process might take several hours depending on the number of nodes and their workload configurations.</p><h3>GKE auto upgrade lifecycle strategy</h3><p>We recommend using GKE auto upgrades where possible. GKE auto upgrades prioritizes convenience over control. Even so, GKE auto upgrades provide many ways to influence when and how your clusters get upgraded within certain parameters. You can influence the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions">maintenance windows and exclusions</a>. <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels">Release channels</a> influence the version selection and <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-upgrades#surge">surge upgrades</a> influence how many nodes at a time to upgrade. Despite these controls and even for regional clusters (with multiple Kubernetes control planes), GKE auto upgrades does not guarantee services’ uptime.</p><p>You may choose not to utilize the GKE auto upgrade feature if you require one or more of the following:</p><ol><li>Control the exact version of GKE clusters</li><li>Control the exact time to GKE upgrade</li><li>Control the upgrade strategy (discussed in the next blog) for your GKE fleet. For example, using multi-cluster as part of the cluster lifecycle management.</li></ol><p>If any of the above reasons exist for you, you cannot use the GKE auto-upgrade strategy. In the next part, I will dive into few multi cluster upgrade strategies as well as the planning associated with each.</p><blockquote>Up next… <a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-43ed97142c47">Part IV: GKE multi-cluster lifecycle management</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a73185392345" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/gke-multi-cluster-life-cycle-management-series-a73185392345">GKE multi-cluster life cycle management series</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GKE Multi cluster life cycle management series]]></title>
            <link>https://medium.com/google-cloud/gke-multi-cluster-life-cycle-management-series-b0d39080ac6b?source=rss-91845b0cfdde------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0d39080ac6b</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Ameer Abbas]]></dc:creator>
            <pubDate>Tue, 21 Apr 2020 22:34:36 GMT</pubDate>
            <atom:updated>2020-04-22T01:25:05.077Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Engine (GKE) multi-cluster life cycle management series</h3><h4>Part II: Multi-cluster and Distributed Services</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYXYKYNjX0FHzt8kKiwMxQ.png" /><figcaption>Distributed Services on GKE</figcaption></figure><p>In <a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-17ff892b85c8">part I</a>, I talked about various reasons for multi-cluster Kubernetes architectures. In this part, I’ll introduce the concept of a Distributed Service. A distributed service is a Kubernetes Service that is deployed to multiple Kubernetes clusters. Distributed services are stateless services and act identically across multiple clusters. This means a distributed service has the same Kubernetes Service name and is implemented in the same namespace across multiple clusters. Kubernetes Services are tied to the fate of the Kubernetes cluster they run on. If a Kubernetes cluster goes offline, so does the Kubernetes Service. Distributed services are abstracted from individual Kubernetes clusters. If one (or more) Kubernetes clusters are down, the distributed service may very well be online and within the desired <a href="https://en.wikipedia.org/wiki/Service-level_objective">SLO</a> (Service Level Objective).</p><p>In the following diagram, Foo is a distributed service running on multiple clusters (with the same Service name and namespace). As such, it is not tied to a single cluster and hence represented conceptually as a layer above the Kubernetes cluster infrastructure layer. If any of the individual clusters (running Service Foo) were to go down, the distributed service Foo still remains online. Meanwhile, Services Bar and Baz are Kubernetes Services running on single individual clusters. Their uptime (and availability) are dependent on the uptime of the specific Kubernetes cluster on which they reside.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K8ARuPpeG2AOJdFfGL5rRg.png" /><figcaption>Distributed Service Foo and Kubernetes Services Bar and Baz on GKE</figcaption></figure><p>Resiliency is one of the reasons for multi-cluster deployments. Distributed services create resilient services on multi-cluster architecture. Stateless services are prime candidates for distributed services in a multi-cluster environment. There are a few requirements and considerations when working with distributed services.</p><p><strong>multi-cluster networking </strong>— Traffic destined to a distributed service can be sent to clusters running that service. This is accomplished via a multi-cluster ingress technology like <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-for-anthos">Ingress for Anthos</a> or rolling your own external load balancer/proxy solution. Nonetheless, the networking solution must give the operator control over when, where (i.e. which cluster) and how much traffic is routed to a particular instance of a distributed service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PE5rV1PLkj6Oz1TFDnYSoA.png" /><figcaption>Multi-cluster Ingress to Distributed Service Foo</figcaption></figure><p><strong>Observability</strong> — Tools must be in place to observe at a distributed services layer. SLOs (typically for availability and latency) should be measured collectively for a distributed service. This provides a global view of how each service is performing across multiple clusters. While distributed service is not a well defined resource in most observability solutions, the intended outcome can be achieved by collecting and combining individual Kubernetes Service metrics. Solutions like <a href="https://cloud.google.com/monitoring">Cloud Monitoring</a> or open source tools like <a href="http://grafana.com/">Grafana</a> and many others provide Kubernetes Service metrics. Service mesh solutions like <a href="http://istio.io/">Istio</a> and <a href="https://cloud.google.com/anthos/service-mesh">ASM</a> also provide Service metrics out of the box without any instrumentation required.</p><p><strong>Service Placement</strong> — Kubernetes Services provide node level fault tolerance within a single Kubernetes cluster. This means that a Kubernetes Service can withstand node outages. During node outages, a Kubernetes master automatically reschedules Pods to healthy nodes. A distributed service provides cluster level fault tolerance. This means a distributed service can withstand cluster outages. When capacity planning for a distributed service, this must be taken into account. A distributed service does not need to run on every cluster available. Which clusters a distributed service runs on may depend upon the following requirements:</p><ol><li>Where, or in which regions, is the service required?</li><li>What is the required SLO for the distributed service?</li><li>What type of fault tolerance is required for the distributed service? Cluster, zonal or regional. For instance, multiple clusters in a single zone, or multiple clusters across zones in a single region or multiple regions.</li><li>What level of outages should the distributed service withstand in the worst-case? At a cluster layer</li><li>N+1, meaning a distributed service can withstand a single cluster failure</li><li>N+2, meaning two concurrent failures. For instance a planned and an unplanned outage of a Kubernetes Service in two clusters at the same time.</li></ol><p>The placement of a distributed service depends on the above requirements.</p><p><strong>Rollouts/Rollback</strong> — Distributed services, like Kubernetes Services, allow for gradual rollouts and rollbacks. Unlike Kubernetes Services, distributed services enable clusters as an additional unit of deployment as a means for gradual change. Rollouts and rollbacks also depend upon the service requirement. In some cases, it may be required to upgrade the service on all the clusters at the same time (for example a bug fix). In other cases, it may be required to slowly roll out (or stagger) the change one cluster at a time. This lowers the risk to the distributed service by gradually introducing changes to the service. However, this might take longer depending on the number of clusters. Typically, there is no one size fits all solution. Often, multiple rollout/rollback strategies are used depending upon the distributed service requirements. The important point here is that distributed services must allow for gradual and controlled changes in the environment.</p><p><strong>Business Continuity/Disaster Recovery (BCDR)</strong> — These terms are often used together. Business continuity refers to continuation of (critical) services in the face of a major (planned or unplanned) event, whereas disaster recovery refers to the steps taken or needed to return business operations to its normal state after such events. There are many strategies for BCDR that are beyond the scope of this guide. Suffice it to say, BCDR requires some level of redundancy in systems and services. The key premise of distributed services is they run in multiple locations (clusters, zones, regions). BCDR strategies are often dependent upon the rollout/rollback strategies discussed earlier. For example, if rollouts are performed in a staggered/controlled manner, the effect of a bug or a bad configuration push can be caught early without affecting a large number of users. At large scale coupled with rapid rate of change (e.g. in modern CI/CD practices), it is common that not all users are served the same version of a distributed service. BCDR planning and strategies in distributed systems and services differ from traditional monolithic architectures. In traditional systems, a change is made wholesale affecting a large number of or perhaps every user and thus must have a redundant/backup system in place in case of undesirable effects of a rollout. In distributed systems/services, almost all change is done in a gradual manner affecting a small number of users.</p><p><strong>Cluster lifecycle management</strong> — Like controlled rollouts/rollbacks, distributed services allow for controlled cluster lifecycle management. As mentioned before, distributed services provide cluster level resiliency. This means clusters can be “taken out of rotation” for maintenance. This can be done in multiple ways. Some of these strategies are described later in this guide. Suffice it to say here, cluster lifecycle management is a tenet of distributed services that does not apply to Kubernetes Services.</p><p>The remainder of this series focuses on the cluster lifecycle aspect of distributed services.</p><blockquote>Up next… <a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-a73185392345">Part III: GKE Cluster lifecycle management</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0d39080ac6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/gke-multi-cluster-life-cycle-management-series-b0d39080ac6b">GKE Multi cluster life cycle management series</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Engine (GKE) multi-cluster life cycle management series]]></title>
            <link>https://medium.com/google-cloud/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-17ff892b85c8?source=rss-91845b0cfdde------2</link>
            <guid isPermaLink="false">https://medium.com/p/17ff892b85c8</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ameer Abbas]]></dc:creator>
            <pubDate>Tue, 21 Apr 2020 22:19:24 GMT</pubDate>
            <atom:updated>2020-04-22T22:26:32.181Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part I: The need for multi-cluster</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UOmhMcwGcwO1Dn7s9CeSZQ.png" /><figcaption>Distributed Services on GKE</figcaption></figure><h3>Introduction</h3><p>This 5-part blog series covers how to design, plan for, and execute upgrades in a multi-cluster GKE architecture. This final part of this series is a walk-through step-by-step tutorial that uses <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-for-anthos">Ingress for Anthos</a> for multi cluster upgrades. Even though the tutorial uses Ingress for Anthos as an example implementation, the concepts can be applied using other solutions as well.</p><p>The series is divided into following parts:</p><ul><li>Part I: The need for multi-cluster (this one)</li><li><a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-b0d39080ac6b">Part II: Multi-cluster and Distributed Services</a></li><li><a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-a73185392345">Part III: GKE cluster life cycle and auto-upgrades</a></li><li><a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-43ed97142c47">Part IV: GKE multi-cluster lifecycle management planning and strategies</a></li><li><a href="https://medium.com/@ameericus/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-ee0f583d9b10">Part V: GKE multi-cluster upgrades using Ingress for Anthos step-by-step tutorial</a></li></ul><blockquote>A very special thanks to my friends: an SRE extraordinaire <a href="https://medium.com/u/40155432ed42">Steve McGhee</a> and Kubernetes expert and enthusiast <a href="https://thekev.in/blog/index.html">Kevin James</a> for their invaluable inputs and feedback with this series.</blockquote><h3>Part I: The need for multi-cluster</h3><h4>Kubernetes as infrastructure</h4><p>Before discussing the need for multi-cluster, it is important to note that this series considers Kubernetes clusters as pieces of “infrastructure”. In cloud-native parlance, infrastructure is disposable. No special treatment must be given to any infrastructure component. It is there to serve a purpose. The purpose of Kuberenetes is to provide automation (and orchestration) to developers and operators to serve container-based applications and services to consumers. Consumers can be internal teams, other services or external customers. A Kubernetes cluster is therefore an infrastructure component and hence should be considered disposable. More on this later.</p><h4>Common multi-cluster scenarios</h4><p>Despite the “kubernetes as infrastructure” argument, there are a number of reasons for having a multi-cluster environment.</p><p><strong>Geography</strong> — Many Services require to be in multiple regions. Placing a Service closer to the consumer (in their region) provides a better experience due to lower latency than if the Service is served out of a single region. A Kubernetes cluster runs in a single region. For multiregional deployments, multiple Kubernetes clusters in multiple regions are required. Likewise, multi cloud or hybrid cloud environments also require multiple clusters in each environment. Kubernetes clusters are also often co-located with the Services’ (stateful) data sources. Certain applications may be required to be in the same location (region and zone) as its backend, for example an RDBMS.</p><p><strong>Tenancy and environments</strong> — Kubernetes clusters are designed for multi-tenancy. This means a single cluster may be shared by multiple teams for their respective Services. Kubernetes provides standard resources (namespaces, RBAC, network policies, authentication) to properly configure access controls in multi-tenant environments. In some cases, certain Services might not be able to co-reside on a cluster with other Services. This could be due to company policy, privacy, security or industry regulation. In such cases, multiple clusters are required to separate certain tenants into their own clusters. Likewise, environments (dev, stage and prod) are also often created as separate clusters. The scope of access and types of applications installed in different environments vary tremendously and hence it is best to keep them as separate clusters.</p><p><strong>Composition and function</strong> — Sometimes, a cluster is created with a specific composition to perform a particular function. For instance, ML workflows using <a href="http://kubeflow.org/">kubeflow</a> or data analytics jobs may require nodes with GPUs or other specific hardware requirements for instance clusters made up of <a href="https://cloud.google.com/preemptible-vms">preemptible VMs</a> for the purpose of batch analytics workloads.. These hardware requirements might not apply to other Services. These workflows may not be crucial to running the business and may require ephemeral clusters (clusters created for short periods of time to perform a specific task before shutting them down). Likewise, there may be shared services such as observability (logging, metrics, traces) and CI/CD tooling that are better suited in their own “platform_admin” cluster. Separate function-specific clusters are often seen for non business critical workflows.</p><p><strong>Resiliency</strong> — Multiple clusters are often used to increase resiliency in an environment. Each cluster has a blast radius. In this context, a blast radius is the number of Services that are adversely affected due to a cluster malfunction, misconfiguration or simply cluster going offline (planned or unplanned maintenance). Having a large number of smaller-sized clusters means having a large number of smaller-sized blast radii. If a Service is deployed on two clusters sharing the load equally, and one cluster were to go offline, that would affect 50% of the users (assuming no changes are made to the load balancer splitting the traffic). In contrast, assume a Service is being served out of a single cluster. Any event on that cluster would cause 100% outage for that Service. Along the same lines, multiple clusters are also often used for disaster recovery.</p><p>For the remainder of this series, I’ll focus on the resiliency aspect of multi-cluster which is a key component in multi cluster GKE life cycle management.</p><blockquote>Up next… <a href="https://medium.com/@ameericus/gke-multi-cluster-life-cycle-management-series-b0d39080ac6b">Part II: Multi-cluster and Distributed Services</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17ff892b85c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-engine-gke-multi-cluster-life-cycle-management-series-17ff892b85c8">Kubernetes Engine (GKE) multi-cluster life cycle management series</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>