<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gokul Chandra on Medium]]></title>
        <description><![CDATA[Stories by Gokul Chandra on Medium]]></description>
        <link>https://medium.com/@gokulchandrapr?source=rss-e57e2ec3b649------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3mNJ6-tl16PFpPTwJP-e8Q.jpeg</url>
            <title>Stories by Gokul Chandra on Medium</title>
            <link>https://medium.com/@gokulchandrapr?source=rss-e57e2ec3b649------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:09:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gokulchandrapr/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Amazon EKS Anywhere & EKS Connector]]></title>
            <link>https://gokulchandrapr.medium.com/amazon-eks-anywhere-eks-connector-600953aaa42d?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/600953aaa42d</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ek]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Thu, 16 Sep 2021 23:28:35 GMT</pubDate>
            <atom:updated>2021-09-22T19:54:45.555Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oG9VwKMCCt7-z6_7bA_0Wg.png" /><figcaption>Create and Manage Kubernetes Clusters on Premises and Connect any Kubernetes Cluster to AWS</figcaption></figure><p>Amazon EKS Anywhere is a new deployment option for Amazon EKS that enables users to easily create and operate Kubernetes clusters on-premises. EKS Anywhere uses EKS Distro, a Kubernetes distribution based on and used by Amazon Elastic Kubernetes Service which was announced back in 2020.</p><p>EKS Anywhere provides an installable software package for creating and operating Kubernetes clusters on-premises and automation tooling for cluster lifecycle support. EKS provides several deployment options for Kubernetes clusters managed and hosted by AWS — EKS on AWS, EKS on AWS Wavelength (infrastructure offering optimized for mobile edge computing applications), EKS on AWS Local Zones (extension of an AWS Region in geographic proximity to the end-users).</p><p>EKS on AWS Outposts was one of the options where users were able to run Kubernetes in their own data centers but on AWS infrastructure. With EKS Anywhere users can deploy Kubernetes on their own infrastructure and is managed by the customer with a consistent AWS management experience in their data center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DVtimPeDWfgFg6hs" /><figcaption>AWS EKS Deployment Options</figcaption></figure><p>EKS Anywhere can create and manage Kubernetes clusters on multiple providers. Currently, creating development clusters locally with Docker (*only for testing) and production clusters using VMware vSphere are supported. As per the documentation, other deployment targets will be added in the future, including bare metal support in 2022 (users can deploy EKS Anywhere on any supported OS).</p><p>EKS Anywhere clusters are created and managed by the eksctl CLI, with the <a href="https://anywhere.eks.amazonaws.com/docs/reference/eksctl/">EKS Anywhere</a> plugin added. While a cluster is running, most EKS Anywhere administration can be done using kubectl or other native Kubernetes tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uRDxSPYfgcecOj0p" /><figcaption>eksctl — anywhere extension</figcaption></figure><p>EKS Anywhere provider, VMware vSphere, is implemented based on the <a href="https://github.com/kubernetes-sigs/cluster-api-provider-vsphere">Kubernetes Cluster API Provider vsphere</a> (CAPV) specifications.</p><h3>Cluster Creation — Stages</h3><p>Cluster creation involves two steps, generating a configuration file template using eksctl anywhere generate command, adding required information to the template followed by eks anywhere create cluster command which involves a sequence of steps starting with validating the vSphere assets to creating a workload cluster. Users can run these commands from macOS or Linux host with docker installed on it, this is termed as an admin machine.</p><p>The cluster creation starts with authenticating to vSphere and validating the assets there and creates a temporary bootstrap cluster to direct the workload cluster creation. Once the workload cluster is created, the cluster management resources are moved to the workload cluster and the local bootstrap cluster is deleted.</p><p>With a fully functional workload cluster, users can use the eksa-controller and specific CRD’s provisioned on it to manage the cluster itself.</p><p>Steps involved in a successful cluster creation process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*grCBoCIBIzfh-rYO" /><figcaption>EKS Anywhere Cluster Creation Steps</figcaption></figure><p>A bootstrap cluster is created using <a href="https://kind.sigs.k8s.io/docs/user/quick-start/">kind</a> command to build a single-node Kubernetes bootstrap cluster on the admin machine. This includes pulling the kind node image, preparing the node, writing the configuration, starting the control-plane, installing CNI, and installing the StorageClass on the kind (Kubernetes in Docker) based single node cluster.</p><p>The sequence of steps from creating a configuration file template to a functional bootstrap cluster on the admin machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e6K83HLZH0Uk4jal" /><figcaption>EKS Anywhere — Bootstrap</figcaption></figure><p>Once the temporary bootstrap cluster is functional on the admin machine, it starts creating a workload cluster (creates virtual machines for control-plane, etcd, and nodes) using vSphere provider and EKS-D as Kubernetes distro. Once the infrastructure provisioning is done addition of CNI (Cilium is the supported CNI today), addition of storage (vSphere CSI) are performed on the workload cluster. The bootstrap cluster then moves the CAPI objects over to the workload cluster, so it can take over the management of itself followed by deletion of the bootstrap cluster on the admin machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SifwXRbd0fTxVV3C" /><figcaption>EKS Anywhere — Bootstrap</figcaption></figure><p>At this point, the workload cluster is ready to use, both to run workloads and to accept requests to change, update, or upgrade the cluster itself. Users can continue to use eksctl anywhere to manage the cluster, with EKS Anywhere handling the fact that CAPI management is now being fulfilled from the workload cluster instead of the bootstrap cluster.</p><h3>Configuration</h3><p>Each EKS Anywhere cluster is built from a cluster specification file, with the structure of the configuration file based on the target provider for the cluster. Currently, VMware vSphere is the recommended provider for supported EKS Anywhere clusters in production. EKS Anywhere uses EKS-D (EKS Distro) for bootstrapping the Kubernetes cluster.</p><p>Similar to CAPV and CAPI specifications there involves a global cluster configuration with references to machine configs. With EKS Anywhere, this configuration is condensed and made simple so that users can provide a base configuration and EKS Anywhere will translate the same to the required specs.</p><p>General CAPV — CAPI objects and relation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0IBecyj2LWVy6lst" /><figcaption>CAPV and CAPI Resource Relation and ClusterCtl</figcaption></figure><p>In EKS Anywhere, instead of using CAPI directly with the clusterctl command to manage the workload cluster, users can use the eksctl anywhere command which abstracts that process, including calling clusterctl under the covers.</p><p>Cluster <a href="https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/vsphere/">configuration</a> includes — CNI parameters (pod subnet, service subnet CIDR’s), controlPlaneConfiguration (number of control-plane nodes, endpoint- VIP HA) and reference to machine group for control-plane machine configuration, DatacenterConfig which includes the vSphere URL, network in which the machines should be deployed, thumbprint for self-signed certs, etc., Kubernetes version specification and workerNodeGroupConfiguration (number of Kubernetes nodes) and reference to machine group for worker-node configuration.</p><p>There are two types of etcd topologies supported for configuring a cluster: Stacked (etcd members and control plane components are colocated (run on the same node/machines) or Unstacked/External (unstacked or external etcd topology, etcd members have dedicated machines and are not colocated with control plane components.</p><p>There are other optional configurations: <a href="https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/oidc/">OIDC</a>, <a href="https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/etcd/">etcd</a>, <a href="https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/proxy/">proxy</a>, <a href="https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/gitops/">gitops</a> which can be included.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*V9UQnJEYICwyMLWN" /><figcaption>EKS Anywhere — Configuration</figcaption></figure><p>Each VSphereMachineConfig refers to the host configuration of control plane, worker nodes and etcd hosts. Apart from the resource parameters (cpu, memory, disk) other configuration sections include — Operating system on virtual machines (permitted values: ubuntu, bottlerocket (default: bottlerocket)), vSphere resource pools for the VMs, folder to group the machines and SSH public keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eyXnlp-Icp_LEj8o" /><figcaption>EKS Anywhere — Configuration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8yfv6hbOsBOTwmvX" /><figcaption>EKS Anywhere — Configuration</figcaption></figure><h3>Operating System — OVA’s and Management</h3><p>EKS Anywhere today supports two operating system families: Ubuntu and Bottlerocket (default), users can use the released OVA’s from <a href="https://anywhere.eks.amazonaws.com/docs/reference/artifacts/">artifacts</a> or <a href="http://eks%20anywhere%20today%20supports%20two%20operating%20system%20families:%20Ubuntu%20and%20Bottlerocket%20(default)/">build</a> them with a custom base image.</p><p>To specify an OVF template, users can import OVA files into vSphere before it can be used in EKS Anywhere cluster. Users can import and deploy the OVA files using the console or Govc-CLI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ekwUVUzZi9C4wrW7" /><figcaption>OVA — Tagging</figcaption></figure><p>The installer identifies which OVA to use for installation based on the Kubernetes specification in the cluster configuration and two tags on the template: OS (os:bottlerocket, os:ubuntu) and <a href="https://anywhere.eks.amazonaws.com/docs/reference/artifacts/">eksdRelease</a> (eksdRelease:kubernetes-1–20-eks-*).</p><p>Users can use the vSphere console or Govc CLI (govc tags.category.create -t VirtualMachine os) to create tag categories and associate them with the templates (govc tags.attach os:ubuntu &lt;Template Path&gt;).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g2rp7aJoxgCYWfsW" /><figcaption>OVA — Tagging</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yCXAxw1g1P1qYv1H" /><figcaption>OVA — Tagging</figcaption></figure><p>Sample bottlerocket template with associated tags:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TDsjMHkgBcvqHeGL" /><figcaption>OVA — Tags</figcaption></figure><p>If a template is not defined in the cluster spec file, EKS Anywhere will use the proper default one for the Kubernetes minor version and OS family you specified in the spec file. If the template doesn’t exist in vSphere environment, an appropriate OVA will be imported into vSphere and the necessary tags are added.</p><h3>Cluster Creation Process — In Detail</h3><p>Like Cluster API, EKS Anywhere runs a kind cluster on the local Administrative machine to act as a bootstrap cluster. A CLI utility container is deployed on the admin machine which installs a single node kind cluster and performs other actions such as installing CNI, StorageClass, etc. on the bootstrap cluster. The bootstrap cluster also uses EKS-D for Kubernetes (all images from EKS-D).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XR9kC8SAXpdiMeOR" /><figcaption>EKS Anywhere — Bootstrap Cluster — Creation</figcaption></figure><p>The bootstrap cluster includes all CAPI and CAPV components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ixRl7wCWlxMpD8fC" /><figcaption>EKS Anywhere — Bootstrap Cluster — Pods</figcaption></figure><p>CRD’s (custom resource definitions) on the bootstrap cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5RyLH41wKqU6FfgS" /><figcaption>EKS Anywhere — Bootstrap Cluster — CRD’s</figcaption></figure><p>EKS Anywhere formulates the cluster configuration and configures all the CAPI &amp; CAPV objects on the bootstrap cluster. For example, the output of vsphereclusters, clusters, etcdadmclusters, machines, vspheremachines and vspherevms are as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wDz68wnG3ATgsmy7" /><figcaption>EKS Anywhere — Bootstrap Cluster — CAPI &amp; CAPV Custom Objects</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AhbG_6SpeQV7uywX" /><figcaption>EKS Anywhere — Bootstrap Cluster — CAPI &amp; CAPV Custom Objects</figcaption></figure><p>With all the configuration above and using vSphere provider, all the virtual machines required to host control plane, etcd and worker nodes are created on the target vSphere environment. The Kubeadm bootstrap provider uses EKS distro to bootstrap the Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hnUW1NlD86qesslA" /><figcaption>vSphere — Creation of Virtual Machines</figcaption></figure><p>All actions such as configuring the network, storage and other aspects of the virtual machine are handled by the automation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uff8a5cC8JcECIAs" /><figcaption>vSphere — Creation of Virtual Machines</figcaption></figure><p>Once the cluster creation is complete and followed by few validations, users can access the workload cluster using the kubeconfig written to a folder named after the cluster name on the admin machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bb1xMQXs3KKJ-AH4" /><figcaption>EKS Anywhere — Workload Cluster</figcaption></figure><p>The bootstrap cluster will push all the CAPI &amp; CAPV components to the workload cluster, with this users can use the CRD’s and eksa-controller running in the workload cluster to manage the cluster itself.</p><h3>GitOps — Cluster Management and Deploying Applications</h3><p>GitOps can be enabled during cluster creation, EKS Anywhere will automatically commit the cluster configuration to the provided GitHub repository and install a GitOps toolkit (Flux) on the cluster which watches that committed configuration file.</p><p>Users can then manage the cluster by making changes to the version-controlled cluster configuration file and committing the changes. Once a change is detected by the GitOps controller running in the cluster, the scale of the cluster will be adjusted to match the committed configuration file.</p><p>Flux components deployed in flux-system namespace, the flux component configuration files are also committed to the repository and can be managed using GitOps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_fI7dthRFusTXPH7" /><figcaption>EKS Anywhere — GitOps — Flux</figcaption></figure><p>Once the cluster creation is complete, the git repository is cloned to the cluster folder on the admin machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qqyARabiJeS9ib14" /><figcaption>EKS Anywhere — GitOps</figcaption></figure><p>Cluster and flux-system configuration committed to git repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xEGZJXFBovtsEQBF" /><figcaption>EKS Anywhere — GitOps</figcaption></figure><p>Version-controlled cluster configuration file committed to git repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i5j9bmMthwgdEfrh" /><figcaption>EKS Anywhere — GitOps</figcaption></figure><p>Flux component configuration files committed to git repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HTp8h3gh1wEK_q9T" /><figcaption>EKS Anywhere — GitOps</figcaption></figure><p>Users can deploy applications using the same repository having multiple folders with Kubernetes manifests and a Kustomization file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bG7tUgpXbYW3GMob" /><figcaption>EKS Anywhere — GitOps</figcaption></figure><h3>EKS Connector</h3><p>Amazon Elastic Kubernetes Service (Amazon EKS) now allows users to connect any conformant Kubernetes cluster to AWS and visualize it in the Amazon EKS console. EKS Connector is a new capability that allows users to connect any Kubernetes clusters to the EKS console (not limited to EKS Anywhere) securely, eks-connector-agent installed on the connected cluster communicates to AWS through a secure data channel using Session Manager.</p><p>The cluster can be registered in multiple ways, using AWS CLI, SDK, eksctl, or the console. When registering a cluster through eksctl or the console, a manifest file is auto-populated, whereas additional manual steps are required when using the AWS CLI and SDK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PcvGHxW1Aovjnj12" /><figcaption>EKS Connector — Cluster Registration</figcaption></figure><p>Regardless of where the cluster is running, users can use the Amazon EKS console to view all connected clusters and the Kubernetes resources running on them.</p><p>EKS console now includes a register option along with create cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_676-C0AHeQeME6T" /><figcaption>EKS Connector — Cluster Registration</figcaption></figure><p>The cluster registration process involves two steps: registering the cluster with Amazon EKS and applying a connector YAML manifest file in the target cluster to enable connectivity. The cluster configuration requires a unique name for the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nvt_sX-sz986-KLj" /><figcaption>EKS Connector — Cluster Registration</figcaption></figure><p>Users can choose a specific Kubernetes provider from the list of available providers or choose ‘Other’ for a generalized configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f33Vze8yFLYXE7Bf" /><figcaption>EKS Connector — Cluster Registration</figcaption></figure><p>An eks-connector-agent role with permissions to access systems manager — ssmmessages is required which enables the connector-agent to access ssmmessages. This IAM role is used by the EKS Connector agent on the Kubernetes cluster to connect to the Systems Manager service on AWS. This role can be scoped down further to include only the EKS cluster name if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XHWnPLVQO5PjcvPp" /><figcaption>EKS Connector — IAM</figcaption></figure><p>Amazon EKS uses the service-linked role named <a href="https://docs.aws.amazon.com/eks/latest/userguide/using-service-linked-roles-eks-connector.html">AWSServiceRoleForAmazonEKSConnector</a>, this role allows Amazon EKS to connect Kubernetes clusters.</p><p>Connector ServiceLinkedRole allows Amazon EKS to call other services to set up resources required for connecting a cluster to Amazon EKS. This includes managing activation in Systems Manager for the EKS Connector agent and for receiving events from EventBridge whenever the EKS Connector agent is registered and deregistered with the Systems Manager service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H-lT04zNRUmH8suI" /><figcaption>EKS Connector — IAM</figcaption></figure><p>Users can download the YAML file using the console instruction or get the same <a href="https://amazon-eks.s3.us-west-2.amazonaws.com/eks-connector/manifests/eks-connector/latest/eks-connector.yaml">here</a> when using the CLI. This manifest contains the configurations for the EKS Connector and a proxy agent which are deployed as a StatefulSet on the target Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DilhEBawPj8Tff2u" /><figcaption>EKS Connector — YAML</figcaption></figure><p>When the manifest is applied to a Kubernetes cluster, the EKS Connector agent connects to the Systems Manager service, which sends notifications to EKS through Amazon EventBridge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*NuLowCXVjjtjgOJB" /><figcaption>EKS Connector — Sequence of Registration Steps</figcaption></figure><p>EKS Connector agent registered and deregistered with the Systems Manager service event on EventBridge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2D0AHO0LtViXw90w" /><figcaption>EKS Connector — Event Bridge</figcaption></figure><p>EKS Connector acts as a proxy and forwards the EKS console requests to the Kubernetes API server on the connected cluster, so there is a need to associate the connector’s service account with an EKS Connector Role, which gives permission to impersonate AWS IAM entities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OO6K0WlznWPxRh-x" /><figcaption>EKS Connector — IAM</figcaption></figure><p>Here users can control full access (all resources) or restricted access (specific namespace or resources) by replacing references of %IAM_ARN% with the Amazon Resource Name (ARN) of a specific IAM user or role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kq_PDu0J4WFUBcxW" /><figcaption>EKS Connector — RBAC — Kubernetes</figcaption></figure><p>With the above Service Account created on the target cluster user can visualize the workloads from EKS console. AWS Systems manager invokes a proxy to the agent container (ssm-agent) and the proxy container using the service account impersonates the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fznUaJXPhF18Y6Kx" /><figcaption>EKS Connector — View Request from EKS Console to Kubernetes Cluster</figcaption></figure><p>The ‘Overview’ section shows all the nodes in the cluster. As of now, all the objects are read-only and the user cannot edit or delete an object in the registered cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LVSgUNf_beH9k0IV" /><figcaption>EKS Console — Cluster Visualization</figcaption></figure><p>The ‘Workloads’ section displays all objects of Type: Deployment, DaemonSet and StatefulSet. Users can select these objects to select a pod-level overview. As of today, other objects of the cluster (like services, ingress, secrets, etc.) are not available for visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gDEaauvyYh83rnEV" /><figcaption>EKS Console — Cluster Visualization</figcaption></figure><p>The eks-connector is deployed as a StatefulSet and consists of two containers — amazon-ssm-agent (connector-agent) and eks-connector (connector-proxy).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2YL8_2RCr7aqwpXZ" /><figcaption>EKS Console — Cluster Visualization</figcaption></figure><p>eks-connector on a connected cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nu3MZOtC7NUjZDui" /><figcaption>EKS Connector on Kubernetes Cluster</figcaption></figure><p>The EKS Connector agent enables connectivity to AWS, the proxy agent interacts with Kubernetes to serve AWS requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m-LGcBNxKE47XULW" /><figcaption>EKS Connector on Kubernetes Cluster — SSM</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L4XLAoTNKNg5CkaT" /><figcaption>EKS Connector on Kubernetes Cluster</figcaption></figure><p>connector-agent container — eks-connector pod:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-A9NTqg4EfiRzrrk" /><figcaption>EKS Connector on Kubernetes Cluster — connector-agent</figcaption></figure><p>connector-proxy container — eks-connector pod:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KCemtgFwBtn91fIf" /><figcaption>EKS Connector on Kubernetes Cluster — connector-proxy</figcaption></figure><p>Amazon EKS leverages AWS Systems Manager’s agent to connect to AWS services. SSM configuration (registration key, fingerprint, instance-id, etc.) stored as secret in eks-connector namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OOG6J99BCfLGr7-G" /><figcaption>EKS Connector on Kubernetes Cluster — SSM Secret</figcaption></figure><p>Systems Manager uses ssmmessages endpoint to make calls from SSM Agent (agent-container) to Session Manager. The connector configuration file includes the ssm agent parameters and logging configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QL2WbjIoxD9oJh2X" /><figcaption>EKS Connector — Cluster Configuration</figcaption></figure><p>AWS Systems Manager hybrid activations created by eks-connector-agent, this provides secure access to the AWS Systems Manager service from the target cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f_98OgueyJ2d1EwE" /><figcaption>AWS Systems Manager — Hybrid Activations</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tGYn77t2toEpXNrj" /><figcaption>AWS Systems Manager — Hybrid Activations</figcaption></figure><p>The service-linked-role (service-linked roles are predefined by the service and include all the permissions that the service requires to call other AWS services on the user’s behalf) is used to create sessions and command executions on the ssm-agent on the target cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qgcDxMq5rs0yl0FR" /><figcaption>EKS Connector — Service Linked Role</figcaption></figure><p>Sessions in Session Manager:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8uB-yw-tIHfUduaC" /><figcaption>AWS Sessions Manager</figcaption></figure><h3>Local Cluster for Testing</h3><p>EKS Anywhere supports a Docker provider for development and testing use cases only. This allows users to try EKS Anywhere on a local system before deploying to a supported provider. This is implemented using CAPD (Cluster API Provider Docker) which uses docker installed on the host to bootsrap multi-node Kind (Kubernetes in Docker) cluster with EKS Distro.</p><p>Similar to vSphere users can use eksctl generate command with provider flag set to docker to create a config file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uf9IyvRzikZ9PkvP" /><figcaption>EKS Anywhere — Cluster Configuration — Docker Provider</figcaption></figure><p>Here, based on the capacity of the underlying host users can edit the node count and can even create a HA Kind cluster.</p><p>Similar to the other provider a temporary bootstrap cluster is created first which then creates the docker based workload cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ixwPzzOOc8yQoJgA" /><figcaption>EKS Anywhere — Docker</figcaption></figure><p>The bootstrap cluster now includes all the components and CRD’s of CAPI and CAPD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p-raPxDK6s7hEe-N" /><figcaption>EKS Anywhere — Docker</figcaption></figure><p>Once the workload cluster is created the bootstrap Kind cluster is deleted from the host enabling users to operate the workload cluster using the kubeconfig file written to the host.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zGGk5mrnQ9DTghYw" /><figcaption>EKS Anywhere — Docker</figcaption></figure><h3>Cluster Integrations and Partners</h3><p>EKS Anywhere offers AWS support for certain third-party vendor components, namely Ubuntu TLS, Cilium, and Flux. It also provides flexibility for users to integrate with their choice of tools in other areas as they do today with any Kubernetes environment.</p><p>Some of the tools in the list are as below, these are not covered by the EKS Anywhere support subscription but users can use the documentation to get an idea on how to implement these on EKS Anywhere clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8OLgEFmP5srwgyWs" /><figcaption>EKS Anywhere — Integrations</figcaption></figure><p><a href="https://aws.amazon.com/eks/eks-anywhere/partners/">Partner integrations</a> include various products under different categories like Monitoring &amp; Logging, Cost Management, DevOps, Security &amp; Networking, etc.</p><p>EKS Anywhere and EKS Connector are part of a clear play for businesses embracing hybrid cloud and private infrastructure setups. Apart from hybrid cloud, these play a phenomenal role in building a concrete telco cloud portfolio for 5G enablement and edge computing. Both EKS Anywhere and EKS Connector are still new (initial releases) and based on the documentation, users can see many more capabilities added in future releases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=600953aaa42d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go Crypto and Kubernetes — FIPS 140–2 & FedRAMP Compliance]]></title>
            <link>https://gokulchandrapr.medium.com/go-crypto-and-kubernetes-fips-140-2-fedramp-compliance-66d852ccccd2?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/66d852ccccd2</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Tue, 13 Apr 2021 19:16:32 GMT</pubDate>
            <atom:updated>2021-04-13T19:31:13.727Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PtgtGRtJyYeDdfNvsQiGEw.png" /><figcaption>Go Cryptography — FIPS 140–2, CAVP and CMVP</figcaption></figure><h3>Go Crypto and Kubernetes — FIPS 140–2 and FedRAMP Compliance</h3><p>Encryption is key to keeping sensitive data protected. However, there are a number of algorithms available as well as varying capabilities for providing encryption. As a result, it is inevitably challenging to know which algorithm and security standard to use. To establish standards vendors can rely on, the U.S. government and its National Institute of Standards (NIST) have established FIPS 140–2. FIPS stands for Federal Information Processing Standard, and the FIPS-140 series is a collection of computer security standards set by the National Institute of Standards &amp; Technology (NIST) for the United States government. FIPS 140–2 defines the critical security parameters vendors must use for encryption before selling into the U.S government.</p><p>If a vendor wants to supply cloud-based services to the US Federal Government, then they have to get <a href="https://www.fedramp.gov/">FedRAMP</a> approval. This certification process covers a whole host of security issues, but is very specific about its requirements on cryptography: usage of FIPS 140–2 validated modules wherever cryptography is needed, these encryption standards protect the cryptographic module from being cracked, altered, or otherwise tampered with. FIPS 140–2 validated encryption is a prerequisite for FedRAMP.</p><p>Most of the public cloud vendors already have GovCloud (<a href="https://aws.amazon.com/govcloud-us/?whats-new-ess.sort-by=item.additionalFields.postDateTime&amp;whats-new-ess.sort-order=desc">AWS</a>, <a href="https://azure.microsoft.com/en-us/global-infrastructure/government/">Azure</a>, <a href="https://www.oracle.com/industries/public-sector/government-cloud/">Oracle</a>, <a href="https://cloud.google.com/solutions/government">GCP</a>, etc.) which are isolated from general cloud setting and designed to allow customers and the U.S government agencies to move their confidential data into the cloud to address their compliance and specific regulatory requirements (ITAR regulations, the FedRAMP requirements, Defense Federal Acquisition Regulation Supplement (DFARS), etc.).</p><p>Managed Kubernetes service like Amazon Elastic Kubernetes Service (EKS) recently <a href="https://aws.amazon.com/blogs/publicsector/announcing-amazon-elastic-kubernetes-service-eks-aws-govcloud-us/">launched in AWS GovCloud (US)</a>, Azure Kubernetes Service (AKS) <a href="https://azure.microsoft.com/en-us/updates/azure-kubernetes-service-is-now-available-in-azure-government/">now available in Azure Government </a>enabling public sector customers with FedRAMP, DoD CC SRG, ITAR, etc to run fully-managed Kubernetes clusters. AWS and other providers facilitate FIPS endpoints for all supported services in AWS GovCloud (US) which operate using FIPS 140–2 validated cryptographic modules.</p><p>Several operating system vendors have obtained FIPS 140–2 Level 1 validation for the OpenSSL Cryptographic Module shipped with their respective operating systems: <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/Certificate/3016">Red Hat, Inc.: Red Hat Enterprise Linux OpenSSL Cryptographic Module</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/Certificate/2888">Canonical Ltd.: Ubuntu OpenSSL Cryptographic Module</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search?SearchMode=Basic&amp;ModuleName=amazon&amp;CertificateStatus=Active&amp;ValidationYear=0">Amazon Linux: Amazon Linux Cryptographic Modules</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/Certificate/3017">Oracle Corporation: Oracle Linux OpenSSL Cryptographic Module</a>. Even if the vendor supports using a Linux OS with FIPS enabled crypto, the components that make up the Kubernetes platform should be able to take advantage of the FIPS validated crypto without bypassing them or the whole distribution should be compiled with a custom compiler that uses FIPS validated cryptography.</p><h3>Go Cryptography and Kubernetes — FIPS 140–2</h3><p><a href="https://pkg.go.dev/k8s.io/kubernetes">Kubernetes</a> is a Go project, as are most of the Kubernetes subcomponents and ecosystem. Golang has a crypto standard library, <a href="https://golang.org/pkg/crypto/">Golang Crypto</a> which fulfills almost all the application crypto needs (TLS stack implementation for HTTPS servers and clients all the way to HMAC or any other primitive that are needed to make signatures to verify hashes, encrypt messages.). Go has made a different choice compared to most languages, which usually come with links or wrappers for OpenSSL or simply don’t provide any cryptography in the standard library (Rust doesn’t have standard library cryptography, JavaScript only has web crypto, Python doesn’t come with a crypto standard library).</p><p>In a Kubernetes cluster apart from user to kube-apiserver interactions, the core components and other components of the stack such as etcd, contained, etc. work collectively and need to communicate with each other. In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, where TLS plays a significant role. The TLS protocol supports the use of numerous cryptographic protocols. Knowing how to use and or configure these protocols can be difficult if not near impossible to understand, but this knowledge is key to providing the level of security required to thwart attacks.</p><p>The native go crypto is <a href="https://github.com/golang/go/issues/21734">not</a> FIPS compliant and there are few open <a href="https://github.com/golang/go/issues/33281">proposals</a> to facilitate Go code to meet FIPS requirements. Users can use prominent go compilers/toolsets backed by FIPS validated SSL libraries provided by Google or Redhat which enables Go to bypass the standard library cryptographic routines and instead call into a FIPS 140–2 validated cryptographic library. These toolsets are available as container images, where users can use the same to compile any Go based applications.</p><h3>BoringCrypto</h3><p><a href="https://go.googlesource.com/go/+/dev.boringcrypto/README.boringcrypto.md">BoringCrypto</a> (<a href="https://boringssl.googlesource.com/boringssl/">BoringSSL</a> based crypto) maintained by Google is an open-source, general-purpose cryptographic library that provides <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/2964">FIPS 140–2 approved cryptographic algorithms</a> to serve BoringSSL and other user-space applications.</p><p><a href="https://boringssl.googlesource.com/boringssl/">BoringSSL</a> is Google’s forked version of OpenSSL cryptographic library and BoringSSL is used in all Google website product’s TLS stacks since June 2014 — including Google Android OS and Google Chrome. Google initially forked the OpenSSL code with BoringSSL, but have now formally released Google <a href="https://github.com/google/tink">Tink</a>.</p><p>Upstream <a href="https://github.com/golang/go">golang/go</a> repository maintains a separate dev branch <a href="https://github.com/golang/go/branches/all?query=dev.boringcrypto">dev.boringcrypto.go&lt;go-version&gt;</a>. This branch holds all the patches to make Go using BoringCrypto and the community has specified that there is no intention to merge this branch to main.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AysHtJyDL_uriIwE" /><figcaption>BoringCrypto — Go Dev Branch</figcaption></figure><p>Go+BoringCrypto release has the form <em>&lt;GoVersion&gt;b&lt;BoringCryptoVersion&gt;</em>, where <em>&lt;GoVersion&gt;</em> is the Go version the release is based on, and <em>&lt;BoringCryptoVersion&gt;</em> is an integer that increments each time there is a new release with different BoringCrypto bits. Google maintains <a href="https://github.com/golang/go/blob/dev.boringcrypto.go1.12/misc/boring/README.md">docker images with patched go toolchain</a>, similar to golang:x.y.z.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O2fNxa3VwkSTUbIP" /><figcaption>BoringCrypto — Go Version Parity</figcaption></figure><p><a href="https://landscape.cncf.io/?selected=rke-government">RKE2</a> a open source CNCF project, also known as RKE Government uses the <a href="https://hub.docker.com/r/goboring/golang">goboring/golang</a> which replaces the standard Go crypto libraries with the FIPS validated BoringCrypto module to statistically <a href="https://github.com/rancher/rke2/blob/master/BUILDING.md">compile</a> most of the Kubernetes, etcd and other runc components ensuring all levels of stack from Kubernetes daemons, database to container orchestration mechanics are FIPS 140–2 compliant. RKE2 also <a href="https://docs.rke2.io/assets/fips_engagement.pdf">revalidated</a> this module as Rancher Kubernetes Cryptographic Library for the additional platforms and systems supported by RKE2.</p><h3>Redhat’s go-toolset</h3><p><a href="https://developers.redhat.com/blog/2019/06/24/go-and-fips-140-2-on-red-hat-enterprise-linux/">Go Toolset</a> is a Red Hat offering for developers on the Red Hat Enterprise Linux platform. Go Toolset requires a valid subscription (paid) and is distributed as a collection of RPM packages (yum module install go-toolset).</p><p>When a RHEL system is booted in <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/chap-federal_standards_and_regulations">FIPS mode</a>, Go will instead call into OpenSSL via a new package that bridges between Go and OpenSSL. This also can be manually enabled by setting GOLANG_FIPS=1. The Go Toolset is available as a <a href="https://access.redhat.com/documentation/en-us/red_hat_developer_tools/2019.3/html/using_go_toolset/chap-image">container image</a> that can be downloaded from Red Hat Container Registry. Red Hat mentions that this as a new feature built on top of existing upstream work (BoringSSL).</p><p>The latest versions of <a href="https://www.openshift.com/blog/introducing-red-hat-openshift-4-3-to-enhance-kubernetes-security">Red Hat Openshift use this go-toolset,</a> when OpenShift runs on Red Hat Enterprise Linux booted in FIPS mode, OpenShift calls into the Red Hat Enterprise Linux FIPS validated cryptographic libraries.</p><h3>Differences — BoringCrypto | Redhat’s go-toolset</h3><p>As of now <a href="https://github.com/golang/go/branches/all?query=dev.boringcrypto">dev.boringcrypto</a> is being kept up-to-date against master and actively maintained as a dev branch part of the official Go repository by the community and there is no guarantee of support in the future. Red Hat offers support contracts and <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sect-security_guide-federal_standards_and_regulations-federal_information_processing_standard">FIPS mode for RHEL</a> and recent releases of Openshift use the toolset to run in FIPS mode, so they are committed to supporting their FIPS complaint go- toolset, this comes with a subscription cost. With Red Hat, FIPS mode can be enabled at runtime.</p><p>If a user needs to pass a FIPS audit or validation they have to show FIPS compliance in both the OS and Kubernetes level, so even with using BoringCrypto for Kubernetes components, the underlying OS should be running in FIPS mode.</p><h3>Traditional Go Crypto vs FIPS Validated Go Crypto</h3><p>To be FIPS 140–2 compliant, the module must use FIPS 140–2 complaint algorithms, ciphers, key establishment methods, and other protection profiles. FIPS-approved algorithms do change at times; not extremely frequently, but more often than they come out with a new version of FIPS 140. Users can use the latest publications to gather information on what is supported: Documents — Annexes <a href="https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/fips1402annexa.pdf">A</a> (approved security functions), <a href="https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/fips1402annexb.pdf">B</a> (approved protection profiles), <a href="https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/fips1402annexc.pdf">C</a> (approved random bit generators), and <a href="https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/fips1402annexd.pdf">D</a> (approved key establishment methods) provide a list of the approved functions applicable to FIPS 140–2.</p><p>Some of the fundamental requirements (not limited to) are as follows:</p><ul><li>Support for TLS 1.0 and TLS 1.1 is now deprecated (only allowed in certain cases). TLS 1.3 is the preferred option, while TLS 1.2 is only tolerated.</li><li>3DES is not tolerated anymore.</li><li>DSA/RSA/ECDSA are only approved for key generation/signature.</li><li>Allowed Hash Standards: SHA-1, SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224, SHA-512/256</li><li>Allowed Message Authentication: CCM, GCM, GMAC, CMAC, HMAC</li><li>Cipher suites with CBC+HMAC must use the Encrypt-then-MAC extension.</li><li>More TLS extensions should be avoided, like max_fragment_length, ec_point_formats and early_data.</li><li>Session resumption should use key exchange.</li><li>The 0-RTT option in TLS 1.3 should be avoided.</li><li>FIPS 140–2 compliant ciphers: <a href="https://www.ibm.com/docs/en/ibm-http-server/9.0.5?topic=options-ssl-cipher-specifications">cipher-specifications</a></li></ul><p>Using a sample Go based server application, to access the compatibility/compliance from the information above. Below, the same application is built/compiled using traditional upstream Go and the other is using goboring which replaces the standard/native Go crypto with FIPS 140–2 compliant boringcrypto, the patch here is a drop-in replacement, where no changes to the code are needed if required users can make changes/configure to filter specific ciphers, etc.</p><p>The app below is built using traditional/official <a href="https://hub.docker.com/layers/golang/library/golang/1.16/images/sha256-cc7c2f66a969d0905e8f2b66060224a0139d5688c51924796586a7975b843f92?context=explore">golang:1.16.2</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*GoVhks3OFxEmnQOA" /><figcaption>Test Application — Go 1.16</figcaption></figure><p>Testing the above application using <a href="https://testssl.sh/">testssl.sh</a> which tests a service on the given port for the support of TLS/SSL ciphers, protocols and recent cryptographic flaws.</p><p>As shown below upstream Go crypto:</p><ul><li>Allows unsupported/deprecated TLS 1 and TLS 1.1. TLS 1.0 is prohibited and TLS 1.1 is deprecated for government sites and not supported. Recently <a href="https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-fips-endpoints/">AWS removed support for 1.0 and 1.1 and enabled 1.2 for all AWS FIPS Endpoints</a>.</li><li>Allows Triple DES (a TLS 1.2 standard) which has a really “small” blocksize with only 64 bits, which led to attack such as <a href="https://sweet32.info/">Sweet32</a> against TLS sessions.</li><li>Allows usage of obsoleted CBC ciphers which are proven to be vulnerable, these can expose the TLS connection to <a href="https://en.wikipedia.org/wiki/Padding_oracle_attack#Padding_oracle_attack_on_CBC_encryption">padding oracle attacks</a>/<a href="https://en.wikipedia.org/wiki/Lucky_Thirteen_attack">lucky thirteen attack</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YyGwo0kFHj2oYcCX" /><figcaption>TLS Scan — Test Application — Go 1.16</figcaption></figure><p>As shown below Go crypto allows unsupported TLS v1 and v1.1 and also TLS v1.2 ciphers such as ECDHE-RSA-DES-CBC3-SHA and DES-CBC3-SHA based on Triple DES which are not FIPS compliant. Go crypto also allows multiple non-compliant CBC ciphers with TLS v1.2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mH2avQLMh84ooz_e" /><figcaption>TLS Scan — Test Application — Go 1.16</figcaption></figure><p>The Forward Secrecy below supports some non-compliant ciphers such as ECDHE-RSA-AES256-SHA, ECDHE-RSA-CHACHA20-POLY1305.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UiPyHAOvjArlzqCD" /><figcaption>TLS Scan — Test Application — Go 1.16</figcaption></figure><p>The same application above is compiled using <a href="https://hub.docker.com/layers/goboring/golang/1.16.2b7/images/sha256-bf12e9d59e1ae2810873f1b4d9c603918fe3b403ed1577f0def30c9a7f75edeb?context=explore">goboring/golang</a>, this replaces standard Go crypto with FIPS validated cryptography. An assert step is added to assure that the build is using boringcrypto instead of native golang crypto.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*PFsZGIBaUKhPXRaF" /><figcaption>Test Application — GoBoring (boringcrypto)</figcaption></figure><p>Users can check whether a given binary is using BoringCrypto by running <a href="https://golang.org/pkg/cmd/nm/">go tool nm</a> on the binary and check for <em>*_Cfunc__goboringcrypto_*</em>. For example in case of the sample application above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OKH3gFf78KZFcryZ" /><figcaption>Test Application — boringcrypto validation</figcaption></figure><p>As shown below the boringcrypto which replaces the standard Go crypto library:</p><ul><li>TLS 1 and 1.1 are not offered</li><li>Triple DES is dis-allowed/not offered</li><li>Obsoleted CBC ciphers are dis-allowed/not offered</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pf36vB3BdZ87xClf" /><figcaption>TLS Scan — Test Application — GoBoring</figcaption></figure><p>As shown below the app compiled with boringcrypto only allows supported TLSv1.2 &amp; v1.3 and only allow FIPS 140–2 compatible ciphers. The forward secrecy also only allows complaint ciphers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iZkFCP5ODhWn55iO" /><figcaption>TLS Scan — Test Application — GoBoring</figcaption></figure><p>The same scenario applies to all Kubernetes components and other entities of the ecosystem, wherewith boringcrypto all the components only allow TLS connections using the FIPS 140–2 compliant ciphers and algorithms.</p><h3>CAVP and CMVP (Cryptographic Algorithms Validation Program &amp; Cryptographic Module Validation Program)</h3><p>ISVs (Independent Service Vendors) looking to obtain FedRAMP accreditation must comply with FIPS 140–2 encryption standards. FIPS Compliance means that different components of a product have received FIPS validation, but the product in its entirety has not passed testing or has not been tested at all. FIPS Validation means a product has undergone and passed detailed conformance testing at an accredited national laboratory.</p><p>In order to perform FIPS 140 conformance testing, a laboratory must become an accredited CST (Cryptographic and Security Testing) laboratory under the <a href="https://www.nist.gov/nvlap">National Voluntary Laboratory Accreditation Program (NVLAP)</a>. The current list has 12 accredited laboratories in USA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BxSSNPlUf-nefmoX" /><figcaption>NVLAP Laboratories</figcaption></figure><p>The <a href="https://csrc.nist.gov/Projects/cryptographic-module-validation-program">Cryptographic Module Validation Program</a> (CMVP), a joint effort of the U.S. National Institute of Standards and Technology (NIST) and the Canadian Centre for Cyber Security (CCCS), validates cryptographic modules to the Security Requirements for Cryptographic Modules standard (i.e., FIPS 140–2) and related FIPS cryptography standards. The FIPS 140–2 security requirements cover 11 areas related to the design and implementation of a cryptographic module. The NIST Information Technology Laboratory operates a related program that validates the FIPS approved cryptographic algorithms in the module (Cryptographic Algorithms Validation Program (CAVP)).</p><p>The CMVP process consists of four entities namely Vendor, CMT (Cryptographic Module Testing) Labs, CMVP Validation Authorities, and the Users. The cryptographic module of the vendor is submitted to CMT Lab where testing is performed. After the testing by CST Lab, the report is given to CMVP Validation Authorities for final validation and certification. Users, in turn, verifies the cryptographic module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*GxvH1RJsdHRARpIN" /><figcaption>CMVP Process</figcaption></figure><p>Obtaining a FIPS 140–2 Certification can take several months. The time frame is typically between 2–4 months with the lab (depending on the type of module to be tested and the overall security level) after the lab receives all the required documentation and module(s) to be tested from the vendor. These are high-end time frames and can be shorter depending on the completeness of the documentation, product meeting, and passing all the requirements.</p><p>Cryptographic Module is validated and tested with the help of two programs namely: <a href="https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program">Cryptographic Algorithms Validation Program (CAVP)</a> and the <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program">Cryptographic Module Validation Program (CMVP)</a>.</p><p>Cryptographic algorithm validation is a prerequisite of cryptographic module validation.</p><p>In CAVP, Algorithms in Cryptographic Module are checked (whether the Cryptographic Module is using FIPS approved algorithm or not) whereas CMVP checks the whole Cryptographic Module as per the FIPS 140–2 Derived Test Requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KhKDY2xLVRe79ZZr" /><figcaption>Cryptographic Algorithm Validation (CAVP)</figcaption></figure><p>The CMVP validates commercial cryptographic modules to Federal Information Processing Standard (FIPS) 140, NIST-recommended standards, and other cryptography-based standards. The CMVP is a government validation program that is jointly managed by NIST and CCCS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3VBTD3fMQo9zdlHW" /><figcaption>Cryptographic Module Validation Program (CMVP)</figcaption></figure><h3>Fips 140–2 validation Process — Analysis using Rancher’s Kubernetes Cryptographic Library as an example</h3><p><a href="https://docs.rke2.io/">RKE2</a>, also known as RKE Government, is Rancher’s open source next-generation Kubernetes distribution. It is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector.</p><p>RKE2 is built with FIPS validated cryptographic libraries using <a href="https://hub.docker.com/u/goboring">goboring Go compiler</a> maintained by Google. In RKE2 almost all the components (cluster components, runtime) are built with the version of Go from the goboring compiler which replaces the standard Go crypto libraries with the FIPS validated <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3678">BoringCrypto</a> module. In order to support additional platforms, Rancher opted for <a href="https://docs.rke2.io/security/fips_support/#use-of-fips-compatible-go-compiler">revalidation of the module as Rancher Kubernetes Cryptographic Library</a> through one of the NVLAP accredited laboratories (all information procured from publicly available sources).</p><p>The following are the steps that are involved for Rancher to procure a FIPS 140–2 validation for their cryptographic module:</p><ol><li>CMT Lab Engagement: As shown below Rancher submitted their module to <a href="https://www.acumensecurity.net/">Acumen Security</a> (NVLAP Code: <a href="https://csrc.nist.rip/groups/STM/testing_labs/">201029–0</a>). Document Source: <a href="https://github.com/rancher/rke2/blob/master/docs/assets/fips_engagement.pdf">rke2_fips_engagement</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4JFk-MRvP0gsJ3cR" /><figcaption>CMVP — CMT Lab Engagement</figcaption></figure><p>2. CAVP (Cryptographic Algorithm Validation Program): The lab above tests the algorithm implementations using test tools provided by NIST and CCCS and an algorithm implementation successfully tested by a lab and validated by NIST is added to an appropriate <a href="https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?validation=33463">validation list</a>, which identifies the vendor, implementation, operational environment, validation date and algorithm details (<a href="https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?product=13364">source</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6DrOVu09oT3gEP2_" /><figcaption>CMVP — CAVP</figcaption></figure><p>3. CMVP (Cryptographic Module Validation Program): Once CAVP is complete, the lab sends the reports to CMVP where it will be going through module pending, module review stages and reaches the finalized stage (~4 months). On completion of CMVP a certificate number is associated to the module and validation dates, sunset dates are specified. (<a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3836">source</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IpfYz9iNVJVL-OyT" /><figcaption>CMVP</figcaption></figure><p>The tested configuration section includes a list of environments where the module is tested and also includes different configurations in the same environment. The FIPS and Allowed Algorithms sections specify the list of algorithms used in the module or under the scope of validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rNqmglc0PsxA7sbg" /><figcaption>CMVP</figcaption></figure><p>The last section includes the vendor details, the NVLAP lab name &amp; details, a link to module-specific <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3836.pdf">security policy</a>, and a <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/certificates/March%202021_010421_0724_signed.pdf">consolidated certificate</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AfX2bTaG5joISzC5" /><figcaption>CMVP</figcaption></figure><p>5. Once CMVP is complete the module will be added to the list of modules (some of the most recently validated) in a <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/certificates/March%202021_010421_0724_signed.pdf">consolidated certificate</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tof0H8Ju8_GfdNE4" /><figcaption>CMVP — Consolidated Certificate</figcaption></figure><h3>FIPS 140–2 Levels and Kubernetes</h3><p>The <a href="https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-2.pdf">FIPS 140–2</a> standard provides four increasing, qualitative levels of security: Level 1, Level 2, Level 3, and Level 4. These levels are intended to cover a wide range of potential applications and environments into which cryptographic modules may be employed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*snDXMf3vvoI4GH7y" /><figcaption>FIPS 140–2 Certification Levels</figcaption></figure><p>A particular level requires that the previous levels also be met, but not every product must reach FIPS Level 4. For example, Level 1 provides the most basic security with practically no physical requirements, such as a personal computer encryption board, which is a validated Security 1 cryptographic module. In order for a system to be Security Level 2 validated, it would need to comply with all the standards outlined in Level 1, and additionally meet role-based authentication requirements to account for tamper-evidence required in FIPS Level 2. Certain levels are only appropriate for certain products or solutions.</p><p>As Kubernetes is a software stack, modules specific to Kubernetes will be provided with a Level-1 certification. Here cryptography is the entity that is tested and other levels 2–4 are non-applicable in this scenario as they are more related to physical and hardware-based security. For example, the following are the links to some of the validated modules from different vendors: <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3702">D2iQ</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3836">Rancher</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3636">Oracle</a>.</p><p>Taking <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3702">D2iQ BoringCrypto Cryptographic Security Module</a> as a specific example:</p><p>The ‘Overall Level’ field specifies the certification level provided to the module (other modules: <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3836">Rancher</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3636">Oracle</a>). Every module will have a certification expiration date (Sunset Date).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Qiret3sLwUD0oSW" /><figcaption>CMVP — Certification Level</figcaption></figure><p>The caveat section contains information required by the CMVP for the cryptographic module. Common caveats describe operational requirements for “FIPS mode” and entropy. For the module above it mentions ‘section 12.1 of the security policy’, the <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3702.pdf">Security Policy</a> document link will be provided with every validated module and section 12.1 above provides the information on how D2iQ compiles the FIPS enabled module and guidance on the installation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uz6qIpm_OCXyhp3W" /><figcaption>CMVP — Security Policy/Caveats</figcaption></figure><p>Taking another example <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3836">Rancher Kubernetes Cryptographic Library</a> which is awarded Level-1 certification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5zPOjDZ2B5IpSORk" /><figcaption>CMVP — Certification Level</figcaption></figure><p>The caveat section here specifies ‘Section 9.1’ of <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3836.pdf">Security Policy</a>, this provides installation guidance and instruction on compiling the module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u2XpNBZfTvT25RMV" /><figcaption>CMVP — Security Policy/Caveats</figcaption></figure><p>The ‘Security Level Exceptions’ specifies the list of entities that are not applicable for the module and not in the scope of validation. As shown below the <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3702">D2iQ BoringCrypto Cryptographic Security Module</a> lists physical security and migration of other attacks as not applicable (the same with <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3836">Rancher</a>, <a href="https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3636">Oracle</a>), the Level-1 considers the software components (specifically tamper-proof cryptography) and doesn’t require any hardware or other physical security requirements (levels 2–4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C4lIIRm19eQE7uDz" /><figcaption>CMVP — Security Level Exceptions, Test Configurations, FIPS Algorithms</figcaption></figure><p>The ‘Module Type’ specifies the module category ‘Software/Hardware/Firmware/Hybrids’. ‘Tested Configuration’ lists the different distros, versions, compilers and hardware on which the module is tested. A module can contain multiple algorithms, ‘Fips Algorithms’ list all the individual algorithms used in the module and their certificate links. ‘Allowed Algorithms’ specify a list of encryption algorithms that the module allows.</p><p>The last section includes details of the vendor, the NVLAP (National Voluntary Laboratory Accreditation Program) accredited laboratory information, a link to the <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3836.pdf">Security Policy</a> of the module, and a link to <a href="https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/certificates/August%202020_010920_0658.pdf">Consolidated Validation Certificate</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-Rh-3HvQQsxawBR6" /><figcaption>CMVP — Vendor Information and Consolidated Certificate</figcaption></figure><p>Many government organizations and commercial organizations in government-regulated industries are adopting Kubernetes as their standard for orchestrating containers, vendors by maintaining FIPS 140–2 compliance enables government agencies to adopt a DevSecOps transformation to build on a solid base of government-ready defaults, removing the heavy lifting required to meet a lot of government compliance requirements.</p><p>FIPS 140–2 is one of the many requirements to operate in the government sector, there are many other Kubernetes specific security hardening requirements such as PodSecurityPolicies, NetworkPolicies, fine-grained authorization(RBAC), dynamic secret management, CIS Hardening, SELinux enablement, Usage of distroless containers, etc. These entities together enable public sector customers to run some of the most mission-critical workloads in the world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66d852ccccd2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Anthos on Bare Metal and Akri — Managing Leaf Devices on Edge Kubernetes Clusters from Cloud]]></title>
            <link>https://itnext.io/anthos-on-bare-metal-and-akri-managing-leaf-devices-on-edge-kubernetes-clusters-from-cloud-222ff17dd7b8?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/222ff17dd7b8</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[edge-computing]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Thu, 25 Mar 2021 08:02:37 GMT</pubDate>
            <atom:updated>2021-03-25T08:36:33.035Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZBc_Brt_WPqD6fdIjx_GyQ.png" /><figcaption>Anthos Clusters on Bare Metal and Akri as Kubernetes Resource Interface for the Edge</figcaption></figure><h3>Anthos on Bare Metal and Akri — Managing Leaf Devices on Edge Kubernetes Clusters from Cloud</h3><p>The millions of devices that currently make up the Internet of Things (IoT) reside not in the cloud but on-premises: from retail stores to factory floors. Many of these devices are tiny edge leaf devices like cameras (IP cameras, USB cameras etc.), sensors (smart heat sensors on equipment etc.). It has become increasingly difficult and impractical for developers to write bespoke solutions to detect and use each of the devices and the core DevOps/deployment teams to manage the applications and clusters which host these applications that are positioned in far edge sites (close to end-users).</p><p>Most of these devices, however, are too small to run Kubernetes themselves. How can they be leveraged by a Kubernetes workload and how a private cluster can be managed from cloud?</p><p><a href="https://cloud.google.com/anthos/clusters/docs/on-prem/1.6/overview">Anthos clusters on VMware</a> also called GKE on-prem was an offering that enabled users to run GKE in customer datacenters but this model needs vCenter Server and VSphere, but <a href="https://cloud.google.com/anthos/clusters/docs/bare-metal/1.6/concepts/about-bare-metal">Anthos on bare metal</a> uses a “bring your own operating system” model. It runs atop physical or virtual instances, and supports Red Hat Enterprise Linux 8.1/8.2, CentOS 8.1/8.2, or Ubuntu 18.04/20.04 LTS.</p><p><a href="https://github.com/deislabs/akri">Akri</a> from Deislabs (Microsoft), an open source project that defines a Kubernetes based approach on representing leaf devices on the edge as native Kubernetes Resources. It provides an abstraction layer like the Container Networking Interface, but instead of abstracting the underlying network details, it removes the work of finding, utilizing, and monitoring the availability of leaf devices.</p><h3>Example Use Case Scenario</h3><p>A nationwide retail store needs to implement camera-based applications (general store monitoring, sentiment analysis, gender analysis, mask detection, self-checkout etc.) in their stores around the nation and enable users like store-managers and local business development teams to access the applications locally and enable the core team to operate and manage applications from HQ over a public cloud platform. In this scenario, the stores can be considered as edge-sites.</p><p>In this setting, the viable option is to use small footprint devices like Intel NUC’s or any other SBC’s with enough capacity to run Kubernetes and use USB cameras connected to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/0*jzvyEDxMwycBUNCG" /><figcaption>USB Camera connected to NUC</figcaption></figure><p>Anthos clusters on Bare Metal here deploys Kubernetes and connects the private bare metal cluster in edge site to Google Cloud. In the example use case above, the HQ can image and pre-deploy/configure the NUC’s with Anthos Bare Metal or use some kind of automation (systemd, rc.local etc.) to bootstrap the cluster. Once powered on the cluster will be automatically registered to users project on GCP using pre-configured SA keys and then the HQ team can takeover to deploy, manage, update applications on the edge site. In this scenario, the local teams need no technical abilities and they can just consume the end-application.</p><p>In this post, <a href="https://cloud.google.com/anthos/config-management">Anthos Config Management</a> is used to deploy Akri components to the remote/private cluster from Anthos portal on GCP. Once the deployment is complete the local users can access the camera’s footage using the local LB IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l6Cpc_m9hPzfJBKB" /><figcaption>Deploying Akri to remote cluster using Anthos</figcaption></figure><p>This is just an example scenario to access the abilities and usage of Anthos Clusters on Bare Metal and Akri to manage Kubernetes and Leaf devices on edge.</p><h3>Anthos Clusters on Bare Metal</h3><p><a href="https://cloud.google.com/anthos/clusters/docs/bare-metal/1.6">Anthos on bare metal</a> is a deployment option to run Anthos on physical servers, deployed on an operating system provided by the user, without a hypervisor layer. Anthos on bare metal ships with built-in networking, lifecycle management, diagnostics, health checks, logging, and monitoring. Users can use supported CentOS, Red Hat Enterprise Linux (RHEL), and Ubuntu — all validated by Google for the cluster deployment.</p><p>With Anthos on bare metal, users can leverage existing investments and use the company’s standard hardware and operating system images, which are automatically checked and validated against Anthos infrastructure requirements. Anthos on bare metal follows the same billing model as all <a href="https://cloud.google.com/anthos/docs/setup/overview#requirements">managed Anthos clusters</a> and are based on the number of Anthos cluster vCPUs, charged on an hourly basis.</p><p>Users can deploy Anthos on bare metal using one of the following deployment models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tiQNWNS0eqiYli7B" /><figcaption>Anthos Clusters on Bare Metal — Deployment Models</figcaption></figure><p>A standalone model allows users to manage every cluster independently. This is a good choice when running in an edge location or if users want their clusters to be administered independently from one another. A multi-cluster model which allows users to manage a fleet of clusters (user clusters) from a centralized cluster called the admin cluster. Suitable for managing multiple clusters from a centralized location.</p><p>To meet different requirements Anthos clusters on bare metal support the following deployment models (all three models include an admin cluster and user cluster construct as specified above).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bvn0_uQkncVpkbN5" /><figcaption>Anthos Clusters on Bare Metal — Deployment Models</figcaption></figure><p>The topology in the post comprises three Intel NUC’s, two with 16 GB RAM, 4 vCPU’s, 250 GB SSD which are used as a Master and Node (Anthos provides specific hardware requirements). The deployment model here is Hybrid, where the cluster is an admin cluster that also allows running user workloads, this cluster can be re-used as an admin cluster to create user clusters if needed. The third NUC (2 vCPU’s, 8GB RAM) is used as a workstation where all the required deployment operations are performed, this hosts the temporary KIND (Kubernetes in Docker) bootstrap cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xpMshIdbrCROJIao" /><figcaption>Anthos Clusters on Bare Metal —Test Cluster Topology</figcaption></figure><p>All the machines in the cluster are connected to a home network (private) with outbound connectivity to the internet. All the NUC’s procure IP’s from the home network (node_ip and master_host_ip). Anthos clusters on bare metal use <a href="https://metallb.universe.tf/">MetalLB</a> running in L2 mode for data plane load balancing. Users can carve out a block of contiguous IP addresses on the private network for allocating to K8s service of type “LoadBalancer”. Users can select a “start” IP address in the block above to be allocated to IngressVIP and another IP, not in the range for controlPlaneVIP.</p><p>Anthos clusters on bare metal deploy L4 load balancers that run on either a dedicated pool of worker nodes or on the same nodes as the control plane. The above topology deploys LB in ‘bundled’ mode where a load balancer will be installed on load balancer nodes during cluster creation.</p><p>Anthos provides overlay networking and L4/L7 load balancing out of the box. Customers can also integrate with their own load balancers such as F5 and Citrix. For storage, users can deploy persistent workloads using <a href="https://kubernetes-csi.github.io/docs/drivers.html">CSI integration</a> with their existing infrastructure.</p><h3>Bootstrapping a Cluster using bmctl</h3><p>The install command of Anthos clusters on bare metal is <a href="https://cloud.google.com/anthos/clusters/docs/bare-metal/1.6/installing/creating-clusters/create-clusters-overview">bmctl</a>, the utility enables users to create a cluster with minimal effort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZsZGpuhtfbpmrlyi" /><figcaption>Anthos Clusters on Bare Metal — bmctl utility</figcaption></figure><p>The <em>bmctl</em> utility reads all the configuration from a <a href="https://cloud.google.com/anthos/clusters/docs/bare-metal/1.6/quickstart#edit-config">config file</a>, a template can be created using ‘bmctl create config’. Users can pass ‘<em> — enable-apis — create-service-accounts</em>’ to automatically enable service accounts and APIs automatically required for authentication to project in Google Cloud. This configuration file includes the key paths and ssh_keys to enter the master and node to deploy the components. The configuration file includes NodePools, AddressPools, and Storage-related configurations.</p><p>During installation, Anthos on Bare Metal creates a temporary bootstrap cluster (KIND, a Kubernetes cluster where required components are deployed to create the first admin/user cluster), in the above scenario the cluster is deployed on the workstation host. After a successful installation, the bootstrap cluster is deleted, leaving users with an admin cluster and user cluster. Generally, you should have no reason to interact with this cluster, if needed the same can be left without deleting (<em> — cleanup-external-cluster=false</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_k4CGrry1_gNE5WB" /><figcaption>KIND Bootstrap Cluster</figcaption></figure><p>Cluster creation follows the cluster-api pattern which uses a custom GKE bare metal provider, the bootstrap KIND cluster running on the workstation uses the generated CRD’s to bootstrap the hybrid cluster (or any other deployment model). The CRD’s are also created on the Admin cluster which then enables users to create user clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3Va3inB2eL-ciZoI" /><figcaption>Anthos Clusters on Bare Metal — Custom CRD’s for cluster creation</figcaption></figure><p>Cluster deployment steps :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sXCoAtJY8j8wqWM7" /><figcaption>Anthos Clusters on Bare Metal — Cluster deployment stages</figcaption></figure><p>All configuration files along with preflight checks, cluster creation logs and kubeconfig required to access the main cluster are created on the workstation in a specific directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/0*ZtA4UJ7K6K7YeduZ" /><figcaption>Anthos Clusters on Bare Metal — Cluster deployment</figcaption></figure><p>Once the cluster is ready, the bootstrap cluster will be deleted and users can start using the created hybrid cluster, users can use the hybrid cluster to create multiple user clusters as required. If the deployment model is standalone then a user cluster is created without any machinery (admin cluster) to create multiple user clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HP5TIJzaTVgNwGD6" /><figcaption>Anthos Clusters on Bare Metal — Cluster deployment</figcaption></figure><p>As the cluster above is deployed using hybrid mode the cluster comprises Admin cluster machinery.</p><p>Cluster API components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HESRoZr3BN5dsUfB" /><figcaption>Admin Cluster — CAPI Components</figcaption></figure><p>BareMetalCluster CRD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UledtScjUkhnktvW" /><figcaption>Admin Cluster — BareMetalCluster CRD</figcaption></figure><p>BareMetalMachine CRD (as this is a hybrid cluster, the NUC’s are added as objects):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4e2TF_zbuRMCyib9" /><figcaption>Admin Cluster — BareMetalCluster CRD</figcaption></figure><h3>Connecting Anthos Clusters on Bare Metal to Google Cloud</h3><p>Anthos clusters on bare metal connects the clusters to Google Cloud. This connection lets users manage and observe the isolated clusters from the Cloud Console by using <a href="https://cloud.google.com/anthos/multicluster-management/connect/overview">Connect</a>. As part of cluster deployment, a connect agent is deployed in the cluster and ‘<em>bmctl</em>’ utility automatically creates required service accounts (if more control is needed users can manually create connect-agent, connect-register and logging-monitoring service accounts) required for establishing a connection and registers the cluster to Google Cloud GKE Hub.</p><p>Connect is an Anthos feature and not specific to Anthos bare metal which allows users to connect any of the Kubernetes clusters to Google Cloud irrespective of where it is deployed. This enables access to cluster and workload management features, including a unified user interface, <a href="https://cloud.google.com/cloud-console">Cloud Console</a>, to interact with the cluster. Connect allows Anthos Config Management to install or update the Connect in-cluster agent and observe the sync status. Connect also lets the metering agent observe the number of vCPUs in a connected cluster.</p><p>This is pivotal for a topology where applications on far edge-sites in isolated network spaces are managed from a central location (HQ/Regional IT operations center etc.)</p><p>The Connect Agent can be configured to traverse NATs, egress proxies, and firewalls to establish a long-lived, encrypted connection between the isolated bare metal cluster’s Kubernetes API server and Google Cloud project.</p><p>connect-agent deployed on the bare metal cluster :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rsr5XsnWim0jpi8F" /><figcaption>GKE Connect Agent on Anthos Cluster on Bare Metal</figcaption></figure><p>The anthos-creds namespace holds required secrets to connect with GCP services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wg7Xm0Ic7SmZczRg" /><figcaption>Secrets for GCP Services Connectivity</figcaption></figure><p>Anthos portal showing the registered cluster (before sign in), the cluster information on the right panel will not be shown until the users log-in :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NkPvH6_Bz_tfOcNS" /><figcaption>Anthos Portal Showing Registered Cluster</figcaption></figure><p>Kubernetes Engine portal showing the registered cluster (before sign in) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VSZlRDMhz1NfLHwr" /><figcaption>Kubernetes Engine Showing Registered Cluster</figcaption></figure><p>Users can use the Google Cloud Console to sign in to registered clusters. This can be done in three ways: using basic authentication or a bearer token or using an OIDC provider. In the scenario below an admin user is created associated with cluster-admin role and the token is used for log-in (users can create a cluster read-only role based on the level of operation required).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bssgnibrgZxNITgq" /><figcaption>Logging-in to Registered Cluster using Service Account</figcaption></figure><p>Logging-in to cluster using token :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*noPho0P58dEqWMvM" /><figcaption>Logging-in to Registered Cluster using Service Account Token</figcaption></figure><p>As shown below, once the log-in is successful all the management options will be enabled and cluster information will be displayed. The info also shows the GKE Hub Membership ID, users can use ‘<em>gcloud container hub memberships list</em>’ to list all clusters connected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tEoatAD3h92OahO_" /><figcaption>Kubernetes Engine Showing Registered Cluster</figcaption></figure><p>Anthos portal showing cluster information and list of cluster features :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jZmn70N373OrtIol" /><figcaption>Anthos Portal Showing Registered Cluster</figcaption></figure><h3>Managing Workloads on Anthos Clusters on Bare Metal</h3><p>Once the connect agent connects the cluster, users can use Kubernetes Engine portal as a dashboard to access all Kubernetes components and manage workloads on the private Anthos on bare metal cluster as if they are on GKE.</p><p>Anthos dashboard enables users to deploy Anthos related components (Features) like ACM, Service Mesh, etc. (provides a catalog of features and instructions) there is limited functionality to just verify the connectivity of the cluster and high-level details like cluster_size and nodes in the portal. Kubernetes Engine portal is the main cluster management entity that enables users to operate on core Kubernetes functionalities like create, edit, update and delete objects, access configuration objects (secrets/configmaps), deploy applications from the marketplace, browse objects and manage/list storage elements like PV/PVC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jdvb-Sv8QzxTGjwf" /><figcaption>Workloads — Kubernetes Engine</figcaption></figure><p>Users can edit any Kubernetes object and the same will be reflected on the bare metal cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6uun3sHAkO00nBKX" /><figcaption>Edit/Update Workloads — Kubernetes Engine</figcaption></figure><p>Accessing cluster Services and Ingress :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZWnNcuQcgGBajm-6" /><figcaption>Services and Ingress — Kubernetes Engine</figcaption></figure><p>Anthos clusters on bare metal uses the local volume provisioner (LVP) to manage local persistent volumes. Three types of storage classes are created for local PV’s in an Anthos clusters on bare metal: LVP Share (creates a local PV backed by subdirectories created during cluster creation in a local, shared file system), LVP node mounts (creates a local PV for each mounted disk in the configured directory), Anthos system (creates pre-configured local PV’s during cluster creation that is used by Anthos system pods).</p><p>Accessing PVC’s and available Storage Classes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vDV-FM4NrXe7KUX2" /><figcaption>Storage — Kubernetes Engine</figcaption></figure><p>Anthos portal facilitates users to enable supported features on the connected clusters :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l2tl8jd-nqDiIOG_" /><figcaption>Enabling Features on Registered Cluster — Anthos</figcaption></figure><h3>Monitoring and Logging</h3><p>Logging and metrics agents are installed and activated in each cluster when users create a new admin or user cluster. Stackdriver operator manages the lifecycle for all other Stackdriver related components (log-aggregator, log-forwarder, metadata-agent and prometheus) deployed onto the cluster. Users can setup a Cloud Monitoring Workspace within the Cloud project using the monitoring portal to access logs of all system and Kubernetes components in the remote bare metal clusters on the console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uQEk42iovEkX-D4r" /><figcaption>Monitoring and Logging — Stackdriver</figcaption></figure><p>The Stackdriver collector for Prometheus constructs a Cloud Monitoring <a href="https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.uptimeCheckConfigs#MonitoredResource">MonitoredResource</a> for the Kubernetes objects from well-known Prometheus labels. A separate entity ‘<em>stackdriver-prometheus-app</em>’ is deployed with the stack and can be configured to monitor the applications in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LUokEKKTI6xYDZu_" /><figcaption>Monitoring and Logging — Stackdriver Prometheus</figcaption></figure><p>Google Cloud’s operations suite is the built-in observability solution for Google Cloud. It offers a fully managed logging solution, metrics collection, monitoring, dashboards, and alerting. Cloud Monitoring uses Workspaces to organize and manage its information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8QMiSlIYMwmFfnEM" /><figcaption>Monitoring and Logging — Workspaces</figcaption></figure><p>Integrated logging:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aqJa13ghDrUKDqpg" /><figcaption>Monitoring and Logging — Workspaces</figcaption></figure><h3>Akri — A Kubernetes Resource Interface for the Edge</h3><p>Akri enables discovery and exposing heterogeneous leaf devices as resources and creates services for each device in Kubernetes clusters. This enables applications running on Kubernetes to consume the inputs from the devices. Akri handles the automatic inclusion and removal of devices, as well as the allocation and deallocation of resources to better optimize the clusters.</p><p>Akri is a standard Kubernetes extension implemented using two custom resource definitions (configuration and instances), an agent, a controller. Once installed, users can write an Akri definition for the device, the device plug-in acts as an agent, finding hardware that matches the definitions using supported discovery protocols, before using the Akri controller to manage it through a broker running in a Kubernetes pod that exposes the device and its APIs to your application code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/0*-yCsIvjT6UvjzTRO" /><figcaption>Akri Architecture</figcaption></figure><p>Akri is based on the <a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md">Kubernetes device plugin framework</a>, which provides vendors a mechanism called device plugins which can be used to advertise devices, monitor devices (like health checks), hook the devices into the runtime to execute device specific instructions (e.g: Clean GPU memory, capture video etc.) and make the device available in the container. This enables vendors to advertise their resources and monitor them without writing additional code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/0*wW7Dtx4vAnkHDxQi" /><figcaption>Device Plugin Framework</figcaption></figure><p>Akri currently supports protocols: udev (to discover anything in the Linux device file system), ONVIF (to discover IP cameras) and OPC UA (to discover OPC UA Servers). Protocols like Bluetooth, Simple scan for IP/MAC addresses, LoRaWAN, Zeroconf etc. are in the roadmap.</p><p>In this post, udev (userspace /dev) protocol is used to discover USB cameras connected to two nodes of the Anthos on Bare metal Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xwFGHXocETYNv4GR" /><figcaption>USB Camera connected to NUC</figcaption></figure><p>Udev manages device nodes in the <em>/dev</em> directory, such as microphones, security chips, usb cameras, and so on. Udev can be used to find devices that are attached to or embedded in nodes. Akri’s udev discovery handler parses udev rules listed in a Configuration, searches for them using udev, and returns a list of discovered device nodes (ie: /dev/video0). User can configure Akri which device(s) to find by passing <a href="https://wiki.archlinux.org/index.php/Udev">udev rules</a> into a Configuration spec.</p><p>A user can also allow multiple nodes to utilize a leaf device, thereby providing high availability in the case where a node goes offline. Furthermore, Akri will automatically create a Kubernetes service for each type of leaf device (or Akri Configuration), removing the need for an application to track the state of pods or nodes.</p><p>For example, the following is the information of the attached USB camera using ‘udevadm’ on one of the nodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RUq1VgXMOLZLqa2T" /><figcaption>Udevadm — Listing attached camera information</figcaption></figure><p>Users can use Akri’s pre-defined grammar <a href="https://github.com/deislabs/akri/blob/main/discovery-handlers/udev/src/udev_rule_grammar.pest">here</a> to define specific udev rules in the protocol section of Akri’s configuration spec to define what devices should be discovered on a specific node by a udev rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T3bw2lZjc9elKHz_" /><figcaption>Udev — Akri Rules</figcaption></figure><p>Akri agent deployed as a daemonset handles resource availability changes and enable resource sharing, these tasks enable Akri to find configured resources (leaf devices), expose them to the Kubernetes cluster for workload scheduling, and allow resources to be shared by multiple Nodes. The Agent discovers resources via <a href="https://github.com/deislabs/akri/blob/main/docs/agent-in-depth.md#resource-discovery">Discovery Handlers (DHs)</a>.</p><p>Akri Controller deployment on the master Node in the cluster enables cluster access to leaf devices and handle node disappearances. The broker pods are created automatically based on the protocol chosen and the capacity (<a href="https://github.com/deislabs/akri/blob/main/docs/resource-sharing-in-depth.md">resource sharing</a>) defined.</p><h3>Deploying Akri to Anthos on Bare metal Cluster using ACM (Anthos Config Management)</h3><p>The topology below illustrates a classic example of how GitOps can be configured and deployment of applications can be performed from cloud to edge. Akri is deployed on Anthos Bare Metal cluster in isolated network space using Anthos Config Management from cloud. Akri then takes care of all other required aspects to discover the connected USB cameras and enable the streaming application to consume them. End users can use LB_IP (local network) to view the footage accessing the streaming-app running on far edge sites (in this scenario the far edge site is the NUC running Anthos Bare Metal Kubernetes cluster)</p><p>In the topology below an admin user provides ‘ConfigManagement’ spec to Anthos Config Management, the spec contains the cluster_selector (in this case the Anthos bare metal cluster is selected) and git information which holds all the deployment manifests of Akri.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8hSXJmw9n07ad5Zs" /><figcaption>Deploying Akri Using ACM (Anthos Config Management)</figcaption></figure><p>ACM takes care of deploying the manifests to the private bare metal cluster defined in the specified git_repo and maintain sync with the git branch. The Akri machinery detects the USB cameras connected to the nodes (here both the hosts serve as nodes — master taint removed) and local users can access the footage using the LB_IP on the local network.</p><p>ACM can be configured from Anthos dashboard on GCP console, usually all the clusters part of the project will be considered for update, users can configure specific clusters or apply the config to all.</p><ol><li>Git repository authentication:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ixyKxBJ4nOu9v6Sm" /><figcaption>ACM Configuration</figcaption></figure><p>2. Conifg Sync — holds the information like git_url and the directory with the configuration manifests. A sample <a href="https://github.com/gokulchandrap/akri-edge">repo</a> used here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l4KcpUPTTgKpXVA6" /><figcaption>ACM Configuration</figcaption></figure><p>The <a href="https://github.com/gokulchandrap/akri-edge">repo</a> configured above includes all manifests required to deploy Akri on Kubernetes. The format follows <a href="https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/concepts/repo">ACM specified layout</a> where the cluster directory includes all cluster level objects (like CRD’s, ClusterRole, ClusterRoleBinding etc.), the namespaces directory include a sub-directory with the name of the namespace which holds the namespace object and all other namespaced objects (like Deployments, Daemonsets, Services etc.), the system directory contains configs for the Operator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gc2xl1OVf7EuGRBA" /><figcaption>ACM Configuration — Git Source</figcaption></figure><p>Once the ACM spec is submitted with a cluster_selector (in this case the Anthos bare metal cluster is selected) the controller on the bare metal cluster creates required git-importer and monitor pods in config-management namespace which performs all GitOps related tasks on the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h3gzRyHv7vMLGOaE" /><figcaption>ACM Components — importer and monitor</figcaption></figure><p>Once the installation is complete users can check the sync status from Anthos Config Management section of the Anthos dashboard or use ‘nomos status’. As shown below once the sync is complete all Akri related components are deployed on the private Anthos on Bare Metal cluster and can be verified from Workloads section of Kubernetes Engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FT-Y4Xu5qsIkuagl" /><figcaption>Akri Components</figcaption></figure><p>Akri implementation installs two CRD’s in the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i9Z4YCGO2eJt_Usc" /><figcaption>Akri CRD’s</figcaption></figure><p>Akri Configuration specifies Akri controller the kind of leaf device to discover. In the sample below (akri-udev-video), udev protocol is used to discover USB cameras.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*hlrD2Zx4G03QbKii" /><figcaption>Akri CRD’s</figcaption></figure><p>Akri Agent, a Kubernetes device plugin framework implementation searches for the leaf devices, checking for the availability based on the rules specified. Once the device has been discovered, the Akri Controller which sees each Akri Instance as a representation of a leaf device, deploys a “broker” pod that facilitates connection to leaf devices and utilize it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qqXSD2Zr7nJ-CysS" /><figcaption>Akri CRD’s</figcaption></figure><p>Apart from the core Akri components, an ‘<em>akri-vedio-straming-app</em>’ is deployed which streams video frames collected from the broker pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*myWUKymG4_-sJMaX" /><figcaption>Akri — Video Streaming Application</figcaption></figure><p>The streaming application can be configured using ‘<em>akri-configuration</em>’ object:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YTF0xX8n0quq87sh" /><figcaption>Akri — Video Streaming Application — Configutration</figcaption></figure><p>Configurations like format, resolution and frames can be configured using broker pod spec:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Oy8j3WViLwZZ-9d" /><figcaption>Akri — Configuration</figcaption></figure><p>The admin can manage all the Akri components from workloads section of Kubernetes Engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bEK2EYU-PYe-yOON" /><figcaption>Managing Akri Components from Cloud</figcaption></figure><p>Pod logs can be accessed form Kubernetes Engine portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6GFKuPYuNJf3ZYWi" /><figcaption>Monitoring Akri Components from Cloud</figcaption></figure><p>The stackdriver installed on the cluster streams all the logs to operations &amp; logging unit enabling users to filter specific logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nX2K3fLLw1hYWOPQ" /><figcaption>Monitoring Akri Components from Cloud</figcaption></figure><p>Anthos clusters on bare metal use <a href="https://metallb.universe.tf/">MetalLB</a> running in L2 mode for data plane load balancing. The MetalLB uses the pool of IP’s configured while bootstrapping the cluster to attach them to services of type LoadBalancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EndMOAVsRa7ykTTb" /><figcaption>Anthos Clusters on Bare Metal — Metallb Components</figcaption></figure><p>Users can edit a service from Kubernetes Engine — Services &amp; Ingress portal if required:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9TGcd9bA8zGHoHKN" /><figcaption>Exposing Video Streaming Application Service as LoadBalancer</figcaption></figure><p>As shown below the akri-video-streaming-app is exposed as a type LoadBalancer which can be used to access the footage using Load balancer IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BoQkpq5lmb7CFA0S" /><figcaption>Exposing Video Streaming Application Service as LoadBalancer</figcaption></figure><p>In the topology above two cameras are attached to two nodes and Akri controller deploys two broker pods for each instance (camera). As shown below, the cameras are pointed to two different objects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LSIeBxaLzBekmsCo" /><figcaption>Two camera connected to the same cluster pointing to two different objects</figcaption></figure><p>The akri-video-streaming-app collects frames from both broker pods and displays them in a unified view. The resolution and frames are configured based on the akri-configuration spec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Xoghj74UsQ0HtWJt4C89A.png" /><figcaption>Video Streaming Application — Relaying frames from the connected cameras</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DDaSUkxpaTusTWvSSuCn1w.png" /><figcaption>Video Streaming Application — Relaying frames from the connected cameras</figcaption></figure><p>The combination provides users with automation required to create clusters, connect them to cloud and enables leaf devices to be represented as Kubernetes objects, together making Kubernetes to be a versatile edge computing solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=222ff17dd7b8" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/anthos-on-bare-metal-and-akri-managing-leaf-devices-on-edge-kubernetes-clusters-from-cloud-222ff17dd7b8">Anthos on Bare Metal and Akri — Managing Leaf Devices on Edge Kubernetes Clusters from Cloud</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Kubernetes Engine (GKE) Autopilot]]></title>
            <link>https://faun.pub/google-kubernetes-engine-gke-autopilot-7d6ef338d464?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d6ef338d464</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Thu, 25 Feb 2021 13:43:38 GMT</pubDate>
            <atom:updated>2021-02-25T15:16:01.798Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IOxrHiHQtDDF6zWkzBTTyg.png" /><figcaption>Managed Control-Plane and Node-Pools — Serverless option for GKE</figcaption></figure><p>Autopilot is a new mode of operation for creating and managing Kubernetes clusters in Google Kubernetes Engine (GKE). In this mode, GKE configures and manages the underlying infrastructure, including nodes and node pools enabling users to only focus on the target workloads and pay per pod resource requests (CPU, memory, and ephemeral storage). In addition to GKE’s SLA on hosts and the control plane, Autopilot also includes an SLA on Pods.</p><p>With this new addition, users can now create GKE cluster in either ‘<em>Standard</em>’ mode where users can customize their configurations based on the requirements, GKE manages the control plane and users manually provision and manages their node infrastructure (or) ‘<em>Autopilot</em>’ mode where GKE manages the nodes, pre-configures the cluster with Google’s production-ready best practices baked in and adds auto-scaling, auto-upgrades, maintenance, Day 2 operations and security hardening. With Autopilot, Google specifies to use them as users/organizations SRE for both nodes and control plane (“Google SRE already handles cluster management for GKE; with Autopilot, Google SREs manage your nodes as well, including provisioning, maintenance, and lifecycle management.” ).</p><p>GCP with its <a href="https://cloud.google.com/run">Cloud Run</a> already enabled users to create containerized applications on managed serverless platform and compete with other cloud provider products such as <a href="https://aws.amazon.com/fargate/">AWS-Fargate </a>and <a href="https://azure.microsoft.com/en-us/services/container-instances/">Azure Container Instances</a>. With Autopilot addition it enables users to use Kubernetes as a container orchestrator to deploy applications on managed platform and adopt pay per use model competing with other homologous products such as <a href="https://itnext.io/eks-fargate-extensibility-of-kubernetes-serverless-benefits-77599ac1763?source=friends_link&amp;sk=52071656488524c9f2a1bcc13f492374">EKS+Fargate</a>. There is a substantial difference in the implementation when compared with EKS/Fargate where the user has to provision the EKS cluster control plane and the Fargate profile attached to the EKS cluster associates one or more namespaces and labels with the Fargate control plane, When a pod definition is submitted to EKS that doesn’t target a namespace or doesn’t contain the labels, it never gets scheduled by Kubernetes. In Autopilot mode users can create a cluster and use the same as they were using the standard Kubernetes cluster without having to worry about any other extra configuration and also adds concrete automation for handling auto-scaling, auto-upgrades, and other Day-2 operations.</p><p>From a developer’s perspective, nothing really changes here, but this new mode does free up teams to focus on the actual workloads and less on managing Kubernetes clusters. With Autopilot, users still get the benefits of Kubernetes, but without all of the routine management and maintenance related to the cluster and resources.</p><p>In Standard mode, GKE only manages the control plane and the user has to manage configuring the nodes or node-pools (in simpler terms the nodes will be on the user’s account running as VM instances) and configure other features such as automation, security hardening, etc. The billing here is per node based on the node-type (CPU, memory, disk, etc.) where the user has to pay the designated amount whether they use the node resources or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CLl71b_gdHEsHoHY" /><figcaption>GKE — Standard Mode</figcaption></figure><p>In Autopilot mode, GKE manages the entire underlying infrastructure of the clusters, including the control plane, nodes, and all system components. Apart from the general management, it adds a rich set of automation required for day-2 operations such as node auto-upgrades, node auto-repair, maintenance windows, etc., which are optional for standard mode. The billing here is per pod resource requests (CPU, memory, and ephemeral storage), this provides users with pay per use model where they only have to pay for the resources they (applications) really use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pOYdvtoh6ZHe7lG3" /><figcaption>GKE — Autopilot Mode</figcaption></figure><p>In Autopilot clusters, users can view the nodes with the traditional ‘<em>kubectl describe nodes’</em> command but there is no need of managing the same. Unlike Standard mode, the nodes are not visible through Compute Engine or the CLI.</p><p>Users now will be presented with an option to choose the mode when creating the GKE cluster from Kubernetes Engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yIvKuZzWC1-7tmzy" /><figcaption>GKE — Cluster Mode Selection</figcaption></figure><p>A comparison chart is provided to compare the modes. The documentation here provides a detailed comparison between both the modes in each aspect: <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview?authuser=3&amp;_ga=2.4785142.-452500305.1608062772#comparison">comparing autopilot and standard modes</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sqP7e3jxUNIavboZ" /><figcaption>GKE — Cluster Mode Comparision</figcaption></figure><p>Autopilot enables most of the production-grade features, user can customize based on the nature of the setting which involves <strong>Pre-configured:</strong> user cannot change this and it is built-in, <strong>Default:</strong> the setting is turned on and can be overridden by the user and <strong>Optional:</strong> the setting is turned off but the user can enable the same.</p><p>The cluster-type of Autopilot mode is pre-configured to be regional (standard mode allows users to choose between zonal/regional). In a regional cluster multiple replicas of the control plane, run in multiple zones within a given region. Nodes also run in each zone where a replica of the control plane runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nY5672Z_rW0tMzaA" /><figcaption>GKE — Creating Autopilot Cluster</figcaption></figure><p>Networking is pre-configured to use <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips?authuser=3">VPC-native</a> (alias IP). The quota is pre-configured to a maximum of 32 pods per node and intranode visibility is pre-configured. User-configurable parameters include private/public cluster (In a private cluster, nodes do not have public IP addresses, and the control plane is inaccessible by default), enabling control-plane authorized networks to block untrusted non-GCP source IPs from accessing the Kubernetes master through HTTPS and pod, service CIDR’s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M9eXOYnAnBiOFJke" /><figcaption>GKE — Creating Autopilot Cluster</figcaption></figure><p>Users can change the default Regular release channel to Rapid Channel. Regular channel includes versions that have passed internal validation and are considered production-quality whereas Rapid channel offers early access for users who want to test new releases before they are qualified for production use or general availability. Rapid Channel may be subject to unresolved issues with no known workaround and are not for use with production workloads or subject to any SLAs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G7hOaQcJQNilxOsv" /><figcaption>GKE — Creating Autopilot Cluster</figcaption></figure><p>The Automation section enables users to schedule custom maintenance windows or leave them to GKE. Normally, routine Kubernetes Engine maintenance may run at any time on the cluster. In this section, users can specify specific times for maintenance, enable maintenance windows or add a maintenance exclusion. Autopilot also supports pod disruption budget for availability and maintenance flexibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MVU8GXN-MzJpGb2m" /><figcaption>GKE — Creating Autopilot Cluster — Maintenance</figcaption></figure><p>Adding maintenance exclusion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rMEtPEyD6eRkfDjJ" /><figcaption>GKE — Creating Autopilot Cluster — Maintenance Exclusion</figcaption></figure><p>With the configuration above, a GKE cluster will be created in Autopilot mode. The cluster information page shows section-wise settings and few of them can be changed and others are immutable. The cluster basics section includes the region, release and name of the cluster which are immutable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qNEdlbP8OXd7WHBZ" /><figcaption>GKE — Creating Autopilot Cluster</figcaption></figure><p>The Automation section includes day-2 operations where maintenance window and maintenance exclusions can be edited after the cluster creation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mzm2b7juzEtTp6Jv" /><figcaption>GKE — Autopilot Cluster — Automation</figcaption></figure><p>Users can enable upgrade notifications by publishing them to Pub/Sub from the automation section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*snb2mp0P59IQ3hm4" /><figcaption>GKE — Autopilot Cluster — Automation — Upgrade Notifications</figcaption></figure><p>Vpc-native traffic routing, Intranode visibility, NodeLocalDNSCache, HTTP load balancing are pre-configured. The only editable parameter in this section is enabling control plane authorized networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yOxG_9wsptxAlzPr" /><figcaption>GKE — Autopilot Cluster — Networking</figcaption></figure><p>Autopilot enables usage of Shielded GKE Nodes by default which are built on top of Compute Engine <a href="https://cloud.google.com/security/shielded-cloud/shielded-vm">Shielded VMs</a>. Without Shielded GKE Nodes an attacker can exploit a vulnerability in a Pod to exfiltrate bootstrap credentials and impersonate nodes in your cluster, giving the attackers access to cluster secrets.</p><p>Workload Identity is also enabled by default, with this users can configure a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/">Kubernetes service account</a> to act as a <a href="https://cloud.google.com/iam/docs/understanding-service-accounts">Google service account</a>. Pods running as the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs. This enables you to assign distinct, fine-grained identities and authorization for each application in your cluster. No parameters in this section can be edited.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dNrVaj2G6d4MPHSw" /><figcaption>GKE — Autopilot Cluster — Security</figcaption></figure><p>The features section includes different add-ons which seem to be still in work-in-progress. No field in the section is editable. Autopilot by default enables Cloud Operations for GKE (monitoring and logging) and the CSI driver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h--5V9hT8Qb10Ar1" /><figcaption>GKE — Autopilot Cluster — Features</figcaption></figure><p>Operations Logging:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hmo2PsJCqJC2ihkc" /><figcaption>GKE — Autopilot Cluster — Logging</figcaption></figure><p>Monitoring:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LiwNNPbayIUuUk27" /><figcaption>GKE — Autopilot Cluster — Monitoring</figcaption></figure><p>As seen below the number of nodes is empty and the user cannot see any of the nodes in the VM instances section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wo8JBk2ODt3kR9ex" /><figcaption>GKE — Autopilot Cluster — Cluster Info</figcaption></figure><p>Autopilot by default creates a default pool with two nodes to accommodate all system components like kube-dns, kube-proxy, metrics-agent, stackdriver etc., Autopilot nodes use cos_containerd (Container-Optimized OS with Containerd) image for the managed which uses Containerd as the container runtime that is directly integrated with Kubernetes.</p><p>Users can list the nodes using Kubectl as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kw7FYMeRziDctGMM" /><figcaption>GKE — Autopilot Cluster — Kubectl</figcaption></figure><p>Users cannot connect to the Autopilot nodes using the external-ip above. In order to <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/autopilot-support">troubleshoot Autopilot nodes</a> a user should contact <a href="https://cloud.google.com/support">Cloud Customer Care</a> to obtain a member name that is required to access the cluster.</p><p>As shown below system pods are deployed on the two Autopilot nodes created above and the resource usage of these components are not added to the overall vCPU and memory usage (simply not billed). The kube-system namespace is managed, meaning that all resources in this namespace cannot be altered and new resources cannot be created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2cwrRjMWTv8owimk" /><figcaption>GKE — Autopilot Cluster — System Pods</figcaption></figure><p>Deploying a demo/sample application that includes multiple services with different resource requirements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AhlvVCTAXIqXDyQW" /><figcaption>GKE — Autopilot Cluster — Demo Application</figcaption></figure><p>As there are currently two nodes with resources that are insufficient for the application the pods fail to be scheduled initially as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X4oKt5otRPfRJFgd" /><figcaption>GKE — Autopilot Cluster — Demo Application</figcaption></figure><p>Node resource capacity of existing default-pool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ycf5LULou7uevocV" /><figcaption>GKE — Autopilot Cluster —Node Capacity</figcaption></figure><p>Pod scheduling failures because of insufficient resources:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hmo1gD0k57Ki9v2z" /><figcaption>GKE — Autopilot Cluster — Autoscaling</figcaption></figure><p>As shown below from the autoscaler logs Autopilot automatically scales the cluster’s resources (nodes) based on the pod specifications/requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WFsi5tkdrig5fzB8" /><figcaption>GKE — Autopilot Cluster — Autoscaling</figcaption></figure><p>As shown below the number of nodes is scaled up to 8 to accommodate the demo application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2PA2aYbqspZa9F-" /><figcaption>GKE — Autopilot Cluster — Autoscaling</figcaption></figure><p>Once the autoscale creates the required resources, the application components are scheduled normally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OsD0PVU46WYIQWXJ" /><figcaption>GKE — Autopilot Cluster — Autoscaling</figcaption></figure><p>To automatically increase or decrease the number of Pods, users can implement <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler?authuser=3">Horizontal pod autoscaling</a> using the standard Kubernetes CPU or memory metrics, or using custom metrics through Cloud Monitoring.</p><p>Autopilot clusters use resource requests to determine the infrastructure to provision. Autopilot does its best to make the Pods schedulable. If a user supplies a pod without requests or with values that stretch the constraints, Autopilot adjusts them to fit automatically. Google recommends providing values that best match the needs of user’s workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SMssc-W0TLUNI1ZN" /><figcaption>GKE — Autopilot Cluster — Resource Requests</figcaption></figure><p>The CPU, memory increment, ratio requirements and potential scale-up of resource requests are calculated after the defaults are applied to containers with missing resource requests. Autopilot relies on what the user specifies in the deployment configuration to provision resources. If resource requests are not specified for any container in the Pod, Autopilot applies default values. These defaults are designed to give the containers in your Pods an average amount of resources, which are suitable for many smaller workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6pEbadqyxHPkGkX9" /><figcaption>GKE — Autopilot Cluster — Default Resource Values</figcaption></figure><p><strong>Allowed resource ranges:</strong> Pod vCPU are available in increments of 0.25 vCPU. In addition to the minimum values, the CPU:memory ratio must be in the range of 1:1 to 1:6.5. Resources outside of the allowable ratio ranges will be scaled up. For ephemeral storage, the available range is between 10 MiB and 10 GiB. Ephemeral storage has a minimum request per container, so if the ephemeral storage requests for a container are less than the minimum, Autopilot increases the request to the minimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gksWRTOad7IBuNCH" /><figcaption>GKE — Autopilot Cluster — Default Resource Ranges</figcaption></figure><p>As shown below the service can be accessed using the LB. With Autopilot, the user deployed a multi-tier application without having to manage any nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dTMbasjcW2A4XqQ0" /><figcaption>GKE — Autopilot Cluster — Demo Application</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Pogvrzjv-0T1eii" /><figcaption>GKE — Autopilot Cluster — Autoscaling</figcaption></figure><p>The vCPU and memory requests of all the pods are shown in the clusters section which are the resources that will be billed apart from a flat fee of $0.10/h per cluster for each cluster after the <a href="https://cloud.google.com/kubernetes-engine/pricing#cluster_management_fee_and_free_tier">free tier</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E5d7zPnbSNNVHD5f" /><figcaption>GKE — Autopilot Cluster — Pricing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G_ET473FPmn7db1m" /><figcaption>GKE — Autopilot Cluster — Resource Evaluation</figcaption></figure><p>Autopilot integration with the marketplace is still in progress and there might be multiple integrations in the future with other GCP features. Autopilot currently supports Datadog for logging and monitoring and GitLab for CI/CD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cv6Y8Hxlpc2CSZVu" /><figcaption>GKE — Autopilot Cluster —Marketplace</figcaption></figure><p>Autopilot implements <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster">GKE hardening guidelines</a> and security best practices, utilizing GCP unique security features like <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes">Shielded GKE Nodes</a> and <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity">Workload Identity</a>. In addition, Autopilot blocks certain features deemed as less safe such as External IP Services or legacy authorization, disabling CAP_NET_RAW, and restricting specific cipher suite usage. By locking down individual Kubernetes nodes, Autopilot further helps reduce the cluster’s attack surface and minimizes ongoing security configuration mistakes. There are <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#other_limitations">limitations</a> and <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#unsupported_cluster_features">unsupported features</a> with Autopilot mode which are available in the standard mode of operation.</p><p>GKE in Autopilot mode provides strong security capabilities, ops-friendly configuration, improved resource utilization, and reduced Day-2 operational and maintenance costs. This feature-set makes Autopilot an easy-to-use packaged solution that can drastically reduce operations and costs. With Autopilot, users can make use of the same cutting-edge tooling for the nodes and pods used by Google SRE for managing the internal GKE environments.</p><p>Autopilot significantly reduces the costs incurred by under-usage of cluster resources. Google offers a 99.95% SLA for the control plane of its Autopilot clusters and a 99.9% SLA for Autopilot pods while reducing the overall operational load required for managing the clusters provides a hands-off fully managed Kubernetes experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p>👋 <a href="https://faun.dev/join"><strong>Join FAUN today and receive similar stories each week in your inbox!</strong></a><strong> </strong>️ <strong>Get your weekly dose of the must-read tech stories, news, and tutorials.</strong></p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d6ef338d464" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/google-kubernetes-engine-gke-autopilot-7d6ef338d464">Google Kubernetes Engine (GKE) Autopilot</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SuperEdge, OpenYurt — Extending Native Kubernetes to Edge]]></title>
            <link>https://itnext.io/superedge-openyurt-extending-native-kubernetes-to-edge-cc59094f92c?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc59094f92c</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[edge-computing]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Thu, 04 Feb 2021 12:21:16 GMT</pubDate>
            <atom:updated>2021-02-04T13:18:57.673Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*labLQp70gBrilfIWDzoZ7Q.png" /><figcaption>Extending Cloud-Native Capabilities to Edge — Cloud-Edge Tunnel, Edge Autonomy, Application Fleet Management</figcaption></figure><h3>SuperEdge, OpenYurt — Extending Native Kubernetes to Edge</h3><p>Enterprises need a robust management layer to effectively manage edge processes that provide dynamic orchestration and automation. However, the entire edge architecture consists of different hardware and software resources. The edge eco-system contains heterogeneous hardware scattered across multiple regions and positioned based on the requirement. According to the leading global research and advisory organizations, edge architectures have a hierarchical structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*54u9ImT0R0kbVoZH" /><figcaption>Gartner and IDC — Edge Topology Hierarchy</figcaption></figure><p>Whatever may be the kind of hardware in the hierarchy, Kubernetes can provide an agonistic infrastructure capable of seamlessly managing diverse workloads running on different compute resources providing a common set of services that can be centrally managed but executed in the edge environment. Kubernetes can orchestrate and schedule resources as they bounce from the cloud to the edge and vice versa.</p><p>An effective edge computing model should address the limitations of latency and bandwidth, deploying to heterogeneous environments using a unified management system, fleet management, network autonomy of edge sites, etc. There are considerable complexities involved in designing a centrally managed edge topology:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*pXkJCy-yCnC2GzBd" /><figcaption>Edge — Complexities</figcaption></figure><p>Kubernetes with specific tooling can efficiently address most of the issues. Managing edge nodes as Kubernetes nodes rather than hosting a full-fledged cluster with a control plane at each site provides advantages like minimizing the edge site resources by only deploying worker components, not having to use any other federation mechanism to control the edge sites centrally, control edge sites the same way as native Kubernetes nodes, etc. But moving the nodes to the edge with the current upstream design (Kubelet on each node will cache information in memory) has significant limitations and majorly depends on the connectivity between the nodes and the master hosted on some cloud over the unstable public internet.</p><p><a href="https://openyurt.io/en-us/">OpenYurt</a> (Alibaba’s first open-source, cloud-native project for edge computing, CNCF <a href="https://www.cncf.io/sandbox-projects/">sandbox project</a>) and <a href="https://github.com/superedge/superedge">SuperEdge</a> (Lead by Tencent along with Intel, VMware, Huya, Cambricon, Captialonline, and Meituan) are two homologous projects built based on native Kubernetes and targets to extend it to support edge computing seamlessly by extending the cloud native ability to the edge side. In a nutshell, enables users to manage applications that run in the edge infrastructure as if they were running in the cloud infrastructure. Both the projects provide the necessary controllers, components, and custom resources for cloud-edge tunneling, edge autonomy, intelligent and unified application management, large-scale application delivery, etc.</p><h3>Operations and Maintenance Tunnels</h3><blockquote><strong>Tunnel between kube-apiserver on the cloud and kubelet on edge</strong></blockquote><p>In the process of application deployment, operation, and maintenance, users often need to obtain application logs or directly log in to the running applications for debugging. This generally involves kubectl — logs, exec, edit, patch, get, describe actions.</p><p>There are two primary communication paths from the control plane (apiserver) to the nodes. The first is from the apiserver to the kubelet process which runs on each node in the cluster. The second is from the apiserver to any node, pod, or service through the apiserver’s proxy functionality. In general setting on a kubectl request, kubelet will act as the server side, responsible for processing the requests forwarded by kube-apiserver (KAS), which requires a network path between KAS and kubelet to allow KAS to actively access kubelet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/0*RNa2FyvBX_QgqsYW" /><figcaption>kube-apiserver — kubelet communication</figcaption></figure><p>However, in the edge node scenario, the edge nodes are often located in the local private network. Although this ensures the security of the edge nodes, this might pose problems for the KAS located in the cloud control node to directly access the kubelet located at the edge node. Therefore, to support the operation and maintenance of edge applications from cloud nodes, the requests have to be proxied to the edge node to solve the problem that the cloud cannot directly access the edge node.</p><h3>OpenYurt — YurtTunnel</h3><p><a href="https://github.com/alibaba/openyurt/blob/master/docs/tutorial/yurt-tunnel.md">Yurt-tunnel</a> consists of <a href="https://pkg.go.dev/github.com/alibaba/openyurt@v0.0.0-20200917165522-b447d2ebcca1/cmd/yurt-tunnel-server?tab=overview">TunnelServer</a> in the cloud and <a href="https://pkg.go.dev/github.com/alibaba/openyurt@v0.0.0-20201022001436-cdc6acf7c8ec/cmd/yurt-tunnel-agent">TunnelAgent</a> running on each edge node. TunnelServer establishes a connection with the TunnelAgent daemon running in each edge node through a reverse proxy, and establishes secure network access between the control plane of the public cloud and the edge node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONXosQeIQ2WLYDY0JhpGxw.png" /><figcaption>OpenYurt — YurtTunnel</figcaption></figure><p>Yurt-tunnel uses the upstream project <a href="https://github.com/kubernetes-sigs/apiserver-network-proxy">apiserver-network-proxy (ANP</a>) to enable the communication between server and agent. ANP is an alpha feature in 1.16 to address <a href="https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/egressselector/egress_selector.go">EgressSelector</a>, which aims to realize the cross-intranet communication of Kubernetes cluster components (for example, the master is located in the control VPC, and other components such as kubelet are located in the user VPC). A core design idea of ANP is to use gRPC to encapsulate all external HTTP requests of KAS. The choice of gRPC here is mainly because it supports streams and clear interface specifications. Also, the strongly typed client and server can effectively reduce runtime errors and improve system stability.</p><p>Compared to native TCP connections, ANP can not only provide higher transmission stability but also greatly reduce public network traffic. Considering the scale of tens of thousands of nodes in an edge cluster.</p><h3>SuperEdge — TunnelCloud and TunnelEdge</h3><p>SuperEdge supports self-built tunnels (currently supports TCP, HTTP, and HTTPS) to solve cloud edge connection problems in different network environments. Tunneling consists of <a href="https://github.com/superedge/superedge/blob/main/docs/components/tunnel.md">tunnel-cloud</a> and <a href="https://github.com/superedge/superedge/blob/main/docs/components/tunnel.md">tunnel-edge</a>, which enable maintaining a persistent cloud-to-edge network connection. SuperEdge uses <a href="https://github.com/superedge/superedge/blob/main/pkg/edgeadm/constant/manifests/tunnel-coredns.go">tunnel-dns</a> (coredns) component to forward specific requests addressed to the edge node to tunnel-cloud service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FOxEBqSROd23053x" /><figcaption>SuperEdge — TunnelEdge &amp; TunnelCloud</figcaption></figure><p>The tunnel-cloud is responsible for maintaining the network tunnel with the edge node tunnel-edge and tunnel-edge is responsible for establishing a network tunnel with the cloud edge cluster tunnel-cloud, accepting API requests, and forwarding them to the edge node component (kubelet).</p><p>Once tunnel-edge establishes a grpc connection with tunnel-cloud, tunnel-cloud will write the mapping of its podIp and nodeName of the node where tunnel-edge is located into DNS. If the grpc connection is disconnected, tunnel-cloud will delete the podIp and node name from the mapping. When APIServer or other cloud applications access kubelet or other applications on edge nodes, tunnel-dns forwards the request to the tunnel-cloud Pod through DNS hijacking (resolving the node name in the host to podIp of tunnel-cloud by replacing kube-apierver’s DNS nameservers with tunnel-coredns’s Cluster IP).</p><h3>Edge Node Autonomy</h3><blockquote><strong>Handling services on edge nodes in the event of network Disconnect and other disruptive scenarios (Local Autonomy)</strong></blockquote><p>In Kubernetes extended to the edge computing scenario, edge nodes will be connected to the control plane through the public network and the cloud. Considering factors such as the instability and cost of the public network, the edge requires that the edge business can continue to operate in a disconnected state or a weak network state.</p><p>In traditional Kubernetes system architecture, container information in the agent (kubelet) is saved in the memory. When the network is disconnected, business data cannot be obtained from the cloud. If a node or the kubelet is restarted, service containers cannot be restored. The Node Lifecycle Controller of Kubernetes decides whether to evict the pod on the node or set the taint according to the node’s lease and node status update time, and set the node ready condition to unknown.</p><p>There is a need for an agent in the edge nodes to continuously poll the KAS on the cloud master and cache the information locally on the node where the same can be used as a shadow apiserver in the event of a network outage, this also provides an enables the pods on the edge nodes to retrieve information from the agent rather reaching the control plane which significantly reduces the round-trips over public network.</p><p>The usual flow when the node is marked not-ready or unknown vs an autonomy enabled node is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/0*JsPN3U5TDCkK2asf" /><figcaption>Node Management and Autonomy</figcaption></figure><h3>OpenYurt — YurtHub</h3><p><a href="https://www.alibabacloud.com/blog/deep-dive-into-openyurt-yurthub-extended-capabilities_597190">YurtHub</a> is a transparent gateway with a data caching feature, kubelet on the edge nodes communicates with the cloud through YurtHub. This acts as a temporary configuration center on the node, continues to provide data configuration services for all devices and customer services on the node when the network connection is interrupted.</p><p>YurtHub provides support for a large number of native Kubernetes API’s and can provide ‘Shadow Apiserver’ capabilities in the dimensions of nodes and edge units, which is particularly valuable in edge computing scenarios with weak network links. When the network connection to the cloud is disconnected and the node or kubelet is restarted, data of service containers are obtained from YurtHub to ensure edge autonomy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*agCfsGaeu1brZkqM" /><figcaption>OpenYurt — YurtHub</figcaption></figure><p>YurtHub along with other resources caches network configuration resources, such as services, in the local storage enabling the services to be intact when accessed locally by providing the statuses and configuration information of objects from before the network interruption. When YurtHub takes over the traffic between edge nodes and the cloud, it can also take over node certificate management.</p><p>YurtHub can take over the cloud access traffic from both system components and service containers and implement node-based throttling on the cloud. In this case, when the number of concurrent requests to the cloud from a single node exceeds 250, YurtHub rejects new requests.</p><p><a href="https://www.alibabacloud.com/blog/introducing-main-feature-of-openyurt-local-autonomy_596322">Yurt-controller-manager</a> component provides few controllers such as the node controller and the unit controller (to be released) for different edge computing use cases. Users can label the edge nodes with ‘<em>openyurt.io/autonomy: true’ </em>where<em> </em>the pods in the nodes that are in the autonomy mode will not be evicted from the API server even if the node heartbeats are missing (disconnect from control plane). The <a href="https://github.com/alibaba/openyurt/blob/b6b02d77eb1f94964f1b700cf45b538ff0c16302/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L1493">nodelifecycleController</a> component manages the autonomy and controls the eviction of pods.</p><h3>SuperEdge — lite-apiserver</h3><p>The <a href="https://github.com/superedge/superedge/blob/main/docs/components/lite-apiserver.md">lite-apiserver</a> is a light-weight version of the kube-apiserver running on the edge nodes. It acts as a proxy for requests from all components and pods on the edge node to the cloud apiserver, and request-cache-controller of lite-apiserver caches the responses to achieve edge autonomy in case of the disconnected cloud -edge network. Kubelet on the edge nodes communicates with the cloud through lite-apiserver.</p><p>SuperEdge provides L3 level edge autonomy. When the edge node is unstable or disconnected from the cloud network, the edge node can still operate normally without affecting the edge services that have been deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*00bOSgbcVjroGzG_" /><figcaption>SuperEdge — Lite-Apiserver</figcaption></figure><p>SuperEdge provides <a href="https://github.com/superedge/superedge/blob/main/docs/components/edge-health.md">end-end distributed health check</a> capabilities. Each edge node will be deployed with an edge-health agent. Edge nodes in the same edge cluster will perform health checks on each other and vote on the status of the nodes. In this way, even if there is a problem with the cloud edge network, as long as the connection between the edge nodes is normal, the node will not be evicted. Every edge node runs <a href="https://github.com/superedge/superedge/blob/main/docs/components/edge-health.md">edge-health</a> by default and introduces distributed health check to achieve independent health check for each node group, which allows each node to detect and monitor nodes within the same group and eliminates the false alerts caused by the unstable cloud-to-edge network.</p><p>The entire design avoids migrations and reconstructions caused by the instability of the cloud-side network and ensures the stability of the service on the edge site.</p><h3>Application fleet-management across heterogenous edge nodes</h3><blockquote><strong>Intelligent and topology-aware application scheduling and maintenance</strong></blockquote><p>In the edge scenario, the distribution of worker nodes is relatively scattered. There may only be one worker node at the same physical location, and there may also be multiple worker nodes at the same physical location. Edge nodes usually have regional, zonal, hardware specific, or other logical grouping characteristics (such as the same CPU architecture, the same operator, cloud provider).</p><p>In the perspective of edge nodes, there is a need to divide the nodes into different groups/pools to represent the same set of features. This grouping can be achieved by Kubernetes native way of adding same label, the annotation, and taint. Both OpenYurt and SuperEdge provide a robust application fleet management system where users can add a logic to deploy applications to specific sites based on the requirement.</p><h3>SuperEdge — application-grid-controller</h3><p>The <a href="https://github.com/superedge/superedge/blob/main/pkg/application-grid-controller/controller/service/controller.go">application-grid-controller</a> is responsible for managing the ServiceGroup which constitutes <a href="https://github.com/superedge/superedge/blob/main/pkg/application-grid-controller/controller/common/deploymentgrid-crd.go">DeploymentGrids</a> and <a href="https://github.com/superedge/superedge/blob/main/pkg/application-grid-controller/controller/common/servicegrid-crd.go">ServiceGrids</a>, the two CRD’s implemented by SuperEdge to manage application deployments across edge nodes.</p><p>SuperEdge enables users to use NodeUnit and NodeGroup to segregate nodes into a pool, where the NodeUnit is the fundamental object which differentiates the nodes based on the labels and the NodeGroup can group multiple NodeUnits. These NodeGroup’s are used in ServiceGroup which provides two CR’s: deployment-grid and service-grid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G4wZRA55gKg2u8Ye" /><figcaption>SuperEdge — Application Management</figcaption></figure><p>The deployment-grid and service-grid provided by ServiceGroup can be used to deploy and manage edge applications, control service flow, and ensure the number of services in each region and disaster recovery.</p><p>The format of the DeploymentGrid is similar to that of deployment. The ‘&lt; deployment template &gt;’ field is the template field of the Kubernetes original deployment object, the specific addition is the <em>gridUniqKe</em>y field, which indicates the key value of the label of the node group created above. For example, if a user creates a NodeGroup using ‘region:us-**’ then the value of gridUniqKey will be region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5ow751IsvnyBuWyV" /><figcaption>SuperEdge — DeploymentGrid &amp; ServiceGrid</figcaption></figure><p>The format of ServiceGrid is similar to that of service. The ‘&lt; service template &gt;’ field is the template field of the Kubernetes original service. The <em>gridUniqKe</em>y can be used in the ServiceGrid spec to group specific NodeGroups as endpoints under a service. For example, if a user selects the value of the key to be ‘region’ then the service endpoints only constitute the pods deployed in specific NodeGroups differentiated by region label. When a service tries to access the application using the ClusterIP of the service created by ServiceGrid from a specific node in the NodeGroup then the traffic will be only distributed to the endpoints in the same NodeGroup.</p><p>ServiceGrid enables users to create a local domain, avoiding cross-regional access of services, controlled access to specific services, control connection thresholds, and also facilitate easier release and management of edge cluster services.</p><p>If new NodeUnit’s are added to the cluster after the DeploymentGrid and ServiceGrid resources are deployed, the system will automatically create the corresponding deployment and service for the new NodeUnit.</p><h3>OpenYurt — Yurt-App-Manager</h3><p><a href="https://github.com/alibaba/openyurt/blob/master/docs/tutorial/yurt-app-manager.md">Yurt-App-Manager</a> is a standard extension of Kubernetes. It can be used in conjunction with Kubernetes to provide two custom resources: <a href="https://github.com/alibaba/openyurt/blob/54daca0801a8f645c2832390076fac5e0bb70455/pkg/yurtappmanager/client/clientset/versioned/typed/apps/v1alpha1/nodepool.go">NodePool</a> and <a href="https://github.com/alibaba/openyurt/blob/54daca0801a8f645c2832390076fac5e0bb70455/pkg/yurtappmanager/client/clientset/versioned/typed/apps/v1alpha1/uniteddeployment.go">UnitedDeployment</a>.</p><p>NodePool is a pool, a group, or a unit of nodes. The hosts are classified and managed using labels to group the worker nodes that have the same attributes. A NodePool groups the nodes based on similar labels in different edge areas. When a node is added to a NodePool, it inherits the annotations, labels, and taints defined in the nodepool Spec, and a tag ‘<em>apps.openyurt.io/nodepool:&lt;nodepool-name&gt;’</em> will be added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HB6e8zY3HlMuNjGa" /><figcaption>OpenYurt — NodePool</figcaption></figure><p>The UnitedDeployment controller provides a template to define the application and match different regions based on the NodePool labels. The workloads in each region under each UnitedDeployment are called Pools. Currently, pools support two types of workloads: StatefulSet and Deployment workloads. The controller creates a sub-workload resource object based on the pool configuration of the UnitedDeployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Dm1wxtiaBZ-ahNrO" /><figcaption>OpenYurt — UnifiedDeployment</figcaption></figure><p>UnitedDeployment enables k8s native Deployment, Statefulset, and Daemonset resources to deploy intelligently according to specified NodePool. Through a UnitedDeployment instance, multiple Deployment or Statefulset resources can be automatically maintained, and it can also have differentiated configurations such as replicas. Users can make changes to a specific UD object spec to manage the configuration of applications across multiple edge sites without having to change multiple traditional deployment or statefulset objects.</p><h3>Tryout</h3><h3>Cluster Topology</h3><p>The topology comprises a master node running on a public cloud provider with public_ip and three edge nodes (udoo-x86 boards) connected to a home network (private). The same cluster is used for both SuperEdge and OpenYurt as both are native Kubernetes extensions and users can re-purpose the clusters by tearing down the control plane and edge components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rGg79vbLvmg48mNj" /><figcaption>Master on Public Cloud</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VRYt3wH_396pypUW" /><figcaption>Nodes on Private Edge and Cloud Master Node</figcaption></figure><p>As shown below the public-cloud-master serving Kubernetes apiserver on public_ip and all edge nodes are in the private network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uSBS1FV_aQySD5wv" /><figcaption>Nodes on Private Network and Master on Public Network</figcaption></figure><h3>OpenYurt</h3><p>The components of OpenYurt on the cloud side and edge are as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8s0dT4z082iqKvuc" /><figcaption>OpenYurt — Architecture</figcaption></figure><p>Users can use <a href="https://github.com/alibaba/openyurt/blob/master/cmd/yurtctl/yurtctl.go">Yurtctl</a>, a centralized installation and management tool to quickly convert any Kubernetes cluster to OpenYurt enabled cluster (or) all the components can be installed manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*MK4sZ9ALBI-2gxgn" /></figure><p>OpenYurt depends on a node label ‘<em>openyurt.io/is-edge-worker</em>’ to install the components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Iw7NeMm6OTN_vFys" /><figcaption>OpenYurt — Node Labels to Identify Edge Nodes</figcaption></figure><p>The controller-manager, app-manager, and tunnel-server are deployed on the cloud and all the edge nodes are deployed with tunnel-agent and YurtHub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*myAE80_Qn6fE6Ej6" /><figcaption>OpenYurt — TunnelServer and TunnelAgents</figcaption></figure><p>The tunnel-server service on public-cloud master is exposed as a NodePort to enable connectivity to all the tunnel-agents on the edge nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7_1g7UTgp5Vt7PRo" /><figcaption>OpenYurt — TunnelServer Service-NodePort</figcaption></figure><p>A sample application that shows the node information is deployed on one of the edge nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2xlBEiYzk_YiaiIc" /><figcaption>Sample Application on Edge Node</figcaption></figure><p>The service can be accessed using the Service:ClusterIP from the local network as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DMVm_L7KTW9aaUtx" /></figure><p>As there are no OpenYurt components installed yet on the edge nodes (no tunnels-agents), Kubectl operations like exec and logs fail when accessed from cloud master.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*99VEx1X_Ew6UFoDr" /><figcaption>Failure — Accessing Application on the Edge Node</figcaption></figure><p>Once the nodes are labeled with OpenYurt required labels, tunnel-agents are installed on the edge node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0gfsGU1P_79ww0IJ" /><figcaption>OpenYurt — Tunnel</figcaption></figure><p>The tunnel-server on the cloud master accepts and approves the tunnels from tunnel-agents on the edge nodes verifying the certificate identity. Users can generate specific certs for the reverse proxy connection or can use the apiserver and kubelet certs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fPEcPmDI4YpOu8ZR" /><figcaption>OpenYurt — TunnelServer</figcaption></figure><p>Once the tunnel-agent successfully connects to tunnel-server on the cloud master all the Kubectl management operations on the edge node objects can be performed from the cloud master where the edge node is located in a totally isolated private network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WUjY0rRal2Ubqjkc" /><figcaption>Accessing Application on the Edge Node Through Tunnel</figcaption></figure><p>The edge nodes are deployed with the YurtHub component which continuously caches the cluster state information from the cloud master. All the Kubelet to KAS communication will be forwarded through the hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yIIsYAEr_Yr2qhOm" /><figcaption>YurtHub on the Edge Node</figcaption></figure><p>The yurt-controller-manager on the cloud master maintains the list of registered nodes and also manages other functions related to autonomy and app management through OpenYurt CRD’s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZIRb9Xi3lEM3MFsw" /><figcaption>Yurt-Controller-Manager on Cloud Master</figcaption></figure><p>OpenYurt installs two custom resources in the cluster: NodePool and UnitedDeployment which are used for application deployment and management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UDM2dpG0eLLTUf7W" /><figcaption>CRD’s — OpenYurt Application Management</figcaption></figure><p>A NodePool resource is created where the NodePool groups the nodes with labels ‘apps-pool-r’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E3WuC_6ssB6dKp38" /><figcaption>OpenYurt — NodePool</figcaption></figure><p>Nodes to be grouped under the NodePool can be added to the pool by labeling the nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y78_swQ2IYeexnca" /><figcaption>OpenYurt — NodePool — Node Labels</figcaption></figure><p>The status section of the NodePool CR shows all the nodes which are part of the pool as shown below. In the sample configuration, the NodePool only includes edge-node-1 and edge-node-3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h75GYhxiBGl-2Npk" /><figcaption>OpenYurt — NodePool</figcaption></figure><p>The yurt-app-manager on the cloud master maintains the information of all NodePools and the United Deployments created in the cluster and manage the resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kYBKaUmg9D6sMvjE" /><figcaption>OpenYurt — Yurt-App-Manager on Cloud Master</figcaption></figure><p>A UnitedDeployment resource is created with a workloadTemplate to deploy a sample Nginx application on the NodePool created above as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i7unICjU3rUyAbkY" /><figcaption>OpenYurt — UnitedDeployment</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OQZjZiptexnBiUrj" /><figcaption>OpenYurt — UnitedDeployment Controller</figcaption></figure><p>The UD creates required Kubernetes deployment and positions the pods on the nodes which are part of the NodePool above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V-UVYWOuJXc-kk-M" /><figcaption>OpenYurt — UnitedDeployment</figcaption></figure><p>As shown below the pods (replica: 3) are only placed on edge-node-1 and edge-node-3 which are part of the ‘<em>NodePool:apps-pool-r</em>’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ehMZnoY474DFj2o9" /><figcaption>OpenYurt — UnitedDeployment</figcaption></figure><p>All deployments created by the UD resource can be centrally managed using UD, for example editing the replica count and image in the UD object as shown below. The same will be applied to all the pods part of the deployment created by UD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XwFO1ogiP208voJf" /><figcaption>OpenYurt — UnitedDeployment</figcaption></figure><p>A network outage scenario is created by disconnecting the edge-node-3, as shown below the application pods are not evicted when a node transitions to NotReady state, the YurtHub component in the edge-node continues to provide necessary network information enabling the service to be available for the applications/users in the local network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j89q2Js2CnFYOzGz" /><figcaption>OpenYurt — Edge Node Autonomy</figcaption></figure><h3>SuperEdge</h3><p>The components of SuperEdge on the cloud side and edge are as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*fsxwWITCnE9M0AYc" /><figcaption>SuperEdge — Architecture</figcaption></figure><p>SuperEdge can be provisioned using <a href="https://github.com/superedge/superedge/blob/main/docs/installation/install_via_edgeadm.md">edgeadm</a> CLI (prerequisite is to have a cluster bootstrapped using Kubeadm) or users can manually generate required certs and deploy the components using the manifests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z4plzPI9NVt2VRn4" /><figcaption>SuperEdge — Components</figcaption></figure><p>SuperEdge deploys tunnel-edge on all the edge-nodes and tunnel-cloud on the cloud master node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-rL9Onyshk4heazA" /><figcaption>SuperEdge — TunnelEdge and TunnelCloud</figcaption></figure><p>A secure connection will be used for tunneling, tunnel-edge-conf, and tunnel-cloud-conf Config Maps hold configuration and certificate information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zxJVIeQ5u80TMEZG" /><figcaption>SuperEdge — TunnelEdge and TunnelCloud</figcaption></figure><p>Kubelet and apiserver certs for tunnel-edge and tunnel-cloud:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jfkC2KlW6JvmEHyu" /><figcaption>TunnelEdge and TunnelCloud Certificates</figcaption></figure><p>The other Config Maps include tunnel-coredns config which is used in tunneling the proxies from kube-apiserver to edge nodes. tunnel-nodes config includes all the registered edge nodes which is used by tunnel-cloud to track and manage the connectivity of tunnel-edge sites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rTM_J1EPIT5HYxEh" /><figcaption>SuperEdge — Tunnel-Coredns</figcaption></figure><p>When the tunnels are established applications deployed on the edge nodes can be managed using the Kubectl operations as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SZf4E0Y2mEUyAygh" /><figcaption>SuperEdge — Accessing applications on edge nodes</figcaption></figure><p>SuperEdge installs two custom resources: DeploymentGrid and ServiceGrid used for application deployment and management. Both resources are namespace scoped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0G7lNGBynZXGOlt4" /><figcaption>SuperEdge — CRD’s — Application Management</figcaption></figure><p>To test the application management functionality, node-1 and node-2 are labeled with ‘<em>region1:us-west</em>’, and node-3 is labeled with ‘<em>region1:us-east</em>’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DXHcVKWpk5Fs0PzY" /><figcaption>SuperEdge — Node Labels</figcaption></figure><p>The <em>gridUniqKey </em>in the sample<em> </em>DeploymentGrid resource is mapped to ‘region1’ which is the key used in the node labels above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EwOuXfoqJ8-PK4De" /><figcaption>SuperEdge — DeploymentGrid</figcaption></figure><p>The DeploymentGrid creates a Kubernetes deployment with 2 replicas on each region as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5K5lJl-5BypQ4dt_" /><figcaption>SuperEdge — DeploymentGrid</figcaption></figure><p>A sample ServiceGrid resource is created with the same <em>gridUniqKey:region1 </em>as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KFz6SQznHN8f19n5" /><figcaption>SuperEdge — ServiceGrid</figcaption></figure><p>ServiceGrid creates a Kubernetes Service object which maps all the four pods (endpoints) running in two regions (3 nodes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tznhILtyxJfWh1Ba" /><figcaption>SuperEdge — ServiceGrid</figcaption></figure><p>When the service is accessed using the ClusterIP from edge-node-1 or edge-node-2 the requests are only served by pods in ‘region:us-west’ (pods on node-1 and node-2 which are part of the NodeGroup), similarly, traffic from edge-node-3 will be only catered by the pods in ‘region:us-east’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VZxsAHrqbsgEnEJA" /><figcaption>SuperEdge — ServiceGrid</figcaption></figure><p>A network outage scenario is created by disconnecting the edge-node-2, as shown below the application pods are not evicted when a node transitions to NotReady state, the lite-apiserver component in the edge-node continues to provide necessary network information enabling the service to be available for the applications/users in the local network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d5E1YQ3fIYmYMdn_" /><figcaption>SuperEdge — Edge Node Autonomy</figcaption></figure><p>The edge-health runs on every edge node by default for accurately determining the actual operation status of edge nodes and edge-admission: assists Kubernetes controllers by providing real-time health check status from edge-health services distributed on all edge worker nodes to identify the real status of the node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0nEPdSaJoecqIscm" /><figcaption>SuperEdge — Edge Node Autonomy</figcaption></figure><p><a href="https://gokulchandrapr.medium.com/kubeedge-extending-kubernetes-to-edge-dcfedd91f5f9"><strong>KubeEdge</strong></a><strong>, </strong>initially from Huawei-the first Kubernetes based edge computing project hosted under the CNCF also has overlapping features with the projects above, the main difference is that these projects are native to Kubernetes and are not intrusive to Kubernetes source code, so it is not necessary to consider the compatibility with Kubernetes version and ecology in theory. Apart from edge-specific capabilities, these projects provide edge autonomy and topology-aware application management capabilities.</p><p>Gartner predicts that by 2025, more than 75% of enterprise-generated data will be created and processed outside the data center or cloud. These projects can greatly address a multitude of edge use-cases and empower Kubernetes usage in edge. The projects are still in development mode and contributions from the community can greatly improve the adoption and usage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc59094f92c" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/superedge-openyurt-extending-native-kubernetes-to-edge-cc59094f92c">SuperEdge, OpenYurt — Extending Native Kubernetes to Edge</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Virtual Application Networks (VAN) for Multi-Cloud, Multi-Cluster and Cloud-Edge Interconnect]]></title>
            <link>https://itnext.io/virtual-application-networks-van-for-multi-cloud-multi-cluster-and-cloud-edge-interconnect-1f63a8081f41?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f63a8081f41</guid>
            <category><![CDATA[multicloud-management]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[edge-computing]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Thu, 14 Jan 2021 20:07:09 GMT</pubDate>
            <atom:updated>2021-01-14T21:38:35.807Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cfTOvnj6dhlTgLywFa39Ew.png" /><figcaption>Virtual Application Networks — Connecting Distributed Applications Across Clouds, Clusters and Networks</figcaption></figure><p>Modern software development is moving away from client/server toward more flexible architectures with services located everywhere. There are various reasons for deploying or replicating an application in multiple locations: geographically distributed applications for enhanced performance and availability, maintain compliance, connected vehicles, local breakouts in 5G, remote edge sites, etc. This requirement makes application-oriented multi-cloud and multi-cluster connectivity an inevitable trend of cloud computing.</p><p>VPN technologies like OpenVPN, Strongswan, Wireguard etc. are widely used to create secure site-to-site connectivity between sites, although VPN is conformant in facilitating connectivity between remote-sites there are caveats and and complexities involved. Some key caveats: VPNs can obfuscate the network path, increased network latency, monitoring the network path through a VPN tunnel is a challenge, as new sites are added the number of site-to-site connections to manage grows exponentially — scaling sites becomes impractical to maintain, cloud-native applications change the topologies based on the advanced scheduling framework where the connectivity has to auto detect the same and lastly the configuration and management of VPN is complex.</p><p>Virtual Application Network (VAN) enables advanced communication capabilities between distributed software components in a cloud-native application setting without having to deal with complex layer 3 network controls like VPNs, firewall rules, and access policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*32PUizW1tnQR_To_" /><figcaption>Virtual Application Network (VAN)</figcaption></figure><p>VAN is an application-layer network that is overlaid on the TCP/IP-based internet. A VAN can be as small as a single laptop or as big as a global network spanning time zones and continents. A VAN can be set up quickly by a non-privileged developer and its topology can be added to and changed just as easily. Applications written to use existing protocols like HTTP, gRPC, AMQP, or anything that runs over TCP or UDP can run on a VAN without modification. VAN addressing uses arbitrary strings to refer to endpoints directly without depending on the underlying host and port information.</p><p>The <a href="https://github.com/skupperproject">Skupper Project</a> is a free and open-source implementation of a VAN built for Kubernetes. Skupper can enable connectivity between applications running in different clusters with simple configuration without having to worry about elevated privileges, cluster access, or admin access (user just need access to specific namespaces that he is allowed to use). Skupper can cater wide range of use cases, some of them are as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w2ZzcIYCXf0a87B1" /><figcaption>Use Cases — VAN</figcaption></figure><p><strong>VAN address is a DNS-style name and references a process</strong></p><p>Traditional TCP/IP addressing uses a ‘<em>host:port</em>’ pair to refer to an endpoint, legacy TCP/IP internet is fast and reliable at moving data from one point to another, but it is not well suited to support the modern needs of cloud-native software systems. In the modern setting of containers and serverless computing, the notion of a host for a port adds an extra overhead and cannot scale. Coordinating port allocations across multiple developers or teams that provide containers is very difficult to do at scale, and exposes users to cluster-level issues outside of their control.</p><p>VAN addressing follows DNS style naming where the address is a name. The VAN routes application traffic based on the VAN address (simply ‘name’) without regard for the underlying IP addresses that are used by the systems hosting the application. A VAN address references a running process whereas an IP address references the host on which the process runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HX2skcqB6AC4IL75" /><figcaption>TCP/IP vs VAN Addressing</figcaption></figure><p><strong>VAN addresses are multi-access</strong></p><p>IP addressing is primarily unicast, with each address referring to a single host. IP does have a notion of multicast addressing where IPV4 and IPV6 support multicast and IPV6 supports anycast, but it is not used in practice beyond the scope of a local area network. VAN addressing is either <em>anycast</em> or <em>multicast</em>. It builds in the idea that multiple software components can attach to the network with the same address for multicast delivery or for load balancing. VAN multicast and anycast is not constrained by component location and therefore provides network-wide multicast delivery and anycast load balancing.</p><p><strong>Secured site-to-site connections</strong></p><p>An application running on a VAN benefits from the security mechanisms inherent to the network. All of the inter-site connections in a VAN are locked down using mutual TLS (Transport Layer Security) with a private, dedicated certificate authority. Access from the outside world only occurs where the developer wishes to provide public access to a portal or web front-end. There is no incidental exposure of services to the internet.</p><h3>VAN Router — Apache Qpid Dispatch Router</h3><p>A VAN is established by deploying a VAN router (dispatch router) in each site that will be part of the distributed application namespace and by establishing secure connections between pairs of sites. A site can be a LAN in a data center, a single host, or a namespace in a Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*si5qCIkRGqsbN5Sv" /><figcaption>VAN Routers Connectivity</figcaption></figure><p>Skupper uses the <a href="http://qpid.apache.org/components/dispatch-router/">Apache Qpid Dispatch Router</a> as its VAN router. The router is based on the AMQP protocol and is lightweight and stateless. In addition to the base router, Skupper provides tools to facilitate network setup and to provide mappings from commonly used protocols like HTTP, gRPC, and TCP onto the VAN network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s6nf4GOCzB4ocLjz" /><figcaption>Apache Qpid Dispatch Router</figcaption></figure><p>Dispatch Router is a lightweight AMQP message router for building scalable, available, and performant messaging networks. Dispatch Router flexibly routes messages between any AMQP-enabled endpoints, including clients, servers, and message brokers. AMQP apart from being an open standard protocol that allows messaging interoperability between systems, it is a general-purpose networking protocol that has real advantages over HTTP for service and API delivery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*RvFxouDoihZTZobe" /><figcaption>Calculating the least-cost path — LSRP</figcaption></figure><p>The network of routers uses Link-State routing protocol, similar to OSPF or IS-IS, to detect topology and compute least-cost paths between each pair of sites. The routing protocol quickly and automatically reacts to changes in the topology. This serves to provide resilience against failure and also makes it easy to add locations, and remove locations from the network as expansion or changes are needed.</p><p>In the sample topology below Private Network-A and Private Network-B are two different sites with two different private networks. Client (C1) is in Private Network-B and backend (B2) is in Private Network-A, with dispatch routing client C1 can access queues, topics and other AMQP services that are in Private Network-A using the connectivity established between routers in Private Network-B and Public Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DGwkJH2GHdom2NLM" /><figcaption>Qpid dispatch router connectivity</figcaption></figure><p>For example, The router Ra and Rp can be configured to expose queues in broker B2 and broker B1 to the network. Client C1 can then establish a connection to Rb, the local router, open a subscribing link to “b1.event-queue” and “b2.event-queue”, and receive messages stored on that queue in broker B1 &amp; B2.</p><p>In this scenario, neither B1, B2 nor C1 have been modified in any way and neither need to be aware of the fact that there is a message-router network between them.</p><h3>Skupper — Components</h3><p>The control-plane components of Skupper include a service-controller and router. Both the components are deployed as deployments in namespace of interest. The router component is actual Apache Qpid Dispatch Router and the service-controller facilitates syncing of exposed services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HhNxddNEiZEbBw73" /><figcaption>Skupper — Components</figcaption></figure><p>The router facilitates connections to other routers across sites and depends on the bridge configuration (configmap which will be programmed when connectors are configured) to retrieve any configured connectors. A connection is established using a token generated and the token is namespace scoped using ‘<em>set-context — current — namespace</em>’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3sOfg_GteNDe77Kl" /><figcaption>Skupper Router</figcaption></figure><p>Service controller tracks information of exposed services. The skupper-services configmap in the namespace is programmed with list of exposed services when the user configures the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kb7SS6KsarmEolmC" /><figcaption>Skupper Service Controller</figcaption></figure><p>All router related config is provided as config map to the router deployment. skupper-site comprise site-name and other global configuration parameters. skupper-internal and skupper-services programmed by the controller when a user exposes a service to be available in remote sites, this can be configured using skupper CLI (skupper expose deployment) or providing annotations (<em>skupper.io/proxy and skupper.io/port</em>) to the deployments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jNf7MQ9PSkWOZRbr" /><figcaption>Skupper — Configmaps</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*59BMCFGv8r_ziB26" /><figcaption>Skupper — Site Config</figcaption></figure><p>All inter-site connections to other routers in a VAN are locked down using mutual TLS (Transport Layer Security) with a private, dedicated certificate authority provided as secrets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_WDFLqGT5Iv7R89h" /><figcaption>Skupper — Secrets</figcaption></figure><p>In public cloud setting the skupper-controller (router) service is exposed as a service-type load balancer with public connectivity as it is required for private sites to peer. In private/edge setting the only requirement is to have egress internet connectivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WvVxzJeOn3NkBsVa" /><figcaption>Skupper — Services</figcaption></figure><p>The skupper-controller provides two consoles one being the Apache Qpid Dispatch Console and other is Skupper console (both of these can be activated with skupper-site configmap or passing the parameters with ‘<em>skupper init</em>’). Apache Qpid Dispatch Console provides vast information related to the router and the associated connections along with concrete monitoring and schema information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kwVzWHGSwrC0DbvG" /><figcaption>Qpid — Console</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nxAeNfWceXISbv8s" /><figcaption>Qpid — Console</figcaption></figure><p>The connections section provides list of all router-router connections, local connections and AMQP based connections established with other routers in the network. The topology section provides a graphical view of routers in the network and the message flow information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GeFV9HoeD02AzyTV" /><figcaption>Qpid — Console</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sIgxfjEYZERSJpyf" /><figcaption>Qpid — Console — Topology</figcaption></figure><p>Skupper console provides much simple and informative information on the connected sites and detailed graphical view of deployments and services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GvWj--tO8uT52s-8" /><figcaption>Skupper — Console</figcaption></figure><h3>Try Out — Creating a VAN between public cloud and edge Sites with private connectivity</h3><p>The test topology below includes a GKE cluster, EKS cluster and two private clusters bootstrapped using Kubeadm, Skupper just needs a Kubernetes cluster and doesn’t depend on the platform/distro or provisioner.</p><p>In the topology below there are four individual K8’s clusters spanning across cloud platforms and networks. GKE and EKS contain <em>skupper-controller</em> exposed as a LoadBalancer and the routers are peered with each other. The other two private clusters (no public connectivity from internet) peer with the router on the GKE cluster. Private clusters can be in different isolated networks, in the example below although both the private clusters are in the same network they are not peered between each other (simulating remote sites with no connectivity between each other).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dT8uA3tcZ0EQjYjQ" /><figcaption>Test Topology</figcaption></figure><h3>1. Connecting GKE and EKS</h3><p>GKE Cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LON1nPJ6ygirGGEi" /><figcaption>GKE CLuster</figcaption></figure><p>EKS Cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fR1BbuBMuKazoOVZ" /><figcaption>EKS Cluster</figcaption></figure><p>Skupper router and service controller on GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xKN_9AXCiP4TCy5c" /><figcaption>Skupper components on GKE</figcaption></figure><p>Skupper router and service controller on EKS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6K5iz3NVvEOb4tIe" /><figcaption>Skupper components on EKS</figcaption></figure><p>Skupper controller exposed as ‘<em>Type: External Load Balancer’</em> on GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W8h6iCXZo35pKAyN" /><figcaption>Skupper controller service LB</figcaption></figure><p>Connecting routers is authenticated using a namespace scoped connection-token. This can be generated using skupper CLI (skupper connection-token) or by creating a resource — ‘<em>skupper.io/type: connection-token-request’ </em>which will yield the connection-token<em>. </em>In this case, a connection token is created on GKE and used in EKS for establishing the connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NYsuRLieFbp9S_g8" /><figcaption>Skupper — Connection Token</figcaption></figure><p>Once the connection is established the infrastructure AMQP message flow will start between the dispatch routers. Users can use the QPID console to gather various information on the topology, logs and flow information.</p><p>EKS peering with GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Xmbc8FIhrS5omRZ" /><figcaption>Qpid Dispatch Router — Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iW_Sd98dHAAzcv9Q" /><figcaption>Qpid Dispatch Router — Console</figcaption></figure><p>Message flow visualization between EKS and GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qSKHaSNKjkVNBbrA" /><figcaption>Qpid Dispatch Router — Visualization</figcaption></figure><p>Dispatch router on EKS connecting to skupper-controller service exposed as LoadBalancer on GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M44HYv4dHmXUcSPC" /><figcaption>Qpid Dispatch Router — Console</figcaption></figure><p>Skupper console portal with connected site information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dBDivdqPp65VgjWB" /><figcaption>Skupper — Console</figcaption></figure><h3>2. Connecting edge-west and edge-east to GKE</h3><p>Clusters edge-west and edge-east are private clusters running on Udoo X86 connected to home network.</p><p>As shown below the edge-west router peers with the router on the GKE using the public service endpoint. In this case the GKE connection token is used to create the connection between edge-west and gkeuseast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z7ET039c0Bz0EUlO" /><figcaption>Exposing Services — VAN</figcaption></figure><p>A SSL profile is created for each connector and skupper-router in the edge sites establish a secured mutual TLS connection with router on GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kKAdaXsNKmeHseWV" /><figcaption>Exposing Services — VAN</figcaption></figure><p>Skupper console view of connected sites:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pXFWOw1aZBtn339v" /><figcaption>Skupper — Connected Sites Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8u7ZaLbnYCf3M7-Z" /><figcaption>Skupper — Connected Sites Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f1UiOWnmZIixZ2Mq" /><figcaption>Skupper — Connected Sites Topology</figcaption></figure><p>QPID topology view:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rnx8VmNWoJ_H-vgL" /><figcaption>Qpid Router — Connected Sites Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vkjRp9CF_mwKTKna" /><figcaption>Qpid Router — Connections</figcaption></figure><p>Qpid console — flow metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qceNqkMxXmZkvZA1" /><figcaption>Qpid Router — Connections</figcaption></figure><p>Similarly the fourth cluster/site edge-east is also peered with gkeuseast as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9pazbOq-cy_BG7yA" /><figcaption>Skupper — Connected Sites Topology</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kkpxImyv1S1kQVPD" /><figcaption>Qpid Router — Connected Sites Topology</figcaption></figure><p>The topology now has a VAN connectivity between two clusters with public connectivity to Skupper service and two private clusters without ingress connectivity from internet. No specific ports were opened and the TLS connections are established between the routers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GrcNOR2172aXFvIi" /><figcaption>Skupper — Message Flow Monitoring</figcaption></figure><h3>Try Out — VAN connectivity between individual Hipster services distributed across above four connected clusters</h3><p><a href="https://github.com/GoogleCloudPlatform/microservices-demo">Hipster Shop</a> is composed of many microservices written in different languages that talk to each other over gRPC. This application deployment topology includes services which are part of the application (frontend, backends and databases) in different namespaces, on different clusters located in different platforms. Technically, this means that they have no way to communicate unless they are exposed to the public internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HcnAxzevP8AbhTSb" /><figcaption>Sample Hipster application across clusters</figcaption></figure><p>Introducing Skupper into each namespace facilitates creation of a virtual application network that can connect services in different clusters. Any service exposed on the application network is represented as a local service in all of the connected namespaces. When the frontend sends a request to all other the backend entities, Skupper forwards the request to the namespace where the backend is running and routes the response back to the frontend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nSw1wX6NvJANBLHU" /><figcaption>Skupper Router-Service Interaction</figcaption></figure><p>Skupper enables users to configure the protocol (tcp, http, or http2. default ‘tcp’) to be used for communication along with the port. In the above example the connectivity of services to the database is ‘tcp’, so users can configure specific protocol by annotating the service object with ‘<em>skupper.io/proxy=tcp</em> and <em>skupper.io/port=6379’.</em></p><p>The frontend service is deployed on <em>gkeuseast</em> (GKE) cluster, there is no need to expose this service to Skupper as this is user facing and will be accessed using the LoadBalancer IP (public_ip) provided by GKE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xO27waHOmd00kwJ8" /><figcaption>Fronetend App — GKE</figcaption></figure><p>Services: cart, checkout, currency and redis-cart are deployed on EKS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7JdcfHvGfCcGDU-o" /><figcaption>Services — EKS</figcaption></figure><p>Services: productcatalog and recommendations are deployed on edge-west</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XVAMYAUalSJYc6n5" /><figcaption>Services — Edge1</figcaption></figure><p>Services: emailservice, paymentservice and shippingservice are deployed on edge-east</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uzl1Z_O-UOxOc4gN" /><figcaption>Services — Edge2</figcaption></figure><p>As seen below the hipster application is non-functional as other services in different clusters are not exposed to Skupper yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wh563Rp9blHJsuMK" /><figcaption>Hipster — Non-functional application</figcaption></figure><p>Services productcatalog and recommendations are exposed to VAN on edge-west.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A74xSf-iGBWoD53Z" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>Once the services are exposed, services send requests to skupper-router which are then forwarded to other routers in the network as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5tcywQABp0lX1wO_" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>The controller adds the skupper-internal configmap <em>bridges.json and </em>skupper-services configmap file with connector, corresponding address, protocol and port information of the services in the namespace where the specific router is located which will be used by the skupper-router and service-controller for configuring connections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3fskgOGFsWPw65a6" /><figcaption>Skupper — Exposing Services</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0kMX12NP7n6-xuU0" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>Skupper router exposes the services with port and siteID information (router-id):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U2vBAL2ffkC2Auf_" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>Skupper service controller maintains information of all exposed services:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q4sHKfyxQlNVuAHz" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>Services: ads, cart, checkout, currency are exposed on EKS cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K8dQ7w6WsuWN9DhS" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>Services: email, payment and shipping are exposed on edge-east.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rDirg9_0XOeb5Mis" /><figcaption>Skupper — Exposing Services</figcaption></figure><p>By default all the services are annotated with protocol: tcp, for example the topology above comprise multiple service that communicate over ‘http’ and cart service which is a redis database that needs ‘tcp’ connectivity. Annotations can be added to individual services based on the connectivity required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GWtGlrqSOm17Z5nL" /><figcaption>Skupper — Annotations</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NX-wjcAKa0q-FAPI" /><figcaption>Skupper — Annotations</figcaption></figure><p>Once all the services are exposed to VAN, the app is functional and can now communicate with all other integral services located in different clusters on different platforms. With VAN, services in private clusters are accessible for other services located in public and other remote private clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f1MCClVg-Esi-Ebv" /><figcaption>Hipster — Functional Application</figcaption></figure><p>Skupper console deployments tab showing all the deployments across the member sites:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oB3LF5MYk9-qnC_P" /><figcaption>Skupper — Deployment Console</figcaption></figure><p>Graphical view of deployments and sites with service connectivity information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XoAC0CEa2By9Bt3f" /><figcaption>Skupper — Deployment Console</figcaption></figure><p>Monitoring traffic flow between services :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vy7qkpzFh05yzpeg" /><figcaption>Skupper — Deployment Console</figcaption></figure><p>As shown above ‘10.48.2.6’ is the pod_ip of the frontend service located in GKE cluster. With VAN the pod can communicate to other pods (services) in different clusters and network spaces without any site-site VPN connectivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tfrQculS5wD_k0h0" /><figcaption>Frontend — Pod_IP</figcaption></figure><p>Qpid console provides detailed information on the service connections :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fIN6JffUfJ82ymRC" /><figcaption>Qpid Router — Console</figcaption></figure><p>Skupper — graphical view of services with health, flow and protocol information :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1SOJ-x-LyfaOYBiD" /><figcaption>Skupper — Services Console</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jXEaugJprtpwWCOB" /><figcaption>Skupper — Services Console</figcaption></figure><p>Skupper console — graphical view of message flow between sites and services :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q3Ojb1blOmOK3j5_" /><figcaption>Skupper — Message Flow Visualization</figcaption></figure><p>Qpid console provides flow level monitoring and live traffic flow :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T9cfE46uwhXcEDDK" /><figcaption>Qpid Router — Message Flow Visualization</figcaption></figure><p>Redundant paths in the topology provide resiliency against failure of a site or a connection. It is ok for multiple private networks to participate even if they have overlapping CIDR subnetworks and there is no central controller or single point of failure. For example in the topology used, a connection can be made from two edge clusters to EKS cluster for redundancy as shown below :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-4zyg2m_K16ev6V6" /><figcaption>Skupper — Connected Site Topology</figcaption></figure><p>Topology of two edge clusters peered with routers on both EKS and GKE :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5ZagxIGS6ngEdG4D" /><figcaption>Qpid Router — Connected Site Topology</figcaption></figure><p>More and more applications and frameworks are no longer directly tied to any cloud by design and are not constrained to be deployed in a single cluster. With VAN the developer of a cloud-native application does not want to have to think about the hosts or the ports involved in running the application services. VAN can cater a wide set of complex use-cases with a simpler approach and configuration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f63a8081f41" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/virtual-application-networks-van-for-multi-cloud-multi-cluster-and-cloud-edge-interconnect-1f63a8081f41">Virtual Application Networks (VAN) for Multi-Cloud, Multi-Cluster and Cloud-Edge Interconnect</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon EKS Distro (EKS-D): The Kubernetes Distribution Used by Amazon EKS]]></title>
            <link>https://itnext.io/amazon-eks-distro-eks-d-the-kubernetes-distribution-used-by-amazon-eks-fa23b54025fb?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa23b54025fb</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Wed, 16 Dec 2020 21:37:53 GMT</pubDate>
            <atom:updated>2020-12-16T21:47:44.267Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60OI2RlF6taWNcT9wZLsgQ.jpeg" /><figcaption>Kubernetes distribution that powers Elastic Kubernetes Service (EKS) globally</figcaption></figure><p>Amazon at the re:Invent 2020 virtual event made strategic announcements related to container services, the announcement include EKS-A (EKS Anywhere), EKS-C (EKS Console), ECS-A (ECS Anywhere) and EKS-D (EKS Distro). While EKS-A, EKS-C, ECS-A are expected to be available in 2021, AWS open sourced EKS-D — Kubernetes distribution that powers its Elastic Kubernetes Service (EKS).</p><p>A Kubernetes distribution is Kubernetes with a chosen configuration and a chosen set of addons. There are multiple Kubernetes distributions like Rancher Kubernetes, VMware Tanzu Kubernetes Grid, Charmed Kubernetes from Canonical, Red Hat OpenShift etc. available upstream. Kubernetes is not secure by default, nor by itself, a security-focused Kubernetes distribution will configure Kubernetes with restrictive security policies which are mission critical for production environments.</p><p>EKS Distro (EKS-D) combines open-source (upstream) Kubernetes components — Kubernetes control plane components (such as kube-controller-manager, etcd, and CoreDNS), Kubernetes worker node components (kubelet, Kubernetes CSI, and CNI), command-line clients (kubectl and etcdctl), security patches, and third-party tools including configuration database, network, and storage components necessary for cluster creation. It will also include all of the upstream patches used by Amazon EKS including fixes to which Amazon has contributed back to the community. In addition, it will include patches that AWS deem important for operational stability and security fixes.</p><p>With EKS-D, AWS has built an open source bridge to its managed Kubernetes platform and a medium to drive the adoption of its cloud platform. With EKS-D, users can rely on the same versions of Kubernetes and its dependencies deployed on Amazon EKS and other cloud platforms (public/private).</p><p>EKS-D reduces the friction between using a different Kubernetes distribution for on-prem and cloud-based environments by using the same validated versions and configurations of Kubernetes and its dependencies, as deployed globally by Amazon EKS. Since it’s free, customers are more likely to evaluate it before considering other Kubernetes distros and enables AWS to efficiently compete with independent software vendors like VMware, Red Hat, Rancher etc. which leverage Amazon EC2 to run their managed Kubernetes offering.</p><h3>EKS vs EKS-D</h3><p>EKS — In Amazon EKS on Amazon EC2 setting a managed control plane — control plane runs in an account managed by AWS, and the Kubernetes API is exposed via the Amazon EKS endpoint associated with your cluster. Each Amazon EKS cluster control plane is single-tenant and unique, and runs on its own set of Amazon EC2 instances. Users can opt for a self-managed data plane or AWS managed dataplane using managed node groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j050_Xfw4u5OwjcC" /><figcaption>EKS Cluster</figcaption></figure><p>Amazon EKS nodes run in users AWS account and connect to cluster’s control plane via the API server endpoint and a certificate file that is created for the cluster. Cluster creation (bootstrap) is seamless and constitutes multiple entities created by a AWS CloudFormation stack. Users can pick any of the supported versions to create the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Vr2xroeROUTVeja4" /><figcaption>EKS Cluster Creation</figcaption></figure><p>Users can only access the workload components, daemonsets deployed on the nodes and all the control plane components cannot be accessed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ev-6xLVoZEOa-I59" /><figcaption>EKS — Managed Control Plane</figcaption></figure><p>EKS-D — All components are customer managed. Bootstrapping of cluster can be done using any of the supported installation methods. Already a number of <a href="https://distro.eks.amazonaws.com/community/partners/">partners</a> are providing install methods as well a integrations with EKS Distro. Tooling required for lifecycle management and other aspects such as provisioning the nodes, HA control plane, fail over etc. should be taken care by the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OvRiB1Z2vC-9QFVn" /><figcaption>EKS-D</figcaption></figure><p>EKS-D can be used by organizations that wish to standardize and use the exact same Kubernetes distribution as they use with Amazon EKS on their non-AWS environments as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v-WCAFuWk7EPUj5G" /><figcaption>EKS-D vs EKS</figcaption></figure><p>Users can access all the components (control plane and other workloads), this enables users to have control over the core control plane components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nh5nIgUpzOnDmpP3" /><figcaption>EKS-D</figcaption></figure><h3>Release Channels and Releases</h3><p>Each Amazon EKS Distro release follows the <a href="https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar">EKS process</a> (supports at least four production-ready versions of Kubernetes at any given time), verifying new Kubernetes versions for compatibility. The Amazon EKS Distro source code, open-source tooling, binaries, and container images as well as configuration are provided for reproducible builds via public Git and S3 storage locations.</p><p>The EKS Distro releases Kubernetes versions at the same pace as EKS, and updates are issued as releases in release channels. A release channel tracks minor versions (<em>v&lt;major&gt;.&lt;minor&gt;.&lt;point&gt;</em>) of Kubernetes, and a channel will be retired when EKS ceases supporting a particular minor version of Kubernetes.</p><p>New releases and release channels will be announced via an SNS topic when they are launched. Releases and release channels are structured as Kubernetes Custom Resource Definitions (CRDs) and the schema can be found in the <a href="https://github.com/aws/eks-distro-build-tooling/tree/main/release">eks-distro-build-tooling</a> GitHub repository.</p><p>Release Channel and Releases — Custom Resource Definitions (CRD’s)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GKQxPvMrYQ8osT3y" /><figcaption>EKS-D — Release and Release Channel CRD’s</figcaption></figure><p>Release Channel spec with snsTopicARN (Amazon Resource Names)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lxEiV8-J3NEg19M6" /><figcaption>Release Channel CRD — EKS-D</figcaption></figure><p>Releases of the EKS Distro are in-step with versions of individual components (CNI plugins, CoreDNS, etcd etc.) used by or recommended for use with Amazon EKS beginning with Kubernetes v1.18.9. A list of all the components and assets that make up a release including URIs to all the compressed archives, binaries, and container images are available in the release manifests.</p><p>Release spec with list of individual components and the corresponding image versions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EaD1rLShE8-PGbPz" /><figcaption>Releases CRD — EKS-D</figcaption></figure><p>All container images for these components are based on Amazon Linux 2, and are available for users on the ECR public registry for amd64 and arm64 architectures or users can build the images from scratch using the tooling provided. New releases will be created when there is an updated component version.</p><p>The EKS Distro is built using <a href="https://prow.eks.amazonaws.com/">Prow</a>, the Kubernetes CI/CD system. EKS operates an installation of Prow which manages all the build operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qk96E5OPJvOz7jIR" /><figcaption>EKS-D Prow</figcaption></figure><h3>Container Images</h3><p>Users can use <a href="https://gallery.ecr.aws/?searchTerm=eks-distro&amp;verified=verified">Amazon’s public ECR repository</a> to pull the images of specific release. The repository includes all the core control plane components and other add-ons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u_XP2g-52k9Z780R" /><figcaption>EKS-D — Images from ECR Public</figcaption></figure><p>All images are tagged with the components individual version and Kubernetes release version as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L96iyh9TUnxxxe0E" /><figcaption>EKS-D — Images from ECR Public</figcaption></figure><p>Users can build all the images from scratch using the base image provided in <a href="https://github.com/aws/eks-distro-build-tooling">EKS Distro Build Tooling repository</a> (amazonlinux:2) and all container images using <a href="https://github.com/aws/eks-distro">EKS Distro</a> repository. The default target for make files uses buildkit to build containers. The alternative to this is building and pushing containers with Docker.</p><p>An up-to-date Amazon Linux 2 base image can be created using the tooling repo and pushed to ECR. This base will be updated whenever there are any security updates to RPMs contained in the base image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*71MnU3CMBLzViUWe" /><figcaption>EKS-D — Base Image for building container images</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D69jPClPrGXmfFJU" /><figcaption>EKS-D — Base Image for building container images</figcaption></figure><p>All images built can be pushed to private ECR repository and can be used with docker using <a href="https://github.com/awslabs/amazon-ecr-credential-helper">ECR Credential Helper</a>.</p><p>Individual private repositories in ECR (sample):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pfVG_aMTg3oHdNPF" /><figcaption>ECR — Private Repositories</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_hkiazUn7ulIQtpx" /><figcaption>ECR — Private Repositories</figcaption></figure><p>The tooling tags the release version and component version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g2yitVqMRI-AoWMf" /><figcaption>ECR — Private Repository</figcaption></figure><p>ECR auto scans (configuration) all the images pushed for any software vulnerabilities in container images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KEZFyIgLy3mNc5Gr" /><figcaption>ECR — Vulnerability Scan</figcaption></figure><p>All the images are built using amazon-linux as base and include all the patches that AWS deem important for operational stability and security fixes, most of the vulnerabilities are fixed in EKS-D when compared with the official ubuntu based upstream Kubernetes images.</p><p>To test, a vulnerability scan is done below using Snyk on kube-proxy image from upstream (k8s.gcr.io/kube-proxy) and EKS-D (public.ecr.aws/eks-distro/kubernetes/kube-proxy).</p><p>The scan below shows that most of the vulnerabilities are fixed in EKS-D based kube-proxy image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BB6Gcv0ANoU64xDl" /><figcaption>Container Vulnerability Scan</figcaption></figure><p><em>k8s.gcr.io/kube-proxy:v1.18.9 :</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k2UQ1oNVYpKJqUAk" /><figcaption>Container Vulnerability Scan — Upstream</figcaption></figure><p><em>public.ecr.aws/eks-distro/kubernetes/kube-proxy:v1.18.9-eks-1–18–1 :</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WtY-PHUVLObgrGys" /><figcaption>Container Vulnerability Scan — ECR</figcaption></figure><h3>Tryout — Bootstrapping a cluster with EKS Distro</h3><p>EKS Distro can be installed using any of the <a href="https://distro.eks.amazonaws.com/community/partners/">supported installation methods</a> like Rancher, Kubeone, Canonical, Kubestack etc., partners also provide services such as monitoring (datadog, instana), security (alicide, aqua security), GitOps (weaveworks) etc.</p><p>The official <a href="https://github.com/aws/eks-distro/tree/main/development/kops">development guide</a> from the eks-distro repository uses Kops for bootstrapping a cluster. Technically EKS-D can be installed using tools like Kubeadm as this just involves modifications of images (use EKS-D based images from public ECR) used and addition of specific flags/parameters in the manifests derived from upstream. There are other easier ways like EKS Snap to quickly get a feel of EKS-D. Other partners like Rancher are already supporting production grade EKS-D with RKE-2 (RKE Government) and many others providing installation methods, monitoring and security.</p><h3>EKS Snap — Ubuntu</h3><p>With <a href="https://snapcraft.io/eks">EKS Snap</a>, a full EKS experience now comes in a single, immutable package with automatic updates, for low-maintenance and increased security.</p><p>The eks snap packages all the Kubernetes binaries of Amazon EKS Distro (EKS-D) and combines them with a MicroK8s like experience, for an opinionated, self-healing, highly available EKS-compatible Kubernetes anywhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ilTPdh5YI8i1kmWK" /><figcaption>EKS-D Snap — Ubuntu</figcaption></figure><p>This EKS Distro is based on MicroK8s. All the control plane components run as systemd services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yZqRsSe9R8D7gj_P" /><figcaption>EKS Snap — EKS-D</figcaption></figure><p>The installation method provides multiple commands to manage addition and deletion of nodes, manage certs, manage db etc. As shown below the snap install automatically installs and initializes multiple services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*__TWIkvujLICL-nl" /><figcaption>EKS-D Snap — Ubuntu — CLI</figcaption></figure><h3>Kops</h3><p>Kops, short for Kubernetes Operations, is a set of tools for installing, operating, and deleting Kubernetes clusters in the cloud. As of today the official development guide for starting a EKS-D cluster is with Kops. Kops framework provided in the <a href="https://github.com/aws/eks-distro/tree/main/development/kops">eks-distro</a> repository uses EKS-D for bootstrapping the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Knu7ASh3_7TRPOw_" /><figcaption>Kops — EKS-D</figcaption></figure><p>Kops creates required EC2 instances and bootstraps a Kubernetes cluster using EKS-D.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vU_iZbMu763lqLAm" /><figcaption>Kops — EC2 Instances (Master and Nodes)</figcaption></figure><p>As shown below all control-plane components can be accessed by the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JbP9SwNZ8mX1koJz" /><figcaption>Kops — EKS-D</figcaption></figure><p>All the images are derived from the eks-distro:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HyHcbl8ttyTHfDlE" /><figcaption>Kops — EKS-D</figcaption></figure><p>Kops needs a “state store” to store configuration information of the cluster, Kops uses S3 as a state store. In this scenario it stores all the information such as cluster-spec, manifests, pki etc.,The state is stored during the initial cluster creation. Any subsequent changes to the cluster are also persisted to this store as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rzUYSnemr16frX9c" /><figcaption>Kops — State Storage — S3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FUm_HTunhtchiT9Y" /><figcaption>Kops — State Storage — S3</figcaption></figure><p>EKS-D is an investment from Amazon to reduce the friction involved in adopting AWS when using a commercial Kubernetes distribution. Apart from being just a Kubernetes distribution EKS-D lay a strong foundation for future hybrid cloud and multi-cloud services expected from AWS.</p><p>With EKS-A coming in 2021 EKS-D will play a vital role in hybrid cloud setting where users can have the same distro of Kubernetes everywhere with optional connectivity to other AWS services. EKS-A will eventually become the de facto compute environment for multi-cloud, hybrid-cloud and edge (like snowball) environments enabling AWS to efficiently compete with homologous products such as Google Anthos and Azure Arc which are already prevalent in the market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nsrVEU_xVxLIy1ud" /><figcaption>Amazon EKS Anywhere</figcaption></figure><p>EKS-D’s upstream alignment demonstrates a continued commitment to the Kubernetes open source community from AWS and also brings additional engineering strength and ideas to the table. This announcement clearly reinforces that enterprises will run Kubernetes as an abstraction across on-premises and public cloud.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa23b54025fb" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/amazon-eks-distro-eks-d-the-kubernetes-distribution-used-by-amazon-eks-fa23b54025fb">Amazon EKS Distro (EKS-D): The Kubernetes Distribution Used by Amazon EKS</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ingress for Anthos — Multi-cluster Ingress and Global Service Load Balancing]]></title>
            <link>https://itnext.io/ingress-for-anthos-multi-cluster-ingress-and-global-service-load-balancing-c56c57b97e82?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/c56c57b97e82</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[anthos]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Fri, 18 Sep 2020 08:20:10 GMT</pubDate>
            <atom:updated>2020-09-20T02:21:46.435Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_VIymOI2Tut5HSKuhFObg.png" /><figcaption>Geo-Aware Multi Cluster Ingress — Ingress for Anthos</figcaption></figure><h3>Ingress for Anthos — Multi-cluster Ingress and Global Service Load Balancing</h3><p><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-for-anthos">Ingress for Anthos</a> is a Google cloud-hosted multi-cluster ingress controller for Anthos GKE clusters. Ingress for Anthos supports deploying shared load balancing resources across clusters and across regions enabling users to use a same load balancer with an anycast IP for applications running in a multi-cluster and multi-region topology.</p><p>In simpler terms this allows users to place multiple GKE clusters located in different regions under one LoadBalancer. It’s a controller for the external HTTP(S) load balancer to provide ingress for traffic coming from the internet across one or more clusters by programming the external HTTP(S) load balancer using <a href="https://cloud.google.com/load-balancing/docs/negs">network endpoint groups</a> (NEGs). NEGs are useful for Container native load balancing where each Container can be represented as endpoint to the load balancer.</p><p>Taking advantage of GCP’s 100+ Points of Presence and global network, Ingress for Anthos leverage GCLB along with multiple Kubernetes Engine clusters running across regions around the world, to serve traffic from the closest cluster using a single anycast IP address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qLtCiHtksYDuBUk8" /><figcaption>Anycast, Geo-Aware Proximity-Based Routing</figcaption></figure><h3>Global load balancing with Anycast</h3><p>Ingress for Anthos creates an external HTTP(S) load balancer in <a href="https://cloud.google.com/network-tiers/docs/overview">Premium Tier</a>, it uses a global external IP address advertised as Anycast IP and can intelligently route requests from users to the closest backend instance group or NEG (Network Element Group), based on proximity. Compared with using multiple addresses with DNS-based load balancing, dedicated anycast addresses mean that clients anywhere can connect to the same IP address, while still entering Google’s network as fast as possible and connecting to a load balancer at the edge of Google’s network where the traffic entered minimizing the network distance between the client and the frontline load balancer.</p><p>Anycast enables usage of a same IP address and requesting backends with identical content in different geographical zones around the world provides the shortest response times possible. Anycast directs packets to the geographically closest backend based on Border Gateway Protocol (BGP) paths. For example, if a user sets up instance groups/NEGs in North America, Europe, and Asia, and attach them to a load balancer’s backend service, user requests around the world are automatically sent to the VMs/Pods closest to the users, assuming the VMs/Pods pass health checks and have enough capacity (defined by the balancing mode).</p><p>Anycast is linked with the BGP protocol which ensures that all of a router’s neighbors are aware of the networks that can be reached through that router and the topographical distance to those networks. With anycast, the system consistently chooses the shortest path every time. In the event of a node failure, the next shortest route is determined, and traffic is redirected without having to change the IP address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BoWr4bb0I7K2weN_" /><figcaption>Anycast Routing</figcaption></figure><p>Anycast improves throughput and minimizes latency for clients around the world. Further, if <a href="https://cloud.google.com/cdn/docs/">Cloud CDN</a> is added, caching can be enabled at these edge locations. A global anycast IP address enables users to seamlessly change or add regions for deploying application instances and increase capacity as needed. Apart from latency optimization other biggest advantage with this approach is high-availability, if the closest backends are all unhealthy, or if the closest instance group/NEG is at capacity and another instance group/NEG is not at capacity, the load balancer automatically sends requests to the next closest region with capacity.</p><h3>Cold Potato Routing to Google PoPs</h3><p>Requests from clients are routed <a href="https://wikipedia.org/wiki/Hot-potato_and_cold-potato_routing">cold potato</a> to Google PoPs, meaning that internet traffic goes to the closest PoP and gets to the Google backbone as fast as possible.</p><p>Cold Potato technique in contrast to Hot Potato (traffic is sent to the peer at the closest exchange point), carry the customer’s traffic for as long as possible before delivering packets to the peer using the internal network/backbone. Although Cold Potato increase overall operational cost, provides advantages like keeping the traffic under the network administrator’s control for longer, applying sophisticated rules using custom software stacks, allowing operators of well-provisioned networks to offer a higher quality of service to their customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zzx4mX6ToQlGpKb2" /><figcaption>Cold Potato vs Hot Potato Routing</figcaption></figure><p>Google PoPs or GFE’s (<a href="https://cloud.google.com/security/security-design#google_front_end_service">Google Front Ends</a>) are software-defined, distributed systems that are located in Google points of presence (PoPs) and perform global load balancing in conjunction with other systems and control planes. Any internal service which chooses to publish itself externally uses the GFE as a smart reverse-proxy front end. This front end provides public IP hosting of its public DNS name, Denial of Service (DoS) protection, and TLS termination.</p><h3>Container Native Load Balancing with NEG’s (Network Endpoint Groups)</h3><p><a href="https://cloud.google.com/load-balancing/docs/negs">NEGs</a> are lists of IP addresses used by Google Cloud load balancers and IP addresses in a NEG can be primary or secondary IP addresses of a VM, which means they can be Pod IPs. In a GKE VPC-Native cluster (Pod IP addresses are natively routable within the cluster’s VPC network and other VPC networks connected to it by VPC Network Peering) GCP VPC network is aware of alias IPs, so the routing is taken care by VPC without any need for extra route configuration.</p><p>NEGs are useful for Container native load balancing where each Container can be represented as endpoint to the load balancer. This enables container-native load balancing that sends traffic directly to Pods from a Google Cloud load balancer.</p><p>In the old GKE load-balancing approach, traditional instance-groups used to be the backends and these comprise the actual Kubernetes nodes where the workloads run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e6OYosm_KkcFMTfY" /><figcaption>Load Balancer — backends — Instance Groups</figcaption></figure><p>Instance groups with Kubernetes nodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T5-Oe3ned9IGE0O0" /><figcaption>Instance Groups — Kubernetes Nodes</figcaption></figure><p>Users can use a tool called <a href="https://github.com/GoogleCloudPlatform/k8s-multicluster-ingress">kubemci</a> (a tool to configure Kubernetes ingress to load balance traffic across multiple Kubernetes clusters) to create a LoadBalancer spread across clusters and associate an anycast IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZgMbbHw_maQKS2SU" /><figcaption>LB with Anycast IP</figcaption></figure><p>This approach is now deprecated and Ingress for Anthos is the recommended way to deploy multi-cluster ingress moving forward.</p><p>Without Container native load balancing, load balancer would distribute packets to each node (Kubernetes node) and iptables in each node would further distribute the packets to each pod/container and this adds the extra hop. With Container native load balancing, the advantages are better load distribution, removal of extra hop that reduces latency and better health check.</p><p>Ingress for Anthos is a cloud-hosted multi-cluster ingress controller that programs the external HTTP(S) load balancer using <a href="https://cloud.google.com/load-balancing/docs/negs">network endpoint groups</a> (NEGs). This service is Google-hosted and supports deploying shared load balancing resources across clusters and across regions. The controller deploys Compute Engine load balancer resources and configures the appropriate Pods across clusters as backends when user creates a MultiClusterIngress resource. The NEGs are used to track Pod endpoints dynamically so the Google load balancer has the right set of healthy backends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vEy9R7zu6D6hocnE" /><figcaption>Container Native Load Balancing with NEGs</figcaption></figure><p>On GKE the NEGs can be created and managed automatically by adding an annotation to a Kubernetes Service (<em>cloud.google.com/neg</em>). When NEGs are used with Anthos Ingress, the Ingress controller facilitates the creation of all aspects of the L7 load balancer. This includes creating the virtual IP address, forwarding rules, health checks, firewall rules, and more.</p><h3>Tryout</h3><p>Three GKE clusters are created in three different regions and all the three clusters are registered to Anthos Hub. The cluster in us-west2-a is selected as a config cluster, a config cluster is a centralized point of control for the multi-cluster resources — MultiClusterIngress and MultiClusterService. These multi-cluster resources exist in and are accessible from a single logical API to retain consistency across all clusters. The Ingress controller watches the config cluster and reconciles the load balancing infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p2JfFcBRcVig8cc2" /><figcaption>Multi-Cluster and Multi-Region Topology</figcaption></figure><p>The ingress controller is a globally distributed control plane that runs as a service outside of the clusters. Environ provides a unified way to view and manage multiple clusters and their workloads as part of Anthos. Clusters are registered with Anthos using the project’s environ using Connect, ingress uses the concept of environs for how Ingress is applied across different clusters. Clusters that are registered to an environ become visible to Ingress, so they can be used as backends for Ingress. GKE clusters registered as members to the Hub become part of what is conceptually known as an <em>environ</em>.</p><p>Clusters registered to an environ are called member clusters, in the topology above all three clusters are member clusters (config cluster can be isolated with cluster-selector). Member clusters in the environ comprise the full scope of backends that Ingress is aware of.</p><p>MCI (MuliClusterIngress) and MCS (MultiClusterService) are custom resources (CRDs) that are the multi-cluster equivalents of Ingress and Service resources. These are configured on the config cluster and MCS creates required derived services (actual services mapping to endpoints) in the member clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ppGpv4rzPMWyX2a7" /><figcaption>MultiClusterIngress and MultiClusterService</figcaption></figure><p>A MultiClusterIngress resource contains a list of backend services (names of MCS that is deployed in the same Namespace and config cluster as this MCI). A MultiClusterIngress (MCI) resource behaves identically in many ways to the core Ingress resource. Both have the same specification for defining hosts, paths, protocol termination and backends.</p><p>A MultiClusterService (MCS) is a custom resource used by Ingress for Anthos that is a logical representation of a Service across multiple clusters. An MCS is similar to, but substantially different from, the core Service type. An MCS exists only in the config cluster and generates derived Services in the target clusters. An MCS does not route anything like a ClusterIP, LoadBalancer, or NodePort Service does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8FW9wqLTBaaTw2nZ" /><figcaption>MultiClusterIngress and MultiClusterService</figcaption></figure><p>A MCI resource can have a default backend (MCS) and list of rules specifying multiple backends. MCI matches traffic to the VIP on the hosts specified in the rules by sending the traffic to the MCS resource specified in the backend and all other traffic which does not match will be sent to the default-backend MCS.</p><p>Users can selectively apply ingress rules using ‘<em>clusters</em>’ on MCS configuration where the derived services are only created on the list specified. If the “clusters” section of the MultiClusterService is not specified or if no clusters are listed, it is interpreted as the default “all” clusters. This feature provides advantages such as: Isolating the config cluster to prevent MCSs from selecting across them, routing to application backends that only exist in subset of clusters/regions etc., controlling traffic between clusters in a blue-green fashion useful for app migration, facilitates to use a single L7 VIP for different clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AWlJEZIDM_2TDQIc" /><figcaption>MultiClusterIngress and MultiClusterService — Cluster Selector and Ingress Rules</figcaption></figure><p>The following are the steps in the workflow to configure Ingress for Anthos across multiple clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P8hkRL_PfboFNbko" /><figcaption>Workflow — Configuring Ingress for Anthos</figcaption></figure><p>Multi Cluster Ingress API should be enabled in the project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gLGW-H3TwTxTp_7Y" /><figcaption>MulticlusterIngress API</figcaption></figure><p>As shown below three GKE clusters are created in different Zones and added to the hub. Cluster1 is selected as a config cluster and Multicluster Ingress Feature is enabled on the hub level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N9sNcFE_EObBKm6H" /><figcaption>Enabling Multi Cluster Ingress Feature</figcaption></figure><p>All clusters added to Hub using Connect, as all these clusters are GKE managed clusters the type specifies them as GKE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y_KyrkOMI1M2Q5uy" /><figcaption>Clusters in a Hub — Registered using Connect</figcaption></figure><p>Enabling the ingress feature from Anthos portal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TUwnVxiAwiV2pMj1" /><figcaption>Enabling Multi Cluster Ingress Feature — Anthos Portal</figcaption></figure><p>Ingress detail page showing the config cluster information and all other registered clusters (memberships):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4duZj1ngNdiOYk2a" /><figcaption>Cluster Memberships and Config Membership</figcaption></figure><p>A demo application ‘zone-ingress’ is deployed on all three clusters, this application just prints the name of the datacenter where it is running when accessed.</p><p>While deploying applications across clusters users have to consider a concept called “<a href="https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-for-anthos#namespace_sameness"><em>namespace sameness</em></a><em>”, </em>a characteristic possessed by Environs. This assumes that resources with the identical names and same namespace across clusters are considered to be instances of the same resource. In effect, this means that Pods in the ‘zoneprinter’ namespace with the labels ‘app: zoneprinter’ across different clusters are all considered part of the same pool of application backends from the perspective of Ingress for Anthos. This has ramifications for how different development teams operate across a group of clusters.</p><p>In this scenario a deployment called zone-ingress is deployed on all the three clusters in ‘zoneprinter’ namespace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l1kCO6Ikc5vn98YC" /><figcaption>Demo application deployed on all clusters</figcaption></figure><p>MCI (MuliClusterIngress) and MCS (MultiClusterService) are created on the config cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YwfsNU8NZKvGogkc" /><figcaption>MultiClusterIngress and MultiClusterService created in Config Cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OvoXdjCLOHTgnT3B" /><figcaption>MultiClusterIngress and MultiClusterService created in Config Cluster</figcaption></figure><p>As we have not provided any ‘clusters’ in the MCS configuration a derived service is created on all the three clusters including the config cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sfw4Fq2knH2n4SPt" /><figcaption>Derived Services created by MCS on all Member Clusters</figcaption></figure><p>The derived services created in the member clusters (all three clusters in this case) are annotated with NEGs (mapping of respective network element group), multiclusterservice-parent (MCS created on the config cluster) and the port information as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L2_2PjonfQc4RKP8" /><figcaption>Derived Services created by MCS on Member Cluster</figcaption></figure><p>This step creates NEGs in Compute Engine, which begins to register and manage service endpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nIWTT9VYD-_FVqCt" /><figcaption>NEG Attachment</figcaption></figure><p>The controller auto-creates the NEGs. NEGs map the ‘zone-ingress’ pods in all three clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6VK_QQtk4koHDjof" /><figcaption>Network Element Groups</figcaption></figure><p>As shown below the network endpoints in the NEG map to the pod_ip of <em>zone-ingress</em> on cluster1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KrNjsw1jLPsagEWv" /><figcaption>Network Element Groups pointing to Pod_IP</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6IJh8LgwQ_LArbsq" /><figcaption>Network Element Groups pointing to Pod_IP</figcaption></figure><p>Creating MCI resource on config cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*scjDWmA86ez3OJY0" /><figcaption>Creating MCI Resource on Config Cluster</figcaption></figure><p>This step deploys the Compute Engine external load balancer resources and exposes the endpoints across clusters through a single load balancer VIP.</p><p>The MultiClusterIngress resource ‘Status’ field shows the backend services: MCS, firewalls, forwarding rules, health checks and NEGs of individual clusters. The VIP is the anycast IPV4 address which maps to a load balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NudZXS-jXXqRHmke" /><figcaption>MCI resource with Backend Services, Firewalls, Forwarding Rules, Health Checks and NEGs</figcaption></figure><p>Load balancer backends mapping NEGs as backends, this is auto-created by the ingress controller when a MCI object is created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X-N-PUQPjMj4bBLO" /><figcaption>Load Balancer with NEGs as Backends</figcaption></figure><p>The NEGs are configured as backends for the Load Balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CM68yi6vUZPg3n2h" /><figcaption>Load Balancer with NEGs as Backends</figcaption></figure><p>The frontend maps to a single IPV4 Anycast address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_jIDo1QTdTRVVSoc" /><figcaption>Load Balancer with Anycast IPV4 Address</figcaption></figure><p>Forwarding rules are configured to send traffic from LB to ingress resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6iiuGT7qsdJFYaAH" /><figcaption>Forwarding Rules</figcaption></figure><h3>Accessing the demo Service (Zoneprinter) from different Geographical Regions</h3><p>Three VM instances are created in respective zones where the GKE clusters are positioned to replicate access from different regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U52ZDNq1xUYlQ_P3" /><figcaption>Test Instances — Test Access from Different Regions</figcaption></figure><p>Accessing Zoneprinter using the LB’s Anycast IP from US-West, returns response from Zoneprinter pod running in Cluster1 — us-west2-a.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EikC7nvqC6oqBT6e" /><figcaption>Accessing from US-West</figcaption></figure><p>Accessing Zoneprinter using the LB’s Anycast IP from US-East, returns response from Zoneprinter pod running in Cluster2 — us-east4-a.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-S0R4rHSdafWSzc4" /><figcaption>Accessing from US-East</figcaption></figure><p>Accessing Zoneprinter using the LB’s Anycast IP from EU-West, returns response from Zoneprinter pod running in Cluster3 — europe-west2-b.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pm-rhIy8ldO_mZbu" /><figcaption>Accessing from EU-West</figcaption></figure><p>Accessing the application using web page performance tool shows the same response time from both USA and Europe.</p><p>From Virginia:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qP71zhg73Prx_1f3" /><figcaption>Accessing from Virginia (us-east)</figcaption></figure><p>From London:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dCFSGjlfFcQ6ZwbC" /><figcaption>Accessing from London (eu-west)</figcaption></figure><p>Comparing <em>traceroute</em> results of the LB’s Anycast IP and an app serving on a traditional EIP (Elastic IP) shows drastic differences in the response time and number of next-hops.</p><p>The <em>traceroute</em> from different regions to the LB’s Anycast IP shows minimal number of hops and lower response time. This basically proves that the proximity based routing is choosing the application running on the nearest cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D9G9mJSB5W0vW_dA" /><figcaption>Traceroute — Anycast IP from Different Regions</figcaption></figure><p>The <em>traceroute</em> from New York to an application serving on a EIP in London shows significant increase in the number of hops and response time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oTafpWuC6yltLTF1" /><figcaption>Traceroute — EIP from New York to London</figcaption></figure><h3>Monitoring and Metrics</h3><p>Load balancer details section provides backend wise health, utilization and request metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TVUVbDkHExwJ0kRa" /><figcaption>LB — Monitoring</figcaption></figure><p>Map showing requests, traffic and serving backends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*grOzs4kjHZamn9aW" /><figcaption>LB — Monitoring</figcaption></figure><h3>Other Features</h3><p>Ingress for Anthos supports other features such as:</p><ul><li>HTTPS support: The Kubernetes <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/secret">Secret</a> supports HTTPS. Before enabling HTTPS support, user must create a static IP address. This static IP allows HTTP and HTTPS to share the same IP address. The secret can be provided using the ‘tls’ parameter in the MCI resource.</li><li>BackendConfig support: The <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-for-anthos#backendconfig_support">BackendConfig CRD</a> allows you to customize settings on the Compute Engine BackendService resource.</li></ul><h3>WAF with Cloud Armor</h3><p>Google Cloud Armor is deployed at the edge of Google’s network and tightly coupled with the global load balancing infrastructure. Cloud Armor can be used to implement Geo-based access controls, pre-configured WAF rules and Custom L7 filtering policies using custom or pre-configured rules provided by Google Cloud. Google Cloud Armor provides preconfigured complex web application firewall (WAF) rules with dozens of <em>signatures</em> that are compiled from open source industry standards.</p><p>Users can frame security and SSL policies and the policies can be applied to targets, here targets are the load balancers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MZry-yqMLEON-5z6" /><figcaption>Cloud Armor — Security Policy</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w9jguMg8C-te4LKd" /><figcaption>Cloud Armor — Adding LB as Target</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lusPps_oCRF629S8" /><figcaption>Cloud Armor — SSL Policy</figcaption></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Apart from the proximity-based routing, multi-cluster ingress simplifies other operations such as Blue/Green cluster upgrades, disaster recovery, automated fail-overs based on sophisticated health-checks, Run highly available (HA) apps on Kubernetes, Lower DDoS attack risk etc. and has has numerous other benefits, all of which have the potential to translate into a significant competitive edge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c56c57b97e82" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/ingress-for-anthos-multi-cluster-ingress-and-global-service-load-balancing-c56c57b97e82">Ingress for Anthos — Multi-cluster Ingress and Global Service Load Balancing</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Service Mesh — Microsoft’s SMI based Open Source Service Mesh Implementation]]></title>
            <link>https://itnext.io/open-service-mesh-microsofts-smi-based-open-source-service-mesh-implementation-bcf41fd30732?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcf41fd30732</guid>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Thu, 13 Aug 2020 07:48:48 GMT</pubDate>
            <atom:updated>2020-08-13T08:02:41.681Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xV3vih_sK3dUkArkw4gAtw.png" /><figcaption>OSM — SMI Compliant Open Source Service Mesh</figcaption></figure><h3>Open Service Mesh — Microsoft’s SMI based Open Source Service Mesh Implementation</h3><p>Microsoft’s <a href="https://openservicemesh.io/blog/introducing-open-service-mesh/">Open Service Mesh</a> is an SMI-compliant, lightweight service mesh being run as an open source project. Backed by service-mesh partners including HashiCorp, Solo.io, and Buoyant, Microsoft introduced the Service Mesh Interface last year with the goal of helping end users and software vendors work with the myriad choices presented by service mesh technology by providing a set of specification standards. OSM can be considered as a reference implementation of SMI, one that builds on existing service mesh components and concepts.</p><p>Open Service Mesh data plane is architecturally based on the Envoy proxy and implements the <a href="https://github.com/envoyproxy/go-control-plane">go-control-plane xDS v3 API</a>. However, despite the fact that Envoy comes with OSM by default, using standard interfaces allows it to be integrated with other reverse proxies (compatible with xDS).</p><p>SMI follows in the footsteps of existing Kubernetes resources, like Ingress and Network Policy, which also do not provide an implementation where required interfaces to interact with Kubernetes are facilitated for providers to plug their products. The SMI specification instead defines a set of common APIs that allow mesh providers to deliver their own implementations. This means mesh providers can either use SMI APIs directly or build operators to translate SMI to native APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vPQE9VA8wicqTOQv" /><figcaption>SMI Implementation</figcaption></figure><p>With OSM, users can use SMI and Envoy on Kubernetes and get a simplified service-mesh implementation. The SMI ecosystem already has multiple providers like Istio, Linkerd, Consul Connect, now Open Service Mesh etc. some of them have implemented SMI compatibility using adaptors (Istio, Consul Connect) and others (OSM, Linkerd etc.) consume the SMI APIs directly.</p><p>OSM implementation is very similar to Linkerd which also directly consumes SMI APIs without any need for an adaptor like Istio, but one key difference is that OSM uses Envoy for its proxy and communication bus, whereas Linkerd uses linkerd2-proxy (rust based — lighter than Envoy).</p><h3>Architecture &amp; Components</h3><p>OSM control plane comprise four core components. All these four components are implemented as a single controller entity (Kubernetes pod/deployment), this is much lighter in weight when compared with older versions of Istio where there are 4 control plane components (Istio-1.6 introduced istiod which unifies all the control plane components into one binary).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OtanTyoVGipMvGJS" /><figcaption>OSM Architecture — Components</figcaption></figure><p>OSM Data Plane — Uses Envoy as reverse-proxy by default — similar to most other Service Mesh providers (Linkerd is unique in this case which uses ultralight transparent proxy written in Rust). While by default OSM ships with Envoy, the design utilizes <a href="https://github.com/openservicemesh/osm/blob/main/DESIGN.md#interfaces">interfaces</a> (An interface type in Go is kind of definition. It defines and describes the exact methods that some other type must have), which enable integrations with any xDS compatible reverse-proxy. The dynamic configuration of all the proxies is handled by OSM controller using <a href="https://github.com/envoyproxy/go-control-plane">Envoy xDS go-control-plane</a>.</p><p>Mesh Specification is a wrapper around the existing SMI Spec (traffic spec, traffic target, traffic split and traffic metrics). Service mesh interface implementation provides two different approaches for providers to consume SMI spec. One, using adapters: requires a separate entity to convert from the SMI APIs to provider specific configuration. Istio and Consul Connect use adapters. The second approach is to directly consume the SMI API’s alongside the controller implementation where there is no need for an extra layer. Open Service Mesh, Linkerd, Maesh, Meshery etc. use this approach.</p><p>Mesh Catalog is the central component of OSM. MC interacts with all other components gathering individual outputs and provides an aggregated structure which are then transformed into proxy configuration. The proxy configuration is sent to all listening/registered proxies via the proxy control plane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g6etTscg-dCoXL9u" /><figcaption>OSM Architecture — Components</figcaption></figure><p>The Certificate Manager entity provides unique X.509 certificates for individual proxies. A Kubernetes service object is the core entity referenced in SMI spec for enforcing traffic policies. All traffic policies are enforced by the target service’s proxy, all the endpoints under a service are augmented with a proxy. Each service will be provided with a service certificate which will be used for service-proxy control plane mTLS connectivity.</p><p>The component proxy fronts a mesh service process (container or binary running on Kubernetes or a VM), All proxies establish an mTLS gRPC connection to the Proxy Control Plane (OSM Controller) and the controller using Envoy xDS go-control-plane implementation sends configuration updates using xDS protocol buffers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Owcl2aDMf0qEqD7o" /><figcaption>OSM Architecture — Components</figcaption></figure><p>OSM relies on <a href="https://github.com/servicemeshinterface/smi-spec">SMI spec</a> to reference services that will participate in the service mesh. OSM ships out-of-the-box with all necessary components to deploy a complete service mesh spanning multiple compute platforms.</p><h3>SMI Spec — Overview</h3><p>Service Mesh Interface (SMI) specification includes four API objects: traffic spec, traffic target, traffic split, and traffic metrics. Each API object is extended as a Kubernetes CRD (Custom Resource Definition).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LcXYI4tQ8bo9VJUA" /><figcaption>SMI — Traffic Spec and Traffic Target</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q53Dx0EXhUTm_pFr" /><figcaption>SMI — Traffic Split and Traffic Metrics</figcaption></figure><p>A detailed explanation on SMI Spec and Istio SMI implementation is provided in the article below:</p><blockquote><a href="https://itnext.io/smi-flexibility-and-interoperability-with-the-service-meshes-on-kubernetes-8cd88a4d8238?source=friends_link&amp;sk=33e20531be24d8e0600443938bb35f17">https://itnext.io/smi-flexibility-and-interoperability-with-the-service-meshes-on-kubernetes-8cd88a4d8238?source=friends_link&amp;sk=33e20531be24d8e0600443938bb35f17</a></blockquote><h3>OSM Control Plane — Kubernetes</h3><p>The OSM Command Line Interface (CLI) installs the OSM control plan into Kubernetes using this Helm chart. Under the hood, ‘osm’ uses Helm libraries to create a Helm ‘release’ object in the control plane Namespace. The Helm `release` name is the mesh-name (this name is used to label namespaces for enabling sidecar injection ). The ‘helm’ CLI can also be used to inspect Kubernetes manifests installed in more detail.</p><p>The whole ecosystem is controlled by a single controller and other components constitute CRD’s, monitoring and telemetry (Prometheus, Grafana and Zipkin). All the components are deployed in osm-system namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RtBxRHQ0nMbKXlBp" /><figcaption>OSM — Control Plane</figcaption></figure><p>OSM controller and related components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KAtcU75dzAT4P1j5" /><figcaption>OSM — Control Plane</figcaption></figure><p>Open Service Mesh controller initializes all the required informers (The client-go package comes with a subpackage that makes getting events easy: k8s.io/client-go/tools/cache. It allows users to easily add functions that will be called when certain events come in. It also allows users to store all of the objects in memory easily which is called a Store) to watch for creation/changes on the current state of Kubernetes objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AK35YQJhvls7UDnE" /><figcaption>OSM — Controller</figcaption></figure><p>Open Service Mesh controller initializes informers related to the SMI components (TrafficTarget, TrafficSplit, TrafficSpec etc.). This enables the controller to identify and process all the SMI specifications on the cluster. As the logic is embedded in the controller there is no need for a separate adaptor (like Istio) to translate the SMI spec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rKm-Je0vgXcwYkLb" /><figcaption>OSM — Controller</figcaption></figure><p>Open Service Mesh uses mTLS for encryption of data between pods as well as Envoy and service identity. Certificates are created and distributed to each Envoy proxy via the SDS (Secret Discovery Service) protocol by the OSM control plane. Open Service Mesh supports 3 methods of issuing certificates: using an internal OSM package, called <a href="https://github.com/openservicemesh/osm/tree/main/pkg/certificate/providers/tresor">Tresor</a> (This is the default for a first time installation), using <a href="https://github.com/openservicemesh/osm/tree/main/pkg/certificate/providers/vault">Hashicorp Vault</a> and <a href="https://github.com/openservicemesh/osm/tree/main/pkg/certificate/providers/keyvault">Azure Key Vault</a>.</p><p>OSM ecosystem requires two types of certificates:</p><ol><li>Certificates used for Envoy proxies to connect to xDS control plane — identifies the proxy and pod connecting to xDS.</li><li>Certificates used for service to service communication (one Envoy connects to another) — identifies the services connecting to each other.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z_-PSCBiwagIsOg4" /><figcaption>OSM — Controller</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_leSpovqmk5M81e-" /><figcaption>OSM — Controller</figcaption></figure><p>A Prometheus deployment is created alongside the controller to constantly poll for OSM related metrics — Control Plane and Data Plane (service-service, pod-service). It helps gather timing data needed to troubleshoot latency problems in service architectures using a specific <em>trace_id</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DFASm5wqmOfI3QpC" /><figcaption>OSM — Prometheus</figcaption></figure><p>A Zipkin deployment is created in the osm-system namespace which can be used for distributed tracing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*khLYvA0XF3WBb7Vt" /><figcaption>OSM — Zipkin</figcaption></figure><p>A Grafana deployment is created alongside Prometheus to visualize the metrics collected by Prometheus. OSM project provides default dashboards stored in a configmap (osm-grafana-config).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WcpvYbRkPZxHXBQD" /><figcaption>OSM — Grafana</figcaption></figure><p>Required config maps are created in osm-system namespace which include the global OSM config, grafana config along with dashboard definitions and datasources and prometheus configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GYs_LsS-Q0smZXIy" /><figcaption>OSM — System Configmap</figcaption></figure><p>Config Map ‘osm-config’ stores the global configuration used by the OSM controller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vlYIvFxUDv4-TCIS" /><figcaption>OSM — Global Configuration</figcaption></figure><p>Required secrets are created in osm-system namespace. These include the certs required for mutating webhook (for sidecar injection). The helm secret stores the release information of the deployed OSM version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nEzJvFY19XyYzobp" /><figcaption>OSM — System Secrets</figcaption></figure><p>All SMI CRD’s are created as part of control plane boot strap, these are all SMI specific CRD’s where there’s no specific configuration related to OSM and the controller implementation enables OSM to convert and process these configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W1pMrD74jmUlBZGO" /><figcaption>OSM — SMI CRD’s</figcaption></figure><p>Along with the above CRD’s an experimental <em>Backpressure/rate limits</em> CRD is also created, this is an experimental extension of SMI. This enables users to dynamically respond to and control inbound traffic in cases of overload situations with downstream services or performance or capacity issues with the service proxy itself. This will be removed once it becomes part of SMI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hTCvPvvo4GC7Xfjk" /><figcaption>SMI — Experimental CRD — Backpressure</figcaption></figure><h3>Proxy Sidecar — Injection</h3><p>A <em>MutatingWebhook</em> is created with required certs to interact with Kube-API, this watches for namespaces with label “<em>openservicemesh.io/monitored-by:&lt;mesh_name&gt;</em>”, mesh_name refers to the configured mesh name using ‘<em> — mesh-name</em>’ or the helm release name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0vptKaxvXSuMfne0" /><figcaption>OSM — MutatingWebhook for Sidecar Injection</figcaption></figure><p>OSM Controller uses informers to derive all namespaces with the label and injects a sidecar to all pods in the namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fGc1gYhBmgc7LlLO" /><figcaption>OSM — Namespace Labeling for Sidecar Injection</figcaption></figure><p>Controller receiving information on the namespaces and injecting a payload to all the pods with the init container spec and a osm-init job container which configures the proxy container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l8tjqMOMdGpAF4hq" /><figcaption>OSM — Controller Logs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VaPy2sspxtPUPwaB" /><figcaption>OSM — Controller Logs</figcaption></figure><p>A sample container with proxy sidecar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fjD8RECibQIFTFU1" /><figcaption>OSM — Envoy Sidecar</figcaption></figure><p>OSM controller — Envoy uses ADS (Aggregated Discovery Services) to discover all the proxy injected services. ADS enables discovery services for endpoints (here proxy sidecars), clusters, routes, listeners and aggregate stream for all resource types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CLf57aJlz3oQoBpl" /><figcaption>OSM — Envoy Sidecar — Discovery</figcaption></figure><h3>Demo Application</h3><p>Official OSM documentation provided users with a demo application (similar to Istio’s bookinfo application). This enables users to test all the features that can be implemented using OSM. The application comprise bookstore (v1 and v2), Bookbuyer and Bookthief and all the services can be access using their respective UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Oq19xiJv1Gwj5YpD" /><figcaption>OSM — Demo Application</figcaption></figure><p>Each application is deployed in an individual namespace and the namespaces are labeled for sidecar injection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D5k2IkFNCLDQWI48" /><figcaption>OSM — Demo Application</figcaption></figure><p>Sample application stack across namespaces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qiMo36JMha2nW65s" /><figcaption>OSM — Demo Application</figcaption></figure><p>All application namespaces will have an associated Service and ServiceAccount which will be used for configuring the traffic spec. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PtWuCAf9iQT7fw_L" /><figcaption>OSM — Demo Application</figcaption></figure><p>All the applications are injected with proxy sidecar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nf7xfT16mH8Dj-dk" /><figcaption>OSM — Demo Application</figcaption></figure><p>OSM Controller — ADS (Aggregated Discovery Service) discovering all individual services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eKnBhEe05gyQnhCa" /><figcaption>OSM — Controller — Aggregated Discovery Logs</figcaption></figure><p>OSM Controller — SDS (Secret Discovery Service) providing required certs for all individual services which is used for mTLS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oz6Wq9RLmS3o9NzR" /><figcaption>OSM — Controller — Secret Discovery Logs</figcaption></figure><p>OSM use Envoy’s go-control-plane xDS v3 API to dynamically configure the proxy config.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b66HxMvr7hKKz1yQ" /><figcaption>OSM — Dynamic Proxy Configuration</figcaption></figure><h3>Egress</h3><p>In OSM environment enabling egress is done via a global setting (configured using osm-config config map in osm-system namespace). The setting is toggled on or off and affects all services in the mesh. Egress is enabled by default when OSM is installed.</p><p>The configuration requires the mesh CIDR ranges to be specified. The mesh CIDR ranges are the list of CIDR ranges corresponding to the pod and service CIDRs configured in the cluster. The mesh CIDR ranges are required with egress to prevent any traffic destined within the cluster from escaping out as egress traffic and to be able to enforce mesh traffic policies for egress traffic. All the in-mesh traffic is routed based on L7 traffic policies, egress traffic is routed differently and is not subject to in-mesh traffic policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RxxUDDb0jjkvlXzT" /><figcaption>OSM — Enabling Egress</figcaption></figure><p>For example, in the sample application below the services <em>bookbuyer</em> and <em>bookthief </em>constantly performs GET requests to github.com.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0cASSv6vEgUILRg5" /><figcaption>OSM — Demo Application — Outbound connectivity to Github</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ipP26bljidCOQPJF" /><figcaption>OSM — Demo Application — Outbound connectivity to Github</figcaption></figure><p>With the egress feature enabled along with the mesh_cidr_ranges configured both the services can reach github.com as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z94KuvTUILnGWDZS" /><figcaption>OSM — Demo Application — Successful Outbound connectivity to Github</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*opkG550ffhEsfTkK" /><figcaption>OSM — Demo Application — Successful Outbound connectivity to Github</figcaption></figure><p>Setting the egress feature to false as shown below prevents all the traffic destined outside the mesh.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vKGWmaqL-Ic74HXg" /><figcaption>OSM — Disabling Egress</figcaption></figure><p>Services <em>bookbuyer</em> and <em>bookthief</em> failing to reach github.com with egress turned off:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eH75v2bne6B1Vcpe" /><figcaption>OSM — Demo Application — Denied Outbound connectivity to Github</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JSRxk05dW5wl0t3h" /><figcaption>OSM — Demo Application — Denied Outbound connectivity to Github</figcaption></figure><h3>Ingress</h3><p>A service can expose HTTP or HTTPS routes outside the cluster using Kubernetes Ingress along with an ingress controller. Once an ingress resource is configured to expose HTTP routes outside the cluster to a service within the cluster, OSM will configure the sidecar proxy on pods to allow ingress traffic to the service based on the ingress routing rules defined by the Kubernetes Ingress resource.</p><p>By default, OSM configures HTTP as the backend protocol for services when an ingress resource is applied with a backend service that belongs to the mesh. A mesh-wide configuration setting in OSM’s osm-config ConfigMap enables configuring ingress with the backend protocol to be HTTPS. HTTPS ingress can be enabled by updating the osm-config ConfigMap. OSM today has official <a href="https://github.com/openservicemesh/osm/blob/main/docs/patterns/ingress.md">documentation</a> providing steps to integrate solutions like Nginx ingress controller, Azure Application Gateway Ingress and Gloo API Gateway.</p><p>For example an ingress implementation using Nginx Ingress controller requires the configuration below:</p><p>A Nginx ingress controller deployed in the cluster provisioned with OSM’s root certificate to be able to authenticate the certificate presented by backend servers when using HTTPS ingress (root certificate is stored in ‘osm-ca-bundle’ secret in osm-system namespace).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yb5pCZV0MeOwZBUS" /><figcaption>OSM — Nginx Ingress Controller</figcaption></figure><p>A sample Ingress resource (HTTPS) configuration is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oNBTbA-zoL-5Py-t" /><figcaption>OSM — Ingress Definition</figcaption></figure><p>Once the ingress resource is created OSM injects a sidecar and will configure the sidecar proxy on pods to allow ingress traffic to the service based on the ingress routing rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*48aY4EwuSAahGIPZ" /><figcaption>OSM — Ingress Definition</figcaption></figure><p>The ingress controller works (update/sync) the same way as it does in traditional Kubernetes environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TeN_Rw1ah8-3iU-C" /><figcaption>Nginx Ingress Controller — Ingress Configuration</figcaption></figure><h3>Traffic Access Control (Allow/Deny Traffic)</h3><p>Traffic access control has three concepts, each defined through TrafficTarget using three fields:</p><ol><li>Source: The source of the traffic, which is reflected in the specific Pod list (selected/grouped by a selector). Currently, it is only supported by the selector using ServiceAccount. Currently, it is not supported by resource selection (such as specifying Deployment and specifying Service).</li><li>Destination: The target of the traffic is also reflected in the specific Pod list (selected/grouped by a selector). Currently, this also only supports using a selector with ServiceAccount.</li><li>Route: Traffic specification (HTTPRouteGroup, TCPRoute), which is used to distinguish different access modes provided by Destination.</li></ol><p>The first scenario is to configure a TrafficTarget and HTTPRouteGroup to enable communication between Bookbuyer and Bookstore-V1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZwUZ1_kYD2ayoVkh" /><figcaption>OSM — Demo Application — Traffic Access</figcaption></figure><p>When no access policies are defined there is no increase in the counters as all the traffic by default is dropped between services without a policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*1bimY_ZnJGiQk3TA" /><figcaption>OSM — Demo Application — Traffic Access</figcaption></figure><p>As there is only one version of bookstore (v1) in the scenario above, all the traffic is destined to V1, so the traffic split is configured to send all the traffic to bookstore-V1 as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sB1HYd00_Uoe9S4J" /><figcaption>OSM — TrafficSplit</figcaption></figure><p>In the TrafficTargets a configuration is made to enable connectivity from <em>ServiceAccount:bookbuyer</em> to <em>ServiceAccount:bookstore-v1 </em>using source and destination as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CGOvSi-0qJ6IbKTQ" /><figcaption>OSM — Demo Application</figcaption></figure><p>HTTPRouteGroups contain two matching paths/routes that should be served by bookstore-v1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3zjPe2Pc65JG8Y4R" /><figcaption>OSM — Demo Application</figcaption></figure><p>With the configuration above the Bookbuyer service can access Bookstore-v1service and the same can be inferred by increasing counters. The Bookthief counters are not being increased as the <em>ServiceAccount:bookthief </em>is not included in the sources above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*nWq7tsQfLqT0s5ZE" /><figcaption>OSM — Demo Application</figcaption></figure><p>Bookthief service unable to access the bookstore-v1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B4IItskVG3AzVsO2" /><figcaption>OSM — Demo Application — Bookthief denied access to Bookstore</figcaption></figure><p>Adding bookthief service account to the sources in the above configuration enables the service to access bookstore-v1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i6g6KbBEL-reMFWs" /><figcaption>OSM — Demo Application — Bookthief allowed access to Bookstore</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RQaj4Wj2-K6JRkoz" /><figcaption>OSM — Demo Application — Traffic Access</figcaption></figure><p>With the configuration above both bookbuyer and bookthief can access bookstore which results in increase of counters on all three services as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*mIK0h_kf_-k38ijs" /><figcaption>OSM — Demo Application — Traffic Access</figcaption></figure><p>Bookthief service logs showing 200 response on requests to the configured HTTPRouteGroups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wL8Hq43s9EVcLopb" /><figcaption>OSM — Demo Application — Bookthief allowed access to Bookstore</figcaption></figure><h3>Traffic Splitting</h3><p>Traffic Split resource is used to achieve a percentage split of traffic to different versions of a same application.</p><p>Traffic split constitutes three concepts: a root-service: point of traffic origin, backend-services: subset of a root-service and weights: controlling amount of traffic flow to each backend service.</p><p>In the scenario below a new version of service bookstore — bookstore-v2 is introduced and configuration is done such that the two services bookbuyer and bookthief can access the service bookstore and the traffic in bookstore is equally (50%) served by both the versions v1 and v2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*KzS9ueKc6YQOpHkk" /><figcaption>OSM — Demo Application — Traffic Split</figcaption></figure><p>A new deployment added to namespace bookstore with associated service and a service account</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ESITOxMFslmDYBz6" /><figcaption>OSM — Demo Application — Traffic Split</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qfj2c0osqTEjz-6L" /><figcaption>OSM — Demo Application — Traffic Split</figcaption></figure><p>As seen below the counters are not increased as all the traffic is configured towards bookstore-v1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*8-XvvEiYYRRShvpT" /><figcaption>OSM — Demo Application — Traffic Split</figcaption></figure><p>Configuring weights 50% for each service<em> bookstore-v1</em> and <em>bookstore-v2</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1lOSa4fl39D3e_kb" /><figcaption>OSM — Demo Application — Traffic Split</figcaption></figure><p>As seen below the counters start to increase as the traffic is split between the versions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*vf6gnLpADTdUBR6u" /><figcaption>OSM — Demo Application — Traffic Split</figcaption></figure><p>The split can be easily verified/monitored from Zipkin, as shown below using the <em>serviceName:bookbuyer</em> and <em>spanName:bookstore </em>it is seen that the requests are being distributed across versions v1 and v2<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J8nVwN5O4CCjoV6q" /><figcaption>OSM — Zipkin — Verifying Traffic Split</figcaption></figure><p>The dependency graph showing the source services traffic (from bookbuyer and bookthief) being distributed between destination services (bookstore-v1 and v2):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bqpZ2UHiNF7WwY27" /><figcaption>OSM — Zipkin — Verifying Traffic Split</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FgfYTpxKTESlTv3R" /><figcaption>OSM — Zipkin — Verifying Traffic Split</figcaption></figure><h3>Monitoring and Tracing</h3><p>OSM comes pre-installed with Prometheus, Grafana and Zipkin. Open Service Mesh (OSM) generates detailed metrics for all services communicating within the mesh. These metrics provide insights into the behavior of services in the mesh helping users to troubleshoot, maintain and analyze their applications.</p><p>Prometheus facilitates gathering of consistent traffic metrics and statistics across all services in the mesh:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wwwzkybtmiT2ObN8" /><figcaption>OSM — Prometheus</figcaption></figure><p>Grafana for visualizing Prometheus backend timeseries database. OSM provides pre-configured dashboards (service-service, workload-workload, service-workload).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bOuGQ4qvMEmfB5R-" /><figcaption>OSM — Grafana</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2eLGkR2qFR7fvKJS" /><figcaption>OSM — Grafana</figcaption></figure><p>Zipkin for APM and Tracing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mCxM68M605rbfbtM" /><figcaption>OSM — Tracing — Zipkin</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2pzu36Ao4UrbxWtP" /><figcaption>OSM — Tracing — Zipkin</figcaption></figure><p>Using the SMI specification offers the promise of moving more easily to another SMI-configured service mesh and also eliminates the need for a significant configuration change when moving from mesh providers. As of today SMI doesn’t fit in scenarios where advanced configuration such as circuit breaking, location based load balancing etc. are required, but all these are possible if SMI is further developed to consume advances Envoy xDS APIs.</p><p>Microsoft is planning to hand over control of Open Service Mesh to the CNCF and this will help it to score points with the wider open-source community, which has been angered by Google’s decision not to do so with Istio (handing over Istio to <a href="https://openusage.org/">Open Usage Commons</a>).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcf41fd30732" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/open-service-mesh-microsofts-smi-based-open-source-service-mesh-implementation-bcf41fd30732">Open Service Mesh — Microsoft’s SMI based Open Source Service Mesh Implementation</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate for Anthos — Application Migration and Modernization]]></title>
            <link>https://itnext.io/migrate-for-anthos-application-migration-and-modernization-78113264b07a?source=rss-e57e2ec3b649------2</link>
            <guid isPermaLink="false">https://medium.com/p/78113264b07a</guid>
            <category><![CDATA[google-anthos]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[application-modernization]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <dc:creator><![CDATA[Gokul Chandra]]></dc:creator>
            <pubDate>Fri, 07 Aug 2020 07:50:49 GMT</pubDate>
            <atom:updated>2020-08-07T08:14:55.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvNdiXDJHe_RVzU-5dQqEg.png" /><figcaption>Modernize applications with Migrate for Anthos</figcaption></figure><h3>Migrate for Anthos — Application Migration and Modernization</h3><p>Migration involves lifting and shifting virtual machines/legacy applications running on physical servers into public cloud platforms which offer many benefits, including enhanced agility, significantly reduced overhead compared to a data center, and a standard usage and management environment. The first step of modernizing an application is to break an application into a set of container images that include everything needed to run a portion of the application: code, runtime, system tools, system libraries, and settings. With Kubernetes all these containers can be hosted on a “managed” cluster where all the underlying Hardware, OS/Kernel and Cluster SRE (Site Reliability Engineering) become the responsibility of the cloud provider.</p><p><a href="https://cloud.google.com/migrate/anthos/docs/getting-started">Migrate for Anthos</a> addresses both migration and modernization providing an almost real-time solution to take an existing VM and make it available as a Kubernetes hosted pod with all the values associated with executing the applications in a Kubernetes cluster. This framework minimizes the manual effort required to move and convert existing applications into containers on Anthos GKE and Google Kubernetes Engine. Users can target workloads from virtual machines or physical servers running on-premises, in Compute Engine, or in other clouds.</p><p>For containerizing the virtual machines <em>Migrate for Anthos</em> creates a wrapper container image from the source VM, the OS is replaced with the one supported by GKE, the filesystem is analyzed by parsing fstab and then a Persistent Volume is mounted using Migrate for Anthos Container Storage Interface (CSI) driver which streams the data from the source VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y-B1vgL4pE3RyKhg" /><figcaption>VM to Container Coversion</figcaption></figure><p><em>Migrate for Anthos</em> converts the source VMs to system containers running in GKE. System containers when compared to application containers run multiple processes and applications in a single container. Although, the application is not converted to true microservice architecture, there are some basic advantages with containerizing the monolith application which includes portability, better packing and containerization virtualize applications at the operating system level, with multiple containers running atop the OS kernel directly, making them far more lightweight and agile than VMs, providing developers with a sandboxed view of the OS logically isolated from other applications.</p><p>The tool provides an easy-to-use interface to plan, execute and manage the migration using declarative specification. All controllers run on a processing-cluster (GKE Cluster), users can manage all aspects of migration using CRD’s.</p><p><em>Migrate for Anthos</em> works alongside Migrate for Compute Engine (in cloud-to-cloud migration scenario, not needed when the SourceProvider is Google Compute Engine) which uses streaming technology, where the complete VM storage need not be migrated to run the container in GKE, this speeds up the entire migration process. Formerly Velostrata — this cloud migration technology was acquired by Google last year to augment Kubernetes, and it does so by delivering on two significant capabilities — converting existing VMs into Pods (Kubernetes applications) and streaming on-premise virtual and physical machines to generate clones in GCE instances.</p><h3>Migration — Sequence of Steps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NKzUfiek9c68ue9x" /><figcaption>Migrate for Anthos — Sequence of Steps</figcaption></figure><h3>Google Compute Engine (Virtual Machine) to Google Kubernetes Engine (Kubernetes Deployment)</h3><p>A sample workflow below shows sequence of steps and explanation of components involved in migrating a sample Apache Tomcat application running as virtual machine on GCE (Google Compute Engine) to a Kubernetes Deployment running on GKE (Google Kubernetes Engine) using Migrate for Anthos.</p><p>The migration source here is GCE and there is no need of Migrate for Compute Engine (formerly Velostrata).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kgQFV2UMOwDY-bDC" /><figcaption>Migration of Virtual Machine on GCE to GKE Pod</figcaption></figure><p>The migration architecture involves a GKE cluster to host various controllers required for migration and is referred as ‘processing cluster’. Once the migration is complete all the artifacts are stored in GCS (Google Cloud Storage) and the generated container image is pushed to GCR (Google Container Registry).</p><p>Migration (setting up and managing a Migrate) can be performed using <a href="https://cloud.google.com/migrate/anthos/docs/migctl-reference">Migrate for Anthos CLI</a> (migctl) available in cloud shell or using the Anthos portal (Migrate to containers).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mhvbk8vKILfesvXu" /><figcaption>Migrate to containers — Anthos Portal</figcaption></figure><p>The sequence of steps in migrating a VM to a container, followed by its deployment on GKE is as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ooZRziDIwEVLq_Lh" /><figcaption>Migration of Virtual Machine on GCE to GKE Pod</figcaption></figure><p>A sample Tomcat application deployed from marketplace is used for tryout, the application is deployed to a single Compute Engine instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*abyjbi9cJ7QdHAub" /><figcaption>Sample Tomcat application running as GCE instance</figcaption></figure><p>As shown below the application is fully functional displaying default webpage when accessed using the External_IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9S_9IKYVogZ4ZV7n" /><figcaption>Sample Tomcat application running as GCE instance</figcaption></figure><h3>Setup Migrate for Anthos</h3><p>A GKE cluster called ‘processing cluster’ is created, this will host all required Migrate for Anthos controllers, CRD’s and other components required to perform the migration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3CoxKAKZZUpTYu_F" /><figcaption>Migration processing cluster — GKE</figcaption></figure><p>Once the GKE cluster is available, users can use ‘migctl’ (Migrate for Anthos CLI) to bootstrap required infra.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aC7Ngxilb9Vhg48g" /><figcaption>Bootstrap — Processing Cluster — migctl</figcaption></figure><p>As shown below the ‘<em>migctl setup install</em>’ creates an admission-validator, csi-vlsdisk-controller, csi-vlsdisk-node, v2k-controller-manager, v2k-generic-csi-controller, v2k-generic-csi-node. Apart from these, a cert-deploy job is created which creates required certificates for webhook authentication and a controllers-upgrade job which upgrades the controllers to the latest version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s-mS1heiv0b89ck4" /><figcaption>Processing Cluster — Migrate for Anthos — Components</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pg7zxa-4z4GeEfWk" /><figcaption>Processing Cluster — Migrate for Anthos — Components</figcaption></figure><p>As part of bootstrap, multiple CRD’s are installed on the cluster. Every migration unit/step/configuration persists in the cluster as a custom resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I6XPQ95o_86OWBGJ" /><figcaption>Processing Cluster — Migrate for Anthos — CRD’s</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Uq9ZeXNFf5yTAtm" /><figcaption>Processing Cluster — Migrate for Anthos — CRD’s</figcaption></figure><p>As part of bootstrap a bucket is created in GCS (Google Cloud Storage) which stores the artifacts once the migration is complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7WjTmAB3se6queqR" /><figcaption>GCS Bucket for Artifacts</figcaption></figure><h3>Configuring Source Provider and Machine</h3><p>The migration requires users to define a source where the instance is migrated from. Supported sources include VMware, AWS, Azure, or Compute Engine. If the migration is from CE (Google Compute Engine) there is no need for Migrate for Compute Engine, Migrate for Compute Engine is required when migrating virtual machines (VMs) running on other clouds including on-premises (VMware).</p><p>The SourceProvider is used in migration configuration. Migration configuration can be customized: Selection of areas excluded from containerization, Selection of area to be extracted as Persistent Volume, Artifacts storage location information, storage settings, etc. , shown below is the SourceProvider (for CE) and Migration CRD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*NgfBTgI-6CvmweKP" /><figcaption>SourceProvider and Migration — CRD</figcaption></figure><p>Users can use ‘migctl’ to configure the SourceProvider and the cluster can have multiple SourceProvider(s) objects which can be used in different Migration configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*psoM1xKA0Q1B8dCM" /><figcaption>Configuring SourceProvider — migctl</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LlK9LZtIM_D9BVEf" /><figcaption>Controller — SourceProvider Configuration</figcaption></figure><p>As shown above, the controller manager watches for Anthos Migrate related objects and realize the same on the cluster.</p><p>A CRD SourceProvider holds the information provided above, here the SourceProvider is Compute Engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K1COwsMigHpyNLvu" /><figcaption>SourceProvider-CRD</figcaption></figure><h3>Create Migration Plan</h3><p>Migration of a VM is initiated by starting migration from portal or using the CLI. This results in creation of a migration plan file. This is implemented as a Kubernetes Custom Resource Definition (CRD) and is contained along with additional resources such as a Kubernetes PersistentVolumeClaim in the migration plan file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k4cwrSVDmf11UORI" /><figcaption>Migrate for Anthos — Migration Plan</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Qx6nEOpOar93rzC" /><figcaption>Migrate for Anthos — Migration Plan</figcaption></figure><p>The migration plan is created as a custom resource and the same can be edited using Anthos portal or editing the Migration object from the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cK-88B4yudN5YZNH" /><figcaption>Migrate for Anthos — Migration Plan</figcaption></figure><p>A migration plan can be customized based on the nature of the workload using an intent flag. The flag’s value determines the contents of your migration plan, which in turn guides the migration.</p><p>There are three intents supported in the migration schema. Image: Used with stateless workloads, ImageAndData: Used with stateful workloads where the application and user mode system is extracted to a container image, and data is extracted to a persistent volume, Data: Used with stateful workloads where only the data portion of the migration plan will be extracted into a persistent volume.</p><ul><li>Specifying content to exclude from the migration (Intent: Any):</li></ul><p>By default, Migrate for Anthos will exclude typical VM content that isn’t relevant in the context of GKE. The filters field value lists paths that should be excluded from migration and will not be part of either the container image or the data volume.</p><ul><li>Setting handling for the data volume (Intent: Data or ImageAndData):</li></ul><p>The dataVolumes field specifies a list of folders to include in the data volume as part of the migration. This must be edited for a migration with an intent of Data or ImageAndData.</p><ul><li>Setting the migration’s name and namespace (Intent: Any):</li></ul><p>Using the metadata field, users can specify the name and namespace for the migration. By default, Migrate for Anthos uses the values specified in the migration spec.</p><ul><li>Setting the name of the container image (Intent: Image or ImageAndData):</li></ul><p>Name of the image that will be created from the VM files and directories copied from the source platform. This image will not be runnable on GKE because it hasn’t been adapted for deployment there.</p><ul><li>Setting the Cloud Storage location for deployment artifacts (Intent: Any):</li></ul><p>Migrate for Anthos uses Cloud Storage to store files related to migration, bucket and folder names can be customized.</p><p>This step also creates initial disk snapshots/copies and stores the same in a persistent volume and seamlessly creates PV’s and PVC’s required for the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BXmVTS_JKWK5oKtM" /><figcaption>PV and PVC — Initial Snapshots</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YGrO6cvEM0arkNxl" /><figcaption>PV and PVC — Initial Snapshots</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iHfzWtq1BNV0IMIN" /><figcaption>PV and PVC — Initial Snapshots</figcaption></figure><h3>Migration Plan Execution</h3><p>This stage executes the migration plan and the following tasks are executed:</p><ul><li>Copy files and directories representing the VM to a container image registry as images.</li><li>Migrate for Anthos creates two images: a runnable image for deployment to another cluster and a non-runnable image layer that can be used to update the container image in the future. All the images along with the layers are pushed to GCR.</li><li>Generate configuration YAML files that can be used to deploy the VM to Kubernetes cluster.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kkW9X4ednzwzToUS" /><figcaption>Migrate for Anthos — Migration Plan Execution</figcaption></figure><p>The controller creates a job for generating the artifacts and uploading the image to GCR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jpqh11TUb9dnE5D4" /><figcaption>Migrate for Anthos — Controller — Generate Artifacts</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sQ4Ou6IMX0P0Sm8A" /><figcaption>Migrate for Anthos — Job — Generate Artifacts</figcaption></figure><p>Once the migration is complete the migration details — artifacts section will list all the resource URL’s. Generate artifacts creates a Dockerfile, Container image for deployment (ready-to-use image), Container image base layer (non-runnable image layers), Kubernetes deployment spec, Migration plan and a file with all artifact links.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UAZr5snEeL98qvQ5" /><figcaption>Migrate for Anthos — Artifact Links</figcaption></figure><p>All the artifacts generated above are stored in GCS (Google Cloud Storage):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AHYgBImFRIajpwkQ" /><figcaption>Artifacts in GCS</figcaption></figure><p>Separate buckets are created for artifacts and image layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wSDSUyUVQf89R1kt" /><figcaption>Artifacts in GCS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*beedg7xCSgZjykEZ" /><figcaption>Artifacts in GCS</figcaption></figure><p>The buckets are created by PVC’s requesting storage (StorageClass: &lt;GCS&gt;) and the corresponding persistent volumes are created in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NnEMrDVONVmb4r8c" /><figcaption>Persistent Volumes for Storage</figcaption></figure><p>Generated image is pushed to GCR (Google Container Registry) and the same is configured in deployment spec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qy3654VY_kjB9OMO" /><figcaption>Images pushed to GCR</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vOnOgtSysbByHXjT" /><figcaption>Images pushed to GCR</figcaption></figure><p>Generated Dockerfile:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yXTYFjPOOLxQNVCd" /><figcaption>Generated Dockerfile</figcaption></figure><p>File with all artifact links:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BKI2YqUZTSgEeM60" /><figcaption>Generated File with Artifact Links</figcaption></figure><p>Generated Deployment spec:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7BZEN2SdyS6wa9k9" /><figcaption>Generated Kubernetes Deployment Spec</figcaption></figure><p>The generated deployment file can be used to deploy the migrated application as a Kubernetes deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oE__panJ3Pb7558u" /><figcaption>Deploying to Kubernetes — Using Generated Kubernetes Deployment Spec</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u8LEOTE3aPK9e59Y" /><figcaption>Running Tomcat application on Kubernetes</figcaption></figure><p>The deployment spec will be configured with the image pushed to GCR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4G6hOWYRrqYgR6PZ" /><figcaption>Tomcat deployment spec with image source from GCR</figcaption></figure><p>As shown below the application is fully functional and running as a Kubernetes pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ywqibfT1aeAzbkmX" /><figcaption>Functional Tomcat application on Kubernetes</figcaption></figure><p>Users can use all the features of GKE and other supporting operations, monitoring and networking features available on GCE with the migrated application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1u4XWZLRS4oAUf0F" /><figcaption>Managing Tomcat Application from GKE</figcaption></figure><h3>Virtual Machines from other Clouds and On-Premises (VMware) to Google Kubernetes Engine (cloud-to-cloud migrations)</h3><p>In this scenario, Migrate for Anthos uses Migrate for Compute Manager which streams cloud or on-premise virtual and physical machines to generate clones in GCE instances.</p><p>As the other scenario discussed above is within GCP there is no need for this entity, here VM’s from external cloud providers are migrated (cloned) to GCE and follows the same steps to migrate the VM to a container once the VM is available on GCE. Communication between Migrate for Compute Engine and the source cloud will be via VPN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Piixt7DawmZg3AYh" /><figcaption>Migration of Virtual Machine on External Cloud Platforms to GKE Pod</figcaption></figure><p>Apart from setting up VPN and configuring a Migrate for Compute manager, the VM -&gt; Container migration involves the same steps as CE — SourceProvider.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7OpOUqhKtFhi2Qnw" /><figcaption>Cloud-to-Cloud Migration — Sequence of steps</figcaption></figure><p>Migrate for Compute Engine (formerly Velostrata) is available in marketplace and is deployed as a managed instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*prgmP7O_t-X74w_N" /><figcaption>GCP — Migrate for Compute Engine</figcaption></figure><p>Deploying Migration Manager requires users to enable site-to-site VPN and required firewalls to enable communication between the VPC in which the manager is positioned and the VPC of the source VM on the other cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JdB4TV1PYsOTwrpN" /><figcaption>Migrate for Compute Engine — Prerequisites</figcaption></figure><p>As shown in the scenario below, a site-to-site VPN is created between GCP and AWS. The same implies to Azure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gvacn6agfyA0gcq0" /><figcaption>Site-to-Site VPN — AWS-GCP</figcaption></figure><p>As shown below GCP uses Cloud Router that works over Cloud VPN or Cloud Interconnect connections to provide dynamic routing by using the Border Gateway Protocol (BGP) for your VPC networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B7lii-1_hMBBiEg4" /><figcaption>Site-to-Site VPN — AWS-GCP — CloudRouter</figcaption></figure><p>AWS site-to-site VPN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5R65ioYQ8uSb7I_8" /><figcaption>Site-to-Site VPN — AWS-GCP</figcaption></figure><p>The manager requires specific permissions to handle the instances like switch-off, switch-on etc., this can be enabled using policies attached to a user and providing the user keys while configuring the manager. An example on AWS is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oV6-0a2a60L_txsH" /><figcaption>IAM permissions for Migrate for Compute Engine on AWS</figcaption></figure><p>Once the migration manager is deployed users can configure the same using the portal/UI. Users should provide authentication to both source and destination clouds. For GCP it is using the service-account-json file and for AWS it is AcessKey and SecurityKey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hW4KzDVCGiWpexax" /><figcaption>Configuring — Migrate for Compute Engine</figcaption></figure><p>A Cloud Extension is configured which creates required VM’s on GCP (destination). A Cloud Extension uses a dual-node active/passive configuration for high availability; each node serves its own workloads while providing backup for the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xMyWLtmMbO3NVLnj" /><figcaption>Configuring — Migrate for Compute Engine</figcaption></figure><p>Runbooks are configured which define the VMs to be migrated, their order, and additional configuration parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SwdDz1xypyitR2yA" /><figcaption>Configuring — Migrate for Compute Engine</figcaption></figure><p>Migrate for Anthos ServiceProvider spec should be configured with the manager information to enable migration, in this case there are two steps where the first migrates (technically creates a clone as GCE instances) the VM from an other cloud provider and secondly containerizing the same. Spec for migrating to compute engine from other clouds and on-prem is as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tz-6Xz83Oq9hlZZf" /><figcaption>Configuring — Migrate for Compute Engine — SourceProvider</figcaption></figure><p>*******************************************************************</p><p>Accelerating the migration and adoption of the modern platform enables the business to operate more efficiently with unified policy, management, and skill sets across existing and newly developed apps.</p><p>When workloads are upgraded to containers running on managed Kubernetes clusters like GKE, IT departments can eliminate OS-level maintenance and security patching for VMs and automate at scale. Legacy apps that are migrated into GKE can be easily upgraded with modern functionality and a large set of available/maintained add-ons can be used for integration without having to rewrite application code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78113264b07a" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/migrate-for-anthos-application-migration-and-modernization-78113264b07a">Migrate for Anthos — Application Migration and Modernization</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>