One of our customers was undergoing a migration from their existing ECS cluster on AWS to Kubernetes (GKE) for their 20 microservices. One of the key challenges that we faced during this migration was handling the complex rewriting that was configured via AWS API Gateway.

In this blog post, we will discuss how we managed multiple URL redirects while moving a customer from AWS ECS to GKE, using a simple Nginx Proxy deployment and Config Maps in Kubernetes.

Challenge

Our customer had multiple URL rewrites configured at their API Gateway to route requests within AWS ECS. These redirects were tightly integrated with the customer’s applications and their clients' applications, which made it impossible to remove them completely unless code changes at both theirs and their 100s of clients' applications. We needed to find a solution that would enable us to manage these redirects while migrating the customer to GKE without impacting their functionality.

Solution

Initially, we thought of using a Ngnix Ingress on the GKE cluster to solve for the rewrite. This failed as the ingress was not built to support such complex Rewrites (Path based rewrites where the value of the path will not remain the same while rewrites are happening). After evaluating various options, we decided to deploy a simple Nginx Proxy as a service in the Kubernetes cluster, which acted as an entry point for all incoming traffic. We then used old-school Nginx rewrite rules to handle the URL rewrites required by the customer. These rules were stored in Config Maps, which we updated whenever there was a change in the rewrite requirements.

Deployment of Nginx Proxy: We deployed the Nginx Proxy as a Kubernetes Deployment with three replicas(HA). We used a ConfigMap to define the Nginx configuration file and mounted it as a volume in the Nginx container. The Nginx configuration file contained the rewrite rules required by the customer’s applications, which were defined using the Nginx Rewrite module. nginx.conf

user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" trace: "$http_x_trace_id" proxy: "$proxy_host" time: $request_time';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    keepalive_timeout  60;
    proxy_set_header X-Real-IP  $remote_addr;
    #proxy_set_header X-Real-IP-LK  $var;
    proxy_set_header X-Forwarded-For http://cloudside.dev;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_connect_timeout 240;
    proxy_send_timeout 240;
    proxy_read_timeout 240;

    server {

        listen 80;
        server_name cloudside.dev;

        ############ Nginx health checking path
        location /index.html {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
        ############ Service 1 rewrites
        location /xyz/svc1 {
            rewrite ^/xyz/svc1$ /svc1$ break;
            proxy_pass http://svc1;
        }
         ############ Service 2 rewrites
        location = /xyz/pqr/svc2 {
            rewrite ^/xyz/pqr/svc2(.*)$ /svc2/pqr$1 break;
            proxy_pass http://svc2;
        }
         ########### Service 3 rewrites
        location = /rest/svc3/v1/webhook {
            rewrite ^/rest/svc3/v1/webhook(.*)$ /svc3/v1/webhook$1 break;
            proxy_pass http://svc3;
        }
    }
    include /etc/nginx/conf.d/*.conf;
}

The above configuration file shows how we handled the URL rewrite for the customer’s applications. We used the rewrite directive to modify the incoming request URL, and then passed it to the corresponding Kubernetes service using proxy_pass.

Use of Config Maps: We used Config Maps to store the Nginx configuration file as a key-value pair. This enabled us to update the configuration file whenever there was a change in the rewrite requirements, without having to redeploy the Nginx Proxy.

kubectl create configmap nginx-configmap --from-file=nginx.conf

or write the yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;
    events {
        worker_connections  1024;
    }
    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for" trace: "$http_x_trace_id" proxy: "$proxy_host" time: $request_time';
        access_log  /var/log/nginx/access.log  main;
        sendfile        on;
        keepalive_timeout  60;
        proxy_set_header X-Real-IP  $remote_addr;
        #proxy_set_header X-Real-IP-LK  $var;
        proxy_set_header X-Forwarded-For http://cloudside.dev;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_connect_timeout 240;
        proxy_send_timeout 240;
        proxy_read_timeout 240;
    
        server {
    
            listen 80;
            server_name cloudside.dev;
    
            ############ Nginx health checking path
            location /index.html {
                root   /usr/share/nginx/html;
                index  index.html index.htm;
            }
    
            ############ Service 1 rewrites
            location /xyz/svc1 {
                rewrite ^/xyz/svc1$ /svc1$ break;
                proxy_pass http://svc1;
            }
    
            ############ Service 2 rewrites
            location = /xyz/pqr/svc2 {
                rewrite ^/xyz/pqr/svc2(.*)$ /svc2/pqr$1 break;
                proxy_pass http://svc2;
            }
    
            ########### Service 3 rewrites
            location = /rest/svc3/v1/webhook {
                rewrite ^/rest/svc3/v1/webhook(.*)$ /svc3/v1/webhook$1 break;
                proxy_pass http://svc3;
            }
        }
        include /etc/nginx/conf.d/*.conf;
    }

If we want to make any changes to the config map, we need to update the config map and do a rolling restart to the Nginx deployment to pick up the latest config map. We have 3 replicas of the proxy running hence the rolling restart of deployment will update the config map without any downtime. If you make a syntax error in the config map, the rolling update will fail and the new Nginx pod will not come up resulting in your production traffic not being affected.

While this approach is not ideal for managing service rewrites in the long term, it can be of great help during a transition to a full-fledged Gateway implementation. In my opinion, Gateway will soon replace ingress in Kubernetes with some great functionalities in the works.

Happy Engineering on Google Cloud!

Overcoming URL Rewrite Roadblocks in Kubernetes (The Old Nginx way): Journey from AWS ECS to GCP… was originally published in The Cloudside View on Medium, where people are continuing the conversation by highlighting and responding to this story.

Errors & Solutions — Anthos on VMware

This is part of the series of posts for setting up Anthos on VMware environment. Take a look at the step by step guide for the initial deployment of Anthos on VMware. In this post, I have consolidated all the issues and errors I faced during the installation of Admin Workstation VM, Admin Cluster, Seesaw LB deployment, and User Cluster along with debugging steps and solutions.

Installation Errors of Admin Workstation VM

**

Error: ComputeResource is not assignable to type mo.ClusterComputeResource” during deployment

Solutions: Your cluster has only one ESXi host. Please add another host to your cluster and you will be able to move pass this error.

****

Error: resource pool not found

Solutions: You will encounter this error if you have not created a resource pool on vSphere Enterprise Plus license. If you are on vSphere Standard license, you can’t create the resource pool as resource pool creation requires DRS support. DRS support is not availbe in standard license. What can you do now?

You can fill in resourcePool: "your_folder/your_cluster/Resources/"

Note: the slash(/) at the end is about 3 hours troubleshooting if you are not a VMware expert :)

Installation Errors of Admin Cluster

**

Error: [FAILURE] Test VM: VCenter access and permission: unable to curl vSphere session endpoint

Solutions: This errors is usually encountered when your vCenter endpoint (vcenter.cloudside.in) is not reachbale from the test VM created in check-config file. When your endpoint is a private endpoint and you have not defined a local DNS server, you will face this. To solve this, we need to update the dnsServers: field under hostConfig entry in admin-cluster.yaml for our endpoint to be available in the network.

You can also do a nslookup test from the admin workstation VM to check which DNS server your system is using.

nslookup vcenter.cloudside.in

you must update the Admin workstation DNS servers manually at this moment for all operations on the admin workstation to work.

****

Error: CreateError: Failed to check if machine gke-admin-xxx exists: failed to create vsphere client: Post “https://vcenter.cloudside.in/sdk":

dial tcp: lookup vcenter.cloudside.in on 198.51.100.1:53: no such host

Solutions: The solutions for this error is same as above explained error, [FAILURE] Test VM: VCenter access and permission: unable to curl vSphere session endpoint

Installation Errors of Seesaw LB VM for User Cluster

Error: Seesaw LB VM creation timeouts as the creation of the template is taking too long on vCenter.

gkectl create loadbalancer command is failing before the LB VM can be created.

Solutions: This error could occure if you are using HDD in your physical host datastore. In my case, the base OS image ubuntu_containerd was stored in a different datastore on different ESXi host compaired to the user cluster creation host. The transfer time of the OS image from one datastore to another was taking alot of time and failing this operation. I has resource constrains on original host where admin cluster was running. Thats why I had to install the user cluster on anthoer host in vCenter datacenter.

To solve this issue,

1. I updated the datastore value in admin-cluster.yamlto another host datastore(server2_datastore1). This would mean that if the VM template for base OS are not found on vCenter datacenter, gkectl will upload them again based on updated settings.

2. DELETE the old VM template files from old (server1_datastore1) datastore. Now run the validate configs command:

sudo gkectl check-config --config admin-cluster.yaml --alsologtostderr --v=5

This will upload the new VM templates to server2_datastore1 datastore. Now when you rerun the create Seesaw LB VM command, it will create the VM before gkectkl timeout.

General cleanups you have to do before rerunning the admin and user cluster create commands.

• Delete the admin cluster data disk manually from vCenter portal. You can find the location of this under field dataDisk: on admin-cluster.yaml
• ISSUE:data disk admin-cluster-disk.vmdk already exists. After the HDD related timeout(gkectl create loadbalancer failures), the vCenter still creates the VM when the files are copied from one DS to another DS. You should delete that resource from vCenter portal before rerunning the create command.
• The Anthos deployment also creates a local k8s cluster on the admin workstation. Sometimes due to any timeout failures, this cluster doesn’t get deleted automatically. to clean up this, use sudo docker stop gkectl-control-planeand sudo docker rm gkectl-control-planecommands.
• If user-cluster create command results in any error, please delete the already created resources using

sudo gkectl --kubeconfig kubeconfig delete cluster --cluster=user-cluster-name --force

• gkectl creates a snapshot of the local cluster if the admin cluster create command results in a failure. If you are not able to debug the exact issues from gkectl logs, you can tap into the snapshot of the local cluster. (snapshot is saved in the current directory). From the snapshot, you can check the gkectlLogs folder.

A lot of other such issues are already documented in official known issues guide.

I hope this was helpful for you! happy reading and good luck solving your issues.

Issues faced with Anthos Installation to VMware vSphere was originally published in The Cloudside View on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anthos is relatively new and I had to go through a considerable amount of trial and error and a lot of gotcha moments while setting up Anthos in VMWare environment. I am sure the documentation will improve with time and with more customer adoption. Meanwhile, I thought it would be useful to publish my experiences in a series of posts.

In this post, we will cover the following

• VMWare pre-requisites
• Deploy the Admin Workstation
• Create and configure the admin cluster
• Create and configure a user cluster

Let’s get started! :)

Getting your vSphere environment ready

Google defines the minimum hardware requirements for Anthos deployment here. Please make sure you are not short on computing/disk/network resources.

a few additional things that you would miss is how your Vcenter settings are configured.

1. vCenter Server Standard license is supported and you don’t need to have DRS enabled on the vCenter cluster (it's good if you have it).
2. Please create a folder, cluster, and a resourcePool. I know you can’t create a resourcePool if you are using a standard license, don’t worry! I have got you covered with a workaround in the config section below.
3. If you can, use SSD storage for your datastore as HDD slows down the overall performance of your Anthos cluster as the etcd (distributed data store for k8s) operations will be slower, ultimately resulting in slower response time from kube API server.
4. If you have a local DNS server and NTP server, note down the IPs as they will be required during configurations.
5. Make sure your cluster has internet access or bare minimum access to these google APIs.
6. Finally, install gkeadm in your system.

Let's deploy the Admin Workstation

By including the admin workstation in the Anthos setup, google tried to minimize the errors you will make while creating and managing Anthos Admin and User clusters.

This VM is where you will write/edit the Admin and user cluster configurations. I am going to follow Google’s deployment guide for Anthos going forward and highlight a few places where you may have confusion filling in the correct information.

Generate your templates:

./gkeadm create config

Check your current directory, you will have these 2 files (credential.yaml, admin-ws-config.yaml)created for you. Incredential.yaml file, fill in your vCenter username and password. Please make sure these creds have sufficient permissions to create vCenter resources.

admin-ws-config.yaml file is where you set your workstation config. Follow this GCP guide for it.

How do you fill in resourcePool information if you are on Standard License

resourcePool: “your-folder/your-cluster/Resources/”

You can refer my admin-ws-config.yamlfile below

Create your admin workstation, and automatically create service accounts:

./gkeadm create admin-workstation --auto-create-service-accounts

this should get you going and create the admin workstation and the required service account for Admin and User clusters. SSH into the admin workstation and verify that you have admin and user cluster config files plus the service account JSON files.

We will now prepare for setting up the LB configs and subsequently Admin cluster.

Setting up LB configs

We are using a bundled Loadbalancing approach for our setup(Seesaw LB). This is a good approach as we offload complete management of the LB back to Anthos. We, however, have to provide a small network configuration. let’s fill it in

We need the following IPs reserved for the LB setup for both Admin and User Cluster

• Admin Cluster Seesaw LB: Node IPs to be filled in admin-seesaw-ipblock.yaml controlPlaneVIP and masterIP in admin-cluster.yaml

• User Cluster Seesaw LB: Node IPs to be filled in user-seesaw-ipblock.yaml controlPlaneVIP, masterIP and ingressVIP in user-cluster.yaml

We will be deploying the LB VM after the config checks are completed for the Admin cluster. let’s go ahead with Admin cluster creation.

Let's deploy the Admin Cluster

This is the brain of your Anthos cluster. essentially, Anthos is making you create a Kubernetes cluster(Admin Cluster) to manage another Kubernetes cluster (User Cluster) where you deploy your apps.

1. If you are using the ipMode as static, let’s fill in admin-cluster-ipblock.yaml.you can skip this if your IP allocation mode is dynamic. we need at least 5 IPs for admin cluster configuration. If your ipblock file has hostconfig entry, please remove it as this has been moved to the main admin-cluster.yamlfile.

2. Let's fill inadmin-cluster.yaml.Please fill in hostConfig section if your config file doesn’t have it.

I have added my comments with # on top of all important config, please follow along to create the cluster.

3. Its time to validate the configs we have filled:

sudo gkectl check-config --config admin-cluster.yaml --alsologtostderr --v=5

• This will check for all the prerequisites and would let you know of any misconfiguration.
• The –alsologtostderr flag will show you the detailed logs of how and what checks are being performed. The output is actually difficult to read, but you would manage :)
• After the initial tests are passed, this goes ahead and creates a VM and runs the startup script of the admin cluster on this VM to check if the environment is correctly configured.
• While this is running, you can check that a VM named check-config gets created in your vCenter portal and gets deleted automatically once the startup script fails or succeeds.
• Fix any issue you might have encountered.
• This finally results in “All validation results were SUCCESS”.

4. Successful config validations lead us to prepare our environment for creating the Admin cluster. Run gkectl prepare to initialize your vSphere environment:

gkectl prepare --config admin-cluster.yaml --skip-validation-all

• This downloads the uploads VM template gke-on-prem-ubuntu-1.x.x-gke.4 and gke-on-prem-cos-1.x.x-gke.4 to the datastore in your vSphere. These VM templates contain “ubuntu_containerd” and “cos” OS images.
• The default OS image used for admin cluster is ubuntu_containerd but you can choose to use google’s own “cos” i.e container optimized OS with Docker.
• The –skip-validation-all flag will skip the validations tests that you have already performed in the previous step.

5. After prepare, we will create the LB VM for the admin cluster. We are using Seesaw LB.

gkectl create loadbalancer --config admin-cluster.yaml alsologtostderr --v=5

• This would create and configure the Seesaw LB VM.
• A successful creation would save a file named seesaw-for-gke-admin.yaml in your current directory. Admin cluster will use this file to connect and configure the LB part.

6. Finally, let's create the admin cluster

gkectl create admin --config admin-cluster.yaml alsologtostderr --v=5

• This would take about 20–40 min depending upon the storage used in your vCenter environment. I was using HDD and it took about 35–40 min for this command to complete.
• This creates the Admin cluster control plane VM and two VMs to run all k8s addons. Addons include things like GCP’s operations suite, Grafana, Prometheus, etc.(If enabled).

• This automatically configures the Seesaw LB to be used with the Anthos admin cluster.
• Once this successfully creates the admin cluster, it saves the admin kubeconfig file as kubeconfig
• Verify that your admin cluster is up and running by listing the nodes

kubectl get nodes --kubeconfig kubeconfig

Congratulation! you have the brain of your Anthos cluster ready.

Let’s deploy the User Cluster and complete the setup

User cluster is where you run your apps. This is indeed a k8s cluster in itself just that it is operationally managed by the admin cluster.

For every user cluster, the admin cluster creates a User cluster control plane that runs the Kubernetes API server, the scheduler, and several controllers for the user cluster.

1. Now, let’s start with updating the user-cluster-ipblock.yaml.for now, we are creating only a 3 node cluster but it's good to have more IPs reserved for the user cluster as you might need more user clusters to support your deployment.

2. Let’s fill inuser-cluster.yaml.Please fill in hostConfig section if your config file doesn’t have it. hostConfig is no longer supported in user-cluster-ipblock.yamlfile.

• This If the user-cluster is going to be in the same environment as the admin cluster, please skip filling any information under vCenter until you want to make some changes to this global setting. (for ex: using a different datastore)
• Anthos now also supports creating user clusters in different vCenter environments, and for this, you have to make changes in the vCenter config.
• Please make you keep the VM sizes to the min of what is defined in the config file below. config check will return errors if you configured less RAM/CPU.

3. Its time to validate the configs we have filled:

sudo gkectl check-config --kubeconfig kubeconfig --config user-cluster.yaml --alsologtostderr --v=5

• The only difference from the admin cluster config check is that we are using a flag — kubeconfig kubeconfig to point the Admin k8s Cluster.
• It does the same checks as the admin cluster with a startup script of user-cluster related operations.

4. After check-config, there is no prepare command here as we already have the VM templates in vCenter. we will create the LB VM for the user cluster.

gkectl create loadbalancer --kubeconfig kubeconfig --config user-cluster.yaml alsologtostderr --v=5

• This would create and configure the Seesaw LB VM.
• A successful creation would save a file named seesaw-for-your-user-cluster.yaml in your current directory. The user cluster will use this file to connect and configure the LB part.

5. let’s create the user cluster

gkectl create cluster --config user-cluster.yaml alsologtostderr --v=5

• you should use the — skip-validation-all flag to skip the time-taking config checks if you successfully run the check-config command.
• As admin cluster creation, it would take 20–40 min depending upon the storage used in your vCenter environment.
• This will create a User cluster control plane VM which is controlled by the Admin cluster for all operations
• The also create 3 (defined in config file) node pools for running our applications.
• Once this successfully creates the user cluster, it saves the user kubeconfig file as user-cluster-kubeconfig
• Verify that your user cluster is up and running by listing the nodes

kubectl get nodes --kubeconfig user-cluster-kubeconfig 

If you look at the nodes, you would notice it does not show the kube-api master because it is part of the admin cluster. This is listed in nodes of admin cluster as User cluster control plane (host4 in screenshot)

Congratulation! Now you can deploy your apps to the user cluster.

**As a next step, you can connect these clusters to the GCP console using the GCP service “Connect”. I will be writing a post on this shortly and will update the URL here.**

**I will also be writing about how to set up automated CI/CD to your Anthos Cluster using CloudBuild. I will update the URL here.**

I have consolidated the errors I faced during this deployment in this blog — Errors & Solutions — Anthos on VMware

Now is the time to explore more about Anthos services. Apart from official docs, you can take help from this Anthos Series Blogs.

1. Anthos Service Mesh by Anchit Nishant

2. Anthos Config Mangement by Anchit Nishant

I hope this post was helpful in your quest of installing the Anthos cluster on VMware in addition to the official guide.

Cheers!

Anthos Cluster on VMware vSphere was originally published in The Cloudside View on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is a hybrid environment. The back-office applications will remain on-prem for the time being and all the core applications will get migrated to GKE. We are working with 3 environments i.e Dev, stage, and Production created as 3 projects on GCP with each one having its own VPC. The task at hand is to connect these 3 environments with on-prem in a secure way where service level communication should happen privately and with service names and not IP addresses.

We have used Cloud VPN (HA) for network-level communication from each project back to on-prem. Discovery of services by their names will be completed by use of Cloud DNS forwarders.

Let’s get into it

What is already in place? You should check this list and have these details ready

• Network communication between On-prem and GCP via Cloud VPN from all 3 projects
• A local DNS server on-prem (which hosts the on-prem services). In our case, the DNS server used is DNS hosted on Windows server 2016.
• List of on-prem hosted internal domains (For ex: we have umesh.local as the main domain on on-prem)
• The IP address of DNS server on-prem
• List of domains that you will host on GCP (For ex: we will create a private hosted zone dev.umesh.gcp, stage.umesh.gcp and prod.umesh.gcp on Cloud DNS in respective projects)

What do we need to create on GCP?

• Cloud DNS private hosted zone in each project (dev.umesh.gcp in dev, stage.umesh.gcp in stage and prod.umesh.gcp in production. Choose the appropriate VPC while configuring them). Note: I am going to use gcloud commands to create them, you can do this from the GCP console too.

Repeat this step for the other 2 projects by changing the values for project id, name, DNS name, and network name.

• Cloud DNS forwarding zone (Under Option: Forward queries to another server) for on-prem umesh.local domain. This has to be created in only one project(in our case, prod) and not all of the projects. We will be using DNS peeing from this project(Prod) to all other projects (Dev and Stage)

You can’t add records to this forwarding zone as the data comes from your on-prem DNS. Keep in mind that you can add multiple DNS server ips if the on-prem server(s) are hosted that way.

Normally this should be created in all the projects. but this is where we have some limitations (or designed this way) from GCP. Offical from Google: “queries from any VPC network have the same IP range 35.199.192.0/19 as the source. Therefore, responses can’t be routed correctly unless you have separate environments on-premises”. Read more

• Cloud DNS peering zones. Since DNS peering unidirectionally forwards DNS requests, for our case, we have created a forwarding zone in the prod project, so we will have to create DNS peering connections from Dev and Stage projects VPC (called DNS consumer network) to Prod project VPC (DNS producer network). If you want all networks to discover anything and everything in the whole network setup, create a DNS Peering zone in all 3 projects so they all become consumers and producers.

In this example, we will go ahead and create 2 peering zone in dev and stage projects

Repeat this for the stage project.

• The next step is to create DNS policies in all 3 projects for your respective VPC. You can configure one DNS server policy for each VPC network. you have control over both inbound server policy and outbound server policy. In our use case, we are required to create an inbound server policy.

you can read more about DNS policies here

At this stage, by creating private zones in each project, forwarding zone in one of the projects (prod), creating peering zone in dev and stage VPC, and then adding inbound DNS server policy, we have made sure that the GCP side of systems is well connected to route requests to appropriate on-prem DNS servers.

What do we require to configure on on-prem?

• There are a couple of things we need to perform on the on-prem side. The first one is to add DNS forwarders IPs for all 3 GCP networks on 3 GCP projects. How do we get DNS server IPs for hosted Cloud DNS zones?

you run this command in each of the projects and it will list you one IP for each subnet in your VPC. Take note of all these IPs as you have to add this to the On-prem DNS server as DNS forwarder IP.

You can use this article to see how to add forwarders to the windows server

• The second one is to configure your on-premises routers and firewalls to permit traffic from Google’s 35.199.192.0/19 IP address range. Cloud DNS uses the 35.199.192.0/19 source range for all customers. This range is only accessible from a Google Cloud VPC network or from an on-premises network connected to a VPC network.

With this in place, you will be able to discover domains hosted on-prem (umesh.local) from GCPs in any private hosted zone (dev.umesh.gcp, stage.umesh.gcp and prod.umesh.gcp) and vice versa.

Helpful Resources

1. hybrid-architecture-using-multiple-vpc-networks
2. Troubleshooting
3. Issues with DNS Peering

Thanks for reading this, hope it helped you in some way. Happy Networking on Google Cloud

Extending GCP Cloud DNS to On-prem with Multiple VPCs was originally published in The Cloudside View on Medium, where people are continuing the conversation by highlighting and responding to this story.

This time, we are in the process of migrating a huge scale data warehouse off Netezza to AWS Redshift. The data reside in the on-prem data center and is enough in size for us to ask for a Snowball edge.

The data center team received the snowball and its time we connect it to power and switch it on.

Let's get to the next part when the snowball has booted up and shows the LOCKED state. Now we will talk a bit about the networking capabilities the snowball supports. It has 3 ports, 10 Gb — RJ45, 25 Gb — SFP+, 40 Gb — QSFP+. If you would like to use SFP+ or QSFP+, you should have your own SFP transceiver as the Snowball ships with no cable other then power.

The connections tab on Snowball shows the option to choose which port you are configuring. We used SFP+ and configured the interface with a static IP. (you should be ready with static IP, netmask and gateway details upfront for the port you are going to use). Now its time we start with Unlocking the snowball. A detailed guide is available here on AWS docs.

I will walk through the steps/commands we used for unlocking, initializing, setup and configuring the Snowball edge. I hope you have a machine to work with the snowball, part of the local network in the datacenter. The machine is with the latest AWS CLI or Snowball client. More on how to configure the CLI or client is available here.

We will start by configuring a local snowball profile. It's good to have this profile otherwise we have to pass the manifest file and unlock code with every command. We should have the Manifest file and Unlock Code obtained from the Snowball page on the AWS console.

Start with |snowballEdge configure|

Snowball Edge Manifest Path: Path/to/manifest/file

Unlock Code: 29 character unlock code

Default Endpoint: https://10.0.2.197

Note: If you want to configure mutliple profiles for Snowball, use the tag — profile nameoftheprofile and pass the same profile tag for all the following commands)

|snowballEdge unlock-device|

Your Snowball Edge device is unlocking. You may determine the unlock state of your device using the describe-device command. Your Snowball Edge device will be available for use when it is in the UNLOCKED state.

This will unlock your snowball in about 5 mins given your manifest file and unlock code are correctly entered while configuring the profile. To check the status we can look at the snowball screen or run |snowballEdge describe-device|

Once the device unlocks, we can start with the next set of commands which will be used for actual data transfer.

For us to write data to snowball using CLI or SDK, we first need to obtain the access keys and secret access key from the snowball. We would also need the ca-bundle .pem file for the https connection.

|snowballEdge list-access-keys|

{ “AccessKeyIds” : [ “AKIAIOSFODNN7EXAMPLE” ] }

Once we have the access key, we can pass this as in argument to get the secret access key

|snowballEdge get-secret-access-key — access-key-id Access Key|

[snowballEdge]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

It's good to save these outputs in a file as we will need these keys for further use.

Run |snowballEdge get-secret-access-key — access-key-id Access Key >local-access-key.txt|

This local-access-key.txt is saved in the current user directory of the respective OS.

Let's also obtain the ca-bundle file

|snowballEdge list-certificates|

{
“Certificates” : [ {
“CertificateArn” : “arn:aws:snowball-device:::certificate/78EXAMPLE516EXAMPLEf538EXAMPLEa7”,
“SubjectAlternativeNames” : [ “192.0.2.0” ]
} ]
}

This will give us the AWS arn for the certificate of our snowball endpoint. We will now use this arn to obtain the certificate.

|snowballEdge get-certificate — certificate-arn arn:aws:snowball-device:::certificate/78EXAMPLE516EXAMPLEf538EXAMPLEa7|

— — -BEGIN CERTIFICATE — — -
Certificate
 — — -END CERTIFICATE — — -

copy this certificate file with .pem using

|snowballEdge get-certificate — certificate-arn arn:aws:snowball-device:::certificate/78EXAMPLE516EXAMPLEf538EXAMPLEa7 >ca-bundle.pem|

Now we are ready with the credentials for the CLI to work with the snowball. One of the points we missed was about services that run on snowball and what is useful in our case. For us, we are only using the storage part(S3) of snowball and not the compute or IOT part which is also available in some regions.

Amazon S3 and Amazon EC2 services are always on by default and rest other we can start if we would like to use them. You can verify the status of a service by running |snowballEdge describe-device| More on this here.

In this section, we will work with transferring the data to snowball edge using the AWS CLI.

We will start by configuring the AWS CLI profile with access key and secret key obtained in previous steps.

|aws configure — profile demo1|

fill in the access key and secret key you have obtained from the Snowball.

Note: Remember to pass the access key and secret key we have obtained from Snowball device and not use the one you might have downloaded from AWS IAM. We normally configure the AWS Profile with access keys from AWS IAM.

If you have successfully configured the AWS profile, you can run the AWS S3 commands to interact with the snowball.

|aws s3 ls| will return you the name of the bucket the snowball has shipped with. This is not the actual bucket on S3, it is root folder in snowball where you will transfer the data.

Test sending some data to this bucket. It's good to create some test folder and not use the root folder(the name of the bucket).

|aws s3 cp /home/searce.png /bucket-name/test-folder/|

here you go, the file must be copied to the test folder. You can verify this by |aws s3 ls /bucket-name/test-folder|

I wrap this here, We have one more article coming up on how we used AWS DMS and SCT to actually transfer data from Netezza to Redshift. I will add the article link here when it is available.

I hope this reading gives you some confidence going into your first snowball use case. Have fun migrating data to AWS :)

Internals of Snowball Edge | How to Setup, Configure and work with Snowball Edge & Client was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.

The pace of innovation at AWS is rapid. It is difficult to keep up with new features, enhancements, services that come in every day, let alone knowing the best practices associated with them. Simply put, AWS with hundreds of services across 23 different categories is a complex world and AWS knows, to operate with such breadth and depth using best practices is challenging for any customer.

AWS Landing Zone was launched in mid-2018 and has been unnoticed (my guess based on no/low community discussions) by customers/partners until AWS re:Invent 2018 when AWS Control Tower was announced.

AWS Landing Zone is a nice blend of granular, centralized control over your cloud workloads and consistent level of security and compliance across all your AWS accounts.

I am not going to explain how to implement AWS Landing Zone as AWS’s guide is detailed. However, we will discuss the requirements for the landing zone and a simplified view of what it contains. Let's dig in!

When to implement Landing Zone?

Working with a single account becomes difficult as your environment grows because multiple teams in the same account would trespass on one another due to their different responsibilities and resource needs. You may also have completely different business units or products or security profiles or compliance control which require different accounts. In such scenarios, you need to design and configure a base environment that lays out the account structure, security rules, default network settings, and other foundational services. This base environment is called Landing Zone.

Key elements of Landing Zone architecture

A plethora of best practices?

• Organizations master account: It provisions and manages member accounts under AWS Organizations Service.
• Core accounts in an Organizational Unit: It provides essential functions that are common to all the accounts under the Organizations, such as Log Archive, Security management, and Shared Services like the directory service.
• Team/Group accounts in Organizational Units: These are designated for individual business units at the granularity of the team or group level. For example, a set of Team accounts may consist of the team’s Shared Services account, Development account, Pre-Production account, and Production account.
• Developer accounts: These are “sandboxes” for an individual’s learning and experimentation.
• Connectivity: This is set up with default networking patterns between the accounts and with external data centers.
• Security Baseline:

Looking at these best practices, we realize how this “Infrastructure as code” solution can be beneficial when dealing with multiple accounts.

Still, Implementing this solution itself requires a deep understanding of AWS services and AWS stats that in the Landing Zone Implementation Guide.

Don’t worry! AWS was also mindful of these challenges, and they were quick to automate the whole solution as a managed service named AWS Control Tower. (PS: A lot of AWS services started out as a solution to a particular use case in the form of a cloud formation template(s) and then launched as a managed service)

In short, AWS Control Tower automates the setup of all these plethoras of best practices. A few clicks here and there, you are on your way to setup a baseline infrastructure in AWS with best practices at the core.

Resources

1. Get started with AWS Control Tower here
2. If you want to deploy Landing Zone on your own and learn a few more things about AWS services, read the implementation guide and user guide.

Thanks for your time!

AWS Landing Zone — A Plethora of best practices for multi-account setup was originally published in Ankercloud Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWS VPC has been one of the strong networking backbone for EC2 to perform the way it does but VPC capabilities don’t always cater to the networking requirements enterprises ask. They always need a central gateway where a firewall has to know & manage what packets are flowing in the environment and how. Ultimately enterprises used third party solutions like CISCO CSR router to achieve this. Read about the earlier AWS Solution — Transit VPC

let’s discuss AWS Transit-gateway and how it works. Below image explains pretty much what it is .

Let’s get to the working of AWS Transit-Gateway.

We had a customer in the past where we deployed CISCO CSR to achieve the transitive nature of the network. They had 4 different external apps hosted in 4 different VPC which needed to communicate privately with another app on-prem through VPN. Their compliance required was there will be just one endpoint for VPN and all of the traffic between apps has to flow through the firewall sitting at a central location (VPC) on AWS. Let’s see how we would deploy this today using the Transit-Gateway.

1. We have 2 custom VPC’s with each having one subnet. I have created 1 instance in each VPC(subnet). One of the VM is part of 10.0.0.0/16 network (IP:10.0.1.188) and another one being part of 20.0.0.0/16 network (IP:20.0.1.110)

2. Transit-Gateway is available under the VPC-Dashboard. We will start by creating our first gateway. Enabling DNS support will be useful as finding the applications with their hostname/FQDN. Click create on the bottom right.

3. Let’s create Transit-Gateway Attachments now. These attachments are nothing but the connections to VPC’s. We need to create a different attachment for each VPC with the transit-gateway. Each VPC requires to have a separate attachment.

4. Once we have created the attachments for both VPC, they will be available in this windows.

5. Now, If you have checked the “default route table association” in the first step, no need to modify the Default RT at transit gateway end but we need to modify the VPC’s route table and create the route via transit-gateway. Remember to modify the route table for both the VPC.

6. The final step, let’s test out connection and ping the 20.0.0.0/16 network from 10.0.0.0/16 network.

BOOM! No VPC-Perring, We are able to communicate different VPC using the transit-gateway.

Well, interestingly you can share transit gateway with other AWS accounts or AWS Organization using AWS Resource Access Manager

What else about transit-gateway?

BENEFITS

1. Simplified connectivity
2. Better visibility and network control
3. Availability of On-demand network bandwidth

LIMITATIONS

1. Routing between Amazon VPCs with overlapping CIDRs is not supported.
2. Spoke VPC can’t refer security group of other spokes connected to the gateway.
3. Maximum network bandwidth per VPN connection in the transit mesh is limited to 1.25Gbps. You can use ECMP to get higher VPN bandwidth. Bandwidth limit per VPC attachment is 50Gbps burst.

Check this AWSome slide from AWS to understand where transit gateway can be used.

AWS Transit-Gateway Internals & Working was originally published in Searce on Medium, where people are continuing the conversation by highlighting and responding to this story.