In this article, we’ll walk through deploying a highly available Kubernetes cluster on Azure Virtual Machines, using a stacked etcd topology.

I will organize the article into the following four main sections:

1. Introduction
2. Infrastructure Planning
3. Cluster Implementation
4. Securing Access to the Cluster

🔴 Introduction

🔄 What is High Availability in Kubernetes?

High Availability (HA) in Kubernetes ensures that your cluster can tolerate failures and continue operating without downtime, even when components or nodes go offline. This is achieved by redundant deployments of critical control plane components and careful design of infrastructure to avoid single point of failure.

In a highly available Kubernetes setup, the focus is on:

• Control plane redundancy — so the cluster’s brain doesn’t go down with one node.
• etcd quorum — so the cluster state is never lost or corrupted.
• Worker node failover — so workloads can continue running even if a node dies.

🧠 Components That Require HA

1. kube-apiserver

• Purpose: It’s the front-end for the Kubernetes control plane. All interactions with the cluster happen via the API server.
• Why HA?: If this goes down, no users or controllers can make changes to the cluster.
• HA Strategy:
   — Run multiple instances across control plane nodes.
   — Place them behind a load balancer to ensure a single endpoint.
   — Use health checks (e.g., TCP 6443) to monitor availability.

2. etcd

• Purpose: This is the key-value store where Kubernetes stores all cluster data (desired state, configs, secrets, events, etc.).
• Why HA?: If etcd fails or loses quorum, the cluster can’t operate, even if kube-apiserver is running.
• HA Strategy:
  — Use an odd number of etcd members (minimum 3).
   — Distribute across control plane nodes (stacked topology) or dedicated machines (external topology).
   — Ensure etcd data backups are in place.

3. kube-scheduler

• Purpose: Schedules pods to appropriate worker nodes based on availability, resources, and affinity rules.
• Why HA?: If the scheduler is down, no new workloads can be scheduled even if nodes are available.
• HA Strategy:
   — Run it as a replicated static pod on all control plane nodes.
   — Only one active leader schedules at a time, using leader election.
   — Others remain in standby and take over if the leader fails.

4. kube-controller-manager

• Purpose: Manages internal cluster controllers (replication, endpoints, namespace, node monitoring, etc.).
• Why HA?: If this fails, critical operations like self-healing, scaling, and node health management stop.
• HA Strategy:
   — Similar to the scheduler, run multiple instances with leader election.
   — If the active manager goes down, another takes over seamlessly.

⚙️ Options for High Available Topology

Kubernetes gives us two types of etcd topology options for high availability.

1. Stacked etcd topology: With stacked control plane nodes, where etcd nodes are collocated with control plane nodes.
2. External etcd topology: With external etcd nodes, where etcd runs on separate nodes from the control plane.

You can visit Kubernete’s official documentation for more details on this.
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/

🧱 Stacked etcd Topology vs External etcd topology

When designing a highly available Kubernetes cluster, a key decision involves how to deploy the etcd database, which stores all cluster state. There are two common topologies to choose from: stacked and external.

In a stacked etcd topology, the etcd instances run directly on the same nodes as the control plane components. This approach simplifies the setup process, reduces infrastructure cost, and is well-supported by tools like kubeadm. However, it comes with a tradeoff: if a node fails, you lose both an etcd member and a control plane component simultaneously, which can impact cluster availability.

On the other hand, an external etcd topology runs the etcd cluster on separate, dedicated nodes. This improves fault isolation and performance, especially in larger or production environments. But it also introduces additional complexity, requiring more nodes, a separate deployment process, and extra operational overhead to manage etcd independently.

For this tutorial, we choose the stacked etcd topology because it offers a good balance of high availability and simplicity, making it ideal for self-hosted clusters on virtual machines where infrastructure resources may be limited. It also aligns well with kubeadm, which streamlines the bootstrapping process for a highly available Kubernetes control plane.

🔴 Infrastructure plan

To build a robust, highly available Kubernetes cluster from scratch — without using AKS — we’ll first lay out a secure and scalable network architecture on Azure.

1. Virtual Network and Subnet Design

We begin by creating an Azure Virtual Network (VNet) that will serve as the backbone of our cluster. Inside this VNet, we define two subnets.

• Public Subnet: Exposes entry points to the outside world.
• Private Subnet: Host all Kubernetes-related infrastructure, fully isolated from the direct public access.

🔐 The private subnet will only be accessible via the public subnet, ensuring better control and security.

2. Jumpbox Setup

We’ll create a VM in the public subnet, which acts as a secure access point into the private subnet.

• VM Name: jumpbox-vm
• This VM will have a public IP and SSH access.
• From here, we’ll manage and provision all nodes inside the private subnet.

3. Kubernetes Cluster Nodes

Our Kubernetes cluster will follow a high availability control plane pattern with a stacked etcd topology.

• Control Plane Nodes (in private subnet):
   — master-1
   — master-2
   — master-3
• Worker Nodes (in private subnet):
   — worker-1
   — worker-2
   — worker-3

This 3x3 setup ensures quorum on both etcd and control plane and supports failover for workloads.

4. Internal Load Balancer and NAT Gateway

• We’ll create an Internal Load Balancer (ILB) to distribute traffic among the control plane nodes. This will be used as the --control-plane-endpoint in kubeadm, which we will discuss later in the article.
• All private subnet resources will reach the internet through a NAT Gateway, allowing updates and image pulls without exposing any resources directly.

5. Network Security Groups (NSGs)

To enforce strict access control:

• Public Subnet NSG will only allow SSH (port 22) to the jumpbox-vm.
• Private Subnet NSG will allow:
   — Internal communication between cluster nodes
   — Access only from the jumpbox

🔴 Implementation

✅ Prerequisites

Before we dive into building a highly available Kubernetes cluster on Azure, ensure you have the following tools and access configured:

1. Azure Account with Sufficient Permissions

You’ll need an Azure account with permission to create and manage:

• Resource Groups
• Virtual Networks (VNet)
• Subnets
• Virtual Machines (VMs)
• Network Security Groups (NSGs)
• LoadBalancers and NAT Gateways

If you don’t already have an Azure account, you can create one here (comes with a free tier): 🔗 Create Azure Account

2. Azure CLI Installed

We’ll use the Azure CLI (az) extensively to provision resources. Make sure it's installed and configured on your local machine.

• 🔗 Install Azure CLI
• After installation, login with:

az login

3. Visual Studio Code (or any code editor)

Having a good editor will help you manage and write configuration files and commands.

• 🔗 Download VS Code

4. Basic Knowledge Requirements

You should be comfortable with:

• Linux command line and SSH
• Basic networking (subnets, IPs, firewalls)
• Basic Kubernetes concepts

⚪️ Implementation Setup

Now that you’ve got everything ready, it’s time to build the highly available K8s cluster on Azure — from scratch and without relying on AKS.

First, I will create a directory for my project. Inside this directory, I will create a file named vars.sh. You can open the directory in your code editor (e.g., VS Code).

.
└── vars.sh

1 directory, 1 file

# vars.sh
LOCATION="germanywestcentral"
RESOURCE_GROUP="kubernetes"
ADMIN_USERNAME="azureuser"

VNET_NAME="k8sVnet"
ADDRESS_PREFIX="10.0.0.0/16"

PUBLIC_SUBNET_NAME="public-subnet"
PUBLIC_SUBNET_PREFIX="10.0.1.0/24"

PRIVATE_SUBNET_NAME="private-subnet"
PRIVATE_SUBNET_PREFIX="10.0.2.0/24"

NSG_PUBLIC="nsg-public"
NSG_PRIVATE="nsg-private"

NAT_IP_NAME="nat-ip"
NAT_GATEWAY_NAME="nat-gateway"

JUMPBOX_NAME="jumpbox-vm"
VM_SIZE="Standard_B2ms"

ILB_NAME="k8s-ilb"
ILB_FRONTEND_NAME="k8s-ilb-frontend"
ILB_BACKEND_POOL="k8s-ilb-backend"
ILB_PROBE_NAME="kube-api-probe"
ILB_RULE_NAME="kube-api-rule"
ILB_PRIVATE_IP="10.0.2.10"

I’m going to use these variables for the setup. For example, I’ve selected germanywestcentral as my location. You can modify these values as needed. If you want your cluster to be in a different location, feel free to change it.

Then you can run the following command to use it in your terminal session.

⚠️ Reminder

Every time you start a new terminal session, you need to run the following command to load the environment variables:

source vars.sh

You can verify it by running echo $LOCATION .

echo $LOCATION

Then you have to login to your azure account using azure cli.

az login

After login, you can verify you account details.

az account show

Then you can start by creating a resource group.

az group create \
--name $RESOURCE_GROUP \
--location $LOCATION

You can verify it by running.

az group list \
--query "[?name=='$RESOURCE_GROUP']"

1. Create Virtual Network and Subnets.

Create the virtual network k8sVnet with an address prefix of 10.0.0.0/16, along with a public subnet (10.0.1.0/24) and a private subnet (10.0.2.0/24) by running the following commands in order.

virtual-network

az network vnet create \
--resource-group $RESOURCE_GROUP \
--name $VNET_NAME \
--address-prefix $ADDRESS_PREFIX

public-subnet 10.0.1.0/24

az network vnet subnet create \
--resource-group $RESOURCE_GROUP \
--vnet-name $VNET_NAME \
--name $PUBLIC_SUBNET_NAME \
--address-prefix $PUBLIC_SUBNET_PREFIX

private-subnet 10.0.2.0/24

az network vnet subnet create \
--resource-group $RESOURCE_GROUP \
--vnet-name $VNET_NAME \
--name $PRIVATE_SUBNET_NAME \
--address-prefix $PRIVATE_SUBNET_PREFIX

2. Network Security Groups

Create network security groups and attach them to the subnets.

nsg-public to public-subnet

az network nsg create -g $RESOURCE_GROUP -n $NSG_PUBLIC

az network vnet subnet update \
  --resource-group $RESOURCE_GROUP --vnet-name $VNET_NAME \
  --name $PUBLIC_SUBNET_NAME --network-security-group $NSG_PUBLIC

nsg-private to private-subnet

az network nsg create -g $RESOURCE_GROUP -n $NSG_PRIVATE

az network vnet subnet update \
  --resource-group $RESOURCE_GROUP --vnet-name $VNET_NAME \
  --name $PRIVATE_SUBNET_NAME --network-security-group $NSG_PRIVATE

3. Jumpbox Setup

First, let’s create an SSH key pair to connect to the jumpbox-vm. You can customize the command with your email address.

ssh-keygen -t rsa -b 4096 -C "<your-email>" -f azureuser -N ""

Your directory should now look like the following.

.
├── azureuser
├── azureuser.pub
└── vars.sh

Here, I’m assigning the private IP 10.0.1.4 to the jumpbox-vm, as shown in our diagram. If you've used a different setup, make sure to adjust it accordingly.

az vm create \
  --resource-group $RESOURCE_GROUP \
  --name $JUMPBOX_NAME \
  --image Ubuntu2204 \
  --vnet-name $VNET_NAME \
  --subnet $PUBLIC_SUBNET_NAME \
  --private-ip-address 10.0.1.4 \
  --size $VM_SIZE \
  --nsg $NSG_PUBLIC \
  --admin-username $ADMIN_USERNAME \
  --ssh-key-values azureuser.pub

After running the above command, you will see the publicIpAddress of the jumpbox-vm in the output. Save it for later use.

If you try to SSH into the jumpbox-vm immediately after creating it, it won’t work due to the Network Security Group (NSG) rules. You need to add a rule to allow SSH access to the VM.

az network nsg rule create \
  --resource-group $RESOURCE_GROUP --nsg-name $NSG_PUBLIC \
  --name Allow-SSH --priority 1000 \
  --access Allow --direction Inbound \
  --protocol Tcp --destination-port-range 22 --source-address-prefixes Internet

You can now use the private key we created earlier to SSH into the jumpbox-vm.

ssh -i azureuser azureuser@<jumpbox-public-ip-address>

4. Create Kubernetes Master Nodes. (Control-plane nodes)

master-1 (10.0.2.4)

az vm create \
    --resource-group $RESOURCE_GROUP \
    --name master-1 \
    --image Ubuntu2204 \
    --vnet-name $VNET_NAME \
    --subnet $PRIVATE_SUBNET_NAME \
    --private-ip-address 10.0.2.4 \
    --size $VM_SIZE \
    --nsg $NSG_PRIVATE \
    --public-ip-address "" \
    --admin-username $ADMIN_USERNAME \
    --ssh-key-values azureuser.pub

master-2 (10.0.2.5)

az vm create \
    --resource-group $RESOURCE_GROUP \
    --name master-2 \
    --image Ubuntu2204 \
    --vnet-name $VNET_NAME \
    --subnet $PRIVATE_SUBNET_NAME \
    --private-ip-address 10.0.2.5 \
    --size $VM_SIZE \
    --nsg $NSG_PRIVATE \
    --public-ip-address "" \
    --admin-username $ADMIN_USERNAME \
    --ssh-key-values azureuser.pub

master-3 (10.0.2.5)

az vm create \
    --resource-group $RESOURCE_GROUP \
    --name master-3 \
    --image Ubuntu2204 \
    --vnet-name $VNET_NAME \
    --subnet $PRIVATE_SUBNET_NAME \
    --private-ip-address 10.0.2.6 \
    --size $VM_SIZE \
    --nsg $NSG_PRIVATE \
    --public-ip-address "" \
    --admin-username $ADMIN_USERNAME \
    --ssh-key-values azureuser.pub

5. Accessing Control Plane nodes

To access these nodes, you first need to copy the private key to the jumpbox-vm. You can do this using the following command: use your jumpbox vm ip

scp -i azureuser azureuser azureuser@<jumpbox-public-ip>:/home/azureuser/

You can now SSH into the jumpbox-vm and run the ls command to verify that the file is there.

azureuser@jumpbox-vm:~$ ls
azureuser

You should see the private key there.
Now, if you try to SSH into one of the control plane nodes using the copied private key, you should be able to connect successfully.

ssh -i azureuser azureuser@10.0.2.4

By default, Azure Network Security Groups allow access to any resources inside the Virtual Network from other resources within the same network.

So, we’re going to add a rule to block all SSH access initially.

az network nsg rule create \
  --resource-group $RESOURCE_GROUP --nsg-name $NSG_PRIVATE \
  --name Deny-SSH --priority 1200 \
  --access Deny --direction Inbound \
  --protocol Tcp --destination-port-range 22

Now, if you try to SSH into the Kubernetes nodes, you won’t be able to. Let’s allow SSH access from the jumpbox.

az network nsg rule create \
  --resource-group $RESOURCE_GROUP --nsg-name $NSG_PRIVATE \
  --name Allow-SSH-from-jumpbox --priority 1100 \
  --access Allow --direction Inbound \
  --protocol Tcp --destination-port-range 22 \
  --source-address-prefix 10.0.1.4/32

You should now be able to access the control plane nodes from the jumpbox-vm again.

6. Create the Internal Load Balancer.

az network lb create \
  --resource-group $RESOURCE_GROUP \
  --name $ILB_NAME \
  --sku Standard \
  --vnet-name $VNET_NAME \
  --subnet $PRIVATE_SUBNET_NAME \
  --frontend-ip-name $ILB_FRONTEND_NAME \
  --backend-pool-name $ILB_BACKEND_POOL \
  --private-ip-address $ILB_PRIVATE_IP \
  --private-ip-address-version IPv4 \
  --frontend-ip-zone 1 \
  --public-ip-address "" \
  --output table

Health probe

az network lb probe create \
  --resource-group $RESOURCE_GROUP \
  --lb-name $ILB_NAME \
  --name $ILB_PROBE_NAME \
  --protocol Tcp \
  --port 6443 \
  --interval 15 \
  --threshold 4

Loadbalancing rule for the port 6443

az network lb rule create \
  --resource-group $RESOURCE_GROUP \
  --lb-name $ILB_NAME \
  --name $ILB_RULE_NAME \
  --protocol Tcp \
  --frontend-port 6443 \
  --backend-port 6443 \
  --frontend-ip-name $ILB_FRONTEND_NAME \
  --backend-pool-name $ILB_BACKEND_POOL \
  --probe-name $ILB_PROBE_NAME \
  --idle-timeout 15 \
  --enable-tcp-reset true

7. Add Control-plane nodes to the LoadBalancer

Get the network interface names for the newly created nodes, such as master-1.

To add the control plane nodes to the load balancer backend pool, retrieve both the NIC name and the IP configuration name.

az vm show \
  --resource-group $RESOURCE_GROUP \
  --name master-1 \
  --query "networkProfile.networkInterfaces[0].id" \
  --output tsv | xargs az network nic show --ids | jq '{nicName: .name, ipConfigName: .ipConfigurations[0].name}'

Add the master-1 to the backend pool.

az network nic ip-config address-pool add \
  --address-pool $ILB_BACKEND_POOL \
  --ip-config-name <ipConfigName> \
  --nic-name <nicName> \
  --lb-name $ILB_NAME \
  --resource-group $RESOURCE_GROUP

🔁 Repeat the same steps for the other two control plane nodes, master-2 and master-3.

After adding all three nodes, you can verify the changes in the UI as well.

Now, the control plane nodes are created and connected to the load balancer through a load balancing rule.

8. Add a Nat Gateway

We need a NAT Gateway to allow the private nodes (such as Kubernetes nodes) to access the public internet. For example, nodes may need to install packages or fetch updates. This access will be routed through the NAT Gateway.

Step 1: Create a Public IP Address

az network public-ip create \
  --resource-group $RESOURCE_GROUP \
  --name $NAT_IP_NAME \
  --sku Standard \
  --allocation-method Static \
  --location $LOCATION

Step 2: Create the NAT Gateway using the Public IP

az network nat gateway create \
--resource-group $RESOURCE_GROUP \
--name $NAT_GATEWAY_NAME \
--location $LOCATION \
--public-ip-address $NAT_IP_NAME \
--idle-timeout 10

Step 3: Attach the NAT Gateway to the Private Subnet

az network vnet subnet update \
--resource-group $RESOURCE_GROUP \
--vnet-name $VNET_NAME \
--name $PRIVATE_SUBNET_NAME \
--nat-gateway $NAT_GATEWAY_NAME

Jumpbox update

Before configuring the control plane nodes, let’s SSH into our jumpbox-vm and make some changes to simplify the upcoming steps.

ssh -i azureuser azureuser@<public-ip-address-jumpbox-vm>

We previously stored our private key in the azureuser directory to access the control plane nodes. Let’s add it to the SSH agent on the jumpbox-vm so we don’t have to specify it explicitly in our SSH commands.

eval "$(ssh-agent -s)"       
ssh-add azureuser 

Now, if you’re logged in as azureuser on the jumpbox-vm, you should be able to SSH into any control plane node simply by running ssh 10.0.2.4 or ssh master-1.

whoami

Use one of the commands I mentioned to access the master-1 VM.

📌 Tip (for beginners): You can always press Ctrl + D to exit the current SSH session and return to the jumpbox-vm shell.

ssh 10.0.2.4

Once you’ve SSH’ed into a master node, you can run the following command to check whether your node is accessing the internet through the previously created NAT Gateway.

curl ifconfig.me

If this shows the public IP address of the NAT Gateway, it means the nodes are accessing the internet through the NAT Gateway.

📖 Extra Knowledge: NAT Gateway

Imagine you’re running an external service on Server X, which is located behind a firewall in another cloud environment. You only want to give access to VMs or nodes running within your private subnet.

To make this secure and manageable, you need to ensure that all outbound traffic from your private nodes uses a static egress IP address. This way, you can configure Server X’s firewall to allow traffic only from that known IP.

In this architecture, the NAT Gateway is assigned the static public IP address 4.1.35.201. It is attached to the private subnet (10.0.2.0/24) that hosts all your Kubernetes nodes—masters and workers. Although these nodes don’t have public IPs and cannot be accessed directly from the internet, they can still initiate outbound connections (e.g., to Server X) through the NAT Gateway.

Here’s how it works step-by-step:

1. A Kubernetes node (e.g., master-1 or worker-1) sends a request to Server X.
2. Since the node is in a private subnet with no public IP, the request is forwarded to the NAT Gateway.
3. The NAT Gateway translates the source IP of the outgoing packet to its own public IP (4.1.35.201) and sends it to Server X over the internet.
4. Server X receives the request and sees it as coming from 4.1.35.201, which is the only IP it is configured to allow.
5. Server X responds, and the NAT Gateway translates the destination IP back to the internal IP of the original requesting node, ensuring the response reaches the correct machine inside your private network.

This setup provides:

• Security: Your Kubernetes nodes remain private and inaccessible from the internet.
• Simplicity: You only have to manage a single IP (4.1.35.201) in Server X's firewall.
• Reliability: The egress IP remains static, which is essential for services that require IP allow listing.

🔧 Essential Components for Configuring a Kubernetes Node

Before configuring a Kubernetes node, it’s important to understand the key components involved in the setup process.

1. cgroups (Control Groups)

cgroups is a low-level Linux kernel feature that allows you to allocate CPU, memory, disk I/O, and other resources to processes. It forms the foundation for container runtimes like containerd and Docker.

Kubernetes depends on cgroups to enforce pod-level resource limits like cpu and memory.

2. containerd

containerd is a lightweight container runtime used by Kubernetes to run containers. It handles image management, container lifecycle, and storage interfaces. It interacts with the Linux kernel (via cgroups) to isolate and run containers.

Kubernetes uses containerd to create and run pods behind the scenes.

Other than containerd Kubernetes supports some other container runtimes as well. For more details, you can check the official documentation.
https://kubernetes.io/docs/setup/production-environment/container-runtimes/

3. kubelet

kubelet is a daemon that runs on every Kubernetes node. It watches for pod specifications (from the API server), pulls the required container images via containerd, and ensures containers are running as expected.

It’s the core component on each node that keeps things working locally.

4. kubeadm

kubeadm is a one-time setup tool used to initialize a Kubernetes control plane or join a node to an existing cluster. It handles all the low-level configuration of certificates, networking, and kubelet bootstrap configs.

Used only during cluster creation or node joining.

5. kubectl

kubectl is a command-line interface used by users (not the node itself) to interact with the Kubernetes API server. You use it to deploy apps, inspect resources, and manage the cluster.

It is not required on the node itself, but essential for managing the cluster from a user/admin perspective.

⚓️ Configure a Kubernetes Node (Ubuntu)

🔁 Repeat these steps on all worker and control plane nodes before joining them to the cluster. (First do this for control plane nodes)

1. Enable IP Forwarding

Kubernetes networking requires IP forwarding to route packets between pods and services.

sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

2. Install and Configure containerd (Container Runtime)

Kubernetes needs a container runtime to run containers. containerd is a lightweight, recommended option.

sudo apt update && sudo apt install -y containerd

Generate the default containerd configuration:

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Restart containerd to apply the configuration:

sudo systemctl restart containerd

3. Configure systemd as the cgroup driver for containerd

Kubernetes works best when the container runtime uses systemd for managing cgroups, which matches the kubelet's default behavior.

Open /etc/containerd/config.toml and edit the following block:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd-systemd

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

✴️Optional: You can also use the following command to edit that.

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

Save and restart containerd again:

sudo systemctl restart containerd

4. Install Kubernetes Tools: kubelet, kubeadm, kubectl

These are the core tools for running and managing Kubernetes.

• kubelet: Runs on each node and manages containers.
• kubeadm: Bootstraps and manages the cluster.
• kubectl: CLI to interact with the cluster.

Add the Kubernetes APT repository and install the tools:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install the components:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl

Prevent automatic updates that might cause version mismatches:

sudo apt-mark hold kubelet kubeadm kubectl

5. Enable the Kubelet Service

Ensure the kubelet starts at boot and is running now:

sudo systemctl enable --now kubelet

🚧 Control-plane pool

Initialize the Kubernetes Control-plane

To initialize the Kubernetes control plane, we will use kubeadm with a custom kubeadm init configuration file. Knowing the private IP addresses of the three master nodes and the load balancer, we will configure the kubeadm init file accordingly.

In master-1 (10.0.2.4)

1. Create the following kubeadm-init.yaml using the nano editor.

sudo nano kubeadm-init.yaml

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.0.2.4
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  name: master-1
  taints:
  - key: node-role.kubernetes.io/control-plane
    effect: NoSchedule

---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.33.0
controlPlaneEndpoint: 10.0.2.10:6443
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12

In Azure, when using an internal Load Balancer for your Kubernetes control plane (like controlPlaneEndpoint: 10.0.2.10:6443), the hairpin NAT issue prevents a control plane node from accessing the Load Balancer IP that maps back to itself. This is a known limitation in Azure networking.

You can read more about this in following links.

https://github.com/microsoft/Azure-ILB-hairpin

To work around this for the first control plane node, you can use a loopback interface that binds the controlPlaneEndpoint IP locally on that node.

✅ Workaround: Bind the Load Balancer IP to a Loopback Interface

This tricks the first node into thinking the Load Balancer IP is local, allowing it to talk to itself through the Load Balancer VIP (in our case 10.0.2.10).

sudo ip addr add 10.0.2.10/32 dev lo

⚠️ Optional

You don’t have to make this change persist after reboot if you plan to add the other control plane nodes at the same time. Otherwise, it’s better to make it persistent.

2. Initialize the first control plane node (master-1) using the kubeadm-init.yaml file. (This process might take some time to complete.)

sudo kubeadm init --config kubeadm-init.yaml --v=5

At the end, you will see two commands in the output similar to the following:

These are used to join additional control plane nodes.

kubeadm join 10.0.2.10:6443 --token 3yfgob.oczs4ho9aynj0p6g \
        --discovery-token-ca-cert-hash sha256:9cd4043be872b09b8af8b10f06f22922cc21d34abff4628e1999028000834da2 \
        --control-plane 

To join worker nodes

kubeadm join 10.0.2.10:6443 --token fetdv4.ngb7hi792zs4150x \
        --discovery-token-ca-cert-hash sha256:3c25328284a8436a959c51df6942a6c8c9660f0eaba77a3044f8bbb3c841a65c

Save these commands somewhere, as you’ll need them to join the other nodes in the next steps.

⚠️ You should save above commands from your output.

3. Run the following commands from the output to access the cluster.

First switch to the root user.

sudo su

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

export KUBECONFIG=/etc/kubernetes/admin.conf

Check the created node using the following command:

sudo kubectl get nodes

4. Install a CNI Plugin

Purpose of CNI in Kubernetes

In Kubernetes, CNI plugins are responsible for setting up the network for your pods. This includes:

• Assigning an IP address to each pod.
• Setting up the routes so pods can talk to each other.
• Enabling external access to pods if required.
• Optionally enforcing network policies (e.g., deny pod-to-pod communication).

Without a CNI plugin, pods cannot get IP addresses or communicate with each other, and the node stays in a NotReady state.

Install Calico as CNI

sudo kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml

Check if Calico pods come up

sudo kubectl get pods -n kube-system -o wide

Run the following command to get the certificate-key to use when joining the next node.

sudo kubeadm init phase upload-certs --upload-certs

Save the certificate key.

👐 Join other control-plane nodes. (master-2, master-3)

SSH into master-2 (10.0.2.5).

Then, run the previously saved kubeadm join command, appending the certificate key you stored, as shown below. master-2 should be configured as in the previous strep (Configure a Kubernetes Node (Ubuntu)). If not, go back and complete that configuration first.

Append the previously saved kubeadm join command with the certificate key you saved earlier, as follows.

⚠️ Use your own stored commands in the following steps — not the examples shown here.

kubeadm join 10.0.2.10:6443 --token fetdv4.ngb7hi792zs4150x \
        --discovery-token-ca-cert-hash sha256:3c25328284a8436a959c51df6942a6c8c9660f0eaba77a3044f8bbb3c841a65c \
        --control-plane --certificate-key 848a88f680f1050d15f101977799c4bde43d514ed5385333dbfdc304de39dfa9

Then run the command as the root user.

sudo su

sudo kubeadm join 10.0.2.10:6443 --token fetdv4.ngb7hi792zs4150x \
        --discovery-token-ca-cert-hash sha256:3c25328284a8436a959c51df6942a6c8c9660f0eaba77a3044f8bbb3c841a65c \
        --control-plane --certificate-key 848a88f680f1050d15f101977799c4bde43d514ed5385333dbfdc304de39dfa9

You can execute the output commands just like you did before.

mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

You’ll be able to see both nodes listed by running kubectl get nodes.

Follow the same steps for master-3.

👷 Worker Nodes

1. Create three worker nodes in the same private subnet.

worker-1

az vm create \
    --resource-group $RESOURCE_GROUP \
    --name worker-1 \
    --image Ubuntu2204 \
    --vnet-name $VNET_NAME \
    --subnet $PRIVATE_SUBNET_NAME \
    --private-ip-address 10.0.2.7 \
    --size $VM_SIZE \
    --nsg $NSG_PRIVATE \
    --public-ip-address "" \
    --admin-username $ADMIN_USERNAME \
    --ssh-key-values azureuser.pub

worker-2

az vm create \
    --resource-group $RESOURCE_GROUP \
    --name worker-2 \
    --image Ubuntu2204 \
    --vnet-name $VNET_NAME \
    --subnet $PRIVATE_SUBNET_NAME \
    --private-ip-address 10.0.2.8 \
    --size $VM_SIZE \
    --nsg $NSG_PRIVATE \
    --public-ip-address "" \
    --admin-username $ADMIN_USERNAME \
    --ssh-key-values azureuser.pub

worker-3

az vm create \
    --resource-group $RESOURCE_GROUP \
    --name worker-3 \
    --image Ubuntu2204 \
    --vnet-name $VNET_NAME \
    --subnet $PRIVATE_SUBNET_NAME \
    --private-ip-address 10.0.2.9 \
    --size $VM_SIZE \
    --nsg $NSG_PRIVATE \
    --public-ip-address "" \
    --admin-username $ADMIN_USERNAME \
    --ssh-key-values azureuser.pub

2. SSH into the worker nodes (worker-1, worker-2, worker-3) through the jumpbox-vm, and configure each of them as Kubernetes nodes.

Go back to the previous step: ⚓️ Configure a Kubernetes Node (Ubuntu).

3. Run the kubeadm join command to join the worker node to the cluster.

⚠️ Not the command to add the master nodes.

The difference between the kubeadm join command for adding a master node and a worker node is the use of the --control-plane flag and the --certificate-key.
When joining a worker node, omit the --control-plane flag and the --certificate-key from your join command.

sudo kubeadm join 10.0.2.10:6443 --token fetdv4.ngb7hi792zs4150x \
        --discovery-token-ca-cert-hash sha256:3c25328284a8436a959c51df6942a6c8c9660f0eaba77a3044f8bbb3c841a65c 

Now, if you SSH into a master node,
you’ll be able to confirm the availability of the connected worker nodes by running kubectl get nodes.

4. Repeat the same steps for the remaining two worker nodes.

🔴 Accessing the Cluster

After connecting all the nodes, you’ve successfully completed the cluster setup. You now have a stacked etcd cluster with 3 master nodes and 3 worker nodes.
The next step is figuring out how to securely connect to the cluster from your local machine.

How do you connect to your cluster from your local computer?

1. First, SSH into the jumpbox-vm and install kubectl on it.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

2.Since we’re using the same private key to SSH into both our nodes and the jumpbox-vm, I’m going to copy the private key from the jumpbox-vm to my master-1 node (10.0.1.4).

scp -i azureuser azureuser azureuser@10.0.2.4:/home/azureuser

3. ssh into the master-1 node.

📌 Once you’ve SSH’ed into master-1, you can run the ls command to see the azureuser private key you copied in the previous step.

Copy the kubeconfig file to the jumpbox VM (10.0.1.4).

sudo scp -i azureuser /etc/kubernetes/admin.conf azureuser@10.0.1.4:/home/azureuser

4. Return to the jumpbox VM and create the kubeconfig by using the copied admin.conf file from master-1.

mkdir -p ~/.kube
mv ~/admin.conf ~/.kube/config

Then verify by running.

kubectl get nodes

If you see the output listing your nodes, it means you’ve successfully connected to the cluster.

5. Connect from you local machine using kube-proxy

🧩 What is kubectl proxy?

kubectl proxy sets up a local HTTP proxy server that forwards requests to the Kubernetes API server using your kubeconfig credentials. It's authenticated and secure, and only listens locally unless configured otherwise.

1. First copy the kubeconfig from your jumpbox-vm to the local machine.

copy the content of the below config in your jumpbox-vm and create a new config file in your local machine.

cat ~/.kube/config

2. update the config in your local machine.

You should update the server address in your config file to http://localhost:38001. (You can choose any port you prefer; here, I’ve chosen port 38001.)

...
server: http://localhost:38001
...

3. ssh into the jumpbox-vm using the following command.

ssh -i azureuser -L 38001:127.0.0.1:8001 azureuser@<jumpbox-ip-address>

• -i azureuser : specifies the SSH private key used for authentication.
• -L 38001:127.0.0.1:8001 : sets up the local port forwading, 38001 on your local machine and 127.0.0.1:8001 on your remote machine (in this case, jumpbox-vm)
• azureuser@<jumpbox-ip-address> : remote user (azureuser) and the public IP address or DNS name of the jumpbox

4. start the proxy on your jumpbox-vm with the following command. (Flag --port is optional here because port 8001 is the default.)

kubectl proxy --port=8001

It will start the kube proxy server.

5. In your local terminal.

export KUBECONFIG=<path-to-your-updated-config>
kubectl get nodes

As you can see, you are now able to access your Kubernetes cluster from your local machine.
You can also use tools like K8sLens by adding your updated kubeconfig file from your local machine to manage the cluster easily.

If you’ve enjoyed this article, don’t forget to give it a 👏. Your support gives me more energy and enthusiasm to write more articles like this.
Thank you!

Happy Coding!!! 🐧

If you haven’t read the first part of this article, feel free to check it out here:
https://medium.com/devops-dev/securing-microservice-apis-with-oauth2-proxy-a-complete-project-71fabc79147d

This article includes,

• More about OAuth2 flow.
• CSRF Cookie.
• CORS and Preflight requests.
• OAuth2 proxy cookie refresh, and more options.

Our Current Setup and the Auth flow

OAuth2 Proxy authentication flow

1. User Request Access: The user tries to access a protected resource (todo-api) without being authenticated. Our nginx server (reverse proxy) redirects the user to the OAuth2 proxy.
2. OAuth2 Proxy checks for _oauth2_proxy Cookie: The OAuth2 proxy checks if the user has a valid _oauth2_proxy cookie, which indicates that the user is already authenticated.
3. No _oauth2_proxy cookie: If the _oauth2_proxy cookie is not present or is invalid, the OAuth2 proxy redirects the user to the configured authentication (Google).
4. User Authenticates: The user logs in to the authentication provider and grants permission to the application to access their information.
5. Authentication Provider Sends Authorization: After successful authentication, the authentication provider (Google) sends an authorization token or code back to the OAuth2 proxy.
6. OAuth2 Proxy Grants Access: The OAuth2 proxy uses the authorization token to validate the user’s identity and grants access by setting a valid _oauth2_proxy cookie in the user’s browser.
7. User Accesses Protected Resource (todo-api): With the valid _oauth2_proxy cookie, the user is redirected back to the original application and can access the protected resource.
8. Access Granted: If the _oauth2_proxy cookie is valid, the OAuth2 proxy allows the user to access the protected resource without requiring re-authentication.

Let’s break this down to four steps.

we have the normal authentication flow when using OAuth2 proxy with cookie-based authentication.

Let’s discuss how this is applied in our project by breaking it down into four steps.

Step 01: OAuth2 proxy checks for _oauth2_proxy cookie

When a user requests a protected resource, it goes through an authentication check at OAuth2 proxy.

As you might recall from the previous story, this is done using the auth_request directive in our nginx.conf .

OAuth2 proxy verifies whether the cookie is present and valid before granting access. If the check fails, the user is redirected to the configured authentication provider, which in our case is Google OAuth2.

Step 02: OAuth2 proxy authentication flow starts

In our setup, we initiate the OAuth2_proxy authentication process by invoking the /oauth2/start API endpoint. This is the standard starting endpoint provided by OAuth2 proxy.

By doing that, a CSRF cookie is set on the browser, and user is redirected to the configured authentication provider, which in our case is Google.

The user can then log in through the provider’s login page and will be redirected back to the application. If you have followed the previous story, you should have already experienced this.

CSRF Cookie?

OAuth2 proxy uses a CSRF (Cross-Site Request Forgery) cookie as a security measure to protect against CSRF attacks. When a user initiates the OAuth2 authentication flow by accessing the /oauth2/startendpoint, OAuth2 proxy generates a CSRF token and sets it as a cookie in the user’s browser.

When the user is redirected to the authentication provider (e.g., Google) to log in, the CSRF token is included in the request. After the authentication provider validates the user’s credentials, it redirects the user back to OAuth2 proxy along with the CSRF token.

OAuth2 proxy then verifies that the CSRF token received from the authentication provider matches the one set in the user’s browser. This verification ensures that the request is coming from the same user who initiated the authentication flow, helping to prevent CSRF attacks where a malicious website tries to trick a user into performing actions on another website without their consent.

You can read more about CSRF prevention here.

Step 03: Sets _oauth2_proxy cookie and redirects user

After logging in, you’ll be redirected to the redirect URL configured in the oauth2_proxy variables.

If you recall from the previous story, we authorized that redirect URI when creating the Google OAuth2 consent screen and credentials.

OAuth2 proxy then validates whether the authentication was successful. If successful, it sets a valid _oauth2_proxy cookie in the user's browser and removes the _oauth2_proxy_csrf cookie.

Step 04: Sends a request with a valid _oauth2_proxy cookie

Finally, the user will be able to access protected API endpoints using the valid _oauth2_proxy cookie. This cookie can be included in the request header to access any protected microservice authenticated by oauth2_proxy.

When the authentication process is repeated with a valid cookie, invoking the auth_request directive (/oauth2/auth) in the nginx configuration, OAuth2 Proxy will send a 202 Accepted response to the client and grant access to protected micro-services.

OAuth2 Proxy configuration options

Now you should have a better understand about the oauth2 proxy auth flow in our project. Let’s talk about our oauth2 proxy configuration in the docker-compose.yml file.

  oauth2-proxy:
    container_name: oauth2-proxy
    image: quay.io/oauth2-proxy/oauth2-proxy:latest
    environment:
      - OAUTH2_PROXY_CLIENT_ID=<your-client-id>
      - OAUTH2_PROXY_CLIENT_SECRET=<your-client-secret>
      - OAUTH2_PROXY_COOKIE_SECRET=<your-cookie-secret>
      - OAUTH2_PROXY_EMAIL_DOMAINS=*
      - OAUTH2_PROXY_REVERSE_PROXY=true
      - OAUTH2_PROXY_REDIRECT_URL=http://localhost/oauth2/callback
      - OAUTH2_PROXY_COOKIE_SECURE=false
      - OAUTH2_PROXY_UPSTREAM=http://todo-api:8000
      - OAUTH2_PROXY_HTTP_ADDRESS=http://0.0.0.0:4180
      - OAUTH2_PROXY_SET_AUTHORIZATION_HEADER=true
      - OAUTH2_PROXY_SET_XAUTHREQUEST=true
      - OAUTH2_PROXY_WHITELIST_DOMAINS=.localhost:3000
    command:
      - --http-address=0.0.0.0:4180
      - --upstream=http://todo-api:8000
      - --skip-provider-button=true
      - --skip-auth-preflight=true
    networks:
      - microservices

1. As I mentioned in the previous story, OAUTH2_PROXY_CLIENT_ID and OAUTH2_PROXY_CLIENT_SECRET env variables are used to connect Auth provider to the OAuth2 proxy.
2. OAUTH2_PROXY_COOKIE_SECRET is the secret key used to encrypt and sign cookies in our authentication flow. It should be unique and kept secret from the outside world. If you recall from the previous story, you should remember that we generated a cookie secret using the Linux terminal or Windows Power-Shell.
3. OAUTH2_PROXY_EMAIL_DOMAINS specifies the allowed email domains for authentication. Using * allows any email domain.
4. OAUTH2_PROXY_REVERSE_PROXY indicates that OAuth2 proxy is running behind a reverse proxy (in our case its nginx) and should use headers provided by the reverse proxy for client information.
5. OAUTH2_PROXY_REDIRECT_URL is the URL to which the OAuth provider will redirect the user after authentication. (Step 03)
6. Certainly! Here’s a revised version:
7. OAUTH2_PROXY_COOKIE_SECURE determines whether the cookie is sent only over HTTPS. In our project, we’ve set this to false because we’re using our local network and want to avoid complicating things by implementing self-signed certificates.
8. OAUTH2_PROXY_UPSTREAM specifies the upstream server to proxy requests to after authentication. it’s like telling OAuth2 Proxy, “After you’ve checked that the user is authenticated, send their request to this server.”
9. OAUTH2_PROXY_HTTP_ADDRESS The HTTP address on which OAuth2 proxy should listen for incoming requests.
10. OAUTH2_PROXY_SET_AUTHORIZATION_HEADER If you’ve set this to true, OAuth2 proxy will set the Authorization header with the user’s access token for upstream requests.
11. OAUTH2_PROXY_SET_XAUTHREQUEST If you’ve set this to true, OAuth2 proxy will add a X-Auth-Request-Email header with the user’s email address for upstream requests.
12. OAUTH2_PROXY_WHITELIST_DOMAINS This specifies the allowed domains for the OAuth2 proxy. Requests from domains not in this list will be rejected. (This is important because your redirect requests will also rejected if the relavant domain is not in this list.)
    Ex: If we didn’t specify .localhost:3000 in this list. when we’ve invoked http:localhost/oauth2/auth/start?rd=http%3A%2F%2Flocalhost%3A3000 , OAuth2 proxy will reject the redirection to http://localhost:3000 after authentication.
13. --skip-provider-button This option tells OAuth2 Proxy not to show the provider selection page. If you have only one OAuth provider configured, this page is skipped, and users are directly sent to the authentication provider’s login page.
14. --skip-auth-preflight This option skips the authentication preflight check. Normally, OAuth2 Proxy checks if the user is already authenticated before processing a request. This check helps to avoid unnecessary redirects. However, setting this option to true skips this check, which might slightly reduce latency but could potentially lead to more redirects for authentication.

CORS and Preflight Requests

What is a preflight request?

Preflight request is a CORS request that checks to see if the CORS protocol is understood and a server is aware using specific methods and headers.

It is an OPTIONS request, using two or three HTTP request headers: Access-Control-Request-Method, Origin, and optionally Access-Control-Request-Headers.

What is CORS?

CORS (Cross-Origin Resource Sharing) is a way for web servers to tell browsers that they’re okay with allowing certain types of requests from other websites.

Normally, browsers prevent websites from accessing resources on different domains for security reasons, but CORS allows servers to override this restriction. It does this by transmitting special HTTP headers that give browsers permission to allow these cross-origin requests.

The basic request flow of a cross-origin request involves two steps: The main request and the preflight request.

• Main Request: The browser sends the main request (GET or POST request) from the frontend Javascript code to the server on a different domain. This request includes the relevant CORS headers, such as Origin, to indicate the origin of the request.
• Preflight Request: Before sending the actual request, the browser may send a preflight request (using the OPTIONS method) to the CORS headers and asks the server if the main request is allowed based on the CORS policy set by the server.

If the preflight request is successful and the server responds with the appropriate CORS headers indicating that the main request is allowed, the browser will proceed to send the main request.

Otherwise, the browser will block the main request, and the frontend code will not able to access the server’s resources.

!Important

Some requests don’t trigger a CORS preflight request. Those are called simple requests.

That’s why some requests were able to succeed to the backend server without setting --skip-auth-preflight=true. Those requests were able to proceed without a problem because they didn’t have a preflight request at all.

What happened to the other requests?

If you’ve ran the docker compose up -d without setting --skip-auth-preflight=true (You can set it to false or comment it), you’ll see that some simple requests are handled successfully while some are not.

In order to be a simple request, it should meet all the conditions listed here. and It’s decided by the browser.

For example, if you try to edit a todo, the request may fail with a CORS error due to a failed preflight request.

If you examine the preflight request closely, you’ll notice that the _oauth2_cookie is missing, even though it exists in the browser and we’ve included the credentials in our fetch request.

Why?

CORS preflight requests must never include credentials.

Since preflight requests are handled by the browser, they don’t include credentials by default. Therefore, we have to set --skip-auth-preflight=true to process these types of requests in OAuth2 proxy. Otherwise, OAuth2 proxy will attempt to authenticate preflight requests and will cause the request to fail.

Since we’re dealing with requests from different origins using CORS, this step is necessary when the origin of the request is different from the origin of the resource. In our case we have to do this because we’re requesting from the http://localhost:3000 and the resource origin is http://localhost/.

You might be wondering why it’s failing even though we’re on the same network and using the name “localhost.” That’s because CORS also restricts sharing resources with different ports. Even if the origin name is the same, if the port is different, CORS won’t allow sharing resources. That’s the standard behavior of the CORS.

You can learn more about preflight requests and requests with credentials here.

Cookie expiry and Cookie refresh

Now you have a better understanding of CORS, preflight requests, and the OAuth 2 authentication flow with OAuth2 proxy.

If you recall the last story, you should now remember how we handled login/logout in the Next.js middleware and started the auth flow by invoking the/oauth2/start endpoint.

You might be wondering what will happen if the cookie becomes invalid or expires.

Since we haven’t configured any cookie expiry settings in our current setup, we can’t clearly see what’s happening.

Let’s simulate what happens when the cookie expires by updating the docker compose configuration with --cookie-expire=15s, as shown in the following snippet.

oauth2-proxy:
    container_name: oauth2-proxy
    image: quay.io/oauth2-proxy/oauth2-proxy:latest
    environment:
      - OAUTH2_PROXY_CLIENT_ID=<your-client-id>
      - OAUTH2_PROXY_CLIENT_SECRET=<your-client-secret>
      - OAUTH2_PROXY_COOKIE_SECRET=<your-cookie-secret>
      - OAUTH2_PROXY_EMAIL_DOMAINS=*
      - OAUTH2_PROXY_REVERSE_PROXY=true
      - OAUTH2_PROXY_REDIRECT_URL=http://localhost/oauth2/callback
      - OAUTH2_PROXY_COOKIE_SECURE=false
      - OAUTH2_PROXY_UPSTREAM=http://todo-api:8000
      - OAUTH2_PROXY_HTTP_ADDRESS=http://0.0.0.0:4180
      - OAUTH2_PROXY_SET_AUTHORIZATION_HEADER=true
      - OAUTH2_PROXY_SET_XAUTHREQUEST=true
      - OAUTH2_PROXY_WHITELIST_DOMAINS=.localhost:3000
    command:
      - --http-address=0.0.0.0:4180
      - --upstream=http://todo-api:8000
      - --skip-provider-button=true
      - --skip-auth-preflight=true
      - --cookie-expire=15s 
    networks:
      - microservices

Now, sign out from the current session and try to sign in again while opening the network tab.

After logging in, try to interact with todos while monitoring the request status.

After 15 seconds, you’ll see that your request is getting a CORS error. That’s because we’ve set the cookie to expire in 15 seconds.

Now, if you check for cookies in your browser, you’ll see that the oauth2_proxy cookie is unset from the browser and replaced with the _oauth2_proxy_csrf cookie.

This happens because when the _oauth2_proxy cookie expires, OAuth2 proxy responds by removing the _oauth2_proxy cookie in our browser and setting the _oauth2_proxy_csrf cookie to redirect us to the Auth Provider to relogin.

You’ll see that in the above figure. Even if the request has the _oauth2_proxy cookie credential with it, since it’s an invalid cookie, OAuth2 proxy responds by removing it and setting the _oauth2_proxy_csrf cookie to redirect us for the login with the Auth provider.

If you try to refresh the page now, you’ll be redirected to the login because the cookie is not present in your browser. We’ve handled this in our Next.js middleware.

In the current implementation, even if the user is actively engaged with our todo frontend application, they may be logged out due to cookie expiration. This could be problematic in scenarios such as online examination systems or similar cases.

We can handle these kind of scenarios by using cookie refresh.

We can address this issue by using a cookie refresh mechanism.
For example, if we set --cookie-expire=15s, we can set --cookie-refresh=10s in our docker-compose.yml configuration.

oauth2-proxy:
    container_name: oauth2-proxy
    image: quay.io/oauth2-proxy/oauth2-proxy:latest
    environment:
      - OAUTH2_PROXY_CLIENT_ID=<your-client-id>
      - OAUTH2_PROXY_CLIENT_SECRET=<your-client-secret>
      - OAUTH2_PROXY_COOKIE_SECRET=<your-cookie-secret>
      - OAUTH2_PROXY_EMAIL_DOMAINS=*
      - OAUTH2_PROXY_REVERSE_PROXY=true
      - OAUTH2_PROXY_REDIRECT_URL=http://localhost/oauth2/callback
      - OAUTH2_PROXY_COOKIE_SECURE=false
      - OAUTH2_PROXY_UPSTREAM=http://todo-api:8000
      - OAUTH2_PROXY_HTTP_ADDRESS=http://0.0.0.0:4180
      - OAUTH2_PROXY_SET_AUTHORIZATION_HEADER=true
      - OAUTH2_PROXY_SET_XAUTHREQUEST=true
      - OAUTH2_PROXY_WHITELIST_DOMAINS=.localhost:3000
    command:
      - --http-address=0.0.0.0:4180
      - --upstream=http://todo-api:8000
      - --skip-provider-button=true
      - --skip-auth-preflight=true
      - --cookie-expire=15s 
      - --cookie-refresh=10s
    networks:
      - microservices

This setup will refresh the OAuth2 cookie every 10 seconds if the user is actively making requests to the OAuth2-protected API.

However, if the user remains idle during the refresh interval (15–10 = 5 seconds) or for a total of 15 seconds, the OAuth2 cookie will expire, and the application will prompt the user to re-login.

By implementing cookie refresh, we can mitigate such problematic situations and ensure a smoother user experience.

Message to the audience

It took me a while to finish Part 2 of the OAuth2 Proxy article due to personal commitments. I focused on the main points of OAuth2 Proxy for now and left out details about the FastAPI and Todo frontend. I’ll be covering improvements to the Next.js Todo frontend and FastAPI microservice in future posts.

Thank you. ☕

Thank you! If you’ve made it this far, well done.

If you’ve enjoyed the project, don’t forget to give it a 👏. Your support gives me more energy and enthusiasm to write more projects like this. Thank you!

Happy Coding!!! 🐧

Overview

In this hands-on project, we will discuss how to build & secure microservice APIs using OAuth2 Proxy behind a reverse proxy. We will implement a microservice API, a frontend application, a reverse proxy, and OAuth2 Proxy. This setup will add an additional layer of security to our microservices, ensuring that only authenticated and authorized users can access our APIs.

Introduction

We’re going to implement the following architecture on your local machine to learn how to accomplish this task.

Note: If you have a strong understanding of APIs and have your own service ready, you can skip the first two parts of this project and proceed directly to the third part.

As I mentioned in the overview section,

1. First, we’ll establish a microservice API using FastAPI to test this architecture. (I’ll provide a pre-coded solution for you.)
2. Next, we’ll configure a frontend application using Next.js to interact with our FastAPI service.
3. Following that, we’ll implement a reverse proxy between the frontend app and the microservice we’ve created using Nginx.
4. Lastly, we’ll set up OAuth2 Proxy to secure the FastAPI service we’ve developed.

The Goal

The goal of this project is to understand the workings of a modern, secure, and scalable web application architecture and to gain a thorough understanding of the OAuth2 authentication flow.

Prerequisites

A good internet connection.

You need to have docker installed in your computer. If not you can follow the official documentation to download docker. https://docs.docker.com/engine/install/

It’s nice to have a solid grasp of Docker and Docker networking, as I’ll be providing you with a pre-coded API service and a frontend application to work with. However, it’s not mandatory, as I’ll explain what would be necessary for this project.

If you need any help while working on this project, you can refer to my Docker networking project to understand concepts like ‘Host Network’ and ‘Bridge Network’. Feel free to leave a comment below if you have any questions.

Docker networking: https://medium.com/@kesaralive/diving-deeper-into-docker-networking-with-docker-compose-737e3b8a3c8c

With these in place, you should be ready to start working on the project.

Before Starting,

This will be a long project with many steps to complete. If you find anything difficult to understand, please leave a comment in the article. I’ll do my best to help you out. If you get stuck at any point, don’t hesitate to reach out to me. I’m here to help. ✌️

Project Architecture

OAuth2 Proxy

OAuth2 Proxy is a reverse proxy and static file server that provides authentication using OAuth2. You’ll configure OAuth2 Proxy to secure access to your FastAPI service. This will add an extra layer of security by requiring users to authenticate before accessing the API.

Identity Server

You’ll configure OAuth2 Proxy to use Google OAuth 2.0 as the Auth Provider. This means that when a user tries to access your FastAPI service, they will be redirected to Google’s authentication service to log in. Once authenticated, Google will issue an access token that allows the user to access your API.

Microservices

Microservices are small, autonomous services that work together. They’re like individual pieces of a puzzle that work together to create a bigger picture. Each piece (Microservice) has its own job to do and can be developed, deployed, and scaled independently, making the overall system more flexible and easier to manage.

Today, we’re going to create one piece of a puzzle (a microservice) to understand how we can secure this puzzle using OAuth2 Proxy. We’ll create a very simple Todo service API using FastAPI and MongoDB to represent a piece of this puzzle.

Reverse Proxy/ API Gateway

A reverse proxy/ API gateway acts as an intermediary between cleints and servers. It receives requests from clients and forwards them to appropriate backend servers. Reverse proxies can also perform additional functions like load balancing, caching, SSL termination, and request routing.

We’re going to use Nginx as our reverse proxy/ API gateway for this project. Nginx is known for its high performance, stability, and low resource consumption. It’s commonly used as a reverse proxy and API gateway due to its ability to handle a large number of concurrent connections efficiently.

Front-end App / Client

We’ll use a Todo client application implemented using Next.js for this project. It’s a simple one-page frontend application that can be used to create, read, delete, and update todos.

FastAPI Todo-service — Step 01

Create a new folder named oauth2-project and an docker-compose.yml file inside it. I'm going to use the name oauth2-project folder in the rest of the project. It's better if you can use the same name as mine.

Then clone the fastapi-todo-service into that folder using following command:

git clone https://github.com/kesaralive-medium/fastapi-todo-service.git 

Then update the docker-compose.yml file with below code snippet.

version: "3.8"
services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    volumes:
      - mongodb_data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password

  fastapi-todo-service:
    container_name: todo-api
    build:
      context: ./fastapi-todo-service
      dockerfile: Dockerfile
    depends_on:
      - mongodb
    ports:
      - 8000:8000
    environment:
      ATLAS_URI: mongodb://root:password@mongodb:27017
      DB_NAME: todoapi

volumes:
  mongodb_data:
    driver: local

In the above docker-compose.yml configuration, I’ve included a MongoDB container to provide the database service to our FastAPI service. If you have a MongoDB account and a free tier cluster, you can use your MongoDB Atlas URI and a preferred database name as you wish. (In that case you can remove the mongodb service from docker-compose.yml configuration.)

If you’re going to use your mongodb cloud atlas uri docker-compose.yml will look like this,

version: "3.8"
services:
  fastapi-todo-service:
    container_name: todo-api
    build:
      context: ./fastapi-todo-service
      dockerfile: Dockerfile
    ports:
      - 8000:8000
    environment:
      ATLAS_URI: <your-mongodb-atlas-uri>
      DB_NAME: todoapi

Once you’ve selected your way, you can run the following command to up and run the todo-service.

docker compose up -d

If you’ve done everything correctly so far, you should be able to see the docs page of the FastAPI Todo service by accessing the http://localhost:8000/docs URL from your browser.

You will see 6 API endpoints related to the Todo service and one API endpoint to get the headers of our request. (This particular endpoint will help us understand the behavior of our Auth flow later.)

You can try to create a todo using the POST / Create Todo API endpoint and retrieve it using the GET / Get Todos API endpoint to make sure everything is working fine.

You can simply press the “Try it out” button, enter your todo in the request body, and then Execute the request.

If you’ve set up your database URI correctly, you should be able to retrieve the created todo using the GET / Get Todos endpoint. The response body will be shown below after executing the request, as shown in the image below.

Try to access the /headers endpoint from your browser.

http://localhost:8000/headers

Keep the above step in mind; we’re going to compare these headers every time we integrate something into this project.

You can use the following command to stop the running containers.

docker compose down

If you have followed the instructions correctly, you have completed the following part of our project architecture.

Before proceeding to the next step, I’ll share some important details about Docker Compose with you:

• docker compose up — Use this command to up and run all the containers defined in the docker-compose.yml file.
• docker compose up -d — This command starts the containers in the background (detached mode).
• docker compose down — Stops and removes all containers, networks, and volumes created by docker compose up.
• dockse compose build <SERVICE> —Builds or rebuilds a specific service defined in the docker-compose.yml file. Replace <SERVICE> with the name of the service you want to build. If you’ve done some changes in any service after using docker compose up, you must run this command to reflect those changes in the new build.

You should run all of the above commands only inside the project directory.

Some helpful docker-compose.yml specification attributes details

In the docker-compose.yml file of this project, there are three main sections, and under the services section, there will be around 4-5 services. Here are some important attributes you should understand:

• build: Specifies how to build the Docker image for the service.
• depends_on: Specifies that the service depends on another service or services. However, it's important to note that this does not wait for the other service to be "ready" before starting the current service. It simply starts the other service before the current service.
• environment: This attribute is used to set environment variables required by the service.

Front-end App/ Client — Step 02

Now, let’s integrate the front-end application into this setup by cloning the next-todo-app repository into the project folder using the following command.

git clone https://github.com/kesaralive-medium/next-todo-app.git

After cloning the next-todo-app, your directory should resemble the image below.

Now, let’s add the frontend configuration to the docker-compose.yml file.

version: "3.8"
services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    volumes:
      - mongodb_data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password

  fastapi-todo-service:
    container_name: todo-api
    build:
      context: ./fastapi-todo-service
      dockerfile: Dockerfile
    depends_on:
      - mongodb
    ports:
      - 8000:8000
    environment:
      ATLAS_URI: mongodb://root:password@mongodb:27017
      DB_NAME: todoapi

  next-todo-app:
    container_name: todo-client
    build:
      context: ./next-todo-app
      dockerfile: Dockerfile
    ports:
      - 3000:3000


volumes:
  mongodb_data:
    driver: local

Again, run the following command to start and run all three containers.

docker compose up -d

If you’ve integrated the todo-client correctly, you should be able to see the todo frontend application at http://localhost:3000.

You will also see the previously created todo in the todo client application. Try creating, updating, and deleting todos. (Press ‘Enter’ to add todos after typing.)

Now, try accessing the http://localhost:3000/headers page. You will notice that the headers output is different from when we accessed it from the FastAPI Todo service itself.

You will see some newly added values and updated values there.

"accept": "*/*"
"origin": "http://localhost:3000"
"sec-fetch-site": "same-site"
"sec-fetch-mode": "cors"
"sec-fetch-dest": "empty"
"referer": "http://localhost:3000/"

The headers mentioned above are part of the CORS (Cross-Origin Resource Sharing) mechanism in web browsers. Since I’ve already granted access to any origin at the FastAPI Todo service end, our todo-client can access the todo-service using CORS. You’ll be able to see the code regarding that in the /fastapi-todo-service/app/main.py file.

...

origins = [
    "*"
]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

...

• “accept”: “*/*” — Indicates that the client accepts any media type.
• “origin”: “http://localhost:3000" — Specifies the origin of the request, which is the domain where the todo-client is hosted.
• “sec-fetch-site”: “same-site” — Indicates that the request is being made from the same site as the todo-client.
• “sec-fetch-mode”: “cors” — Specifies that the request is a cross-origin request and should be handled using CORS.
• “sec-fetch-dest”: “empty” — Indicates that the request’s destination is not a specific resource type.
• “referer”: “http://localhost:3000/"— Specifies the address of the previous web page from which the current request originated.

MDN web docs is the best place to learn more about the HTTP protocol and HTTP headers.

Accept - HTTP | MDN

If you have followed the instructions correctly, you have completed the following part of our project architecture.

Reverse Proxy/ API Gateway — Step 03

Now that we have a todo-client, todo-api-service, and a database up and running, let’s integrate Nginx as a reverse proxy between the todo-client and the FastAPI todo-service.

Create a folder named todo-api-gateway inside your project folder. Your directory structure should look like this.

After that, create a Dockerfile and an nginx.conf file inside the todo-api-gateway folder, as shown in the below snippet.

.
├── docker-compose.yml
├── fastapi-todo-service
├── next-todo-app
└── todo-api-gateway
    ├── Dockerfile
    └── nginx.conf

Dockerfile

FROM nginx:latest

COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 80
EXPOSE 443

CMD ["nginx", "-g", "daemon off;"]

This Dockerfile is used to build a custom Nginx image for our project. We use nginx:latest as the base image for this custom image. Then, we update the default nginx.confwith our configuration file.

nginx.conf

events {
    worker_connections 1024;
}

http {
    server {
        listen 80;
        server_name localhost;

        location / {
            proxy_pass http://todo-api:8000;
        }
    }
}

This nginx.conf configuration sets up Nginx as a reverse proxy to forward requests from port 80 to the todo-api service running on port 8000.

• events: This block defines parameters that affect the Nginx event loop processing, such as the maximum number of connections per worker process.
• http: This block contains configuration for the HTTP protocol.
• server: This block defines a virtual server for handling HTTP requests.
• listen 80: Configures the server to listen on port 80 for incoming HTTP requests.
• server_name localhost;: Sets the server name to localhost
• location /: Defines how Nginx should process requests for the root URL. When you access http://localhost/, you will be accessing the logic inside the curly braces {} in this location block.
• proxy_pass http://todo-api:8000;: Forwards requests to http://todo-api:8000, which is the hostname of the todo-api service running on port 8000.

Now let’s add this nginx reverse-proxy setup to our docker-compose.yml configuration.

version: "3.8"
services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    volumes:
      - mongodb_data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password

  fastapi-todo-service:
    container_name: todo-api
    build:
      context: ./fastapi-todo-service
      dockerfile: Dockerfile
    depends_on:
      - mongodb
    ports:
      - 8000:8000
    environment:
      ATLAS_URI: mongodb://root:password@mongodb:27017
      DB_NAME: todoapi

  next-todo-app:
    container_name: todo-client
    build:
      context: ./next-todo-app
      dockerfile: Dockerfile
    ports:
      - 3000:3000

  api-gateway:
    container_name: nginx
    build:
      context: ./todo-api-gateway
      dockerfile: Dockerfile
    depends_on:
      - fastapi-todo-service
    ports:
      - 80:80

volumes:
  mongodb_data:
    driver: local

Now, you can use the docker compose down command to stop all the running containers, and the docker compose up -d command to run the newly added API gateway in the project.

docker compose down

docker compose up -d

If you enter the following command, you will see four running containers.

docker ps

In the docker-compose.yml file, we’ve mapped the Nginx service to the host machine’s port 80. Therefore, now you should be able to access it at http://localhost/.

If you have previously added todos, you will be able to see them as well. Similar to before when we accessed http://localhost:8000/docs, now you should be able to access the API documentation at http://localhost/docs.

Since we can now access the todo-api through the Nginx reverse proxy, let’s update our todo-client to send requests using the reverse proxy.

Navigate to the file next-todo-app/actions/todo.ts and update the API_URL tohttp://localhost/. Please note that the trailing slash is important, so keep that in mind.

next-todo-app/actions/todo.ts

...

const API_URL = "http://localhost/"

...

Then, go to the file next-todo-app/app/headers/page.tsx and update the fetch URL inside the useEffect hook as well.

next-todo-app/app/headers/page.tsx

...

const response = await fetch('http://localhost/headers')

...

Since we’ve updated 2 files in the todo-client application, we now need to rebuild it in the Docker Compose. To do that, you can run the following command inside the project folder:

docker compose build next-todo-app

This command will rebuild the next-todo-app service in the Docker Compose, incorporating the changes you made to the files.

docker compose up -d

Now, Let’s check requests headers by accessing the http://localhost:3000/headers page.

You will see some headers are updated there.

"host": "todo-api:8000"
"connection": "close"

• “host”: “todo-api:8000” — indicates the hostname and port number of the server to which the request is being sent (todo-api on port 8000). This information is exposed to the client because we forwarded the client requests directly to the todo-api server without any additional configurations.
  “Revealing internal infrastructure details to the client is not ideal.”
  In the next step of this project, we will change this behavior.
• “connection”: “close” — indicates that the connection will be closed after the current transaction is completed. This means that the client or server intends to close the TCP connection after sending or receiving the current HTTP message.
  In an Nginx reverse proxy, this practice can help manage server resources by limiting the number of open connections. It is a part of the connection management strategy to control the lifecycle of connections between clients and servers, helping in the optimization of resource usage and performance improvement.

If you have followed the instructions correctly, you have completed the following part of our project architecture.

Network Isolation — Step 04

In the current setup, you should be able to access both the Nginx reverse proxy service and the todo-api service. However, when using a reverse proxy, there’s no point in giving direct access to the todo-api. First, let’s see how our current setup works by inspecting the docker-compose.yml file.

...

services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    volumes:
      - mongodb_data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password

  fastapi-todo-service:
    container_name: todo-api
    build:
      context: ./fastapi-todo-service
      dockerfile: Dockerfile
    depends_on:
      - mongodb
    ports:
      - 8000:8000
    environment:
      ATLAS_URI: mongodb://root:password@mongodb:27017
      DB_NAME: todoapi

  next-todo-app:
    container_name: todo-client
    build:
      context: ./next-todo-app
      dockerfile: Dockerfile
    ports:
      - 3000:3000

  api-gateway:
    container_name: nginx
    build:
      context: ./todo-api-gateway
      dockerfile: Dockerfile
    depends_on:
      - fastapi-todo-service
    ports:
      - 80:80

...

Currently, we have 4 services up and running, and all the services are running on the default bridge network created by Docker Compose when we first ran the setup. You can verify this by using the following command.

docker network ls

You’ll see a bridge network named token-handler-pattern_default. This name may vary depending on your project folder name. If you want to understand docker networking deeply, refer my previous article about docker networking.

However, if we visualize the current setup in a diagram, it would look like the image below.

The current setup includes the following mappings and configurations:

• The todo-client container’s port 3000 is mapped to the host’s port 3000.
• The nginx container’s port 80 is mapped to the host’s port 80.
• The todo-api container’s port 8000 is mapped to the host’s port 8000.
• The todo-api container accesses the mongodb container’s port 27017.
• The todo-client accesses nginx through the host network using the host’s port 80.
• nginx proxies requests to the todo-api container using port 8000.

To fix this, we need to remove the unnecessary port mappings and isolate the nginx, todo-api, and mongodb containers to the token-handler-pattern_default network. Here's how you can do it:

version: "3.8"
services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    volumes:
      - mongodb_data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password
    networks:
      - microservices

  fastapi-todo-service:
    container_name: todo-api
    build:
      context: ./fastapi-todo-service
      dockerfile: Dockerfile
    depends_on:
      - mongodb
    environment:
      ATLAS_URI: mongodb://root:password@mongodb:27017
      DB_NAME: todoapi
    networks:
      - microservices

  next-todo-app:
    container_name: todo-client
    build:
      context: ./next-todo-app
      dockerfile: Dockerfile
    ports:
      - 3000:3000
    networks:
      - client-host

  api-gateway:
    container_name: nginx
    build:
      context: ./todo-api-gateway
      dockerfile: Dockerfile
    depends_on:
      - fastapi-todo-service
    ports:
      - 80:80
    networks:
      - microservices

networks:
  microservices:
    driver: bridge
  client-host:
    driver: bridge

volumes:
  mongodb_data:
    driver: local

Updates

I’ve set up two networks: one for hosting microservices (microservices network) and another for hosting the frontend (client-host).

...

networks:
  microservices:
    driver: bridge
  client-host:
    driver: bridge

...

In this setup, the microservices network represents a place to host all your microservices, such as Azure AKS, Azure Container Apps, Google Cloud Run, GKE, or EKS.

The client-host network, on the other hand, represents a different place to host your frontend, like Vercel, Hostinger, or AWS Amplify.

These two networks don’t have direct connectivity with each other.

• Removed the unnecessory port mapping of todo-api container’s port 8000 to the host’s port 8000.
• Added all the containers to the microservices network except for the todo-client .
• todo-client container is added to the client-host network and its port 3000 is mapped to the host’s port 3000.

Diagram

Now, run the following command to restart the containers and observe the changes.

docker compose up -d

You can check these two networks by running the following commands. You’ll see two newly created networks, each prefixed by the project folder name.

For example, if your project folder is named “oauth2-project”, the networks might be named “oauth2-project_microservices” and “oauth2-project_client-host”.

docker network ls

You can inspect the network using the following command. This will allow you to see the network configuration for the containers mentioned above.

docker network inspect <<network-name>>

If you access http://localhost:3000/headers, you’ll still see the host header as “todo-api:8000”. Let’s fix that now.

You can fix this by setting the Host header to the domain of the Nginx API gateway. Add a proxy_set_header directive in the Nginx configuration, like in the snippet below:

todo-api-gateway/nginx.conf

events {
    worker_connections 1024;
}

http {
    server {
        listen 80;
        server_name localhost;

        location / {
            proxy_set_header Host       $host;
            proxy_pass http://todo-api:8000;

        }
    }
}

To rebuild the API gateway image and restart the container, you can use the following commands:

docker compose build api-gateway

docker compose up -d

If you access http://localhost:3000/headers now, you will see that the host has been changed to “localhost” and doesn’t give any information about the underlying architecture.

OK. now we have the setup to implement OAuth2 proxy to secure our microservice API. (fastapi-todo-service)

Securing APIs (OAuth2 Proxy setup)— part 05

Current Setup

First, let’s understand the current setup using the following diagram.

1. User initiates a request to the /headers endpoint.
2. NGINX (API Gateway) proxies the request to todo-api:8000/headers as configured.
3. The resource server (todo-api) returns the response to the user's browser.

Our Goal

What we’re aiming to do is implement an authentication mechanism at the API gateway to inspect incoming requests and determine whether to forward the request to the resource server or not.

This is where we need OAuth2 Proxy. Let’s configure OAuth2 Proxy for our current setup and discuss what happens when we try to access our todo-api service.

Google OAuth 2.0 setup

In order to setup OAuth2 proxy we need to have an auth provider, In this project we’re going to use Google OAuth2.0 as our provider.

1. Go to your Google Cloud Console and search for “Manage Resources.”
2. Click the “Create Project” button to create a new project.

3. Provide a project name and click “Create.”

4. Navigate to the OAuth consent screen by searching for it in the Google Cloud Console.

5. Make sure you are in the correct project, ‘todo-app.’

6. Select ‘External’ and click ‘Create.’

7. Complete all the required details in the form. You can skip non-required ones.

8. Save and continue after entering your email address for the developer contact information.

7. Select all three scopes in the right-hand panel and click ‘Update.’

8. Next, enter two or more Gmail accounts to test the setup later.

9. Click ‘Add’ and then go back to the dashboard.

10. Click the ‘Credentials’ button in the left-side menu and create an OAuth client ID.

11. Select the application type as ‘Web application’ and give it a name to identify it.

12. Add ‘http://localhost' and ‘http://localhost:3000' as JavaScript origins.

13. Add ‘http://localhost/oauth2/callback' as the redirect URL. Since our NGINX OAuth2 Proxy URL is ‘http://localhost/', the callback URL of the OAuth2 Proxy will be that.

14. Create the credential, and it will show you the client ID and client secret. Save it for later usage.

OAuth2 Proxy setup

First, stop all running containers by running the following command.

docker compose down

Let’s add the OAuth2 Proxy to the docker-compose.yml file. Remember, correct indentation is crucial in YAML, so make sure to avoid indentation errors.

  oauth2-proxy:
    container_name: oauth2-proxy
    image: quay.io/oauth2-proxy/oauth2-proxy:latest
    environment:
      - OAUTH2_PROXY_CLIENT_ID=<your-client-id>
      - OAUTH2_PROXY_CLIENT_SECRET=<your-client-secret>
      - OAUTH2_PROXY_COOKIE_SECRET=<cookie-secret>
      - OAUTH2_PROXY_EMAIL_DOMAINS=*
      - OAUTH2_PROXY_REVERSE_PROXY=true
      - OAUTH2_PROXY_REDIRECT_URL=http://localhost/oauth2/callback
      - OAUTH2_PROXY_COOKIE_SECURE=false
      - OAUTH2_PROXY_UPSTREAM=http://todo-api:8000
      - OAUTH2_PROXY_HTTP_ADDRESS=http://0.0.0.0:4180
      - OAUTH2_PROXY_SET_AUTHORIZATION_HEADER=true
      - OAUTH2_PROXY_SET_XAUTHREQUEST=true
      - OAUTH2_PROXY_WHITELIST_DOMAINS=.localhost:3000
    command:
      - --http-address=0.0.0.0:4180
      - --upstream=http://todo-api:8000
      # - --skip-provider-button=true
    networks:
      - microservices

1. Update the OAUTH2_PROXY_CLIENT_ID and OAUTH2_PROXY_CLIENT_SECRET with your Google OAuth credentials.
2. To generate a cookie secret,

Linux terminal

dd if=/dev/urandom bs=32 count=1 2>/dev/null | base64 | tr -d -- '\n' | tr -- '+/' '-_' ; echo

COPY the output value for your cookie secret.

Windows Powershell

# Add System.Web assembly to session, just in case
Add-Type -AssemblyName System.Web
[Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes([System.Web.Security.Membership]::GeneratePassword(32,4))).Replace("+","-").Replace("/","_")

Follow the documentation for more details.

Overview | OAuth2 Proxy

Before running the services, you need to update your NGINX configuration as well.

nginx.conf

events {
    worker_connections 1024;
}

http {

    server {
    listen       80;
    server_name  localhost;

        location / {
            proxy_set_header Host       $host;
            proxy_set_header X-Real-IP  $remote_addr;

            proxy_pass http://oauth2-proxy:4180/;
        }
    }

    server {
        listen  80;
        server_name localhost;

        auth_request /internal-auth/oauth2/auth;

        error_page 401 = http://localhost/oauth2/start?rd=$scheme://$host$request_uri;

        location / {
            proxy_pass http://todo-api:8000;
        }


        location /internal-auth/ {
            internal;

            proxy_set_header Host       $host;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-Uri $request_uri;
            proxy_set_header Content-Length   "";
            proxy_pass_request_body           off;
            proxy_pass http://oauth2-proxy:4180/;
        }
    }
}

Now rebuild the NGINX API gateway service using the following command.

docker compose build api-gateway

Let’s run all the containers again to see the effect.

docker compose up -d

If you’ve followed all the instructions correctly, when you access `http://localhost/headers`, you should see the OAuth2 Proxy “Sign in with Google” button, similar to the image below.

Try to log in with one of your Gmail accounts that you added as a test user when you were creating the Google OAuth 2.0 consent screen.

You will be asked to continue. Click the “Continue” button.

Now, you will see the headers page with a large cookie header.

Let’s discuss what’s happening here.

1. When a user requests access to a service, it first goes through the NGINX reverse proxy, which checks if the user is authenticated. If you check the nginx.conf, you’ll see that before proxying to the todo-api service, we’ve added an auth_request to verify whether the user is authenticated or not.

2. Next, OAuth2 Proxy will validate the user authentication via an OAuth2 request with the auth provider as configured. If you look at the docker-compose.yml, you’ll see all the settings that you’ve configured there.

You can find all the configuration options in the documentation.
https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview/#command-line-options

3. Then, OAuth2 Proxy will return the authenticated state to NGINX.

4. Depending on the authenticated state (if successful), the user will be allowed to access the service. Otherwise, they will be redirected to the login portal.

If you try to access http://localhost:3000/headers now, you’ll see a 403 Forbidden status for our request. Let’s fix that.

Before moving to the frontend, let’s ensure that users are redirected to the Google login if they try to directly access our API URLs through NGINX. To do this, uncomment the comment and add the — skip-provider-button=true option under the command in the docker-compose.yml file.

Rerun the command docker compose up -d to apply the updates made to the docker-compose.yml file.

docker compose up -d

Now, if you try to access http://localhost/headers, you’ll still be able to see the headers because you’ve already authenticated and the _auth2_proxy cookie is stored in your browser.

You can right-click and manually delete the cookie or use the http://localhost/oauth2/sign_out endpoint to clear the cookie.

Now, if you try to access http://localhost/ again, you will be redirected to the Google login without showing the provider button.

Okay, now let’s fix our frontend to work as before.

Fix Todo Client

In order to fetch data correctly from our secured API, we need to pass the _oauth2_proxy cookie with our request. We can achieve that by setting credentials: “include” in the fetch request.

To learn more about the Fetch API, refer to the MDN Web Docs documentation.

Request: credentials property - Web APIs | MDN

next-todo-app/app/headers/page.tsx

...

const response = await fetch('http://localhost/headers', {
    credentials: "include"
})

...

next-todo-app/actions/todo.ts (update 6 todo fetch requests)

...

// create a todo
const response = await fetch(API_URL, {
    method: 'post',
    credentials: 'include',
    headers: {
        'Content-Type': 'application/json',
    },
    body: JSON.stringify(data)
})

...

...

// get todos
const response = await fetch(API_URL, {
    credentials: 'include'
})

...

...

// delete todo
const response = await fetch(`${API_URL}${id}`, {
    method: 'delete',
    credentials: 'include'
})

...

...

// complete todos
const response = await fetch(`${API_URL}${id}/complete`, {
    method: 'post',
    credentials: 'include'
})

...

...

// withdraw todos
const response = await fetch(`${API_URL}${id}/incomplete`, {
    method: 'post',
    credentials: 'include'
})

...

...

// update todos
const response = await fetch(`${API_URL}${id}`, {
    method: 'put',
    credentials: 'include',
    headers: {
        'Content-Type': 'application/json',
    },
    body: JSON.stringify(data)
})

...

Now, rebuild the todo-client service in Docker Compose and rerun the service again to see the updated changes.

docker compose build next-todo-app

docker compose up -d

Now, if you access http://localhost:3000/headers or http://localhost:3000/, and you have a valid _oauth2_proxy cookie stored in your browser, you will be able to see todos rendered in the todo-client application as you saw before implementing the authentication. (But still you won’t be able to create todos or delete them. We’ll fix that later in the project.)

How can we login from the Next.js todo-client?

In the current setup, to set the _oauth2_proxy cookie, we have to access http://localhost/.

However, in a real scenario, redirecting users out of your client application like this may not be ideal.

In OAuth2 Proxy, to start the authentication flow, they’ve given us the /oauth2/start endpoint. Invoking that with the rd (redirect) URL parameter will help us achieve this task.

Next.js middleware

We can create a file named middleware.ts inside the /next-todo-app/ folder and add the following code snippet to achieve that.

/next-todo-app/middleware.ts

import {NextRequest} from "next/server";

export async function middleware(request: NextRequest) {
    const oauthCookie = request.cookies.get('_oauth2_proxy')

    console.log('Current path', request.url)


    if (!oauthCookie && request.nextUrl.pathname !== '/login') {
        return Response.redirect(new URL('/login', request.url));
    }
}

export const config = {
    matcher: ['/', '/headers', '/login']
}

Next Middleware allows you to run code before a request is completed. Then, based on the incoming request, you can modify the response by rewriting, redirecting, modifying the request or response headers, or responding directly.

Routing: Middleware | Next.js

I’ve already added a simple login screen with a Login with Google button. From middleware.ts we will be redirected to that page if we don’t have the oauth2 cookie stored in the browser. we can start the auth flow from there.

If you want to have a simple logout button, add the following code snippet in your next-todo-app/app/page.tsx file.

next-todo-app/app/page.tsx

...

<main className="mx-auto max-w-7xl px-4 sm:px-6 lg:px-8 py-10 bg-slate-50 space-y-2">
    {/* add logout button here */}
    <div className="flex justify-end">
        <a href="http://localhost/oauth2/sign_out?rd=http%3A%2F%2Flocalhost%3A3000%2F"><p
            className="text-blue-500 underline transition cursor-pointer">Logout</p></a>
    </div>
    {/* logout button end */}

...

Now, rebuild the todo-client service in Docker Compose and rerun the service again to see the updated changes.

docker compose build next-todo-app

docker compose up -d

Now, If you access http://localhost:3000 with _oauth2_proxy cookie stored in the browser. You’ll be able to see your previously entered todos with the newly added logout button.

If you try to interact with the app, you’ll notice that you cannot create new todos. This is because OAuth2 Proxy is trying to authenticate preflight requests. You can change that behavior by adding — skip-auth-preflight=true in your docker-compose.yml file.

...
 
     - OAUTH2_PROXY_SET_XAUTHREQUEST=true
      - OAUTH2_PROXY_WHITELIST_DOMAINS=.localhost:3000
    command:
      - --http-address=0.0.0.0:4180
      - --upstream=http://todo-api:8000
      - --skip-provider-button=true
      - --skip-auth-preflight=true
    networks:
      - microservices

...

Rerun all the services to see the effect.

docker compose up -d

If you have followed all the instructions correctly, you should now be able to log in and log out of your todo application using the todo-client app. You have also completed the following project architecture:

Let’s have a look at the docker container diagram with network boundaries too.

Auth flow — Step 06

In the diagram below, we explain how a protected API route request is handled through the OAuth2 proxy flow. The user starts by requesting the /headers API endpoint. Below diagram is inspired by the 1438th issue in the OAuth2-proxy GitHub repository.

Document/sketch OAuth2-Proxy authentication flow · Issue #1438 · oauth2-proxy/oauth2-proxy

Congratulations. ☕

Congratulations! If you’ve made it this far, well done. While I couldn’t cover everything I had hoped to in this article, I promise to provide an extension to it. In the next part, I plan to discuss:

• How to handle sign-in, sign-out, and protected routes in the Next.js middleware.
• How to leverage our current setup in the Next.js front-end.
• How to handle authorization in the FastAPI back-end.
• CORS and preflight requests.
• Cross-Site Request Forgery (CSRF) and CSRF cookies.
• OAuth2 proxy refresh cookies, and more options.
• How to add more services to our current setup?

Following that, I plan to continue writing about how to host this kind of architecture in managed services and Kubernetes in the near future.

If you’ve enjoyed this project, don’t forget to give it a 👏. Your support gives me more energy and enthusiasm to write more projects like this. Thank you!

Happy Coding!!! 🐧

Securing Microservice APIs with OAuth2 Proxy: A Complete Project was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the world of making software, things happen really fast nowadays. One important thing developers do is put their software inside containers, like virtual boxes. This helps the software run well no matter where it goes. Docker is a popular tool for doing this.

In this article, we will show you how to put a simple php-api application into a Docker container and host it on the cloud.

Prerequisites:

Before we start the process of dockerizing the application, make sure the following requirements are satisfied:

1. Docker: Install Docker to create, manage, and run containers.
2. Git & GitHub: Having the basic idea behind local repositories & remote repositories.

I’m suggesting you to read the below article before beginning. It will give you a better idea about the local deployment of the same application.

Building a Simple 3-Tier Architecture with Docker Compose: A Hands-On Project Journey

Steps to Dockerize the simple-php-todo-api Application

1. Create the Application

• Create a project folder named simple-php-todo-api .
• Inside that folder create another folder calledsrc .
• Inside the src folder create a new file named index.php and add the below code to it.

<?php
// Set the response content type to JSON
header('Content-Type: application/json');
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Methods: GET, POST, OPTIONS');

// Create a response array
$response = [
    [
        "_id" => 1,
        "todo" => "Todo 1"
    ],
    [
        "_id" => 2,
        "todo" => "Todo 2"
    ],
    [
        "_id" => 3,
        "todo" => "Todo 3"
    ],
    [
        "_id" => 4,
        "todo" => "Todo 4"
    ]
];

// Encode the response array as JSON
$jsonResponse = json_encode($response);

// Output the JSON response
echo $jsonResponse;

• Above code will create a simple hard-coded todo api server.

2. Create the Dockerfile

In the root directory of your project, create a file named Dockerfile (without any file extensions) and add the following content:

# Use the official PHP image with Apache as the base image
FROM php:8.0-apache

# Copy your application files to the appropriate directory
COPY ./src /var/www/html/

# Expose the default HTTP port (port 80)
EXPOSE 80

# Enable the rewrite module (assuming your PHP application uses URL rewriting)
RUN a2enmod rewrite

# Install any necessary PHP extensions or dependencies here (if needed)
# For example, if you use database connections, you might need to install relevant extensions.

# Set the working directory to the web root
WORKDIR /var/www/html/

# Start the Apache web server when the container starts
CMD ["apache2-foreground"]

3. Build the Docker Image

Run the terminal on your project folder and then, Build the Docker image using the following command, replacing simple-php-todo-api with a suitable image name:

docker build -t simple-php-todo-api .

4. Run the Docker Container

After building the Docker image, run a container from it:
Container port 80 will be mapped to the host port 8000 in the below command.

docker run -p 8000:80 simple-php-todo-api

5. Access the simple-php-todo-api

Open your web browser and navigate to http://localhost:8000/ to see your simple-php-todo-api application running inside a Docker container.

Steps to Host the Dockerized Next.js Application in Google Cloud Run

This part of the tutorial explains how to put the created app on Google Cloud Run using GitHub. This way is smooth and makes it easy to get apps up and running. It will enable the collaboration between developers as well.

Going further you can automate the deployment process and ensure consistent updates across environments like dev , test & prod .

1. Set Up GitHub Repository

First we have to make sure our code is on GitHub.

1. Create a new repository in your GitHub account.
2. You can copy the code from your GitHub repository Quick Setup and push your code to that repository.

Quick setup will only add the README.md file. After that follow the below commands to push your project code to the repository.

git add .
git commit -m "feat: add simple-todo-api"
git push

Okay, Now we’ve done setting up our GitHub repository.

2. Getting Google Cloud Run Ready

1. Go to the cloud run https://cloud.google.com/run and create a new service.
2. Select the option “Continuously deploy new revisions from a source repository” like in the below image.

3. Then click the “SET UP WITH CLOUD BUILD” button.

• Select “GitHub” as Repository provider and connect your GitHub account.
• Click the “Repository” dropdown.

4. If you see your repository there, select it and continue. If not click the “Manage connected repository” link and authorize your GitHub repository from there.

5. Then click the “Next” button to continue.

6. In the build configuration main branch is already selected, That means Every time when you push any changes to the main branch, Cloud Run will build a new image and deploy from that branch. You can change it as you want. If you’re maintaining an existing repository with a master branch. You can change it to ^master$ or else you can choose a specific branch that you want to deploy your application.

7. Since we’re using Docker to build the project, select the “Build Type” as Dockerfile.

8. Then save it and go back to the “Cloud Run” configuration.

9. You can select the region as you want, I select asia-southeast1 (Singapore) because It gives me better performance.

Lower the distance, higher the performance.

10. End of the configuration, Select the “Allow unauthenticated invocations” option.

11. Click the “Container, Networking, Security” drop down and change the container port to “80”. Since we’re serving the application in PORT 80 it should be changed.

Note: I’m recommending you to click “Learn more” links and discover a lot about Cloud Run. This will help you to understand why each option is there.

12. Then click the “Create” button.

It will take some time to deploy your container on Cloud Run. In the end it will show you an output like in the below figure.

13. Now you should be able to access your simple-php-todo-api using the URL there.

Note:
Please remember that Cloud Run is designed for stateless applications, so It’s well-suited for serving static content or APIs like applications. If your application requires dynamic content or server-side processing, Cloud Run might not be the best choice. In such cases, you might consider other Google Cloud services like Compute Engine or Google Kubernetes Engine (GKE).

Experience the CI/CD process

• CI: Continuous Integration
• CD: Continuous Deployment/Delivery

1. Replace the index.php with below code.

<?php
// Set the response content type to JSON
header('Content-Type: application/json');
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Methods: GET, POST, OPTIONS');

// Create a response array
$response = [
    [
        "_id" => 0001,
        "todo" => "Buy groceries for the week"
    ],
    [
        "_id" => 0002,
        "todo" => "Complete the presentation slides"
    ],
    [
        "_id" => 0003,
        "todo" => "Call the utility company to resolve billing issue"
    ],
    [
        "_id" => 0004,
        "todo" => "Schedule a dentist appointment"
    ]
];

// Encode the response array as JSON
$jsonResponse = json_encode($response);

// Output the JSON response
echo $jsonResponse;

2. Push changes to the GitHub repository.

git add .
git commit -m "update todos"
git push

3. You’ll be able to see the build happening in the Cloud Run and will be able to access the updated todo-api after that.

Resources

Github repository for this tutorial: https://github.com/kesaralive/simple-php-todo-api

Conclusion

So, to sum it all up: first, we made a basic “simple-php-todo-api” program. Then, we made it even cooler by putting it into a special Docker image. We made sure it worked on our computer and then stored it on GitHub.

But that’s not all! We brought in Google Cloud Run. It let our program live on the internet.

Using Docker, GitHub, and Google Cloud Run together we’ve experienced the power of CI/CD in the end.

Some other docker articles by me

• From OS Hassles to Team Harmony: How Docker Saved Our Group Project
• Diving Deeper into Docker Networking with Docker Compose

Connect with me

• LinkedIn: https://www.linkedin.com/in/kesaralive
• GitHub: https://github.com/kesaralive
• Twitter: https://twitter.com/kesaralive

In a recent project, I had to use Livekit, but I had trouble finding up to date resources on how to deploy a Livekit server on Google Cloud Platform (GCP).

So, I decided to write an article to help others who might be facing the same issue. The article will be a step-by-step guide to set up a Livekit server on GCP, making it easier to use this useful tool for real-time communication applications.

What is Livekit?

Livekit is an open-source toolkit designed for building scalable, real-time video and audio applications.

It provides a set of APIs and SDKs that developers can use to create applications such as video conferencing, live streaming, and real-time communication tools. Livekit aims to make it easier for developers to integrate live video and audio features into their applications without having to handle the complexities of video encoding, streaming, and signaling protocols.

Let’s see how can we deploy a Livekit server on GCP.

Prerequisites

1. You need to have a Google Cloud Account.
2. A domain that you own.
3. The ability to add DNS records for subdomains for your new LiveKit server

For this tutorial I’m gonna use my own domain (kesaralive.com)

Virtual Machine Setup

First we need to have a virtual machine provisioned in google cloud. In order to do that, go to the cloud console and navigate to the VM instances like in the below image.

1. Click the Hamburger icon in the top left corner of the cloud console.
2. In the side menu, go to “Compute Engine”.
3. Select “VM Instances”.

Click the button “Create Instance” to configure a new Virtual Machine (VM) Instance.

VM Instance Configuration

1. Enter a name you prefer to identify your VM Instance.
2. Select the appropriate region to you.
   “Closer the region, better performance.”
3. I’m selecting a low cost E2 series Virtual Machine.
   (Not recommended for a production server)

4. Click the “Change” button to configure the boot disk.

5. Select the “Ubuntu” operating system from the dropdown menu.

6. Click the “Select” button to save the boot disk configuration.

7. Allow both HTTP & HTTPS traffic to access port 80 & 443 from the Internet. (This will allow you to access the VM through your browser.)
8. Click the “Create” button to start your Virtual Machine.

DNS Configuration

In order to setup Livekit, you need to have two domain names. You can use subdomains for this. One for the Livekit server and other one for the turn server. Since Livekit has built in TURN, we can use the same virtual machine for TURN. In this tutorial I’m using following sub domains for the setup.

1. For LiveKit: livekit.kesaralive.com
2. For TURN: turn.kesaralive.com

You can use your domain as below,

1. livekit.<your domain name>
2. turn.<your domain name>

Note: I’m going to use Cloud DNS to manage these subdomains but you can create an “A record” to your subdomain directly from you main domain authority. (This can be much simpler for you.)

First, Navigate to the “Cloud DNS” like in the blow image.

If you haven’t enable “Cloud DNS API”, enable it.

We have to create two domain zones for this.

1. First one is for the livekit.<your domain name>
2. Second one is for the turn.<your domain name>

In my case, I will use live.kesaralive.com and turn.kesaralive.com for the tutorial.

After creating a domain zone we have to give the sub domain authority to the Cloud DNS to resolve any requests on that domain. In order to do that, we have to setup custom nameservers for our subdomains in the main domain name’s authoritative DNS server provider. In my case It’s cloudflare.

After creating a domain zone, the next step is to allow “Cloud DNS” to manage the subdomains ( livekit.kesaralive.com and turn.kesaralive.com ) so that it can handle requests for them. To do this, we need to set up custom nameservers for our subdomains in Cloudflare, which is the service I use to manage my main domain’s (“kesaralive.com”)settings.

In a nutshell, setting up custom nameservers in Cloudflare is essential for giving authority to Cloud DNS

Sub Domain Setup

Before going to create nameserver records in “Cloudflare”,
Let’s set “A Records” in both subdomains for the previously created virtual machine’s ip address.

When you create an “A record” in DNS (Domain Name System), you are essentially associating a specific IP address with a domain name. This enables the DNS system to translate human-readable domain names into machine-readable IP addresses, allowing computers to locate and communicate with the correct server hosting the website or service associated with that domain.

Go to the domain zone by clicking it. (Ex: livekit.kesaralive.com domain zone)

1. Click to “Add standard”.
2. In Cloud DNS, when working within the same GCP Project, you have the convenience of selecting an IP address for a resource without the need for manual input. This is especially helpful when the desired IP address belongs to a resource within the same GCP Project. As we have already created our virtual machine within this project, we can easily avail ourselves of this feature by simply clicking the “Select IP Address” button. This streamlined process ensures a smooth and efficient experience when configuring our DNS settings, eliminating the burden of manual entry and facilitating seamless integration within the project’s infrastructure.
3. Select the appropriate “IP address” which has directed to the previously created Virtual Machine.
4. Then click the “Create” button to create the A record.

Do that for the both livekit and turn sub domains like in the below image.

Nameservers

Like we discussed earlier, to give domain authority to the “Cloud DNS” we have to create NS records in our Authoritative DNS server provider. In my case It’s “CloudFlare”, This can be your domain name provider (Where you bought the domain name).

COPY all NS records one by one and add them into the authoritative DNS server provider like in the below image. (In my case, It’s Cloudflare.)

Do the same for both livekit and turn subdomains. In the end you should have a recordset like in the below image.

Livekit deployment on the Virtual Machine

1. Navigate to the “VM Instances” like we did in the previous step.

2. Click the “SSH” button, this will give you a dialog box to authorize. Click “Authorize” button and continue.

3. It will give you the access to your Virtual Machine terminal.

COPY the provided code and paste it into your Cloud VM terminal to initiate a clean installation of Docker in your virtual machine.

You can use “ctrl + shift + v” to paste the code, or simply do a right-click and select “paste.”

This installation process will take some time as it sets up Docker on your “Cloud VM”, ensuring a smooth and efficient deployment of Docker’s containerization capabilities within your virtual environment.

for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
sudo docker run hello-world

RUN docker --version to check whether you have installed the docker correctly.

  docker --version

Now let’s add the default user to the docker user group

sudo usermod -aG docker $USER
newgrp docker

Verify that you can run docker commands using the below command

docker run hello-world

Then RUN the below command to generate the starter files for the Livekit server.

docker run --rm -it -v$PWD:/output livekit/generate

We’ll go with “LiveKit Server only” option in this tutorial.

$ docker run --rm -it -v$PWD:/output livekit/generate
Generating config for production LiveKit deployment
This deployment will utilize docker-compose and Caddy. It'll set up a secure LiveKit installation with built-in TURN/TLS
SSL Certificates for HTTPS and TURN/TLS will be generated automatically via LetsEncrypt or ZeroSSL.

For the primary domain name, give livekit.<your domain name> . In my case, it’s livekit.kesaralive.com .

For the TURN domain name, give turn.<your domain name> . In my case, it’s turn.kesaralive.com .

Choose “Let’s Encrypt (no account required)” option.

Use the “latest” version of LiveKit.

Select “no — (we’ll bundle Redis)” option.

Select “Startup Shell Script” for this tutorial.

It will generate an API KEY, API SECRET and a TEST TOKEN.
Save it somewhere for later use.

Firewall Setup

We’ll use “Linux UFW” to configure the firewall.

Linux UFW (Uncomplicated Firewall) is a user-friendly front-end for managing iptables, the built-in firewall tool for Linux systems. It is designed to simplify the process of configuring a firewall and makes it easier for users to control network traffic and secure their systems.

First switch to the root user using sudo su command.

sudo su

Enable the UFW in your virtual machine.

ufw enable

Say yes (y) to this and enable the “OpenSSH” asap in your virtual machine. If you close the window without enabling SSH, you will lose the access to the VM.

ufw allow "OpenSSH"

In order to deploy Livekit server successfully, we need to make sure that the following ports are open on our firewall and accessible on the instance:

• 443 — primary HTTPS and TURN/TLS
• 80 — TLS issuance
• 7881 — WebRTC over TCP
• 443/UDP — TURN/UDP
• 50000–60000/UDP — WebRTC over UDP

Let’s Configure these ports using following commands.

ufw allow 443
ufw allow 80
ufw allow 7881
ufw allow 50000:60000/udp

RUN the command ufw status . You’ll get an output like in the below figure.

List the current directory using ls command to see generated folder.

cd to the generated folder and set permissions to execute for init_script.sh file. Folder name will be different according to your domain name.

cd livekit.kesaralive.com/
chmod +x init_script.sh

Then run the script using below command.

./init_script.sh

It will take some time for the DNS to propagate and for everything to be set up.

After a successful deployment, you will be able to see an “OK” message when you browse your Livekit URL.

Then install the livekit-cli to generate a room token.

curl -sSL https://get.livekit.io/cli | bash

Use your API credentials generated by the previous steps to create the token.

livekit-cli create-token \
    --api-key <project_api_key> --api-secret <project_secret_key> \
    --join --room my-first-room --identity user1 \
    --valid-for 24h

You can test the server using Livekit’s example app

Click the “Connect” button and join to the room. 🧨

LiveKit provides official client SDKs for JavaScript, React, iOS/MacOS (Swift), Android (Kotlin), Flutter, Unity (WebGL). You can use the Livekit server you built in your projects using these client SDKs.

You can see all the available SDKs using the below link.

https://docs.livekit.io/references/client-sdks/

If you want another tutorial creating a simple app using the server we’ve built, Let me know in the comments section.

Let’s meet with another informative article soon.

Until then, Happy Coding!

Follow me on LinkedIn: https://www.linkedin.com/in/kesaralive

Resources

Livekit CLI Setup: https://docs.livekit.io/getting-started/cli-setup/
Livekit Deployment to a VM: https://docs.livekit.io/oss/deployment/vm/

In my previous article, I made a commitment to explore Docker networking, and now I’m delivering on that promise. In this new article, we’ll dive into the workings of Docker networking and gain a better understanding of how it operates.

Network Modes

In Docker, We have several Docker network types available:

1. Bridge : The default type of network in Docker. It provides isolation between containers, but also allows containers to communicate with each other.
2. Host : A special type of network that allows containers to connect directly to the host machine’s network stack.
3. Overlay : A more complex type of network that allows containers to communicate with each other even if they are not running on the same host machine.
4. Macvlan: This network allows you to specify a MAC address for the container, allowing the container to be a physical device in the network.
5. Network plugins: You can install and use third-party network plugins.
6. None: This won’t create any new network environment. If the none driver is used, the container can only use the local network (Your computer network).

In this article, I’m planning to cover the Bridge and the Host Network modes.

The Default Bridge Network

When you install Docker on your local computer, it will create a new network called Docker Bridge. You can check this by running the following command in your terminal.

In Linux Terminal:

ip address ls

You’ll see the docker0network interface there.

Additionally, you can run the following Docker command to list all the Docker networks available on your computer. (Windows & Linux)

docker network ls

You will get an output like the one below.

If you’ve worked with Docker for some time, it might show you some additional networks alongside it. We’ll discuss those in more detail later in this article.

NETWORK ID          NAME                DRIVER              SCOPE
17c1d0d4956b        bridge              bridge              local
f04d3c0691be        host                host                local
cea56a468e79        none                null                local

In simple terms, the above output tells us about a connection between two entities known as a “bridge network” and a “bridge driver.” It’s essential to understand that the network and the driver are linked, but they are not same components. The output also indicates that the bridge network has a local scope, meaning it exists only on this Docker host.

All the bridge networks offer a single host network, It means that the network is limited to enabling communication and connectivity among containers running on the same host. It does not extend to other hosts or computers in the network. This approach provides isolation and enhances security for the containers, allowing them to interact exclusively within the confines of the same system.

However, containers running on different hosts cannot directly communicate with each other using the bridge network. For multi-host communication, Docker offers alternative network drivers, such as overlay networks. These overlay networks enable containers across multiple hosts to communicate with each other over a distributed network, facilitating seamless communication in a clustered and distributed Docker environment.

Now, let’s see how this behavior works by doing a simple tutorial.

Docker Compose: Default Bridge Network Tutorial

Prerequisites

I hope you’ve installed docker in your computer. If not follow the official documentation. https://docs.docker.com/engine/install/

In this article, I will frequently refer to the following terms:

1. Mars: Planet in our solar system, used for a network name.
2. Phobos: One of the moons of Mars, used as a network node (container service).
3. Deimos: Another moon of Mars, used as a network node (container service).
4. Earth: Our planet, used for a network name.
5. Moon: Our moon, used as a network node (container service).

First, we need to have a docker-compose.yaml file to get started.
In this tutorial, we will create the following "Mars" network by the end of the process.

1. First, Create a new project folder in your local computer.
2. Within the project folder, create a new file named docker-compose.yaml.
3. Now, copy and paste the following code snippet into your docker-compose.yaml file

docker-compose.yaml

version: "3"
services:
  phobos:
    container_name: phobos
    image: busybox
    command: sleep infinity
  deimos:
    container_name: deimos
    image: busybox
    command: sleep infinity

We will use ‘Busybox’ docker image in this tutorial. It’s a linux docker image coming in somewhere between 1 and 5 Mb in on-disk size. It’s small and it contains ‘iputils-ping’ package. Simply, It comes with the ping command.

With the ping command, you can determine whether a remote destination IP is active or inactive. You can also find the round-trip delay in communicating with the destination and check whether there is a packet loss. Basically, You can check the connectivity between two destinations (computers/ nodes).

Let’s see what’s inside our docker-compose.yaml file.

version: "3": This line specifies the version of the Docker Compose file format being used, which is version 3 in this case.

services: This section defines the individual services that make up the application.

phobos: This is the name of the first service, which corresponds to a container named "phobos". Phobos is one of moons of the Mars.

container_name: phobos: This specifies the name of the Docker container that will be created for the "phobos" service. The container's name will be "phobos".

image: busybox: This line indicates that the "phobos" service will use the "busybox" image as its base image. BusyBox is a lightweight and minimal container image often used for debugging purposes or as a base image for other containers.

command: sleep infinity: This command is executed when the "phobos" container starts. In this case, the command is "sleep infinity", which means the container will do nothing and remain running indefinitely, essentially acting as a background process.

• deimos service is also as same as the the phobos.

Now Let’s open the terminal in the project directory and execute the docker compose up -dcommand to build and run the both services.

docker compose up -d

We add -d at the end of the command to run all the containers in detach mode. You can check the running containers by using the docker-compose ps command.

$ docker compose ps
NAME                IMAGE               COMMAND             SERVICE             CREATED             STATUS              PORTS
deimos              busybox             "sleep infinity"    deimos              17 hours ago        Up 17 hours         
phobos              busybox             "sleep infinity"    phobos              17 hours ago        Up 17 hours         

Now you should be able to see two running containers.

If you run docker network ls now, you'll notice that it creates a new network named <your-project-folder>_default. In my case, it's docker-networking_default because I named my project folder as docker-networking.

$ docker network ls
NETWORK ID     NAME                        DRIVER    SCOPE
17c1d0d4956b   bridge                      bridge    local
eeadb6a2dc43   docker-networking_default   bridge    local
f04d3c0691be   host                        host      local
cea56a468e79   none                        null      local

Like I said in my last article,

Docker Compose understands the idea behind running services for one application on one network.

This is precisely what happens when you build and run your containers using Docker Compose. Docker Compose’s standard behavior involves creating a new bridge network specifically for its containers. This bridge network enables seamless communication and connectivity between the containers defined in the docker-compose.yaml file, facilitating a well-organized and isolated environment for the applications to run.

Connectivity

First of all, Let’s inspect our new network created by docker compose using the following command.

⚠️ Note: When you follow this tutorial, you might receive different IP addresses for your containers. If you prefer copy and paste, please carefully use the appropriate IP addresses from your output while continuing with this tutorial. The IP addresses may differ depending on your system and configuration, so it’s important to handle them carefully to ensure a successful implementation of the tutorial.

docker network inspect <newly-created-network>

$ docker network inspect docker-networking_default
[
    {
        "Name": "docker-networking_default",
        "Id": "1bdb95b66268640cf272174afa966913db2ad40ac38553dc6511028b51952652",
        "Created": "2023-07-22T10:34:07.181685944+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "687a21b31cf4715391f3858e86c4c43c4a254359faa4aa05bf002194e036e5d0": {
                "Name": "phobos",
                "EndpointID": "8039fdaa2c7bdb87b2afbb2fad48bed9aa1b42d9e1c3f7ce1f31773a70c32e31",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "9fa4b494b685eb3557cbeca86c40cb4dbabf9a6bd4b8a6e62e36f9f6fe2b50ea": {
                "Name": "deimos",
                "EndpointID": "e529ce9c375d41d7c9717f9abb3820a553d39da327f3b79126f0eafc0ef81cd3",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "docker-networking",
            "com.docker.compose.version": "2.18.1"
        }
    }
]

You will receive a JSON output similar to the snippet above. As you can see under the “Containers” section, we have our “phobos” and “deimos” containers, and each of them has been assigned to an individual IP address from the same network. This illustrates that the containers are successfully connected and can communicate with each other using the designated network.

Now let’s check the connectivity between the two containers.

Run the docker exec -it <container> /bin/sh command to start an interactive shell session.

docker exec -it phobos /bin/sh

The exec sub-command of Docker is used to run a command inside a running container.

-it These are two separate options provided together.

The -i represents --interactive and it keeps the standard input (stdin) open even if not attached, and the -t represents --tty which enables an interactive shell session by allocating a pseudo-terminal.

It will then execute the specified command /bin/sh inside that container, which starts an interactive shell session. In our case, It’s /bin/sh because Busybox image has “sh” installed by default. This allows you to work directly with the command-line interface inside the container, just as if you were logged in to a virtual machine or a remote server.

Just for the sake of easiness, I extracted the below part from the previous output. The results might vary for you, so please keep that in mind.

"Containers": 
{
"687a21b31cf4715391f3858e86c4c43c4a254359faa4aa05bf002194e036e5d0": 
  {
    "Name": "phobos",
    "EndpointID": "8039fdaa2c7bdb87b2afbb2fad48bed9aa1b42d9e1c3f7ce1f31773a70c32e31",
    "MacAddress": "02:42:ac:12:00:03",
    "IPv4Address": "172.18.0.3/16",
    "IPv6Address": ""
  },
"9fa4b494b685eb3557cbeca86c40cb4dbabf9a6bd4b8a6e62e36f9f6fe2b50ea": 
  {
    "Name": "deimos",
    "EndpointID": "e529ce9c375d41d7c9717f9abb3820a553d39da327f3b79126f0eafc0ef81cd3",
    "MacAddress": "02:42:ac:12:00:02",
    "IPv4Address": "172.18.0.2/16",
    "IPv6Address": ""
  }
}

Ping operates by means of Internet Control Message Protocol (ICMP) packets. Pinging involves sending an ICMP echo request to the target host and waiting for an ICMP echo reply.

For networking enthusiasts: ICMP packets pass through the Gateway of 172.18.0.1. The above figure is simply to help you understand our next steps and what we aim to achieve.

First, let’s run the command ip address ls to validate the above output obtained from using the docker network inspect command. As you can see in Terminal 1.0, you will see the same IP addresses displayed as shown in the above output.

phobos : 172.18.0.3/16

Now, let’s test the connectivity between two containers by calling “deimos” at IP address 172.18.0.2/16 from “phobos” at IP address 172.18.0.3/16, just like in the figure shown in Terminal 2.0.

ping 172.18.0.2

As shown in the figure Terminal 2.0, if you have followed all the steps correctly so far, you should now be able to successfully transmit packets between the two containers, “phobos” and “deimos.” This demonstrates the successful establishment of communication between the containers over the created bridge network.

Docker DNS

When you create a Docker network, Like we discuss earlier, Docker automatically sets up a bridge network, which acts as a virtual network switch connecting all containers in that network. The Docker daemon acts as the DHCP server for this bridge network, providing IP addresses to containers and handling their DNS resolution.

Docker daemon:

AKA “Dockerd,” is a core component of the Docker platform. It is responsible for managing Docker objects like images, containers, networks, and volumes, and it acts as the intermediary between the Docker CLI (Command Line Interface) and the container runtime. The Docker daemon runs as a background process on the host machine and listens for Docker API requests.

DHCP server

A DHCP server is a computer program that automatically assigns IP addresses to devices on a network, simplifying the process of connecting to and communicating with other devices.

DNS server

A DNS server is a server that translates human-readable domain names (like medium.com) into numerical IP addresses, allowing devices to find and access websites and services on the internet.

If you run ns lookup medium.com command in your terminal, you’ll be able to see it’s numerical IP address. You can use it to inspect ip address of a domain name.

Let’s call deimos again!

If you call “deimos” by its name in the same interactive terminal, you will notice that it resolves to the same IP address as before (Terminal 3.0). This behavior occurs because Docker’s internal DNS resolution allows containers to communicate with each other using their assigned names, which are mapped to their respective IP addresses within the Docker network.

ping deimos

Since we’ve given the container_name as deimos in our docker-compose.yaml we can call our container using it’s name.

Note: you don’t have to worry about the ip configuration of a container in a docker network. You can call the container by its name.

You can exit from the interactive terminal by entering the exit it it.

exit

Now let’s call phobos from deimos

Like we did in the previous step, we can create an interactive shell session in deimos by running the below command (Terminal 4.0).

docker exec -it deimos /bin/sh

Then try to ping phobos from deimos (Terminal 5.0)

ping phobos

You should be able to see packets transmitting without any loss.

Docker Compose: User-Defined Network

So far we didn’t specify any networks in our docker-compose.yaml . Eventhough phobos & deimos can talk to eachother, It’s not yet in the Mars network.

So far, we haven’t specified any networks in our docker-compose.yaml. Even if the "phobos" and "deimos" can communicate with each other, they are not yet part of the Mars network.

Before going to connect them to Mars Network, run the command docker compose down in your project directory to remove phobos & deimos from current configuration.

$ docker compose down
[+] Running 2/2
 ✔ Container deimos  Removed                                                                                                                                                                      10.3s 
 ✔ Container phobos  Removed  

Let’s add “phobos” and “deimos” to the Mars network. To do that, update the docker-compose.yaml file with the following code snippet:

docker-compose.yaml

version: "3"
services:
  deimos:
    container_name: deimos
    image: busybox
    command: sleep infinity
    networks:
      - mars
  phobos:
    container_name: phobos
    image: busybox
    command: sleep infinity
    networks:
      - mars
networks:
  mars:
    name: mars
    driver: bridge

Then run the command docker compose up -d again in the terminal inside your project directory.

$ docker compose up -d
[+] Running 3/3
 ✔ Network mars      Created                                                                                                                                                                       0.1s 
 ✔ Container phobos  Started                                                                                                                                                                       0.6s 
 ✔ Container deimos  Started  

Now, If you run docker network ls command, You’ll see our newly created mars network there (Terminal 6.0).

docker network ls

If we look at the mars network using docker network inspect mars you’ll see that phobos & deimos inside that network.

$ docker network inspect mars
[
    {
        "Name": "mars",
        "Id": "9dc9407778df9b2bc336a9d4c912c869592889aeb46ae08050a09c81d7e1828d",
        "Created": "2023-07-22T13:42:23.084152643+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "a82fa15fa31548cf3f92e6c90174f1d4a78d7b743186e6f05922c0d1a7d58eb7": {
                "Name": "phobos",
                "EndpointID": "1045e43d6c87e9e245dadaa7d62aed56268320820848b2fd330e9dfe4cd2b4bc",
                "MacAddress": "02:42:ac:15:00:03",
                "IPv4Address": "172.21.0.3/16",
                "IPv6Address": ""
            },
            "aadefe0c9ada96fbf61e1f0f5c3d68dfc4e3685f500f771f147c46bb1e37fb16": {
                "Name": "deimos",
                "EndpointID": "a4dbcb28acc4025438201ce55133a3effe1f6695583d92882a721f6a255dadfa",
                "MacAddress": "02:42:ac:15:00:02",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "mars",
            "com.docker.compose.project": "docker-networking",
            "com.docker.compose.version": "2.18.1"
        }
    }
]

If you try to check the connectivity between phobos and deimos . It will work like before we tested in the default bridge network.

If you check the connectivity between “phobos” and “deimos” now, it will work just like before when we tested in the default bridge network. This is because both containers are now part of the “mars” bridge network, which enables them to communicate with each other within this designated network.

Earth Network

Let’s add “Moon” and “Earth” networks to our docker-compose.yaml file. Below is the updated code snippet:

docker-compose.yaml

version: "3"
services:
  deimos:
    container_name: deimos
    image: busybox
    command: sleep infinity
    networks:
      - mars
  phobos:
    container_name: phobos
    image: busybox
    command: sleep infinity
    networks:
      - mars
  moon:
    container_name: moon 
    image: busybox 
    command: sleep infinity
    networks:
      - earth
networks:
  mars:
    name: mars
    driver: bridge
  earth:
    name: earth 
    driver: bridge

1. Use docker compose down to stop & remove previous configuration.
2. RUN docker compose up -d to build and run all the containers.
3. RUN docker compose ps see the running containers.

$ docker compose ps
NAME                IMAGE               COMMAND             SERVICE             CREATED             STATUS              PORTS
deimos              busybox             "sleep infinity"    deimos              7 seconds ago       Up 5 seconds        
moon                busybox             "sleep infinity"    moon                7 seconds ago       Up 5 seconds        
phobos              busybox             "sleep infinity"    phobos              7 seconds ago       Up 5 seconds    

4. RUN docker network ls to list all the docker networks. You’ll see that our mars & earth networks listed there.

$ docker network ls
NETWORK ID     NAME                        DRIVER    SCOPE
1d540324eaff   bridge                      bridge    local
1bdb95b66268   docker-networking_default   bridge    local
ad19e866e8ad   earth                       bridge    local
f04d3c0691be   host                        host      local
31a102c179b7   mars                        bridge    local
cea56a468e79   none                        null      local

5. Let’s try to talk to the moon (Earth Network) from the phobos (Mars Network).

Create a interactive shell session in phobos and ping to the moon. (Terminal 7.0)

docker exec -it phobos /bin/sh

You will notice that you can’t communicate with “moon” from “phobos,” but you can still communicate with “deimos.” This is because “deimos” is part of the same “Mars Network” as “phobos,” while “moon” is in a separate network, “Earth Network,” which is not directly accessible from “Mars Network.”

5. Let’s inspect the “Earth Network” using docker network inspect earth command.

$ docker network inspect earth
[
    {
        "Name": "earth",
        "Id": "ad19e866e8add2936f047fd8ba6315cbded2b5ba8ca544afbc6df0cb5e79ecf4",
        "Created": "2023-07-22T14:26:47.928073948+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.23.0.0/16",
                    "Gateway": "172.23.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "7cc84d8f2854565ab2a63d7aee508861b73539494187b504b4b70f520f7d7af3": {
                "Name": "moon",
                "EndpointID": "86f87b3f0c6a79ade3f301d3811ea4c916e9104821bb7636914172a6f1194960",
                "MacAddress": "02:42:ac:17:00:02",
                "IPv4Address": "172.23.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "earth",
            "com.docker.compose.project": "docker-networking",
            "com.docker.compose.version": "2.18.1"
        }
    }
]

6. Let’s inspect “Mars Network” too. command: docker network inspect mars

$ docker network inspect mars
[
    {
        "Name": "mars",
        "Id": "31a102c179b70a6d9b855da6422e0797fa527069f0c324bdc622de5f88805e0c",
        "Created": "2023-07-22T14:26:47.842542312+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.22.0.0/16",
                    "Gateway": "172.22.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "3a20082a7fafca313d04c397ef2389831dc4cf5224e3ca9c0c05bd062e631d9d": {
                "Name": "deimos",
                "EndpointID": "c61ea9206b1f2406402401ee0be8b1ba7e6d540116d21a51cff0c061371a1d10",
                "MacAddress": "02:42:ac:16:00:03",
                "IPv4Address": "172.22.0.3/16",
                "IPv6Address": ""
            },
            "89ebfa3426dd31e6b466a86e8f26796e736f093dbef13f10799c869b242a8202": {
                "Name": "phobos",
                "EndpointID": "ecf6b66097b19985ed8467dc6056911c0fb16e0eb0938c0dfa66b6e2f10bb025",
                "MacAddress": "02:42:ac:16:00:02",
                "IPv4Address": "172.22.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "mars",
            "com.docker.compose.project": "docker-networking",
            "com.docker.compose.version": "2.18.1"
        }
    }
]

You’ll clearly see that these two networks are two different subnets.

Mars Network: 172.22.0.0/16

Earth Network: 172.23.0.0/16

Mars doesn’t know how to talk to Earth. That’s why we couldn’t talk to the moon from phobos.

Phobos sends ICMP requests to the Deimos through the Mars Network, Since there’s no connection between Earth Network and Mars Network Phobos can’t send any ICMP requests to the Moon.

To communicate between two networks, It’s better to use overlay network mode.

By using a user-defined bridge network, You can achieve the isolation.

Containers connected to a user-defined bridge network are isolated from other networks by default. This ensures that communication between containers is limited to the network they belong to, enhancing security and preventing unintended access.

Host Network Explanation (Only works on Linux)

⚠️ Note: Host network driver only works on Linux Hosts
✅ 1st Part will work in any OS.

Before going to talk about host networks, Let’s see how we mapped our container ports to the host machine’s ports previously in a default bridge network.

To understand the docker Host Network, we’ll use the nginx docker image in this tutorial.

Nginx Image

Nginx (pronounced “engine-x”) is an open source reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer, HTTP cache, and a web server (origin server).

By default, Nginx server will listen for incoming HTTP requests on port 80. When a client (such as a web browser) sends an HTTP request to the server’s IP address or domain name, Nginx will receive the request on port 80, process it, and return the appropriate response back to the client.

Like that in nginx docker image also listens for incoming HTTP requests on port 80. So when we build an nginx container in a docker bridge network, we have to map it’s port 80 to a port in our host machine (your computer).

In the below example, I’m going to map nginx container’s port 80 to the host machine’s port 8000.

1. Create a new project folder. (You can name it as you want, I’m naming it docker-host-network)
2. Create a new docker-compose.yaml file and add the bellow code snippet to it.

docker-compose.yaml

version: "3"
services:
  web:
    container_name: hello_world
    image: nginx
    ports:
      - 8000:80

3. RUN the command docker compose up -d to build and run the new container.

As we learned before, this will create a default bridge network for the nginx container. Our container will be named “hello_world”. In the port mapping, we’re exposing our container’s port 80 on the Host machine’s port 8000.

If you run docker compose ps in your terminal, you’ll see that port mapping there.

$ docker compose ps
NAME                IMAGE               COMMAND                  SERVICE             CREATED             STATUS              PORTS
hello_world         nginx               "/docker-entrypoint.…"   web                 5 seconds ago       Up 3 seconds        0.0.0.0:8000->80/tcp, :::8000->80/tcp

If you visit http://localhost:8000 you’ll see the nginx welcome page there.

By using port mapping we can access containers in bridge networks from our local computer (Host machine).

Bridge network container -> Host network container

Now, let’s see what happens when we move our “hello_world” container (Nginx container) to a host network.

1. First, Let’s stop and remove the previous container by running docker compose down command.
2. Then, You have to update your docker-compose.yaml like in the below code snippet.

docker-compose.yaml

version: "3"
services:
  web:
    container_name: hello_world
    image: nginx
    network_mode: host

3. By setting network_mode to host you can add a container to the host network in docker compose.

4. Let’s build & run the “hello_world” container again by running docker compose up -d.

docker compose up -d

5. If you run the command docker compose ps now, You won’t see the port mapping there.

$ docker compose ps
NAME                IMAGE               COMMAND                  SERVICE             CREATED             STATUS              PORTS
hello_world         nginx               "/docker-entrypoint.…"   web                 31 seconds ago      Up 29 seconds       

6. But if you access the http://localhost/ or http://127.0.0.1/ in your computer, You’ll be able to access the same page that we’ve accessed before in port 8000.

In a Host Network, It allows a Docker container to share the network namespace with the host system like in the below figure. When a container is run in host network mode, it bypasses Docker’s virtual network interface and directly uses the network interfaces of the host machine. This means that the container will have the same network stack as the host, including the same IP address and network interfaces.

You might be new to the networking world, and if you don’t have a networking background, you might not know that port 80 is a well-known port for the HTTP protocol. In the networking realm, ports can be categorized into three main types based on their numbers.

1. Well-Known Ports (0–1023)
2. Registered Ports (1024–49151)
3. Dynamic or Private Ports (49152–65535)

Well-Known ports are standardized ports that have been assigned by the Internet Assigned Numbers Authority (IANA) to specific services and applications. As mentioned earlier, they include ports like 21 (FTP), 22 (SSH), 80 (HTTP), 443 (HTTPS), etc. They are commonly used for widely recognized protocols and services on the internet.

That’s why we didn’t have to enter it like http://localhost:80/ or http://127.0.0.1:80/ in the browser. If you do that, it will automatically redirects you to http://localhost/ or http://127.0.0.1/ respectively.

If you run the command docker network inspect host you’ll see that our “hello_world” container there.

$ docker network inspect host
[
    {
        "Name": "host",
        "Id": "f04d3c0691be5113708f0facee709111d1a34567287e0fd13811349ff67fd599",
        "Created": "2023-06-19T00:21:55.411437884+05:30",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "f071b8951047855ab9a3225455e0fd4d459e60a510f41ba921e7277ac886b41c": {
                "Name": "hello_world",
                "EndpointID": "0fc2921ec1503464a153459c470f3a35247a5cdbe323553f44bb1199f099c37f",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

That’s how the Docker Host Network Magic works!

If you like to take a further step and try to change the Nginx’s default welcome page, You can do it by using volumes.

1. RUN docker compose down to stop and remove the previous container.
2. UPDATE your docker-compose.yaml by adding a volumes: field like in the below code snippet.

docker-compose.yaml

version: "3"
services:
  web:
    container_name: hello_world
    image: nginx
    network_mode: host
    volumes:
      - ./src:/usr/share/nginx/html/

3. Then add a folder called src inside your project directory.

4. Add a index.html file inside it.

index.html

<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Hello, World!</title>
  </head>
  <body>
    <h1>Hello, World</h1>
  </body>
</html>

5. Now you should have a project directory like below.

.
├── src                # source directory. (.html/.js/.css)
│   ├── index.html          # html page 
└── docker-compose.yml

6. RUN the command docker compose up -d and see the changes in your browser.

Cheers!!! ✨

Conclusion

In conclusion, this article has focused on two fundamental Docker networking modes: the bridge network and the host network. By exploring the unique characteristics of each, we gained valuable insights into how containers communicate and connect within these network configurations. Whether you’re looking for isolated container communication using bridge networks or opting for direct host network integration, understanding these concepts is crucial in building efficient and secure containerized applications. I hope this article has provided you with the foundational knowledge to navigate Docker networking effectively and confidently.

Happy networking! 🚀

Resources:

GitHub Repository: https://github.com/kesaralive/docker-networking

Find me on LinkedIn: https://www.linkedin.com/in/kesaralive

It’s been a long time since I posted my last article about Docker. If you haven’t had the chance to read it yet, I would like to extend an invitation for you to explore it (From OS Hassles to Team Harmony: How Docker Saved Our Group Project). This article offers valuable insights into Docker and how it played a pivotal role in the successful containerization of our university group project.

Initially, my intention was to provide a tutorial outlining the process of containerizing our project. However, I have taken this opportunity to incorporate additional details that were not available to me at the time of doing my project. I hope you find this enhanced version enjoyable and informative as you navigate through it.

Prerequisites:

Before proceeding with this tutorial, please ensure that you have met the following prerequisites:

1. An active Internet connection: You will need a reliable Internet connection to access the necessary resources and documentation throughout the tutorial.
2. Docker installation: Make sure that Docker is installed on your machine. Docker enables the creation and management of containers, which is crucial for following along with the tutorial. You can follow the below instructions to install it.

Docker Installation

Please run the following commands in your terminal (for Linux and MacOS) or PowerShell (for windows) in order to check the presence of Docker installation on your computer.

$ docker version
$ docker compose version

Docker is a platform that allows you to create and manage containers, while Docker Compose is a tool specifically designed to simplify the management of multi-container applications.

Docker Compose comes with the docker installation. You don’t have to install it separately.

In case Docker is not installed on your computer, kindly refer to the instructions provided in the official Docker documentation for proper installation guidance. Docker Installation (Official Docker Documentation)

A Simple 3-Tier Architecture

Today, our objective is to build a basic 3-tier architecture using Docker containers. This architecture has three components.

1. A frontend container
2. A backend container (With an API layer)
3. A MySQL database container

Frontend App Container

To start our project, Let’s create a new folder to maintain it as our main project folder (You can name it as you want. Ex: docker-tutorial). Within the main folder, we’ll create another folder called frontend to handle our frontend application. After that we can add a simple index.html file inside the frontend folder. This will help us to check whether the frontend app is working or not once it is deployed.

index.html

<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>HTML Web Page</title>
  </head>
  <body>
    <h1>Hello, World!</h1>
  </body>
</html>

Next, we will create a new file named docker-compose.yml in our main project directory as our configuration file for deploying the frontend application within a Docker container.

docker-compose.yml

version: "3"
services:
  frontend:
    image: httpd:latest
    volumes:
      - "./frontend:/usr/local/apache2/htdocs"
    ports:
      - 3000:80

Let’s discuss a little bit about our docker-compose.yml configuration.

• version: “3” : This tells us the version of the Docker Compose file format.
• services : This section contains the services to be deployed. (In our case, frontend container)
• frontend : This is the name of the service. You can choose any name you prefer. Ex: frontend-service
• image: httpd:latest : This line tells the Docker image to be used for the frontend service. In our case, we use the httpd image with the latest tag. httpd image is the official image for the Apache HTTP Server. You can find more images in Docker Hub.
• volumes: : This section is used to mount the local filesystem directory (File directory in your computer) to the container.
• - “./frontend:/usr/local/apache2/htdocs” : This line specifies the volume mapping. It maps the ./frontend directory from the local filesystem to the /usr/local/apache2/htdocs directory inside the container. This allows the container to access the files in the frontend directory.
• ports: : This section defines the port mapping between the host (Your computer) and the container.
• - 3000:80 : This line maps port 3000 of the host machine (Your computer) to port 80 of the container. It means that when the container is running, you can access the frontend application on http://localhost:3000 in your web browser.

Main Project Directory

After all the settings, you should have a folder structure like this.

.
├── frontend                # Frontend files. (.html/.js/.css)
│   ├── index.html          # html page 
└── docker-compose.yml

What is docker-compose?

Docker compose is a tool that was developed to help define and share multi-container applications. docker-compose reads configuration data from a YAML file. You can learn more about docker-compose from here.

Next, Open up the terminal(Linux, Mac), or powershell(Windows) in your project directory, and run the command.

$ docker compose up

Note!

You need to have an Internet connection to perform above command for the first time.
First time setup will take some time because it has to pull the httpd container image from the docker hub.

At this point, you should be able to see your frontend application at http://localhost:3000/. We’ve used httpd docker container image to serve this simple frontend app.

You can press Ctrl + c in the terminal to stop all the containers.

What is Docker Hub?

Docker Hub is the world’s largest container image library and community for container images. It has over 100,000 container images from various software vendors. And It allows developers to store, manage, and distribute Docker container images.

What is a Docker Image?

A Docker image is a lightweight, standalone, and executable package that contains all the necessary components, including the code, runtime environment, system tools, libraries, and dependencies, required to run a specific application. It serves as a blueprint for creating Docker containers.

Backend API Container

Now Let’s create our backend api container,
To create the backend API container, we will follow these steps:

1. Create a new folder named api inside the main project directory.
2. Inside the api folder, create a new PHP file named index.php.
3. COPY & PASTE the below code snippet to your index.php file.

Then let’s add a simple todos api endpoint in our index.php file.

index.php

<?php
// Set the response content type to JSON
header('Content-Type: application/json');
header('Access-Control-Allow-Origin: http://localhost:3000');
header('Access-Control-Allow-Methods: GET, POST, OPTIONS');

// Create a response array
$response = [
    [
        "_id" => 1,
        "todo" => "Todo 1"
    ],
    [
        "_id" => 2,
        "todo" => "Todo 2"
    ],
    [
        "_id" => 3,
        "todo" => "Todo 3"
    ],
    [
        "_id" => 4,
        "todo" => "Todo 4"
    ]
];

// Encode the response array as JSON
$jsonResponse = json_encode($response);

// Output the JSON response
echo $jsonResponse;

Dockerfile

In this case, I’m not going to make the Backend API container using the approach we’ve used to create our frontend container. We used a pre-existing Docker image directly from the Docker Hub to create the frontend container.

Instead, we will create the Backend API container by writing the Docker Build process in a Dockerfile located in the main project directory.

What is a Dockerfile?

Dockerfiles are used in Docker to define the configuration and instructions necessary to build a Docker image. Dockerfiles provide a simple and declarative way to automate the creation of Docker images, ensuring consistency and reproducibility across different environments.

We’ll talk more about Dockerfiles in my upcoming articles. For now, Let’s add the below Dockerfile inside our main project directory by naming it Dockerfile and you don't need to have an file extension for that file.

Note!
If you’re a beginner to Docker, You don’t have to worry about Dockerfile implementation.
For the purpose of this tutorial, you can simply copy and paste the provided code, and it will be sufficient to follow along.

Dockerfile

FROM ubuntu:20.04

LABEL maintainer="kesaralive@gmail.com"
LABEL description="Apache / PHP development environment"

ARG DEBIAN_FRONTEND=newt
RUN apt-get update && apt-get install -y lsb-release && apt-get clean all
RUN apt install ca-certificates apt-transport-https software-properties-common -y
RUN add-apt-repository ppa:ondrej/php

RUN apt-get -y update && apt-get install -y \
apache2 \
php8.0 \
libapache2-mod-php8.0 \
php8.0-bcmath \
php8.0-gd \
php8.0-sqlite \
php8.0-mysql \
php8.0-curl \
php8.0-xml \
php8.0-mbstring \
php8.0-zip \
mcrypt \
nano

RUN apt-get install locales
RUN locale-gen fr_FR.UTF-8
RUN locale-gen en_US.UTF-8
RUN locale-gen de_DE.UTF-8

# config PHP
# we want a dev server which shows PHP errors
RUN sed -i -e 's/^error_reporting\s*=.*/error_reporting = E_ALL/' /etc/php/8.0/apache2/php.ini
RUN sed -i -e 's/^display_errors\s*=.*/display_errors = On/' /etc/php/8.0/apache2/php.ini
RUN sed -i -e 's/^zlib.output_compression\s*=.*/zlib.output_compression = Off/' /etc/php/8.0/apache2/php.ini

# to be able to use "nano" with shell on "docker exec -it [CONTAINER ID] bash"
ENV TERM xterm

# Apache conf
# allow .htaccess with RewriteEngine
RUN a2enmod rewrite
# to see live logs we do : docker logs -f [CONTAINER ID]
# without the following line we get "AH00558: apache2: Could not reliably determine the server's fully qualified domain name"
RUN echo "ServerName localhost" >> /etc/apache2/apache2.conf
# autorise .htaccess files
RUN sed -i '/<Directory \/var\/www\/>/,/<\/Directory>/ s/AllowOverride None/AllowOverride All/' /etc/apache2/apache2.conf

RUN chgrp -R www-data /var/www
RUN find /var/www -type d -exec chmod 775 {} +
RUN find /var/www -type f -exec chmod 664 {} +

EXPOSE 80

# start Apache2 on image start
CMD ["/usr/sbin/apache2ctl","-DFOREGROUND"]

After above change, your folder structure should look like this.

.
├── api                     # Backend api files. 
│   ├── index.php           # php page 
├── frontend                # Frontend files. (.html/.js/.css)
│   ├── index.html          # html page 
├── docker-compose.yml
└── Dockerfile

Adding the backend service to the docker-compose.yml

Now Let’s update the docker-compose.yml file with below code.

You should add the below code snippet under the services section in your docker-compose.yml file. If you have any difficulties, feel free to visit the Github repository to the finalized version. You can find the GitHub repository link and all other resource links in the end of this article.

  backend:
    container_name: simple-backend
    build:
      context: ./
      dockerfile: Dockerfile
    volumes:
      - "./api:/var/www/html/"
    ports:
      - 5000:80

• backend:: This is the name of the service. It can be any name you choose.
• container_name: simple-backend: This line sets the name of the container to be created as "simple-backend". You can use any desired name for your container.
• build:: This section indicates that we will be building the Docker image for the backend service.
• context: ./: It specifies the build context, which is the path to the directory containing the Dockerfile. Since we have our docker file in the main project directory it is set to the current directory (./).
• dockerfile: Dockerfile: This line specifies the filename of the Dockerfile to be used for building the image. In this case, it is named "Dockerfile".
• volumes:: This section is used to mount the local filesystem directory to the container.
• - "./api:/var/www/html/": This line maps the ./api directory from the local filesystem (Your computer) to the /var/www/html/ directory inside the container. (/var/www/html/ path is the default web root in apache web servers) It allows the container to access the files in the api directory.
• ports:: This section defines the port mapping between the host and the container.
• - 5000:80: This line maps port 5000 of the host machine to port 80 of the container. It means that when the container is running, you can access the backend API on http://localhost:5000 in your application.

After adding it, Let’s rerun the terminal command docker compose up to build and run our frontend and backend containers.

If everything works well, You should be able to see your simple todos API at http://localhost:5000/ .

Accessing the Backend from the Frontend

In order to call our backend API from frontend. We should allow access to our frontend in our backend code. We have already put it in the code. Below line helps frontend to call our simple todo api.

./api/index.php

header('Access-Control-Allow-Origin: http://localhost:3000');

Note!
If you’re trying to expose the frontend from a different port instead of port 3000. Please change the above line in below example.
Example: If you're trying to expose it in port 3001.
Update it as below.

header('Access-Control-Allow-Origin: http://localhost:3001');

Let’s update our frontend code as below to call the backend API

<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>HTML Web Page</title>
  </head>
  <body>
    <h1>Hello, World!</h1>
    <div id="todos"></div>

    <script>
      fetch("http://localhost:5000/")
        .then((response) => response.json())
        .then((data) => {
          const todos = document.getElementById("todos");

          data?.forEach((item, index) => {
            const todo = document.createElement("h2");
            todo.textContent = item._id + ". " + item.todo;
            todos.appendChild(todo);
          });
        })
        .catch((error) => {
          console.log("Error: ", error);
        });
    </script>
  </body>
</html>

Now you can check the result at http://localhost:3000/

Upto this point we’ve created our frontend container, backend container and called the backend API from the frontend. Next, Let’s see how we can add a database container to this.

Adding a MySQL Database container

First, Let’s add the sql dump file that I’ve created for this tutorial. Create a folder named db in your main project directory and dump.sql file inside it. Then pase the below code to the dump.sql file and save it.

dump.sql

-- phpMyAdmin SQL Dump
-- version 5.2.1
-- https://www.phpmyadmin.net/
--
-- Host: database:3306
-- Generation Time: Jul 08, 2023 at 05:46 PM
-- Server version: 8.0.33
-- PHP Version: 8.1.17

SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
START TRANSACTION;
SET time_zone = "+00:00";


/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8mb4 */;

--
-- Database: `todo_app`
--

-- --------------------------------------------------------

--
-- Table structure for table `todos`
--

CREATE TABLE `todos` (
  `_id` int NOT NULL,
  `todo` varchar(255) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

--
-- Dumping data for table `todos`
--

INSERT INTO `todos` (`_id`, `todo`) VALUES
(1, 'I will wake up at 4 in the morining.'),
(2, 'I will practice docker for 1 hour.'),
(3, 'I will give time for 2 hours javascript.'),
(4, 'Then I will have breakfast.'),
(5, 'I will give time for 3 hours php.');

--
-- Indexes for dumped tables
--

--
-- Indexes for table `todos`
--
ALTER TABLE `todos`
  ADD PRIMARY KEY (`_id`);

--
-- AUTO_INCREMENT for dumped tables
--

--
-- AUTO_INCREMENT for table `todos`
--
ALTER TABLE `todos`
  MODIFY `_id` int NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=6;
COMMIT;

After above change, your folder structure should look like this.

.
├── api                     # Backend api files. 
│   ├── index.php           # php page 
├── db                      # SQL dump files. 
│   ├── dump.sql            # sql dump file 
├── frontend                # Frontend files. (.html/.js/.css)
│   ├── index.html          # html page 
├── docker-compose.yml
└── Dockerfile

Let’s append below code snippet to our docker-compose.yml. You should append it under services section.

  database:
    image: mysql:latest
    environment:
      MYSQL_DATABASE: todo_app
      MYSQL_USER: todo_admin
      MYSQL_PASSWORD: password
      MYSQL_ALLOW_EMPTY_PASSWORD: 1
    volumes:
      - "./db:/docker-entrypoint-initdb.d"

• database:: This is the name of the service. You can choose any name you prefer.
• image: mysql:latest: This line specifies the Docker image to be used for the database service. In this case, it uses the mysql image with the latest tag. The mysql image is the official image for MySQL.
• environment:: This section allows you to specify environment variables for the container.
• MYSQL_DATABASE: todo_app: This line sets the name of the MySQL database inside the container as "todo_app". You can change it to any desired database name.
• MYSQL_USER: todo_admin: This line specifies the username for the MySQL user inside the container as "todo_admin". You can modify it to any preferred username.
• MYSQL_PASSWORD: password: This line sets the password for the MySQL user as "password". You can choose a different password for your MySQL user.
• MYSQL_ALLOW_EMPTY_PASSWORD: 1: This line allows an empty password for the MySQL user. Setting it to 1 enables this behavior.
• volumes:: This section is used to mount the local filesystem directory to the container.
• - "./db:/docker-entrypoint-initdb.d": This line maps the ./db directory from the local filesystem to the /docker-entrypoint-initdb.d directory inside the container. It is typically used for initializing the database with SQL scripts or other initialization files.

Now rerun the terminal command docker compose up. Then open a new terminal and type docker compose ps in your project directory. If you've done everything correctly, you should be able to see all three containers running on your docker server.

PhpMyAdmin to Access the MySQL Database Container

Next, Let’s add a browser interface to access your database. In this tutorial we use phpmyadmin to access our database. Let's add below code snippet to our docker-compose.yml file.

  phpmyadmin:
    image: phpmyadmin/phpmyadmin
    ports:
      - 8080:80
    environment:
      - PMA_HOST=database
      - PMA_PORT=3306
    depends_on:
      - database

Rerun the terminal command docker compose up

Now you can access the database container through http://localhost:8080/ by using credentials that we've given when creating our database container.

Wiring up the Backend API with Database container

To make it work you need to do some changes in your api folder.

1. Create a folder name app inside the api folder.
2. Add below 3 files inside that app folder.

config.php

<?php 
define("DB_HOST", "database");
define("DB_USERNAME", "todo_admin");
define("DB_PASSWORD", "password");
define("DB_NAME", "todo_app");

Database.php

<?php
class Database
{
    protected $connection = null;
    
    public function __construct()
    {
        try {
            $this->connection = new mysqli(DB_HOST, DB_USERNAME, DB_PASSWORD, DB_NAME);
            if (mysqli_connect_errno()) {
                throw new Exception("Database connection failed!");
            }
        } catch (Exception $e) {
            throw new Exception($e->getMessage());
        }
    }

    private function executeStatement($query = "", $params = [])
    {
        try {
            $stmt = $this->connection->prepare($query);
            if ($stmt === false) {
                throw new Exception("Statement preparation failure: " . $query);
            }
            if ($params) {
                $stmt->bind_param($params[0], $params[1]);
            }
            $stmt->execute();
            return $stmt;
        } catch (Exception $e) {
            throw new Exception($e->getMessage());
        }
    }

    public function select($query = "", $params = [])
    {
        try {
            $stmt = $this->executeStatement($query, $params);
            $result = $stmt->get_result()->fetch_all(MYSQLI_ASSOC);
            $stmt->close();
            return $result;
        } catch (Exception $e) {
            throw new Exception($e->getMessage());
        }
        return false;
    }
}

todos.php

<?php 
require_once "./app/Database.php";

class Todo extends Database
{
    public function getTodos($limit)
    {
        return $this->select("SELECT * FROM todos");
    }
}

To connect all together, let’s update our index.php by replacing with the below code snippet.

index.php

<?php
// Set the response content type to JSON
header('Content-Type: application/json');
header('Access-Control-Allow-Origin: http://localhost:3000');
header('Access-Control-Allow-Methods: GET, POST, OPTIONS');

require "./app/config.php";
require_once "./app/todos.php";

$todoModel = new Todo();
$response = $todoModel->getTodos(10);
// Create a response array

// Encode the response array as JSON
$jsonResponse = json_encode($response);

// Output the JSON response
echo $jsonResponse;

Now, execute the command docker compose up in your terminal to see the changes. This will enable you to see the successful retrieval of todos from the backend to the frontend, as well as the backend’s seamless interaction with the database for data retrieval.

Docker Compose Network

If you’ve followed the tutorial correctly, you should have a docker-compose.yml file like this.

version: "3"
services:
  frontend:
    image: httpd:latest
    volumes:
      - "./frontend:/usr/local/apache2/htdocs"
    ports:
      - 3000:80
  backend:
    container_name: simple-backend
    build:
      context: ./
      dockerfile: Dockerfile
    volumes:
      - "./api:/var/www/html/"
    ports:
      - 5000:80
  database:
    image: mysql:latest
    environment:
      MYSQL_DATABASE: todo_app
      MYSQL_USER: todo_admin
      MYSQL_PASSWORD: password
      MYSQL_ALLOW_EMPTY_PASSWORD: 1
    volumes:
      - "./db:/docker-entrypoint-initdb.d"
  phpmyadmin:
    image: phpmyadmin/phpmyadmin
    ports:
      - 8080:80
    environment:
      - PMA_HOST=database
      - PMA_PORT=3306
    depends_on:
      - database

Docker Compose understands the idea behind running services for one application on one network. When you deploy an app using Docker Compose file, even when there’s no mention of specific networking parameters, Docker Compose will create a new bridge network and deploy the container over that network.

Let’s discuss about docker networking in upcoming articles.

Summary

In this article, we covered several key concepts related to Docker and Docker Compose. We explored the fundamentals of Docker Compose and how it helps in managing multi-container applications. We discussed the role of Docker Hub as a centralized repository for Docker images. Additionally, we had a closer look into Dockerfiles, which allow us to define the configuration of a Docker image.

Furthermore, we examined how to add containers using Docker Compose, demonstrating how to define services within the docker-compose.yml file. We also explored how to establish connections between multiple containers in Docker Compose, enabling communication and interaction between them.

Lastly, we learned how to construct a simple 3-tier architecture using Docker Compose, incorporating frontend, backend API, and database containers. By leveraging the power of Docker Compose, we can easily orchestrate and manage the deployment of this architecture.

By gaining an understanding of these concepts and following the examples provided, you are now equipped to utilize Docker Compose effectively in building and deploying multi-container applications.

Resources

• Docker Introduction Article: https://medium.com/@kesaralive/from-os-hassles-to-team-harmony-how-docker-saved-our-group-project-e37c52ce8509
• Docker Installation Guide: https://docs.docker.com/engine/install/
• Docker Compose : https://docs.docker.com/compose/compose-file/compose-file-v3/
• GitHub Repository: https://github.com/kesaralive/docker-compose-101.git

First Let’s talk about the world before docker’s existence.

Back in the day, developers would create a system and send it to the testers to make sure everything worked perfectly. But there was a problem. The code they sent sometimes didn’t work properly on the tester’s computer because of differences in their computer setups.

It was really annoying! The developers would scratch their heads, trying to figure out why the code worked fine on their own computers but not on the tester’s. The testers had different operating systems, configurations, and software versions, and it caused issues with the code.

Why docker is the solution for this problem?

Docker is a tool which is used to automate the deployment of applications in lightweight containers so that applications can work efficiently in different environments. It’s an open-source platform that enables you to build, package, and run applications in containers.

Don’t you understand what I’ve said above? Don’t worry I’ll tell you my side of the story.

Once upon a time, I had a group project with my awesome university friends. We were four undergraduate students working together on this exciting project. The only catch was that each one of us had a different operating system on our laptops. My friend had a fancy Mac Book with mac-OS, two others had Windows, and I was rocking Ubuntu.

Our project was a web application that included a front-end, a restful API service, and a MySQL database. To set up everything on our laptops, we had to go through the hassle of installing MySQL and the Apache Web server separately, but in different ways for each operating system. You can imagine the headaches we faced due to these differences in our environments.

But then, something magical happened! I came across Docker, and I had an idea. I suggested to my friends that we all install Docker on our laptops. They were curious but willing to give it a shot. So, we went ahead and installed Docker on each of our machines.

With Docker, we were able to containerize our front-end, back-end (the amazing Restful API service), and even the MySQL database separately. It was like putting each part of our application into its own little box, called a container.

What is a container?

Container is a software package that consists of all the dependencies required to run an application. Think of a container as a lightweight, standalone virtual environment that contains everything an application needs to run, including its code, system libraries.

And guess what? We didn’t encounter any of those pesky problems we faced before! Docker made our lives so much easier because it allowed us to run our application in the exact same way, no matter what operating system we were using. It was like having a universal translator for our project.

No more worries about compatibility issues or struggling with different installation methods. Docker provided us with a smooth and consistent experience across all our laptops.

We were able to focus on building our web application together, knowing that it would work flawlessly on any machine. Docker truly saved the day and made our group project journey a whole lot smoother.

What I’ve learned using Docker?

The Major advantage of docker is its ability to ensure consistency across different environments. By packaging an application and its dependencies into a container, you can be confident that it will run the same way on any machine, regardless of the underlying operating system or infrastructure. Docker also allows for easy scalability, as you can run multiple containers simultaneously without any conflicts.

So, if you ever find yourself in a situation where you and your team have different operating systems, remember the magic of Docker. It can make your life so much easier, just like it did for us.

Let’s see how exactly docker helped me to solve the issue in the next article. Until then, Happy coding!

This might help you to learn something well. Some of my friends were asking the question from me.

“How did you learn these things?” and Here I’m going to answer it.

Their question was even if I were not doing much academic work. How do I learn new things? I was thinking about the question for a long time and, I realized something that I can share with you. I will explain how I’m managing it.

What you have to do is…

Let’s say you want to learn something new.

First , what you have to do is. Find a problem in the domain area that you want to learn. Or you can imagine a new problem and then try to solve it.

When you are trying to solve the problem, you may struggle with it. On the other hand, you may get the answer by the first try. If you’re getting it by few attempts. you may lucky or good at the relevant domain area. If it happens, try to find a different problem. A harder one.

In simple words, what I have to tell you is if you’re struggling to solve something. That means you’re learning it well. but to get things done. you have to keep trying to solve it no matter what. Struggle is the key.

You need to understand the learning is not about how fast you take the solution to your problem. It’s about struggling to do something and figuring it out. In this case, time doesn’t matter at all. The more time you take to solve the problem. The more you learn.

So far, we can say that struggle, learning and the time you take to solve the problem is equally proportional.

Let’s say if some is using this type of technique to learn things. Then they will be able to solve more problems in the same domain area. And also, they’ll be able to memorize things related to the domain faster than others. Since it’s an experienced-based learning technique. Your knowledge will last longer for decades.

Be honest to yourself.

Let’s say you’re a typical curriculum learner who’s learning things just to pass an exam. I bet you will forget a lot of things in months or years because you’re training your brain only for the exam. If you’re truly want to learn something you have to do it in a way like this. This is what I’m using to get it done.

Never lie to yourself.

When you’re at a problem always try to figure it on your own. If you need some help, you can read a book or simply you can find help on the internet. But make sure not to get the final answer directly from the reference.

Tree Example

Let’s imagine there’s an infected tree and we have to get rid of it. The tree is very large and it has so many branches and leaves. To get rid of this infection, you have to find a specific leaf in the tree.

First you have to start from the trunk and go to a branch by branch and search every leaf on it. And again, you may have to get down when you tired and try a different branch to find that specific leaf. Let’s assume the leaf you want to find won’t fall.

While you’re doing this you will get so much knowledge on this tree. If you took so much time to find the leaf. You will get to know about so many branches and its leaves. The more time you take to search it the more you learn.

Also, there’re more things to be considered as well.

When you’re doing this, you may take so much time. may be weeks, months or years. because this tree is huge. at that period, the tree can have new leaves and, also it may lose some older leaves.

It describes the changing of the world of knowledge. The world is changing faster and, new things are coming and old ones are going. So, you will have to study those things as well. Also, it describes that you can’t finish learning in a particular domain. You have to understand that as well.

If you’re thinking you don’t know everything about something. It’s okay to be like that. Because you can’t learn everything about anything. As a human being, our lifespan is smaller than what we’re trying to fulfill. but you can do many things by applying what you know.

As I said previously, let’s take the tree as the domain. The problem is the infection. The solution is one specific leaf and to get it. You have to go through many leaves and branches, which means you’re learning about the problem. In the end, you will gain a large amount of knowledge in the domain. You will solve your next in a lesser time.

To get the answer. There may be many paths, After finding it to get the solution. Try to find the best fit for it.

That’s how I’m managing to understand new things. Hope someone will get something from my words. Try to collect more trees. But I think If you can learn one tree well, it would be enough for your success than collecting more trees for your life.

If you can study one tree better, you will be a master of that tree.

Make a Forrest. wink!