K3S cluster arm64 on Oracle Cloud

Create a high availability K3S cluster using Oracle Cloud virtual machines in the always free plan.

RESUME

This text will guide those who want to create a K3S cluster (https://k3s.io/) with high availability taking advantage of Oracle Cloud virtual machines in the always free plan (https://www.oracle.com/cloud/free/#free-cloud-trial).

The instances will be arm64 VM.Standard.A1.Flex. Oracle Cloud offers up to 4 OCPUs, 24 GB of RAM and 200 GB of disk in the always free plan. In our example we will create a cluster with 3 VMs from different accounts, using the maximum of the CPU and memory that each account can use. Despite different accounts, all VMs will be in the same region, this is because we will use embedded etcd (https://docs.k3s.io/datastore/ha-embedded) so there is no additional cost with a external database.

Instances will be created with “Ubuntu 22.04”. To create VMs, the recommendation is to use the “terraform-oci-ampere-a1” project (https://github.com/AmpereComputing/terraform-oci-ampere-a1).

INSTANCES CREATION

In the README of the terraform-oci-ampere-a1 project there is a “Deploy to Oracle Cloud” button:

Already logged into your Oracle Cloud account, clicking the button will open the screen to start creating a VM:

Follow the steps filling the name and the description for the instance and the correct version name of the oci_os_image (“ubuntu2204”):

On the review screen, select the “Run apply” option and click the “Create” button:

Repeat the same operation on the 3 different accounts to create the 3 VMs.

K3S CLUSTER CREATION

Ports to be allowed between cluster nodes (https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes)

On OCI portal go to “Networking > Virtual cloud networks > Virtual Cloud Network Details” and edit the “vcn” of each VM by adding the inbound rules.

VPN WireGuard between master nodes

The master nodes, despite being in the same OCI region and having PING times of less than 1 ms, are on different accounts and therefore do not communicate within the same internal network. For this reason it is necessary to create a VPN between them and thus configure “embedded etcd” through the internal network of that VPN.

To install WireGuard on each node, run:

apt install wireguard
cd /etc/wireguard
wg genkey | sudo tee /etc/wireguard/private.key
cat /etc/wireguard/private.key | wg pubkey | tee /etc/wireguard/public.key

In addition to the ports mentioned above, also open port 51871/UDP between the master nodes and create /etc/wireguard/wg0.conf files on each of them and restart the WireGuard service.

The contents of the configuration files on each of the nodes will be:

Get Root Ex Machina’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Node01 /etc/wireguard/wg0.conf

[Interface]
Address = 192.168.0.1
ListenPort = 51871
PrivateKey = [PRIVATE_KEY_NODE01]
[Peer]
Endpoint = [PUBLIC_IP_NODE02]:51871
PublicKey = [PUBLIC_KEY_NODE02]
AllowedIPs = 192.168.0.2
[Peer]
Endpoint = [PUBLIC_IP_NODE03]:51871
PublicKey = [PRIVATE_KEY_NODE03]
AllowedIPs = 192.168.0.3

Node02 /etc/wireguard/wg0.conf

[Interface]
Address = 192.168.0.2
ListenPort = 51871
PrivateKey = [PRIVATE_KEY_NODE02]
[Peer]
Endpoint = [PUBLIC_IP_NODE01]:51871
PublicKey = [PUBLIC_KEY_NODE01]
AllowedIPs = 192.168.0.1
[Peer]
Endpoint = [PUBLIC_IP_NODE03]:51871
PublicKey = [PUBLIC_KEY_NODE03]
AllowedIPs = 192.168.0.3

Node03 /etc/wireguard/wg0.conf

[Interface]
Address = 192.168.0.3
ListenPort = 51871
PrivateKey = [PRIVATE_KEY_NODE03]
[Peer]
Endpoint = [PUBLIC_IP_NODE01]:51871
PublicKey = [PUBLIC_KEY_NODE01]
AllowedIPs = 192.168.0.1
[Peer]
Endpoint = [PUBLIC_IP_NODE02]:51871
PublicKey = [PUBLIC_KEY_NODE02]
AllowedIPs = 192.168.0.2

Don’t forget to change the variables [PUBLIC_IP_NODE*], [PRIVATE_KEY_NODE*] and [PUBLIC_KEY_NODE*] with the correct values for your environment. The internal network 192.168.0.0/24 used can also be changed to any other network of your choice.

K3S Installing

Start the cluster from Node01

curl -sfL https://get.k3s.io | K3S_TOKEN=[CLUSTER_TOKEN] sh -s - server \
 - cluster-init \
 - tls-san=[PUBLIC_IP_NODE01] \
 - advertise-address=[PUBLIC_IP_NODE01] \
 - bind-address 0.0.0.0 \
 - node-ip 192.168.0.1 \
 - node-external-ip [PUBLIC_IP_NODE01] \
 - disable traefik \
 - flannel-iface=[INTERNAL_INTERFACE_NODE01] \
 - flannel-backend=wireguard-native \
 - flannel-external-ip

Change the variables to match the values of your environment

[CLUSTER_TOKEN] can be created using the command “tr -dc A-Za-z0–9 </dev/urandom | head -c 64” and the same token must be used on all other nodes

[PUBLIC_IP_NODE*] public IP address of the node

[INTERNAL_INTERFACE_NODE*] name of the node’s internal interface e.g. `eth0`, `enp0s6`, etc

Add the remaining nodes to the cluster

Node02

curl -sfL https://get.k3s.io | K3S_TOKEN=[CLUSTER_TOKEN] sh -s - server 
 - server https://[PUBLIC_IP_NODE01]:6443 \
 - tls-san=[PUBLIC_IP_NODE02] \
 - advertise-address=[PUBLIC_IP_NODE02] \
 - bind-address 0.0.0.0 \
 - node-ip 192.168.0.2 \
 - node-external-ip [PUBLIC_IP_NODE02] \
 - disable traefik \
 - flannel-iface=[INTERNAL_INTERFACE_NODE02] \
 - flannel-backend=wireguard-native \
 - flannel-external-ip

Node03

curl -sfL https://get.k3s.io | K3S_TOKEN=[CLUSTER_TOKEN] sh -s - server 
 - server https://[PUBLIC_IP_NODE01]:6443 \
 - tls-san=[PUBLIC_IP_NODE03] \
 - advertise-address=[PUBLIC_IP_NODE03] \
 - bind-address 0.0.0.0 \
 - node-ip 192.168.0.3 \
 - node-external-ip [PUBLIC_IP_NODE03] \
 - disable traefik \
 - flannel-iface=[INTERNAL_INTERFACE_NODE03] \
 - flannel-backend=wireguard-native \
 - flannel-external-ip

ADDING WORKER NODES

curl -sfL https://get.k3s.io | K3S_TOKEN=[CLUSTER_TOKEN]g sh -s - agent 
 - server https://[PUBLIC_IP_NODE01]:6443

ROUND-ROBIN FOR THE MASTER NODES

It’s recommended to create type A entries in a DNS zone pointing the same hostname to the 3 public IP addresses of the master nodes (https://www.cloudflare.com/pt-br/learning/dns/glossary/round-robin-dns /). This way, in the worker nodes initialization command, [PUBLIC_IP_NODE01] can be replaced by the created hostname.

CLUSTER DIAGRAM

CONCLUSION

In the end we will have a K3S HA cluster with embedded etcd. The communication between the master nodes will occur within their exclusive VPN. Communication between worker nodes and the entire CNI (Container Network Interface) will occur within another distinct VPN created by K3S itself.