Along with examining the KubeVirt extensions, there are 2 or 3 other features that came out in 1.34 and 1.33 that I wanted to explore and understand how they are working overall, and this post will explore them in detail. In these examples im using Rancher k3s installed on top of a Ubuntu and for this post we will focus primarily on the KubeVirt.

All of the scripts, CRD examples and other materials that I outline in this post can be found in the github repo here.

• Note! in the examples below I am using a VM to run K8s that will run VMs — make sure you enable Nesting (VHV, VH-T, etc) on your hypervisor if you are trying this at home) or use baremetal nodes.
• However, if you dont have true virtualization on your setup, you can still play with KubeVirt and use Emulation Mode by Patching KubeVirt — see the details below!

What does this post Cover?

• How to install K3s K8s with KubeVirt on a Ubuntu Machine in under 2 minutes and form up a cluster of 1 Control plane and 2 worker nodes.
• How to deploy a simple VM (good ol’ Cirros image for those that remember the QEMU days) via KubeVirt on your K3s cluster.

There are other features outlined within the github repo that we will examine from K8s 1.33 and 1.34 and those will be covered in subsequent posts if this one goes well!

Getting Started — Our Environment

For this exploration, I created three VMs using ESXi, but you can use whatever hypervisor or Cloud Provider you prefer and it should work all the same. The four you see below is my “terminal” that we will run our kubectl and virtctl commands from overall.

The machines are all running Ubuntu 24 and have the following specs

• Control Node: 4 vCPUs, 6GB RAM, 200GB Disk, Single Nic
• Worker Node(s): 4CPUs, 3GB RAM, 200GB Disk, Single Nic

The machines are all operating on a single, flat vSwitch with a 192.168.1.0/24 subnet and router and DNS at 192.168.1.1. and our little setup looks like this. Its assumed for this you have internet connectivity from the machines that you stand up.

Setup SSH Keys Across the Machines

Once you have installed Ubuntu on your Machines, you should login to your Control node, generate a SSH key and copy that to the worker node /home/<your user>/.ssh/authorized_keys file so that password-less SSH between control plane and worker plane works. This is for the install.sh script that we will use for the setup.

Step 1

• Login to Control Node and Create an SSH key with ssh-keygen command. I used the defaults (press enter through), but you can configure as you like.

ssh-keygen -t rsa

Step 2

• Install the id_rsa.pub key created above on the worker nodes by using the SCP command (note: we can do this since we know there is no other keys installed on the workers, if you are doing this on existing node, be careful as this would wipe out things on those node)

scp ~/.ssh/id_rsa.pub <your user>@worker-XX:/home/<your user>/.ssh/authorized_keys

Repeat for both workers and test it that you can SSH to the workers from the control plane without a password as shown below. You will get prompted for key exchange on the first try.

again, make sure you test both nodes and its working, if ssh isnt working from control planes to workers, the install script will fail as its built on SSH keys being setup.

Install Rancher (k3s) K8s Cluster with KubeVirt

I have provided a simple install.sh script that support installing a Single Node or Multi Node K3s Cluster that can be found in the github repo here (install.sh) and is used to install the k8s cluster components on the machines.

The install.sh script uses a simple file called nodes.txt (or whatever you want to call it) that lists the nodes that will form the cluster.

The following Arguments are supported in the script:

• mode: Cluster or Single — Set to cluster if more than one node setup is desired
• node-name: Set the K8s Node Name (if blank, defaults to hostname)
• nodes-file: Path to the nodes list that defines the nodes in the cluster
• control-host: The IP address of the Control Plane Host (192.168.1.23 from our diagram above)

Install.sh Syntax Example

sudo ./install.sh --mode cluster --node-name cp-00 --nodes-file nodes.txt --control-host <control-ip>

Execute the following steps to setup your cluster

Step 1

• Login to your control plane machine and clone the git repo with

git clone https://github.com/dasmlab/k8s-kubevirt-134-primer.git

Example:

Step 2

• Change to the k8s-kubevirt-134-primer/setup/installer directory and examine the files there. You will see the install.sh, the README.md, and a nodelist.sample.txt file there (as shown below)
• Copy nodelist.sample txt to nodeslist.txt and edit it to reflect our cluster layout.

cd k8s-kubevirt-134-primer/setup/installer
cp nodelist.sample.txt nodelist.txt

• Edit your nodelist.txt to match the hostname’s of your machines. The example below uses the same hostnames for SSH, that is worker-00 and worker-01

Step 3

• Run the ./install.sh script to install the k8s cluster. This should be done from the control plane node.

sudo ./install.sh \
--mode cluster \
--node-name control-00 \
--nodes-file nodelist.txt \
--control-host 192.168.1.23

Example Output (start and end shown)

• Starting of script on control plane node:

• Ending of script example:

Step 4

• Use kubectl and check your cluster is up and nodes are present and ready.

kubectl get pods -A
kubectl get nodes

Example Output of our 3 node cluster (1 Control, 2 Workers)

Great! Our cluster is up and has KubeVirt installed and ready to go.

Step 5 — OPTIONAL / EMULATION MODE

If you dont have access to KVM or True Virtulaization, you can run the following command to patch your cluster and use emulation mode.

Run the following command to patch your cluster

    kubectl patch kubevirt kubevirt -n kubevirt -p '{"spec":{"configuration":{"developerConfiguration":{"useEmulation":true}}}}' --type=merge

Examine the KubeVirt CRD, Pods and Services.

With the installation of kubeVirt Extension into our cluster, we can see that the following CRDs are delivered within kubevirt.

We can also see the following KubeVirt Operators (pods/replica sets) that are running to act on any of the CRDs that we submit to the cluster. The layout is straightforward with a API Pod for requests, a controller, handler and the coordinating operator pods as shown below

We can see the following services are setup by default as well, including the support for a prometheus scraping point (we will explore what metrics are exposed by KubeVirt later in this post).

Deploy a Virtual Machine via K8s

We will use the following three Resource Definitions. A Namespace, a Datavolume — this is new! its like a CSI for VMs and our Virtual Machine itself.

Here are the three manifests we will use (also available here)

The Namespace is called kubevirt-lab

apiVersion: v1
kind: Namespace
metadata:
  name: kubevirt-lab
  labels:
    app.kubernetes.io/name: kubevirt-lab
    app.kubernetes.io/part-of: kubevirt-exploration

The DataVolume is called cirros-demo-dv and has a 2Gi size. Notice that this is where we provide the image URL via http (there are other sources that are supported in the spec as well).

I have left out “ volumeMode” from the DataVolume definition below, so that it will use the default from the StorageClass on the cluster. Options include “Block” and “Filesystem” depending on your backing Storage setup.

kind: DataVolume
metadata:
  name: cirros-demo-dv
  namespace: kubevirt-lab
spec:
  source:
    http:
      url: https://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi

The VirtualMachine is called cirros-vm and has 1 CPU Core and 256MB of RAM. We can see we attach our DataVolume from above via the containerdisk disk that provides the mapping to the volume. Very much like doing a CSI or other mount to a K8s pod!

You can see in this manifest, that we are defining Two (2) Disk Devices. The containerdisk, that will be provided via our DataVolume mount, and then a CloudInit disk, where we have a simple inline configuration. This will avoid the 20 iteration time out that the Cirros image has looking for CloudInit Metadata! :)

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: cirros-vm
  namespace: kubevirt-lab
  labels:
    app: cirros-demo
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/domain: cirros-vm
    spec:
      domain:
        cpu:
          cores: 1
        resources:
          requests:
            memory: 256Mi
        devices:
          disks:
            - name: cloudinitdisk
              disk:
                bus: virtio
              bootOrder: 2
            - name: containerdisk
              disk:
                bus: virtio
              bootOrder: 1
      volumes:
        - name: containerdisk
          dataVolume:
            name: cirros-demo-dv
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |
              #cloud-config
              datasource_list: [ None ]
              ssh_pwauth: False

Step 1

Apply the manifests to the cluster and observe the virtual machine is deployed (of course, you can do this in a single file and/or use ‘-k’ to process all the files as you prefer).

kubectl apply -f namespace.yaml
kubectl apply -f data-volume-demo.yaml
kubectl apply -f virtualmachine-cirrors.yaml

Step 2

Check that your Virtual Machine and DataVolume are deployed. You should see your VM is in a Stopped State. Use “kubectl get all -n kubevirt-lab” to see the output as shown below.

We can see that the VM is stopped and doing a “kubectl describe vm cirros-vm -n kubevirt-lab” shows us the message “VMI not started” in the Status section of the output as shown below

Step 3

Use the virtctl command to start your Virtual Machine. Note: either copy /etc/rancher/k3s/k3s.yaml to ~/.kube/config or set KUBECONFIG for this command to work

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
virtctl start cirros-vm -n kubevirt-lab

You should see the following output from your command. You can then check the status of the objects (DataVolume and VirtualMachine) in the namespace and see the changes that have occurred

You can see the DataVolume PROGRESS indicator as it creates the DataVolume with the Cirros Image. Behind the scenes you can also see that a PersistentVolumeClaim is in use, so very much like your standard K8s interfaces.

After a time your Virtual Machine will be provisioned and you can see that things are up and running.

Step 4

Connect to your Virtual Machine using virtctl and check that its running.

virtctl console cirros-vm -n kubevirt-lab

And there you have it, a Virtual Machine running on your K8s Worker that was created using a Declarative Method. You can login with cirros/gocubsgo

Step 5

You can see that the Virtual Machine as a IP address from within our Cluster IP space (10.42.1.0/24), we can use virtctl to connect to that IP over ssh from the Control Plane Node (or you can port forward through as well).

virtctl ssh cirros-vm -n kubevirt-lab -l cirros

Or you can go direct (if you are on a K8s Node) with ssh

# ssh cirros@10.42.1.74

What have we learned?

We have setup a simple 3-node K8s cluster with KubeVirt on it and then using the KubeVirt CRDs, we declared a Virtual Machine and a backing DataStore that contained our QCOW Cirrors image and validated that we were able to get a basic virtual machine to start and its networking stack to come up.

With those simple steps, you can now port virtually (tee hee) any VM into your K8s environment — of course, you should ensure you test in appropriate environments before moving production workloads! :)

Where to go next?

Of course, this is just the starting point, things like adding LoadBalancers to sets of VMs, how to do rolling updates and wiring up your Flux/ArgoCD/GitActions pipelines and start pushing out your VMs.

I hope you enjoyed this article and the examples, i think that you will find that the integration that KubeVirt provides makes deploying Virtual Machines via K8s pretty much like deploying anything else in K8s, in a declarative manner!

Till next time!

Daniel

Check out this and other stuff at https://dasmlab.org