A series of diary entries where I hack away at the Virtual Kubelets, what they stand for and what they don’t stand for and whether they can stand in the first place considering they are ghosts (you’ll figure out what I mean by that soon).

Scheduling Simulations

17th October 2024

I had a very interesting discussion with Anjul Sahu yesterday about the state of Cloud Computing and Kubernetes generally and he was gracious enough to catch me up on what the current state of GPU computing in India looks like.

Thanks for reading Vibhav’s Tech Diary! Subscribe for free to receive new posts and support my work.

After being a little out of touch with the Kubernetes ecosystem, it was nice to get this little crash course back into the system because of it. Today, after that discussion I have decided to learn more about Virtual Kubelets and try them out. I also plan on continue on hacking on something else later today, so I am going to keep it short.

The latest KubeCon video I could find on Virtual Kubelets is the one below.

Notes on SimKube

Why Simulation ?

Simulating `workload scheduling` locally gives us a peek into how the workloads might perform on actual clusters or help us debug through replaying simulations without having to run the workloads on real clusters.

We also talk about simulating the control plane components of the cluster here (not the applications).

SimKube Architecture

SimKube is the project that is being demoed in this talk and below is the high level architecture diagram for SimKube. The components marked in yellow are the 6 components discussed in this talk. SimKube is also written in Rust which is kind of amazing. Would definitely be nice to see more controllers being written in Rust ;)

User Interface to SimKube

• skctl (CLI) — Helps talk to your production cluster for exporting data to persistent storage, and also helps replay that same data on the simulated cluster.

SimKube Components in the Production Cluster (eg: Azure, AWS EKS, GKE etc.)

• sk-tracer (tracing service) — Runs in the production cluster. Collects data which is later stored in persistent storage.
• When we want to use the data being collected by the tracer we can call `skctl export`, which stores the data in some persistent storage from where we it can be used. This data would be used for creating simulations using `skctl run`. `skctl run` creates a Simulation CustomResource.

SimKube Components in Simulated Cluster (eg: kind, minikube etc.)SimKube Simulation Control Plane

• sk-ctrl (kubernetes controller) — listens for the Simulation CustomResource.
• To use the collected data from the production cluster in the simulated cluster, we will have to run `skctl run` which will Simulation CustomResource.
• sk-driver (part of Simulation) — created by sk-ctrl when a Simulation CustomResource is created on the cluster.
• installs a Mutating Admission Webhook and intercepts all the pods and moves them to the target sk-vnode.

SimKube Core Components (the heart of SimKube)

• sk-vnode (Virtual Kubelet based Virtual/Fake Node) — acts as a Kubernetes node and listens for workloads being created but doesn’t actually create it. sk-vnode is that it, only supposed to simulate/mock behavior of resources in the nodes.
• sk-cloudprov (Cloud Provider for Cluster Autoscaler) — the Cluster Autoscaler interfaces will all different cloud providers and sk-cloudprov is a cloud provider here in SimKube. sk-cloudprov scaled the nodes up and down if necessary for it’s simulations. Pod Deletion Cost is used to select nodes to terminate.

I was lucky enough to find drmorr aka David Morrison and his company’s blog here on Substack. if you like this tool and are intrigued by it and would like to hear more about the work Applied Computing Research Labs is doing, go ahead and subscribe to their blog below if you haven’t already.

Applied Computing Research Labs

Open-source research and development in distributed systems

By drmorr

At the end of the talk, David also spoke about possible future plans for the project, one of them stood out to me which was “intergrating with KWOK”. KWOK is Kubernetes WithOut Kubelet, and the point of the project is to be able to simulate nodes and clusters which falls inline with what SimKube is trying to do.

21st October 2024 (morning)

Ghosts of the Node 👻

It’s spooky how Virtual Kubelets work. If a kubelet is the soul of a node then a virtual kubelet is a kubelet without a node (or a real node). Let’s first look at a normal kubelet in an actual node to understand what they are.

Let’s check out a (real) Kubelet (for now)

Kubelet is a cute name. It sounds like it’s a “mini-me” version of a Kube(ernetes). But that’s definitely not the case. It’s a crucial part(s) of a Kubernetes cluster that manages the lifecycle of workloads on Nodes.

Let’s take a look at what the kubelet looks like. I am running Docker Desktop on my Mac and just going to create a kind cluster with the following command and peek into the container created for that cluster to see if I can find the kubelet service running.

kind create cluster

Let’s list the container running for the kind cluster.

docker container list

CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS         PORTS                       NAMES
a035b26721b6   kindest/node:v1.29.2   "/usr/local/bin/entr…"   5 minutes ago   Up 5 minutes   127.0.0.1:51437->6443/tcp   kind-control-plane

Let’s get into the container with the following command

docker exec -it a035b26721b6 bash

root@kind-control-plane:/#

And let’s see if we can find the kubelet service running here.

systemctl status kubelet

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf, 11-kind.conf
     Active: active (running) since Mon 2024-10-21 00:32:04 UTC; 14min ago
       Docs: http://kubernetes.io/docs/
    Process: 701 ExecStartPre=/bin/sh -euc if [ -f /sys/fs/cgroup/cgroup.controllers ]; then /kind/bin/create-kubelet-cgroup-v2.sh; fi (code=exited, status=0/SUCCESS)
    Process: 709 ExecStartPre=/bin/sh -euc if [ ! -f /sys/fs/cgroup/cgroup.controllers ] && [ ! -d /sys/fs/cgroup/systemd/kubelet ]; then mkdir -p /sys/fs/cgroup/systemd/kubelet; fi (code=exited, status=0/SUCCESS)
   Main PID: 710 (kubelet)
      Tasks: 20 (limit: 9310)
     Memory: 36.8M
        CPU: 21.179s
     CGroup: /kubelet.slice/kubelet.service
             └─710 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --node-ip=172.18.0.2 --node-labels= --pod-infra-container-image=registry.k8s.io/pause:3.9 --provider-id=kind://docker/kind/kind-control-plane --runtime-cgroups=/system.slice/containerd.service

and yes we do. Under the above output there are logs of the kubelet which I haven’t pasted here. But the logs pertain to any of the pods and volumes which are being managed by the kubelet here. Let’s look more into the kubelet logs.

journalctl -u kubelet | more

Just going through the logs here, noting down anything that stands out.

• Container Manager is created based on Node Config.
• There is a topology manager here.
• Node is synced with API Server.
• Relevant Kubernetes objects are queried from the API server.
• kubelet initialized iptables rules (I am saying this based on the log below, there seems to be contention online whether if the kubelet even has anything to do with the iptables, and some sources say it doesn’t but the logs might say one thing but the code says another.

kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4"

• After the resource management is setup, we move onto access control related setup with certificate setup and stuff where the node is being registered.
• from here on out the Topology Admit Handler shows up quite a bit along with other things which seem to be part of the reconciliation loop.

As we take a break here having understood that there is a lot which goes into resource allocation and workload creation by the kubelet including communication and syncing with the API Server.

Virtual Kubelets: The ghastly equivalent of a Kubelet

(Here, ghastly is being used to refer to virtual kubelets as being ghost like)

Now that we have some idea of what a kubelet is, let’s focus on what happens when a kubelet doesn’t have a node to live in exactly. It won’t have the resources to handle but it will still contain the interfaces without any endpoints. These interfaces can abstract node capabilities through Container Runtime Interface (CRI), Container Storage Interface (CSI) and Container Network Interface (CNI) without being tied to actual hardware.

Virtual Kubelet is focused on providing a library that you can consume in your project to build a custom Kubernetes node agent.

Using Virtual Kubelets to use VMs as a Provider

Now, I want to be able to use virtual Kubelets to connect a virtual machine to a kind cluster without having to deal with Kubernetes federation. This can be great for use cases where I don’t need to

21st October 2024 (afternoon)

For the use case I have for the virtual kubelet, seems like Interlink is the perfect tool for it and the Target Providers does include “Remote/on-demand virtual machines with any container runtime”. So let’s try this out.

(this discussion is continued in the next batch of diary entries/post …)

Thanks for reading Vibhav’s Tech Diary! Subscribe for free to receive new posts and support my work.

Originally published at https://vibhavstechdiary.substack.com.

TLDR; I started hacking on a Nextcloud Operator.

14th October 2024 (daytime)

So, I started hacking with the GlassKube Operator as I wanted an easy way to create and manage Nextcloud instances on Kubernetes and while searching for this an operator I found GlassKube. There isn’t a lot of documentation on how to run it with Mac. It’s either that or I didn’t search hard enough. Anyways, here’s how you can get start developing with it.

Prerequisites for GlassKube Operator

brew install openjdk gradle

After installing the developer dependencies and making sure you have things like kubectl etc. installed, which you will usually require to work with Kubernetes based tools, let’s go ahead and run the operator in development mode.

This operator is written in Java/Kotlin which gives me the heebie jeebies from back in the Jenkins days when I worked on the openshift-sync-plugin and the openshift-client-plugin for Jenkins, because this operator uses the fabric8io kubernetes client. It was fun working with it then but now it’s nice to see it again, being used in this format.

The preferred distro for development with glasskube is minikube so let’s start the minikube cluster

minikube start

Let’s clone the repo and understand how the build for GlassKube works and how we can get started working on this operator ASAP.

git clone https://github.com/glasskube/operator
cd operator

# Build
gradle
gradle check
gradle operator:installCrd

After this you can start the operator directly with the following command…

gradle operator:bootRun

but you will get cascading failures related the operator’s dependencies.

Let’s take a hacky approach and install the operator helm chart, scale the deployment of the operator to 0 and then run the above command again so that we can ensure the availability of runtime dependencies without having to iterating through these dependencies and installing all of them one by one which would take much more time.

# install via helm
helm repo add glasskube https://charts.glasskube.eu/
helm repo update

kubectl delete crds --all # run this because we installed all the crds in the last command and helm won't work if it cannot install the crds again

# install glasskube operator
helm install go glasskube/glasskube-operator

This above installs the glasskube-operator successfully. Let’s build an image and then push it to minikube using the following command.

gradle operator:loadImage

The above command will ensure that minikube has access to a newly built image from the source code. After running this command, I get the “docker.io/glasskube/operator:0.13.14-SNAPSHOT” image in the list of images accessible to my minikube.

minikube image ls

With the above command you can list these images for yourself.

Deploying Nextcloud using the Glasskube Operator

Let’s use the YAML below to create a Nextcloud instance.

apiVersion: glasskube.eu/v1alpha1
kind: Nextcloud
metadata:
  name: nextcloud
spec:
  host: nextcloud.mycompany.eu
  defaultPhoneRegion: DE
  apps:
    office:
      host: office.nextcloud.mycompany.eu

Create a yaml file with the above contents and call it nextcloud.yaml.

kubectl create -f nextcloud.yaml

I ran this and Nextcloud didn’t work as expected. Had a lot of errors in the operator logs. Gonna get onto debugging this tomorrow now.

Thanks for reading Vibhav’s Tech Diary! Subscribe for free to receive new posts and support my work.

15th October 2024 (extremely early morning, carry over from the last day)

The maintainer and author let me know that this operator is in maintenance mode as you can see below.

The AGPL 3.0 Dilemma

I decided to install GlassKube and see if there is a package which they provide in the current version of the package manager, but seems like they don’t support Nextcloud yet. It might be because of Nextcloud’s AGPL 3.0 license which doesn’t allow people to use Nextcloud as part of a service offering over the network. The newer Glasskube repo on the other hand has moved to Apache 2.0 license and don’t think they would want to support Nextcloud because of that. The Glasskube operator on the other hand is LGPL 3.0 License. AGPL has a stronger copyleft so the LGPL operator will be subject to it probably.

Stronger copyleft refers to a type of licensing that imposes strict requirements on derivative works based on the original licensed software.

Here, the strict requirements are given by AGPL.

I will do more justice to GlassKube in a future post, for now, I need a Nextcloud Operator.

nextcloud/operator

All of us have this one project idea (in my case, many) where you create a placeholder repository on Github to ensure that one day you might just finish that project you always wanted to start. This repository seems like a project like that, created 4 years, ago and waiting to get started.

nextcloud/helm

The Nextcloud Helm Chart seems to be in active maintenance.

A new Nextcloud Operator

If something already has a Helm chart which works extremely well, I would rather create an operator that wraps around that Helm chart and then at that point the Helm chart can be managed by the FluxCD Helm Controller. I have used this method before to create an operator where the target application that needed to be managed had a very comprehensive helm chart and it made sense to write an operator to manage the creation of these Helm charts. The operator here will be an abstraction layer over the creation and management of these charts. Any of the dependencies regarding storage, networking, CI/CD, backups etc. which nextcloud might have, will be managed by the operator itself.

Hacking together an operator

15th October 2024 (daytime)

Let’s start hacking on this. Committing everything to the repository below.

https://github.com/waveywaves/nextcloud-operator.git

Initialize the operator with Operator SDK after installation.

brew install operator-sdk

operator-sdk init --domain waveywaves.com --repo github.com/waveywaves/nextcloud-operator .

This will populate the project with the operator-sdk boilerplate.

Now, let’s add a NextCloud CustomResourceDefinition.

operator-sdk create api --group nc --version v1beta1 --kind Nextcloud

After running the above you should have a scaffold operator and a basic CRD created. Say yes to all options when

make install && make run

The above will run the operator and start listening for resources of kind Nextcloud.

kubectl create -f config/samples/nc_v1beta1_nextcloud.yaml
kubectl get nextcloud

NAME               AGE
nextcloud-sample   7s

Creating a Nextcloud instance through the FluxCD Helm Controller

We need the FluxCD source and helm controllers so that we can have it manage the Helm chart for us. We can later on figure out a way to make sure these are installed along side the operator in a helm chart.

flux install --namespace=flux-system --components="source-controller,helm-controller"

The above will install the necessary CRDs and controllers to which our NextCloud operator will delegate to. To be able to manage these CRDs through our operator let’s add these schemes to to the operator schemes. First add the dependencies below in the cmd/main.go.

fluxhelmv2beta1 "github.com/fluxcd/helm-controller/api/v2beta1"
fluxsourcev1beta2 "github.com/fluxcd/source-controller/api/v1beta2"

And add the code below in the init() function.

utilruntime.Must(fluxhelmv2beta1.AddToScheme(scheme))
utilruntime.Must(fluxsourcev1beta2.AddToScheme(scheme))

Once the above code is added, we can be sure of this operator being able to work well with FluxCD Custom Resource which we need. I am taking a break here as moving forward, we only have a basic integration job to ensure that the Helm Chart for Nextcloud is deployed for the operator once the user creates an instance of the Nextcloud Custom Resouce.

Thanks for reading Vibhav’s Tech Diary! This post is public so feel free to share it. Share

Nextcloud CRD → Nextcloud Helm chart creation

16th October 2024 (daytime)

It has been a nice morning and I need to prep for an interview so we are gonna try and get through this as quickly as possible.

Let’s listen to some Cory Wong while we get through this.

Whenever a Nextcloud Custom Resource is created we would like to create an adjacent HelmRelease which would create a (surprise surprise) a Helm release for Nextcloud. I have gone through a lot of programming blogs in the heyday of blogging so I know for a fact how hard it can be to read through some of these, so I am going to put the whole code below and in the comments you can read what I have done, and why I have done what I have had to do.

Let’s install a helm chart for Nextcloud locally just to be sure.

helm repo add nextcloud https://nextcloud.github.io/helm/
helm install nc nextcloud/nextcloud

I get an output from the above command and reduce it to the commands below to make this testing easier.

export APP_HOST=127.0.0.1
export APP_PASSWORD=$(kubectl get secret --namespace default nc-nextcloud -o jsonpath="{.data.nextcloud-password}" | base64 --decode)

helm upgrade nc nextcloud/nextcloud --set nextcloud.password=$APP_PASSWORD,nextcloud.host=$APP_HOST,service.type=ClusterIP,mariadb.enabled=false

Run the command below and go to https://127.0.0.1:8080 to login into your Nextcloud instance.

kubectl port-forward service/nc-nextcloud 8080:8080

username: admin
password: changeme

Use the username and password above to login.

We get the above page after logging in, this is where we are going to stop. We are going to use the same helm values we used in the last step to intialize Nextcloud.

helm list 

NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
nc      default         2               2024-10-16 12:41:42.116235 +0530 IST    deployed        nextcloud-6.1.0 30.0.0

We need to deploy the v6.1.0 chart through the operator.

Now, let’s make the following changes to the code. First, add a Status field. This will be in a nextcloud_types.go

// NextcloudStatus defines the observed state of Nextcloud
type NextcloudStatus struct {
	HelmReleaseName string `json:"helmReleaseName"`
}

Next, update the controller reconcile loop as I have done in this commit. I will be refactoring everything in a commit later. Not a big fan of editing while writing.

In one terminal instance run …

make run

In another one run …

kubectl create -f config/samples/nc_v1beta1_nextcloud.yaml

Find the Nextcloud Operator code here.

We are also going to set the LICENSE to AGPL 3.0 because Nextcloud is AGPL 3.0 and that extends to us as well.

Next Steps

Lifecycle

We need to make sure that the correct Conditions are set on creation of the Nextcloud instance.

Networking

I need to first make sure that the Nextcloud instance is easily accessible over a Network. Right now we are using ClusterIP, but ideally we should be able to assign a host to the Nextcloud instance and access it like a normal human being instead of port forwarding it.

There is a lot more we can do, but for next time we will focus only on the above two steps. Let me know in the comments if there are some specific next steps you would like to see.

Thanks for reading Vibhav’s Tech Diary! Subscribe for free to receive new posts and support my work. https://vibhavstechdiary.substack.com

Originally published at https://vibhavstechdiary.substack.com.

Introduction

What is an air-gapped environment ?

It’s possible you haven’t heard of air-gapped environments if you haven’t worked in enterprise software. Enterprises like running their workloads in controlled environments where the supply chain is monitored very closely to avoid any unnecessary side effects and vulnerabilites being introduced into the system due to unsecure network exposure.

Air-gapped systems are necessary security pre-requisite in enterprises but what are the tools we need which make a modern air-gapped system ? Getting data in and out of an air-gapped environment is already hard considering you have to transport the data to a server which has access to the system and then inject the data from there. This becomes even harder when we have to manage a complex distributed system like Kubernetes in an air-gapped environment.

Thanks for reading Vibhav’s Tech Diary! Subscribe for free to receive new posts and support my work. https://vibhavstechdiary.substack.com.

Air-gapping Kubernetes clusters from the get-go

In the context of Kubernetes, you will usually need access to a registry which will have all the necessary images you need to run your mission critical software.

• What if we didn’t want the Kubernetes cluster to have any access to a unsecure network or an external network in it’s lifetime ?
• What if we wanted the Kubernetes cluster to be self sufficient in an air-gapped environment from the get go ?

A swiss-knife for air-gapped software delivery

This is where Zarf comes in.

Zarf is a package manager for Kubernetes which specializes in software delivery to air-gapped environments.

Necessities of a air-gapped Kubernetes cluster

Here, we are going to look at critical components required to make an air-gapped environment self-sufficient. When we say self-sufficient, we mean that the environment should have a local artifact registry and a git server which will ensure that the said system is GitOps enabled as well.

We are going to also look at this environment in the context of Kubernetes and what it takes to have a Kubernetes clusters initialize and successfully work and be self-sufficient to a particular extent in an air-gapped environment.

Prerequisites

Necessary Background Knowledge

The reader will need to have an understanding of the following ideas/concepts/tools to be able to get through this article with minimal effort.

• Kubernetes (basic working knowledge)
• - ConfigMaps
• - Pods
• - Mutating Admission Webhooks
• Helm
• - Charts vs. Releases
• Basic working of an HTTP Server
• Container image internals
• Git Server

Setting up a test air-gapped Kubernetes cluster

Tools Used

• minikube / kind / k3d etc.
• zarf

Let’s go !

We first need a Kubernetes cluster which we can “air-gap” i.e. disconnect it from the internet (the most air-gapping we need right now to learn about software delivery on air-gapped Kubernetes clusters).

I am using a Mac so I am going to use Docker Desktop + minikube.

minikube start

😄  minikube v1.31.2 on Darwin 14.2.1 (arm64)
🎉  minikube 1.34.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.34.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

✨  Using the docker driver based on existing profile

💣  Exiting due to PROVIDER_DOCKER_NOT_RUNNING: "docker version --format <no value>-<no value>:<no value>" exit status 1: Cannot connect to the Docker daemon at unix:///Users/vibhavbobade/.docker/run/docker.sock. Is the docker daemon running?
💡  Suggestion: Start the Docker service
📘  Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/

And now minikube runs happily on my system.

Next we need to install Zarf.

brew tap defenseunicorns/tap && brew install zarf

Once installed, contemplate on the nature of security in this world and then remember that you have another train of thought to complete and then create a folder which will be a safe space for this technical exploration.

mkdir zarf-test/explore-init

Contemplate for a bit whether you are a ‘-’ person or a `_` person and move on. Abandon minikube and start using kind.

minikube delete
kind delete cluster && kind create cluster

Deleting cluster "kind" ...
ERROR: failed to delete cluster "kind": error listing nodes: failed to list nodes: command "docker ps -a --filter label=io.x-k8s.kind.cluster=kind --format '{{.Names}}'" failed with error: exit status 1
Command Output: Cannot connect to the Docker daemon at unix:///Users/vibhavbobade/.docker/run/docker.sock. Is the docker daemon running?

Realize you haven’t started Docker Desktop, restart it and rerun the kind cluster creation commands.

kind delete cluster && kind create cluster

Deleting cluster "kind" ...
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.29.2) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️  
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

Let’s initialize the air-gapped environment for usage. Before that we are going to check our zarf version.

zarf version

v0.41.0

Awesome.

Components required for an air-gapped environment

Learning about air-gapped environments through Zarf

Zarf makes it easy for us to dig deeper into deploying an air-gapped Kubernetes cluster and understanding

Initialization

Installing the Init Package

Now that we have Zarf installed, let’s initialize our kind cluster so that it will be a self reliant Kubernetes cluster who does not need help from the outside world when it comes to mission critical services.

zarf init

NOTE  Saving log file to
       /var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/zarf-2024-10-11-00-44-29-2167645169.log

It seems the init package could not be found locally, but can be pulled from
oci://ghcr.io/zarf-dev/packages/init:v0.41.0

 NOTE  Note: This will require an internet connection.
? Do you want to pull this init package? (y/N)

zarf is trying to pull a particular init package from oci://ghcr.io/zarf-dev/packages/init:v0.41.0. Let’s have zarf pull it by pressing `y` and enter and try to see what this init package is. You should see a progress bar like the one below as you go ahead with this.

? Do you want to pull this init package? Yes
    Pulling (63.19 MBs of 325.30 MBs) ███████████████████████████████████████████████████  19% | 12s

Once the package is pulled we will see the below output. For this specifically we needed the init package to be pulled but in an air-gapped environment, we can have this init package be pulled before hand so that it can be used without requiring the user to have an internet connection.

NOTE  Note: This will require an internet connection.                                                                                                                                                                
? Do you want to pull this init package? Yes                                                                                                                                                                          
  ✔  Pulling "oci://ghcr.io/zarf-dev/packages/init:v0.41.0" (325.30 MBs)                                                                                                                                              
  ✔  Validating full package checksums                                                                                                                                                                                
  ✔  Loading package from "/Users/vibhavbobade/.zarf-cache/zarf-init-arm64-v0.41.0.tar.zst"                                                                                                                           
  ✔  Loading package from "/Users/vibhavbobade/.zarf-cache/zarf-init-arm64-v0.41.0.tar.zst"

Understanding the Init Package

In the output we also have a Package Definition which we can go through.

📦 PACKAGE DEFINITION                                                                                                                                                                                               
                                                                                                                                                                                                             
kind: ZarfInitConfig                                                                                                                                                                                                  
                                                                                                                                                                                                                      
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                                                                                                            
metadata:  information about this package                                                                                                                                                                             
                                                                                                                                                                                                                      
  name: init                                                                                                                                                                                                          
  description: Used to establish a new Zarf cluster                                                                                                                                                                   
  version: v0.41.0                                                                                                                                                                                                    
  architecture: arm64                                                                                                                                                                                                 
  aggregateChecksum: a1e37c97567a64c8067e1c3886914bc040f5179c9b6703ce793272eddd80b360                                                                                                                                 
                                                                                                                                                                                                                      
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                                                                                                              
build:  info about the machine, zarf version, and user that created this package                                                                                                                                      
                                                                                                                                                                                                                      
  terminal: fv-az1501-989                                                                                                                                                                                             
  user: runner                                                                                                                                                                                                        
  architecture: arm64                                                                                                                                                                                                 
  timestamp: Thu, 03 Oct 2024 18:40:17 +0000                                                                                                                                                                          
  version: v0.41.0                                                                                                                                                                                                    
  migrations:                                                                                                                                                                                                         
  - scripts-to-actions                                                                                                                                                                                                
  - pluralize-set-variable
  lastNonBreakingVersion: v0.27.0

The first two sections of the Package Definition as we can see about are the metadata of the package and the build section which contains information about the build of this package. The package definition also contains different components that are present in the init package which would be installed. Let’s go component by component.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                                                                                                00:47:51 [293/710]
components:  components selected for this operation                                                                                                                                                                   

- name: k3s
  description: "*** REQUIRES ROOT (not sudo) *** Install K3s, a certified Kubernetes distribution built for IoT & Edge computing. K3s provides the cluster need for Zarf running in Appliance Mode as well as can host
 a low-resource Gitops Service if not using an existing Kubernetes platform."
  only:
    localOS: linux
  files:
  - source: packages/distros/k3s/common/zarf-clean-k3s.sh
    target: /opt/zarf/zarf-clean-k3s.sh
    executable: true
  - source: packages/distros/k3s/common/k3s.service
    target: /etc/systemd/system/k3s.service
    symlinks:
    - /etc/systemd/system/multi-user.target.wants/k3s.service
  - source: https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s2/k3s-arm64
    shasum: 1ae72ca06d3302f3e86ef92e6e8f84e14a084da69564e87d6e2e75f62e72388d
    target: /usr/sbin/k3s
    executable: true
    symlinks:
    - /usr/sbin/kubectl
    - /usr/sbin/ctr
    - /usr/sbin/crictl
  - source: https://github.com/k3s-io/k3s/releases/download/v1.28.4+k3s2/k3s-airgap-images-arm64.tar.zst
    shasum: 50621ae1391aec7fc66ca66a46a0e9fd48ce373a58073000efdc278233adc64b
    target: /var/lib/rancher/k3s/agent/images/k3s.tar.zst
  actions:
    onDeploy:
      before:
      - maxRetries: 0
        cmd: ./zarf internal is-valid-hostname
        description: Check if the current system has a, RFC1123 compliant hostname
      - cmd: "[ -e /etc/redhat-release ] && systemctl disable firewalld --now || echo ''"
        description: If running a RHEL variant, disable 'firewalld' per k3s docs
      - maxRetries: 0
        cmd: if [ "$(uname -m)" != "aarch64" ] && [ "$(uname -m)" != "arm64" ]; then echo "this package architecture is arm64, but the target system has a different architecture. These architectures must be the sam
e" && exit 1; fi
        description: Check that the host architecture matches the package architecture
      after:
      - cmd: systemctl daemon-reload
        description: Reload the system services
      - cmd: systemctl enable k3s
        description: Enable 'k3s' to run at system boot
      - cmd: systemctl restart k3s
        description: Start the 'k3s' system service
    onRemove:
      before:
      - cmd: /opt/zarf/zarf-clean-k3s.sh
        description: Remove 'k3s' from the system
      - cmd: rm /opt/zarf/zarf-clean-k3s.sh
        description: Remove the cleanup script

As we see above, the first component is actually K3s. From the description above, we understand that K3s provides the cluster need for Zarf running in Appliance Mode as well as can host a low-resource Gitops Service if not using an existing Kubernetes platform.

To understand Appliance Mode I had to go to this github issue where Jeff McCoy has described Appliance Mode perfectly.

Zarf deploys a K3s cluster with some pre-configured technologies in a simple, low-resource, declarative way we have been calling Appliance Mode.

This means that we didn’t have to run our own kind cluster and could have just run zarf in Applicance Mode for the purpose of this, but is that even true.

I searched the Zarf docs for Appliance Mode and got 2 hits. Both of them mention using this mode during CLI E2E tests. This is a good way to make development easy for zarf by building in the required components for testing, making it a developer friendly application as well.

Components required to initialize an air-gapped environment

Let’s move on to the next three components. There are a total of five.

- name: zarf-injector                                                                                                                                                                                                 
  description: |                                                                                                                                                                                                      
    Bootstraps a Kubernetes cluster by cloning a running pod in the cluster and hosting the registry image.                                                                                                           
    Removed and destroyed after the Zarf Registry is self-hosting the registry image.                                                                                                                                 
  required: true                                                                                                                                                                                                      
  files:                                                                                                                                                                                                              
  - source: https://zarf-init.s3.us-east-2.amazonaws.com/injector/2024-07-22/zarf-injector-arm64                                                                                                                      
    shasum: b905e647e0d7876cfd5b665632cfc43ad919dc60408f7236c5b541c53277b503                                                                                                                                          
    target: "###ZARF_TEMP###/zarf-injector"                                                                                                                                                                           
    executable: true

- name: zarf-seed-registry
  description: |
    Deploys the Zarf Registry using the registry image provided by the Zarf Injector.
  required: true
  charts:
  - name: docker-registry
    version: 1.0.0
    localPath: packages/zarf-registry/chart
    namespace: zarf
    releaseName: zarf-docker-registry
    valuesFiles:
    - packages/zarf-registry/registry-values.yaml
    - packages/zarf-registry/registry-values-seed.yaml
  images:
  - library/registry:2.8.3

- name: zarf-registry                                                                                                                                                                                                 
  description: |                                                                                                                                                                                                      
    Updates the Zarf Registry to use the self-hosted registry image.                                                                                                                                                  
    Serves as the primary docker registry for the cluster.                                                                                                                                                            
  required: true                                                                                                                                                                                                      
  manifests:                                                                                                                                                                                                          
  - name: registry-connect                                                                                                                                                                                            
    namespace: zarf                                                                                                                                                                                                   
    files:                                                                                                                                                                                                            
    - packages/zarf-registry/connect.yaml                                                                                                                                                                             
  - name: kep-1755-registry-annotation                                                                                                                                                                                
    namespace: zarf                                                                                                                                                                                                   
    files:                                                                                                                                                                                                            
    - packages/zarf-registry/configmap.yaml                                                                                                                                                                           
  charts:                                                                                                                                                                                                             
  - name: docker-registry                                                                                                                                                                                             
    version: 1.0.0                                                                                                                                                                                                    
    localPath: packages/zarf-registry/chart                                                                                                                                                                           
    namespace: zarf                                                                                                                                                                                                   
    releaseName: zarf-docker-registry                                                                                                                                                                                 
    valuesFiles:                                                                                                                                                                                                      
    - packages/zarf-registry/registry-values.yaml                                                                                                                                                                     
  images:                                                                                                                                                                                                             
  - library/registry:2.8.3                                                                                                                                                                                            
  actions:                                                                                                                                                                                                            
    onDeploy:                                                                                                                                                                                                         
      after:                                                                                                                                                                                                          
      - wait:                                                                                                                                                                                                         
          cluster:                                                                                                                                                                                                    
            kind: deployment                                                                                                                                                                                          
            name: app=docker-registry                                                                                                                                                                                 
            namespace: zarf                                                                                                                                                                                           
            condition: Available

You can read the Core Components documentation, but we are trying to read the docs and trying to read these package manifests to understand this more intuitively. So, hang on.

Zarf Injector, Seed Registry and Main Registry : Injecting a Container Registry and the brilliance of a Seed Registry with Zarf

In a Kubernetes cluster you cannot just inject an image into the cluster. First, the cluster needs to have a registry and if you want to run a registry in a Kubernetes cluster you are going to need to have an image already available in a nearby registry to be able to do that.

The Chicken and Egg Problem: I need a registry in the cluster, but I can’t create a registry in the cluster because the cluster doesn’t have a registry which contains an image for the registry.

In our scenario where we have no access to the internet and the Kubernetes cluster is airgapped, you cannot get access to the internet and pull the registry image with docker and then run the image in a Pod on the cluster. Here we have two problems :-

• No registry running in the cluster
• No image available to be be pulled from a registry

A classic chicken and egg problem. But what if the image for the registry didn’t have to be an image.

All images are datablobs and can be represented as tarballs and can be moved around as we please. All kubernetes clusters also have ConfigMap support regardless of whether they are EKS, AKS, K3S or Vanila Kubernetes. We could divide this registry image up between multiple configmaps and install it into the cluster. But once installed these configmaps need to be reassembled. This is where the zarf-injector comes into the picture.

The below is from the documentation itself :-

The zarf-injector binary serves 2 purposes during 'init'.

1. It re-assembles a multi-part tarball that was split into multiple ConfigMap entries (located at ./zarf-payload-*) back into payload.tar.gz, then extracts it to the /zarf-seed directory. It also checks that the SHA256 hash of the re-assembled tarball matches the first (and only) argument provided to the binary.
2. It runs a pull-only, insecure, HTTP OCI compliant registry server on port 5000 that serves the contents of the /zarf-seed directory (which is of the OCI layout format).

The zarf-injector binary’s job is to re assemble the registry’s image basically. But, once the zarf-injector and the registry’s images as ConfigMaps are already present in the cluster what happens now ? We have a total of 11 relevant ConfigMaps here now. 10 for each MB of the registry image and 1 for the zarf-injector binary.

The zarf-injector component unlike the other components is just an executable file and nothing more. It is hence of type Files and is copied from source to target where the source is a AWS S3 bucket and target is ###ZARF_TEMP###/zarf-injector. From this comment we understand that

###ZARF_TEMP### is also a special internal variable that can only be used with file targets and nothing else...

Zarf searches for an image which is already running onto the cluster and uses it as a base from where it can run the zarf-injector binary.

This is where the injection Pod build executes, and this is what the build looks like. As you can see, this piece of code which is run by the CLI generates a Pod manifest which is then created onto the cluster. The zarf-injector binary here is running in an unknown environment and this is what it was created for. The injection here is complete. The injector binary also executes the assembled registry image as you can see here. Cool.

We basically have fragmented binaries which were brought into a Kubernetes Cluster via ConfigMaps. A Pod created to run one of the ConfigMaps as a binary which assembled the other ConfigMaps into a tarball and then unpacked again to get the files that were part of the tarball, and then we start the Axum http server which serves the a docker registry containing the registry image we just assembled.

If you notice the seed-registry and the registry components above, you can see that the only difference in the value files is the seed-registry having an extra values file provided.

image:
  repository: "###ZARF_SEED_REGISTRY###/###ZARF_CONST_REGISTRY_IMAGE###"
  tag: "###ZARF_CONST_REGISTRY_IMAGE_TAG###"

###ZARF_SEED_REGISTRY### is temporarily set to the location of the injector which is serving the seed image in this release. This is then overriden when the zrf-registry component executes and the value is set to ###ZARF_REGISTRY### as you can see here. Once this component runs and executes it’s marked as completed, after making changes to the newly deployed registry chart. At this point the cluster would be using this registry as the default registry and with that the cluster has been initialized with a docker-registry.

Ensuring automatic resource aliasing to avoid operational overhead

Now that the cluster has been initialized with it’s own docker-registry we can run services inside the cluster which are necessary for us to run our GitOps applications and that brings us onto our next component. The Zarf Agent.

- name: zarf-agent                                                                                                                                                                                                    
  description: |                                                                                                                                                                                                      
    A Kubernetes mutating webhook to enable automated URL rewriting for container                                                                                                                                     
    images and git repository references in Kubernetes manifests. This prevents                                                                                                                                       
    the need to manually update URLs from their original sources to the Zarf-managed                                                                                                                                  
    docker registry and git server.                                                                                                                                                                                   
  required: true                                                                                                                                                                                                      
  manifests:                                                                                                                                                                                                          
  - name: zarf-agent                                                                                                                                                                                                  
    namespace: zarf                                                                                                                                                                                                   
    files:                                                                                                                                                                                                            
    - packages/zarf-agent/manifests/service.yaml                                                                                                                                                                      
    - packages/zarf-agent/manifests/secret.yaml                                                                                                                                                                       
    - packages/zarf-agent/manifests/deployment.yaml                                                                                                                                                                   
    - packages/zarf-agent/manifests/webhook.yaml                                                                                                                                                                      
    - packages/zarf-agent/manifests/role.yaml                                                                                                                                                                         
    - packages/zarf-agent/manifests/rolebinding.yaml                                                                                                                                                                  
    - packages/zarf-agent/manifests/clusterrole.yaml                                                                                                                                                                  
    - packages/zarf-agent/manifests/clusterrolebinding.yaml                                                                                                                                                           
    - packages/zarf-agent/manifests/serviceaccount.yaml                                                                                                                                                               
  images:                                                                                                                                                                                                             
  - ghcr.io/zarf-dev/zarf/agent:v0.41.0                                                                                                                                                                               
  actions:                                                                                                                                                                                                            
    onCreate:                                                                                                                                                                                                         
      before:                                                                                                                                                                                                         
      - dir: .                                                                                                                                                                                                        
        cmd: test "v0.41.0" != "local" || make build-local-agent-image AGENT_IMAGE_TAG="v0.41.0" ARCH="arm64"                                                                                                         
        shell:                                                                                                                                                                                                        
          windows: pwsh                                                                                                                                                                                               
        description: Build the local agent image (if 'AGENT_IMAGE_TAG' was specified as 'local')                                                                                                                      
    onDeploy:                                                                                                                                                                                                         
      after:                                                                                                                                                                                                          
      - wait:                                                                                                                                                                                                         
          cluster:                                                                                                                                                                                                    
            kind: pod                                                                                                                                                                                                 
            name: app=agent-hook                                                                                                                                                                                      
            namespace: zarf                                                                                                                                                                                           
            condition: Ready

This component is a Mutating Admission Webhook and it’s job is to intercept any workload creation request to see if the images which are being requested have an alternate image in the internal registry. This way the user doesn’t have to update the name of the image everytime they are deploying an application on Zarf.

The zarf-agent does the same internal renaming of git servers as it does for images. With that let’s move onto the final component which is the git-server.

A reliable Git server for all the GitOps needs

- name: git-server                                                                                                                                                                                                    
  description: |                                                                                                                                                                                                      
    Deploys Gitea to provide git repositories for Kubernetes configurations.                                                                                                                                          
    Required for GitOps deployments if no other git server is available.                                                                                                                                              
  manifests:
  - name: git-connect
    namespace: zarf
    files:
    - packages/gitea/connect.yaml
  charts:
  - name: gitea
    version: 10.1.1
    url: https://dl.gitea.io/charts
    namespace: zarf
    releaseName: zarf-gitea
    valuesFiles:
    - packages/gitea/gitea-values.yaml
  images:
  - gitea/gitea:1.21.5-rootless
  actions:
    onDeploy:
      before:
      - mute: true
        cmd: ./zarf internal update-gitea-pvc --no-progress
        setVariables:
        - name: GIT_SERVER_CREATE_PVC
      after:
      - wait:
          cluster:
            kind: pod
            name: app=gitea
            namespace: zarf
            condition: Ready
      - maxTotalSeconds: 60
        maxRetries: 3
        cmd: ./zarf internal create-read-only-gitea-user --no-progress
        description: Create the read-only Gitea user
      - maxTotalSeconds: 60
        maxRetries: 3
        cmd: ./zarf internal create-artifact-registry-token --no-progress
        description: Create an artifact registry token
      onFailure:
      - cmd: ./zarf internal update-gitea-pvc --rollback --no-progress

Here, zarf installs a git server backed by Gitea which brings GitOps capabilities to this air-gapped Kubernetes clusters. Users can now use GitOps enabled tools like ArgoCD on an air-gapped cluster like this out of the box.

Now that we have understood what each of these components are, let’s move past the constants and variables for now and initialize our Zarf enabled Air-Gapped Kubernetes Cluster.

...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Software Bill of Materials   an inventory of all software contained in this package

 NOTE  This package has 5 artifacts to be reviewed. These are available in a temporary
       'zarf-sbom' folder in this directory and will be removed upon deployment.

- View SBOMs now by navigating to the 'zarf-sbom' folder or copying this link into a browser:
/Users/vibhavbobade/go/src/github.com/waveywaves/zarf-test/zarf-sbom/sbom-viewer-docker.io_gitea_gitea_1.21.5-rootless.html

- View SBOMs after deployment with this command:
$ zarf package inspect /Users/vibhavbobade/.zarf-cache/zarf-init-arm64-v0.41.0.tar.zst

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

? Deploy this Zarf package? (y/N)¸

I was asked if I want to install the git server compoent after saying yes above. I obliged and said yes please.

✔  Zarf deployment complete

     Application          | Username           | Password                                 | Connect               | Get-Creds Key
     Registry             | zarf-push          | P3TgKAGniPUajPzMzcEJ58Jt                 | zarf connect registry | registry
     Registry (read-only) | zarf-pull          | sz4qG~Pe-9eGxDGwJcnySzjD                 | zarf connect registry | registry-readonly
     Git                  | zarf-git-user      | EhtJClcvRzq2jggrlw~dEsgA                 | zarf connect git      | git
     Git (read-only)      | zarf-git-read-user | 0j7Y8zE1S8svuV7ZTpkBV7pr                 | zarf connect git      | git-readonly
     Artifact Token       | zarf-git-user      | ea148bb0776c14424d578703018720805e156c34 | zarf connect git      | artifact

This is the final part of the output after running zarf init and smashing the “y” button over and over again.

Now to test some of these fun connect commands

zarf connect registry

 NOTE  Using config file /Users/vibhavbobade/go/src/github.com/defenseunicorns/zarf/zarf-config.toml

 NOTE  Saving log file to
       /var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/zarf-2024-10-11-03-45-17-208292671.log
  ⠹  Tunnel established at http://127.0.0.1:51632/v2/_catalog, opening your default web browser (ctrl-c to end)

Cool.

The agent, and the gitea images were pushed into the registry after deployed registry from the injector hosted seed-registry.

And these are the components required to run Kubernetes in an air-gapped environment and also have it be GitOps enabled.

Originally published at https://vibhavstechdiary.substack.com.

I stumbled upon XDG while working on an issue for a CLI tool in the past. It required a background on what XDG is and it’s specification. With a little bit of “googling” I was able to find the specification but I was still a bit confused as to what the hell XDG is and was frustrated that there is no concise article for it.

Somewhere, someone had said, “If there is something that you want to read and it doesn’t exist, write it yourself.” So here goes.

What is XDG ?

Quoting from the freedesktop.org wikipedia page,

freedesktop.org was formerly known as the X Desktop Group, and the abbreviation “XDG” remains common in their work.

Now we know that XDG is just another name for the freedesktop.org. But what is freedesktop.org ?

freedesktop.org hosts the development of free and open source software, focused on interoperability and shared technology for open-source graphical and desktop systems. We do not ourselves produce a desktop, but we aim to help others to do so.

— https://www.freedesktop.org/wiki/

So, it is a collaboration zone for “interoperability and shared technology for open-source graphical and desktop systems”. Some of the more popular projects hosted by XDG are Wayland, PulseAudio and systemd.

The Portland Project

Apart from sounding like a great name for a highschool garage band, The Portland Project was an initiative aimed at portable application software between desktop environments and kernels and hence strengthen the appeal of Linux as a Desktop Environment and not just for basement dwellers like some of us who refuse to deal with the capitalist technocrat folks of the Operating Systems world.

xdg-utils is the more well known name for the project and it is basically

a set of tools that allows applications to easily integrate with the desktop environment of the user, regardless of the specific desktop environment that the user runs.

Relevant Specifications by XDG

These specifications are important as they form the base for tools in xdg-utils and are a staple of Linux distributions today.

Some specifications one can start with are the Base Directory and the Desktop Entry Specifications. We are only covering the Base Directory Specification as it helps a ton to understand it while creating consistent applications across Linux Environments (great to know while building Command Line Interfaces for Linux).

XDG Base Directory Specification

Defines the Configuration for Base Directories and the defaulted location of files based on their usage so that not everything is stored in the same place by default.

This specification outlines that the user-specific data, the configuration files and cache files for a particular application need to be segregated and each of their locations need to be defined by an environment variable.

User Specific Data should be in $XDG_DATA_HOME (default to $HOME/.local/share)

Configuration files should be in $XDG_CONFIG_HOME (default to $HOME/.config)

Cache files in $XDG_CACHE_HOME (default $HOME/.cache)

Then there are other environment variables which are more of aggregators which help for searching the locations in different places.

Multiple locations for Configuration files given in $XDG_CONFIG_DIRS (includes $XDG_CONFIG_HOME)

Multiple locations for Data files $XDG_DATA_DIRS (includes $XDG_DATA_HOME)

References

Base Directory Specification https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

https://wiki.archlinux.org/index.php/XDG_Base_Directory

https://askubuntu.com/tags/xdg-open/info

Operator managed Jenkins on Kubernetes

Introduction

Jenkins has been the CI/CD tool used by most companies out there big and small. Before there was DevOps, there was Jenkins and Jenkins ushered the DevOps revolution by helping abstract and automate processes which were too cumbersome to be done manually all the time.

Administration of Jenkins has thus become a real responsibility and with the increasing infrastructure needs of individuals, managing Jenkins can become tough and at times frustrating. While we understand this, Jenkins itself has tried to keep up with the times by being extensible with a massive plugin support and increasing improvements in performance especially with the onset of increased need for containerization. With that in mind, The Jenkins Operator was created to help better manage and automate the Jenkins lifecycle in Kubernetes.

We’ve come a long way

Since the last blog post on Jenkins Operator 0.4.0, we have been working on the Operator here with having a better code quality, architecture and user experience in mind. We have moved the Jenkins Operator from its previously monolith design to a more distributed design, which allows for increased performance and better code readability which makes it easier for contributors in the open source community to contribute.

Working with the Jenkins Operator on Openshift

The Jenkins Operator has been released as a Tech preview on Openshift now which can be consumed by users. In the following examples we will look at how to use the Jenkins Operator on Openshift.

Deploying the Jenkins Operator

Login to the Openshift Console and create a new project.

Install the Operator in the newly created project

Once the Operator has successfully installed please follow the documentation here to perform any of the following tasks.

Building a custom Jenkins Image

The Jenkins Operator provides the ability for the users to create their own Jenkins images which can be used in a reproducible way without having to install their plugins at runtime. This sees to it that the user has a definitive workflow and a good idea of what plugins they need to use and keep documented to have their Jenkins run in a stable manner in Kubernetes.

To be able to use ConfigurationAsCode for Jenkins you would also have to include the plugin in the Jenkins Image you are using.

You can follow the documentation provided here to create your Jenkins Image via the Operator and push it to your repository.

Configuring the Jenkins instance

As you have your Jenkins image ready , you would need to deploy it with the right configuration which fits your needs. You can follow the documentation to understand how to enable configuration as code for your jenkins instance and follow the steps to move your CasC configs from your legacy Jenkins instances to Kubernetes.

If you would like to use Backup and Restore for your Jenkins instances you would also need to enable it as given here.

Deploying Jenkins

Once you have completed the configuration you can deploy the Jenkins by creating an instance of the Jenkins Custom Resource which has been configured above.

After creating the Jenkins instance with Backup enabled you can see that the default BackupConfig as well as the example Jenkins is up.

Backup and Restore for our Jenkins instance

As we now have enabled Backup for this Jenkins instance, we can now use Backup and Restore operations using the respective Custom Resources to secure important data you don’t want to lose.

You can follow the documentation given here to understand how to configure and use the Backup and Restore operations.

Running the Jenkins Operator locally

You can run the Jenkins Operator locally for development purposes. The documentation for the same is provided.

This would be very good for running locally and testing the Operator before deployment.

What’s Next ?

With the current update we have added features and updated the architecture and as we move forward we would be working on improving it further. Our focus currently is for the user to be able to automate their Jenkins lifecycle. So we would be working on fixes and features to make the Operator more robust and have a better automation lifecycle.

We would be focusing on the following:

• BackupSchedule for scheduling backups at regular intervals
• Persistence Support for Jenkins and Backups

Jenkins in the past has been provided as a first class Build option alongside Dockerfile build and s2i build in Openshift. Currently, Jenkins is well integrated in the Openshift ecosystem and with the movement of everything towards Operators, it is only natural that Jenkins make the move as well.

We will go through the current Jenkins Operator specification and how you can migrate from your earlier workflow for Jenkins on Openshift to the Jenkins Operator with minimal changes to your current infrastructure.

What is the Jenkins Operator ?

The Jenkins Operator is a Kubernetes Operator which aims to carry out Day 2 operational tasks with minimal or no intervention allowing for automation of operational tasks which can be repetetive at times. Operations like SeedJobs creation and Backup management are aimed at being automated in this case.

The Jenkins Operator has initially been developed by VirtusLab and is now being adopted by us folks working at Red Hat on Openshift. With it’s strong backing from the community as well as it being a part of the JenkinsCI organization on Github, there lies a lot of potential to make this the defacto Jenkins Operator for people who cannot move to other CI/CD systems.

Your Traditional Jenkins Workflow on Openshift

If you already have Jenkins running in your Openshift Cluster, this is what your workflow might resemble from deploying to kickstarting a Jenkins Job.

• Deploy Jenkins via an Openshift Template
• Create a Jenkins Job from a BuildConfig
• Create Builds as instances of Runs for the Jenkins Job

As we move forward we will carry out the same with a Jenkins instance deployed via a Jenkins Operator and also look at how to install the Operator itself.

Deploying the Operator via Operator Lifecycle Manager

Installation

• Login to your Openshift Cluster.

• Go to Operators sub-menu in the burger menu and select OperatorHub and search for Jenkins Operator.

• Select Jenkins Operator and press theInstall button and in the next page for the Subscription, click on the Subscribe button for the Operator.

• Wait for the Operator to get installed and show up under your Installed Operators

Create Jenkins

• To create a Jenkins instance, click on the Jenkins Operator as shown above and you would be presented with the menu below and then press on the + Create Instance in the Jenkins resource shown below, Provided APIs.

• You would be presented with the following option to make changes to the example. You can configure the size of the Jenkins as well as make other changes to the Instance via the YAML config. Press on Create once you are happy with the settings. Do remove the securityContext under the spec to avoid any problems regarding the same. We are working on getting this removed from example.

• If everything goes well, a Jenkins Instance would be created and be shown here.

• Let’s login and check from the terminal.

$ oc login -u kubeadmin -p password
$ oc get pods -n jenkins-operator-test
NAME                                READY   STATUS    RESTARTS   AGE
jenkins-example                     1/1     Running   0          6m18s
jenkins-operator-5d7bf9c657-8g96b   1/1     Running   0          16m

Configuring additional necessary resources

Considering that the Jenkins instance is up and running let’s set it up to work with the right configuration necessary to run with Openshift.

• Add ClusterRole edit to the ServiceAccount used by the Jenkins instance created by the Operator.

$ oc adm policy add-cluster-role-to-user edit -z jenkins-operator-example -n jenkins-operator-test

• Expose the Service pointing to the Jenkins instance to be used as a Route so we can access the Jenkins instance from outside the Cluster Network.

Get the service which points to the Jenkins instance

$ oc get svc | grep jenkins-operator-http

jenkins-operator-http-example    ClusterIP   172.30.54.123    <none>        8080/TCP            11m

Expose the Http Service pointing to the Jenkins instance via a Route

$ oc expose svc jenkins-operator-http-example

Update the Route to use TLS Edge Temination and set the HA Proxy timout

# TLS Edge Termination
$  oc patch route jenkins-operator-http-example -p '{"spec":{"tls":{"insecureEdgeTerminationPolicy":"Redirect","termination":"edge"}}}'

# HA Proxy Timeout
$  oc annotate route jenkins-operator-http-example "haproxy.router.openshift.io/timeout"="4m"

• Configure Jenkins instance ServiceAccount to be used for the Redirection.

$ oc annotate sa jenkins-operator-example "serviceaccounts.openshift.io/oauth-redirectreference.jenkins"="{\"kind\":\"OAuthRedirectReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"Route\",\"name\":\"jenkins-operator-http-example\"}}"

With the above you should be able to have a Jenkins working that will comply with Openshift Builds with which the jobs can be triggered as well.

Triggering Jenkins Jobs using the Openshift Builds

Simple example to create a BuildConfig with a Jenkinsfile Strategy would be to have a git repo with a Jenkinsfile at the root of the repo as it is given here

$ oc new-app https://github.com/waveywaves/nodejs-ex --context-dir openshift/pipelines

Upon doing the above, a BuildConfig would be created based on the Jenkinsfile strategy.

Above we can see the BuildConfig nodejs-ex has been created by the oc new-app command given above and the Build nodejs-ex-1 has also succeeded which is an instance of the same. This Build is responsible for deploying a NodeJS application including the creation of the BuildConfig, Deployment, Services and Route of the same which are prefixed with nodejs-example in their names.

Let’s check the Build from Jenkins itself and see what exactly happened. We have the Route from before from when we exposed the Service which we are going to use to access the the Jenkins.

$ oc get route | grep jenkins-operator-http

After going to the Route and logging in via our Openshift Credentials, we should be able to see the Jenkins User Interface. Click on the folder with the name of the Namespace.

You should now be able to see the Jenkins Job that resembles the <namespace>/<build-config> name pattern pointing to our BuildConfig.

Once you click on the same you can see the Build and the different stages it has gone through. Click on the Build and then click on the Console Output option given in the left column of the screen.

After doing that you can see the output of your Job Run and in times of failure you can check here, if the Build succeeded or not.

Conclusion

As of now the Operator provides basic functionality and few extra features like creation of SeedJobs and Backing Up Jenkins instances etc. But wouldn’t be considered production ready for Openshift.

The current state of support for the Operator in Openshift is Developer Preview only and is not completely supported yet. A lot of the features and improvements are still Work In Progress. We will keep updating the progress on this and let the world know the story of the Jenkins Operator ;)