Introduction

Over the last 3 months, I’ve been building a Proxmox-based lab centered around a highly available RKE2 Kubernetes cluster. The environment hosts most of my infrastructure services, including GitOps tooling (ArgoCD), monitoring, secrets management (Azure Key Vault), and identity services (Authentik + EntraID).

For remote access, I had been using a traditional OpenVPN deployment with LDAP authentication. The setup was reliable and did exactly what it was supposed to do: provide secure access to the lab from anywhere. However, as the environment continued to grow, managing VPN access separately from the rest of my authentication stack started to feel outdated.

In a previous article, I deployed Authentik inside my RKE2 cluster and federated it with the Microsoft Entra ID as the upstream identity provider. Since basic OpenID Connect federation in Entra ID doesn’t require premium licensing, I decided to use Entra only for authentication while letting Authentik handle application integrations, group management, and Single Sign-On for self-hosted services.

This approach allowed me to build a centralized identity platform without paying for Microsoft Entra ID Premium P1 or P2 licenses while still providing a modern SSO experience across my homelab.

At that point, OpenVPN became the odd one out.

While all other applications were moving toward OpenID Connect and Single Sign-On, VPN access still relied on a separate LDAP-based authentication flow. I wanted a solution that would allow me to authenticate with the same identity provider used across the rest of the environment while keeping the entire stack self-hosted.

That’s when I decided to test NetBird.

NetBird combines WireGuard, identity-aware access control, peer management, and modern authentication mechanisms into a single platform. More importantly, it supports external identity providers, making it possible to integrate directly with Authentik and leverage existing users and groups.

My goal was simple:

• Replace OpenVPN with NetBird
• Remove LDAP authentication from the VPN workflow (by removing OpenVPN)
• Integrate Netbird VPN access with Authentik
• Use Microsoft Entra ID as the authentication source with MFA
• Enable Single Sign-On across the entire environment
• Synchronize users and groups automatically
• Continue running everything on self-hosted infrastructure
• Make it without EntraID license

In this article, I’ll walk through the complete migration from OpenVPN to NetBird, deployment on Proxmox, integration with Authentik, and configuration of Single Sign-On backed by Microsoft Entra ID.

Architecture Before the Migration

The following diagram shows the lab environment before introducing NetBird.

The entire platform runs on a Proxmox hypervisor hosting a highly available RKE2 Kubernetes cluster consisting of three control-plane nodes and four worker nodes. OPNsense is used as the edge firewall and router, providing network connectivity between the internal infrastructure and the Internet.

Remote administrative access is currently provided through OpenVPN running on OPNsense. Administrators connect to the lab through the VPN and authenticate using LDAP credentials.

Inside Kubernetes, the cluster hosts the platform services used across the environment, including monitoring, GitOps, identity management, and security tooling.

Prerequisites

Before deploying NetBird, there are a few components that should already be in place. In my case, the lab was already fully operational, and remote administration was performed through OpenVPN with LDAP authentication. Since NetBird would eventually replace OpenVPN, I still needed an existing way to access the environment during the deployment and testing phase. Also I use Ansible for configuring the VMs. All the code for this article is available here.

If you want to read previous articles on how to set up this lab go to my profile -> https://medium.com/@aleksander.chotecki.

Identity Platform

The most important prerequisite is a working identity provider.

In my setup, Microsoft Entra ID is used only as the authentication source. Users authenticate against Entra ID, while Authentik acts as the central identity platform responsible for application integrations and Single Sign-On.

This approach allows me to benefit from Entra ID authentication without requiring Premium P1 or P2 licenses, while still leveraging Authentik for OpenID Connect, group synchronization, and SSO across self-hosted services.

Infrastructure Access

Since the deployment will be performed inside the existing homelab environment, administrative access is required to several systems.

You should have:

• Access to the Proxmox cluster
• Access to OPNsense
• SSH access to all virtual machines that will be used for NetBird

At the time of deployment, I was still using OpenVPN with LDAP authentication to reach the internal network and administer the environment remotely.

Public Access and DNS

NetBird clients must be able to communicate with the management service over the Internet.

For that reason, you’ll need:

• A public DNS name for NetBird (I use cloudflare for this)
• Port forwarding configured on OPNsense
• TLS certificates for the exposed services (also cloudflare in my case)

In my environment, OPNsense is responsible for publishing internal services to the Internet and forwarding the required traffic to the NetBird server.

Virtual Machines

The lab uses three Ubuntu Server 24.04 LTS virtual machines: one NetBird server hosting the management components and two routing peers used to provide access to internal networks. While one routing peer would be enough for most lab deployments, I chose to deploy two for testing and future scalability.

Initial Server Preparation

The Ansible code used in this article can be found here: https://github.com/heyimjustalex/medium/tree/main/Article_5.

Before running the Ansible automation, I performed a minimal bootstrap configuration on each Ubuntu 24.04 VM. This included verifying SSH connectivity and granting the deployment account passwordless sudo privileges, allowing Ansible to execute administrative tasks without interactive password prompts.

echo "cloud ALL=(ALL) NOPASSWD: ALL" | \
sudo EDITOR='tee' visudo -f /etc/sudoers.d/99-cloud-ansible

This one-time configuration simplifies infrastructure provisioning and enables fully automated deployment of the NetBird components across all servers.

Automating the Deployment with Ansible

Rather than configuring each server manually, I decided to automate the entire deployment with Ansible. This approach made the installation repeatable, reduced the risk of configuration drift, and allowed me to rebuild the environment quickly whenever required.

Inventory and Variables

The project is structured around a simple inventory consisting of one NetBird server and two routing peers.

all:
  children:
    netbird_instance:
      hosts:
        netbird:
          ansible_host: 10.0.0.200 
        
    netbird_agents:
      hosts:
        netbird-router-01:
          ansible_host: 10.0.0.201
        netbird-router-02:
          ansible_host: 10.0.0.202

Global variables define the NetBird version, domain name, installation directories, required packages, firewall settings, and other deployment parameters. Sensitive values such as encryption keys, authentication secrets, and Cloudflare credentials are securely stored in Ansible Vault.

Deploying the NetBird VPN Management Platform

With the inventory and variables prepared, the next step was deploying the NetBird management platform.

Rather than relying on installation scripts executed directly on the server, the entire deployment is driven through Ansible. The playbook performs environment validation, installs all required dependencies, configures the operating system, and deploys the complete NetBird stack using Docker Compose.

One of the first tasks validates that the target operating system is supported and that all required deployment variables are present before any changes are made. This helps catch configuration issues early and prevents partially completed installations.

The server configuration process includes:

• Installing Docker and Docker Compose
• Creating a dedicated Linux account for NetBird
• Preparing application directories and persistent storage
• Configuring TLS certificate storage
• Enabling IPv4 forwarding
• Creating Cloudflare DNS challenge configuration

The use of Infrastructure as Code ensures that the entire deployment can be recreated in a consistent manner whenever required.

To execute the ansible scripts, several sensitive values must be defined in Ansible Vault, including the Cloudflare DNS API token used for automatic certificate issuance, the NetBird authentication secret, and the encryption key used to protect internal NetBird data. The vault file also contains setup keys assigned to individual routing peers, allowing them to register automatically during deployment.

One important detail is that the routing peer setup keys cannot be generated before NetBird is installed. These keys are created manually through the NetBird administration interface after the management platform becomes operational and must then be added to Ansible Vault before deploying the routing peers.

You can create your own vault by executing:

ansible-vault create group_vars/vault.yml

Automatic TLS Certificates with Cloudflare

Since NetBird is exposed to the public Internet, TLS certificates are required.

To avoid manual certificate management, I decided to use Let’s Encrypt with Cloudflare DNS validation. This approach allows certificates to be issued automatically without exposing additional services or opening temporary HTTP validation endpoints.

The playbook creates a dedicated environment file containing the Cloudflare API token used by Traefik:

CF_DNS_API_TOKEN=<secret>

This enables completely automated certificate provisioning and renewal while keeping the deployment fully self-hosted.

Deploying the Stack

After preparing the scripts, Ansible deploys the NetBird components through Docker Compose. This is the script that’s used for the server deployment:

- name: Install self-hosted NetBird from static config
  hosts: netbird_instance
  become: true
  gather_facts: true
vars_files:
    - ../group_vars/all.yml
    - ../group_vars/vault.yml
  tasks:
    ########################################################################
    # PRE-FLIGHT
    ########################################################################
    - name: Validate supported Ubuntu version
      ansible.builtin.assert:
        that:
          - ansible_distribution == "Ubuntu"
          - ansible_distribution_major_version | int >= 22
        fail_msg: "Ubuntu 22.04+ is required"
        success_msg: "Supported Ubuntu version detected"
    - name: Validate required variables
      ansible.builtin.assert:
        that:
          - netbird_domain | length > 3
          - netbird_admin_email | length > 5
          - vault_cloudflare_dns_api_token is defined
          - vault_cloudflare_dns_api_token | length > 10
        fail_msg: "Required variables are missing"
    ########################################################################
    # PACKAGES
    ########################################################################
    - name: Update apt cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600
    - name: Install required packages
      ansible.builtin.apt:
        name: "{{ netbird_required_packages }}"
        state: present
    - name: Ensure docker service is enabled
      ansible.builtin.service:
        name: docker
        enabled: true
        state: started
    ########################################################################
    # USER
    ########################################################################
    - name: Ensure NetBird user exists
      ansible.builtin.user:
        name: "{{ netbird_linux_user }}"
        shell: /bin/bash
        create_home: true
        state: present
    - name: Add user to docker group
      ansible.builtin.user:
        name: "{{ netbird_linux_user }}"
        groups: docker
        append: true
    ########################################################################
    # DIRECTORIES
    ########################################################################
    - name: Create required directories
      ansible.builtin.file:
        path: "{{ item.path }}"
        state: directory
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "{{ item.mode }}"
      loop:
        - { path: "{{ netbird_install_dir }}", mode: "0750" }
        - { path: "{{ netbird_secrets_dir }}", mode: "0700" }
        - { path: "{{ netbird_letsencrypt_dir }}", mode: "0750" }
    - name: Create acme.json
      ansible.builtin.file:
        path: "{{ netbird_acme_file }}"
        state: touch
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "0600"
    ########################################################################
    # SYSCTL
    ########################################################################
    - name: Enable IPv4 forwarding
      ansible.builtin.copy:
        dest: /etc/sysctl.d/99-netbird.conf
        owner: root
        group: root
        mode: "0644"
        content: |
          net.ipv4.ip_forward=1
      when: netbird_enable_ipv4_forwarding | bool
    - name: Reload sysctl
      ansible.builtin.command: sysctl --system
      changed_when: false
    ########################################################################
    # CLOUDFLARE ENV
    ########################################################################
    - name: Create cloudflare env
      ansible.builtin.copy:
        dest: "{{ netbird_cloudflare_env_file }}"
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "0600"
        content: |
          CF_DNS_API_TOKEN={{ vault_cloudflare_dns_api_token }}
    ########################################################################
    # COPY STATIC FILES
    ########################################################################
    - name: Copy docker-compose.yml
      ansible.builtin.copy:
        src: ../files/netbird/docker-compose.yml
        dest: "{{ netbird_compose_file }}"
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "0644"
    - name: Create NetBird config
      ansible.builtin.template:
        src: ../files/netbird/config.yaml.j2
        dest: "{{ netbird_config_file }}"
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "0600"
    - name: Create dashboard env
      ansible.builtin.template:
        src: ../files/netbird/dashboard.env.j2
        dest: "{{ netbird_install_dir }}/dashboard.env"
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "0644"
    ########################################################################
    # PERMISSIONS
    ########################################################################
    - name: Ensure ownership
      ansible.builtin.file:
        path: "{{ netbird_install_dir }}"
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        recurse: true
    - name: Ensure acme.json permissions
      ansible.builtin.file:
        path: "{{ netbird_acme_file }}"
        owner: "{{ netbird_linux_user }}"
        group: "{{ netbird_linux_user }}"
        mode: "0600"
    ########################################################################
    # VALIDATE COMPOSE
    ########################################################################
    - name: Validate compose syntax
      ansible.builtin.command:
        cmd: docker compose config -q
        chdir: "{{ netbird_install_dir }}"
      changed_when: false
    ########################################################################
    # DEPLOY
    ########################################################################
    - name: Pull images
      ansible.builtin.command:
        cmd: docker compose pull
        chdir: "{{ netbird_install_dir }}"
      register: compose_pull
      changed_when: false
    - name: Deploy stack
      ansible.builtin.command:
        cmd: docker compose up -d
        chdir: "{{ netbird_install_dir }}"
      register: compose_up
      changed_when: false
    ########################################################################
    # VALIDATION
    ########################################################################
    - name: Wait for containers
      ansible.builtin.pause:
        seconds: 20
    - name: Get running containers
      ansible.builtin.command:
        cmd: docker ps --format '{{ "{{" }}.Names{{ "}}" }}'
      register: docker_ps
      changed_when: false
    - name: Show containers
      ansible.builtin.debug:
        var: docker_ps.stdout_lines
    - name: Validate required containers
      ansible.builtin.assert:
        that:
          - "'netbird-traefik' in docker_ps.stdout"
          - "'netbird-dashboard' in docker_ps.stdout"
          - "'netbird-server' in docker_ps.stdout"
        fail_msg: "Required NetBird containers are not running"
        success_msg: "NetBird stack is running"
    - name: Validate dashboard through Traefik
      ansible.builtin.command:
        cmd: >
          curl -k -s
          -o /tmp/netbird-check.html
          -w '%{http_code}'
          https://localhost/
          -H 'Host: {{ netbird_domain }}'
      register: dashboard_check
      changed_when: false
    - name: Assert dashboard returns 200
      ansible.builtin.assert:
        that:
          - dashboard_check.stdout == "200"
        fail_msg: "Dashboard validation failed"
        success_msg: "Dashboard validation passed"
    - name: Show compose status
      ansible.builtin.command:
        cmd: docker compose ps
        chdir: "{{ netbird_install_dir }}"
      register: compose_ps
      changed_when: false
    - name: Display compose status
      ansible.builtin.debug:
        var: compose_ps.stdout_lines
    - name: Show server logs
      ansible.builtin.command:
        cmd: docker logs netbird-server --tail 50
      register: server_logs
      failed_when: false
      changed_when: false
    - name: Recreate Traefik
      ansible.builtin.command:
        cmd: docker compose up -d --force-recreate traefik
        chdir: "{{ netbird_install_dir }}"
      changed_when: true
    - name: Print server logs
      ansible.builtin.debug:
        var: server_logs.stdout_lines
    - name: Show traefik logs
      ansible.builtin.command:
        cmd: docker logs netbird-traefik --tail 50
      register: traefik_logs
      failed_when: false
      changed_when: false
    - name: Print traefik logs
      ansible.builtin.debug:
        var: traefik_logs.stdout_lines

Publishing NetBird Through OPNsense

Before any clients could connect, the NetBird management platform needed to be reachable from the Internet.

In my environment, OPNsense acts as the perimeter firewall and is responsible for publishing internal services running on virtual machines and Kubernetes workloads. Since the NetBird management server was deployed on a dedicated Ubuntu virtual machine, I configured the required NAT and firewall rules to expose the service externally.

NetBird requires both HTTPS communication for the management interface and UDP connectivity for peer-to-peer communication. As a result, two port forwarding rules were configured on OPNsense:

For the first log-in you should add the whitelist only for your IP as Netbird requires installation and setting up the initial password when first launched. Also consider adding some WAF rules in cloudflare WAF configuration.

Verifying the NetBird Deployment

Once the playbook completed successfully, the NetBird management platform should become available through the public DNS name configured for the deployment.

At this stage, all core components were running behind Traefik, TLS certificates had been issued automatically through Let’s Encrypt, and the NetBird dashboard could be accessed securely over HTTPS.

The playbook performs several validation checks after deployment, including container health verification and connectivity testing. This helps ensure the environment is fully operational before moving on to client and routing peer configuration.

NetBird is the central location for managing users, peers, routes, setup keys, and authentication settings and should be accessible under netbird.<yourdomain>.com

Creating Setup Keys for Routing Peers

Before deploying the routing peers, NetBird requires a mechanism that allows servers to join the network automatically.

Rather than performing an interactive login on every machine, NetBird provides setup keys that can be used for unattended enrollment. This makes them ideal for infrastructure components such as routers, servers, and automation-managed systems.

After logging into the NetBird administration interface, navigate to Peers and select Add Peer.

Next, choose Agent and generate a new setup key.

The generated key will later be used by the Ansible playbook to perform automatic registration of the routing peers.

Since my deployment uses two dedicated routing nodes, I created a separate setup key for each peer and stored them securely in Ansible Vault:

Deploying Routing Peers

With the setup keys generated, the routing peers can now be deployed.

The routing peer installation is handled by a dedicated Ansible playbook that performs the complete provisioning process automatically. This includes package installation, network configuration, NetBird installation, peer registration, and connectivity validation.

- name: Install NetBird routing agents
  hosts: netbird_agents
  become: true
  gather_facts: true

  vars_files:
    - ../group_vars/all.yml
    - ../group_vars/vault.yml

  vars:
    netbird_deb_file: "netbird_{{ netbird_agent_version }}_linux_amd64.deb"
    netbird_deb_url: "https://github.com/netbirdio/netbird/releases/download/v{{ netbird_agent_version }}/{{ netbird_deb_file }}"
    netbird_deb_path: "/tmp/{{ netbird_deb_file }}"

    netbird_setup_key: "{{ vault_netbird_agent_setup_keys[inventory_hostname] }}"

  tasks:

    #####################################################################
    # VALIDATION
    #####################################################################

    - name: Validate setup key exists
      ansible.builtin.assert:
        that:
          - inventory_hostname in vault_netbird_agent_setup_keys
        fail_msg: "Missing setup key for {{ inventory_hostname }}"

    #####################################################################
    # PACKAGES
    #####################################################################

    - name: Update apt
      ansible.builtin.apt:
        update_cache: true

    - name: Install prereqs
      ansible.builtin.apt:
        state: present
        name:
          - curl
          - wget
          - ca-certificates
          - jq
          - iproute2
          - iptables
          - nftables

    #####################################################################
    # HOSTNAME
    #####################################################################

    - name: Set hostname
      ansible.builtin.hostname:
        name: "{{ inventory_hostname }}"

    #####################################################################
    # HOSTS
    #####################################################################

    - name: Configure NetBird hostname locally
      ansible.builtin.blockinfile:
        path: /etc/hosts
        marker: "# {mark} NETBIRD"
        block: |
          {{ netbird_server_ip }} {{ netbird_domain }}

    #####################################################################
    # ROUTING
    #####################################################################

    - name: Enable IPv4 forwarding
      ansible.builtin.copy:
        dest: /etc/sysctl.d/99-netbird-router.conf
        mode: "0644"
        content: |
          net.ipv4.ip_forward=1
          net.ipv4.conf.all.src_valid_mark=1

    - name: Reload sysctl
      ansible.builtin.command: sysctl --system
      changed_when: false

    #####################################################################
    # CLEANUP
    #####################################################################

    - name: Disconnect old peer
      ansible.builtin.command: netbird down
      failed_when: false
      changed_when: false
      when: netbird_agent_reset_state | bool

    - name: Stop netbird
      ansible.builtin.service:
        name: netbird
        state: stopped
      failed_when: false
      when: netbird_agent_reset_state | bool

    - name: Remove old state
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /var/lib/netbird
        - /etc/netbird
      when: netbird_agent_reset_state | bool

    #####################################################################
    # DOWNLOAD
    #####################################################################

    - name: Download NetBird
      ansible.builtin.get_url:
        url: "{{ netbird_deb_url }}"
        dest: "{{ netbird_deb_path }}"
        mode: "0644"

    #####################################################################
    # INSTALL
    #####################################################################

    - name: Install NetBird package
      ansible.builtin.command:
        cmd: "dpkg -i {{ netbird_deb_path }}"
      register: dpkg_install
      changed_when: true

    - name: Fix dependencies
      ansible.builtin.apt:
        state: fixed

    - name: Start service
      ansible.builtin.service:
        name: netbird
        enabled: true
        state: started

    #####################################################################
    # CONNECT
    #####################################################################

    - name: Connect to NetBird
      ansible.builtin.command:
        cmd: >
          netbird up
          --management-url https://{{ netbird_domain }}
          --setup-key {{ netbird_setup_key }}
      register: netbird_up
      changed_when: true
      no_log: true

    #####################################################################
    # VALIDATION
    #####################################################################

    - name: Wait for initialization
      ansible.builtin.pause:
        seconds: 5

    - name: NetBird status
      ansible.builtin.command:
        cmd: netbird status
      register: nb_status
      changed_when: false

    - name: Show status
      ansible.builtin.debug:
        var: nb_status.stdout_lines

    - name: Validate connected
      ansible.builtin.assert:
        that:
          - "'Management: Connected' in nb_status.stdout"
          - "'Signal: Connected' in nb_status.stdout"

    - name: Validate route table
      ansible.builtin.command:
        cmd: ip route
      register: route_table
      changed_when: false

    - name: Show routes
      ansible.builtin.debug:
        var: route_table.stdout_lines

The setup key automatically registers the node with the NetBird management service and associates it with the correct account. Once the automation finishes, both routing peers should appear in the NetBird dashboard and be ready for usage.

Organizing Access with Groups and Network Resources

At this stage, the NetBird management platform was operational, the routing peers had been successfully enrolled, and connectivity to the management service had been validated.

However, before remote users could access any internal resources, I needed to define how access should be organized. Rather than assigning permissions directly to individual devices, NetBird uses groups, resources, and policies to control access to internal networks.

This model aligns well with the identity-centric approach already used throughout the rest of the platform.

Creating Access Groups

The first step was creating groups that would later be used for routing and authorization.

For this deployment, I created two groups:

• vpn — users who should be allowed to access internal infrastructure resources through NetBird
• routing-peers — dedicated routing nodes responsible for advertising internal networks into the overlay network

Creating dedicated groups makes future access management significantly easier, especially when additional users and networks are introduced.

After creating the routing-peers group, both routing nodes were assigned to it. This allows NetBird to treat both routing servers as a logical routing entity while maintaining high availability and simplifying future network expansion.

Publishing Internal Networks

With the routing peers grouped and operational, the next step was exposing the internal management network to remote users.

NetBird uses a resource-based model, where networks are represented as resources that can later be secured using access policies.

To publish the Proxmox and Kubernetes management network, I created a new network resource called proxmox-network.

Within that network, I added a resource representing my internal management subnet 10.0.0.0/24.

Defining Access Policies

One of the areas where NetBird differs significantly from traditional VPN solutions is access control.

With OpenVPN, authenticated users typically receive unrestricted access to all advertised networks. NetBird, on the other hand, allows access decisions to be based on groups and resources.

To restrict access only to authorized users, I created a policy allowing members of the vpn group to access the proxmox-vnet resource.

We also need to add the routing peers we’ve installed to the proxmox-network.

In the end of this process you should see a network with the routing peers in your configuration.

Integrating NetBird with Authentik

At this point, the network layer was fully operational.

The routing peers were advertising the internal management network, access policies were in place, and users assigned to the vpn group could reach resources inside the 10.0.0.0/24 subnet.

However, one major objective of this migration had not yet been achieved.

Although OpenVPN had been successfully replaced, user authentication was still managed separately from the identity platform used throughout the rest of the environment.

My goal was to provide the same authentication experience regardless of whether a user was accessing GitLab, ArgoCD, Grafana, or connecting through the VPN.

To accomplish this, NetBird needed to be integrated with Authentik using OpenID Connect.

Existing Identity Architecture

In previous articles, I deployed Authentik inside the Kubernetes cluster and configured Microsoft Entra ID as the upstream identity provider.

As a result, all self-hosted applications already authenticated against Authentik, while Authentik delegated authentication to Microsoft Entra ID.

Preparing User Groups

Before configuring the OpenID Connect integration, I created dedicated VPN groups inside Microsoft Entra ID.

These groups are used to control which users should be allowed to access internal resources through NetBird.

For this deployment, two groups were created:

• vpn-admins
• vpn-users

Creating the NetBird Application in Authentik

With the VPN infrastructure fully operational, the next step was integrating NetBird with the existing identity platform.

Since Authentik was already federated with Microsoft Entra ID and used across several self-hosted applications within the environment, it made sense to reuse the same authentication flow for VPN access.

NetBird supports OpenID Connect (OIDC), making the integration relatively straightforward.

The first step was creating a dedicated OAuth2/OpenID application inside Authentik.

I started by creating a new application called netbird and selecting an OAuth2/OpenID Provider.

Configuring the OAuth2 Provider

The provider was configured as a confidential client using the Authorization Code flow. This generates the Client ID and Client Secret that will later be used by NetBird when communicating with Authentik.+1

One important configuration item is the redirect URI. NetBird expects authentication responses to be returned to: https://netbird.<your-domain>/oauth2/callback

https://netbird.<your-domain>/oauth2/callback

This URL must be configured in the Authentik provider settings; otherwise authentication will fail during the OAuth flow.

I also configured the subject identifier to use the user’s email address and enabled the inclusion of claims in the ID token. This allows NetBird to consume user and group information directly from the authentication response.

Mapping User Groups

Authentication alone was not enough.

One of the main goals of the migration was ensuring that NetBird could automatically consume user group information and use it for authorization decisions.

To achieve this, I created a custom Scope Mapping in Authentik that exposes group membership through the groups claim. Instead of publishing every group available inside Authentik, the mapping filters and returns only groups related to NetBird access.

This approach keeps identity information clean while allowing NetBird to make authorization decisions based on existing Authentik groups. The OpenID provider must also be updated to replace the default profile scope with the custom scope mapping created earlier.

Connecting NetBird to Authentik

With the application and OAuth provider configured, the next step was connecting NetBird to Authentik.

Inside the NetBird administration interface, I navigated to Settings → Identity Providers and created a new OIDC identity provider.

The configuration required:

• OpenID Discovery URL
• Client ID
• Client Secret
• Issuer URL

All of these values were obtained directly from the provider configuration generated earlier in Authentik.

After saving the configuration, Authentik should become available as a login method within NetBird.

Enabling Group Synchronization

One feature I particularly wanted was automatic synchronization of user groups.

Instead of manually managing permissions inside NetBird, I wanted access control to remain identity-driven and managed directly from Microsoft Entra ID and Authentik.

NetBird supports this through JWT group synchronization.

After enabling group propagation and JWT group synchronization, NetBird automatically extracts group membership information from the groups claim included in the ID token.

As a result:

• Groups created in Microsoft Entra ID
• Synchronized to Authentik
• Included in OIDC tokens
• Imported automatically into NetBird

Users now inherit access based entirely on their assigned identity groups rather than manually maintained VPN permissions.

Testing the Authentication Flow

Once the integration was complete, I tested the end-to-end login process.

When a user selects Continue with Authentik, NetBird redirects authentication requests to Authentik, which in turn forwards the user to Microsoft Entra ID for authentication.

The authentication process now follows this flow:

From the user’s perspective, VPN authentication now behaves exactly like every other application in the platform.

Single Sign-On is preserved, Multi-Factor Authentication is handled by Entra ID, and user access is controlled through centrally managed groups.

After logging-in the groups from Entra should be present in Netbird

The user should also be present in the Team -> Users tab in Netbird. If you want to have admin rights the vpn-admins group will not work, as you cannot assign the ‘Admin’ role to a group yet in Netbird. What you can do, you can assign it to the user.

Final Network Adjustment

Earlier in the deployment, I created a local vpn group inside NetBird and used it in the access policy for the proxmox-vnet resource. However, the groups synchronized from Microsoft Entra ID are named vpn-users and vpn-admins.

Since group names must match the identities that NetBird receives from Authentik, the policy needs to be updated to reference the synchronized vpn-users group instead of the manually created vpn group.

To do this, edit the access policy associated with the proxmox-vnet resource and replace the original source group with vpn-users.

After saving the changes, users who are members of the vpn-users group in Microsoft Entra ID will automatically receive access to the published network once group synchronization between Entra ID, Authentik, and NetBird has been configured. This removes the need to manage VPN permissions manually inside NetBird and allows access control to be driven entirely through the centralized identity platform.

Installing the NetBird Desktop Client

With the network resources, routing peers, and access policies configured, the final step is connecting a user workstation to the NetBird network.

NetBird provides native clients for Windows, macOS, Linux, Android, and iOS. For this article, I used the Windows desktop client.

The latest client can be downloaded directly from the NetBird download page:

https://app.netbird.io/install

After installing the application, launch the NetBird client and select edit profile

Next, enter the URL of your self-hosted NetBird management service:

https://netbird.<your-domain>.com

After clicking Connect, NetBird redirects the browser to the configured identity provider. Since NetBird is integrated with Authentik, authentication is forwarded to Authentik which then redirects the user to Microsoft Entra ID.

After a successful authentication, the device is automatically registered in NetBird and assigned to the appropriate groups based on the information received from Authentik.

Verifying Connectivity

Once the client connects successfully, NetBird automatically installs the routes associated with the resources and access policies configured earlier.

On Windows, the route table can be verified using:

route print

You should see the internal management subnet advertised through the NetBird interface:

10.0.0.0/24

Summary

At this point, infrastructure resources such as Proxmox, OPNsense, Kubernetes nodes, and other management services should be reachable through the VPN connection.

Unlike the previous OpenVPN deployment, no LDAP authentication is involved in this process. Authentication is performed through Microsoft Entra ID, authorization is managed through Authentik and NetBird groups, and connectivity is provided through the WireGuard-based NetBird overlay network. This results in a significantly simpler user experience while maintaining centralized identity management across the entire platform.

In the previous article, I deployed Authentik as part of the platform services layer and integrated it into the GitOps-managed Kubernetes environment. While the identity platform itself was already running, it was not yet connected to any external identity source and none of the platform services were using it for authentication.

At this stage, users still authenticated directly against individual applications, resulting in separate login flows, isolated user management, and no consistent Single Sign-On experience across the environment.

To solve this, I decided to make Authentik the central identity provider for the entire platform and use Microsoft Entra ID as the upstream authentication source. This approach provides a modern authentication experience, Multi-Factor Authentication support, centralized group management, and Single Sign-On capabilities without requiring Microsoft Entra ID Premium P1 or P2 licenses.

The goal of this article is to build a complete identity architecture that all future platform services can rely on.

By the end of this article:

• Microsoft Entra ID will become the authoritative identity source
• Authentik will act as the central Identity Provider
• Argo CD will authenticate through Authentik using OpenID Connect
• Rancher will authenticate through Authentik using OpenID Connect
• GitLab will be deployed automatically using Ansible
• GitLab will be integrated with Authentik through OpenID Connect (authentication only, as community edition does not support IDP-level authorization)
• User access will be managed through centrally managed groups
• Multi-Factor Authentication will be enforced through Microsoft Entra ID
• A single authentication workflow will be used across the platform

The code used in this article can be found here.

Federating Authentik with Microsoft Entra ID

In the previous article, I deployed Authentik as part of the platform services layer and integrated it into the GitOps-managed Kubernetes environment. While the identity provider itself was operational, it still relied on locally managed users and groups.

Before integrating Authentik with applications such as Argo CD, Rancher, and GitLab, I wanted to establish a centralized identity source that could be reused across the entire platform.

Rather than managing users separately inside every application, I decided to use Microsoft Entra ID as the authentication source and Authentik as the central Identity Provider.

This approach offers several advantages:

• Centralized user lifecycle management
• Single Sign-On across all applications
• Multi-Factor Authentication enforced by Entra ID
• Consistent group management
• Reduced operational overhead
• No requirement for Microsoft Entra ID Premium P1 or P2 licenses

The step-by-step guide for this can be found in Authentik docs.

Creating the App Registration

The first step is creating a new App Registration in Microsoft Entra ID that will represent Authentik.

Inside the Microsoft Entra Admin Center, navigate to Microsoft Entra ID → App registrations → New registration

The application is configured as a standard web application and will be used by Authentik to redirect users to Microsoft Entra ID during the authentication process. Also, the Authentik callback URL must be configured.

This allows Microsoft Entra ID to return authentication responses back to Authentik after a successful login.

Creating a Client Secret

The next step is generating a client secret.

Authentik uses the Client ID and Client Secret to authenticate against Microsoft Entra ID during the federation process.

Certificates & Secrets → New Client Secret

Configuring Microsoft Graph Permissions

Before Authentik can retrieve user information from Microsoft Entra ID, the application registration requires access to Microsoft Graph.

In the App Registration, navigate to:

API Permissions

→ Add a Permission

→ Microsoft Graph

→ Delegated Permissions

and configure all the permissions listed in the docs here.

Configuring Microsoft Entra ID as an Authentik Source

With the App Registration complete, I moved to the Authentik administration interface. I created a new source and provided:

• Tenant ID
• Application (Client) ID
• Client Secret

obtained from the App Registration created earlier.

Authentik requires the Application (Client) ID, Directory (Tenant) ID, and a Client Secret generated from the App Registration. The Service Principal Object ID created under Enterprise Applications is not used for the federation configuration.

Testing the Federation

Once the Microsoft Entra ID source has been configured, Authentik will automatically expose a new login option on the authentication page.

At this stage, I recommend testing the integration before proceeding with any application-specific configuration. This helps verify that the App Registration, redirect URIs, permissions, and credentials were configured correctly.

On the Authentik login screen, select Microsoft Entra ID as the authentication source.

Designing the Authorization Model

Before integrating applications, I wanted to establish a consistent group structure that could be reused across the entire platform.

Rather than creating permissions directly inside individual applications, I decided to manage authorization through Microsoft Entra ID groups and synchronize them through Authentik.

For this deployment, I created the following groups:

These groups will later be synchronized into Authentik and exposed to downstream applications through OIDC claims. This allows permissions to be managed from a single location while maintaining a consistent authorization model across Argo CD, Rancher and future integrations.

Integrating Argo CD with Authentik

With Microsoft Entra ID successfully federated with Authentik, the next step was integrating platform services into the new authentication model.

I decided to start with Argo CD because it serves as the central GitOps engine for the entire platform. Since all deployments are managed through Argo CD, it made sense to eliminate local accounts and move authentication to Authentik as early as possible.

Inside the Authentik administration interface, navigate to Applications → Create and create a new application called argocd-dex

I chose this name because Argo CD uses Dex as its authentication component and all OpenID Connect communication is handled through Dex.

Configuring the OpenID Provider

Next, create an OAuth2/OpenID Provider and associate it with the argocd-dex application.

The provider is configured as a confidential OpenID Connect client and will be responsible for handling authentication requests coming from Argo CD.

One of the most important settings is the redirect URI. Since Argo CD uses Dex, the callback URL must point to the Dex endpoint:

https://argocd.<your-domain>/api/dex/callback

Reusing Existing Client Credentials

Since Argo CD was already deployed and managed through GitOps, I had previously generated the OpenID Connect credentials and added them to my automation code.

Rather than creating a new secret, I reused the existing client credentials already managed through Ansible. The client secret is stored securely in my vault.yml file and injected into Argo CD during deployment.

vault_argocd_oidc_client_secret: <secret>

Configuring a Custom Property Mapping

By default, Authentik uses the standard profile scope to expose user attributes to applications. While this works well for basic authentication, it would also expose every synchronized group to Argo CD.

Since I only wanted Argo CD to receive groups related to Argo CD authorization, I created a dedicated Property Mapping and replaced the default profile scope with my custom implementation.

The expression that has been used for property mapping:

groups = [
    g.name
    for g in request.user.groups.all()
    if "argo" in g.name.lower()
    
]

return {
    "name": request.user.name,
    "given_name": request.user.name,
    "preferred_username": request.user.username,
    "nickname": request.user.username,
    "groups": groups,
}

This ensures that only Argo CD related groups are included in the generated token.

Replacing the Default Profile Scope

Once the Property Mapping was created, I updated the argocd-dex provider configuration and replaced the default profile scope with the newly created custom mapping.

After saving changes the integration with argocd should work. If it does not, make sure that you execute the OIDC configuration script from the second article of this series that can be found here.

Integrating Rancher with Authentik

With Argo CD successfully integrated with Authentik, the next step was extending the same authentication model to Rancher.

Since Rancher acts as the central management platform for my Kubernetes environment, it was another critical service that should not rely on local users or independently managed credentials.

Creating the Rancher Application

Inside Authentik, I created another application under:

Applications
  → Applications
  → Create

This time the application was named rancher and was associated with a dedicated OAuth2/OpenID Provider.

Configuring the OpenID Provider

Similar to the Argo CD integration, Rancher requires an OpenID Connect provider and a redirect URI.

The redirect URI configured for Rancher is:

https://rancher.<your-domain>/verify-auth

After creating the provider, Authentik generated:

• Client ID
• Client Secret
• Issuer URL

which are entered into Rancher’s authentication configuration.

Configuring Rancher Property Mapping

Instead of exposing all synchronized groups to Rancher, I followed the same approach used for Argo CD and created a dedicated Property Mapping.

Configuring Rancher Authentication

Inside Rancher, navigate to:

Users & Authentication
  → Auth Provider
  → OpenID Connect

and enter the details generated earlier by Authentik:

• Client ID
• Client Secret
• Issuer URL
• Auth Endpoint
• Token Endpoint
• UserInfo Endpoint

After saving the configuration, Rancher immediately exposes Authentik as the authentication source.

These groups are synchronized into Authentik and included in the Rancher OIDC token through the custom Property Mapping.

Inside Rancher, group memberships can then be mapped to appropriate roles such as:

rancher-admins → Administrator
rancher-users → Standard User

Configuring Project and Namespace Access

Authentication is only one part of the integration. Once users can successfully authenticate through Authentik, Rancher’s built-in Role-Based Access Control (RBAC) can be used to restrict access to specific clusters, projects, and namespaces.

Rather than granting administrator access to every authenticated user, I created dedicated groups in Microsoft Entra ID.

These groups are synchronized into Authentik and passed to Rancher through the OpenID Connect token. Users can then be assigned to specific Rancher roles based on their group membership.

Project and Namespace Scoping

One feature I particularly like about Rancher is the ability to grant permissions at the Project level.

Instead of giving users unrestricted access to the entire Kubernetes cluster, access can be limited to specific Projects and the Namespaces contained within them.

Secure kubeconfig Generation

An additional benefit of project-scoped access is that Rancher automatically limits the permissions available through generated kubeconfig files.

When a user downloads a kubeconfig from Rancher, the credentials embedded in that configuration inherit the permissions associated with the user’s assigned role.

As a result:

• Project members can only access resources inside authorized namespaces
• Cluster-wide administrative privileges are restricted to administrators
• Access control remains consistent between the Rancher UI and Kubernetes API

This significantly reduces the risk of granting excessive cluster permissions to application teams while maintaining a self-service experience.

Troubleshooting Certificate Validation

During the initial OpenID Connect configuration, I encountered certificate validation issues between Rancher and Authentik.

Since both applications are exposed through Cloudflare and use certificates issued through cert-manager and served by the Traefik ingress controller, certificate trust became one of the first areas to validate.

Rancher stores custom certificate authorities inside the cacerts setting located in the cattle-system namespace.

To verify whether Rancher is using a custom CA bundle, I checked the current configuration:

kubectl -n cattle-system get settings.management.cattle.io cacerts -o yaml

In my case, the setting contained a custom CA bundle that was no longer required. Because both Rancher and Authentik were already using publicly trusted certificates issued by Let’s Encrypt, Rancher could rely on the default system trust store instead of a custom CA chain.

To resolve the issue, I removed the custom certificate configuration by clearing the cacerts value.

After the change, Rancher automatically reverted to using the default trusted Certificate Authorities available inside the container image and was able to validate the Authentik endpoint correctly.

Deploying GitLab with Ansible

With Argo CD and Rancher now integrated into the centralized identity platform, the final application remaining was GitLab.

Unlike the previous integrations, this was not only an authentication project.

At the time, I was already running GitLab inside the lab environment, but the deployment had accumulated technical debt over time. The instance was running an older version, database and application components were deployed together, and upgrading the platform was becoming increasingly difficult.

Rather than performing an in-place upgrade, I decided to rebuild the platform from scratch.

The migration had three primary goals:

• Upgrade GitLab to the latest supported version
• Separate PostgreSQL from the GitLab application server
• Fully automate the deployment using Ansible

This would provide a cleaner architecture and make future upgrades significantly easier to perform.

Target GitLab Architecture

Instead of running all components on a single virtual machine, the new deployment separates the database and application layer.

This separation follows a more production-oriented design while making database maintenance and future migrations considerably simpler.

Since the rest of the platform is already managed through Ansible, I wanted the GitLab deployment to follow the same approach.

The migration workflow is fully automated with Ansible and includes:

1. Linux server preparation
2. PostgreSQL installation and configuration
3. PostgreSQL validation
4. GitLab installation
5. Container Registry configuration
6. OIDC integration with Authentik

Rather than executing commands manually, each phase is validated before the next one begins. This provides repeatable deployments and immediate feedback whenever something fails.

The code for this part can be found here:

https://github.com/heyimjustalex/medium/tree/main/Article_4/gitlab_ansible/playbooks

Deployment Workflow

The entire deployment is fully automated with Ansible and executed in several stages. Each playbook is responsible for a specific part of the migration process and includes validation checks before moving to the next phase.

1. Bootstrap SSH Access

Creates the Ansible user on all target hosts and installs the SSH public key used by the automation platform.

ansible-playbook playbooks/01-upload-ssh-keys.yml

2. Harden SSH Configuration

Applies basic SSH hardening by disabling root logins, reducing the attack surface, and validating the final SSH configuration before reloading the service.

ansible-playbook playbooks/02-harden-ssh-config.yml

3. Prepare Linux Hosts

Performs baseline operating system configuration on both GitLab and PostgreSQL servers. This includes package installation, host configuration, DNS entries, swap removal, and system tuning.

ansible-playbook playbooks/03-prepare-linux.yml

4. Deploy PostgreSQL

Installs PostgreSQL 17 on the dedicated database server, creates the GitLab database, provisions the GitLab service account, configures network access, and validates the deployment.

ansible-playbook playbooks/04-install-postgres.yml --ask-vault-pass

5. Validate Database Connectivity

Before deploying GitLab, this playbook verifies connectivity between the GitLab server and PostgreSQL. Additional checks validate DNS resolution, database authentication, UTF-8 encoding, and PostgreSQL version compatibility.

ansible-playbook playbooks/05-test-postgres.yml --ask-vault-pass

6. Deploy GitLab CE

Installs GitLab CE, configures the external URL, connects GitLab to the dedicated PostgreSQL instance, performs the initial reconfiguration, and validates that the deployment completed successfully. The playbook also verifies that the selected GitLab and PostgreSQL versions are compatible.

ansible-playbook playbooks/06-install-gitlab.yml --ask-vault-pass

7. Configure GitLab OpenID Connect

Integrates GitLab with Authentik by configuring OpenID Connect authentication inside GitLab. The playbook updates the GitLab configuration and validates that the OIDC provider is available after reconfiguration.

ansible-playbook playbooks/07-oidc-conf.yml --ask-vault-pass

8. Enable GitLab Container Registry

Enables the integrated container registry and publishes it through a dedicated endpoint. This registry is later used by Kubernetes workloads and CI/CD pipelines to store container images.

ansible-playbook playbooks/08-configure-container-registry.yml

Preparing the Authentik Application for GitLab

Before executing the OpenID Connect configuration playbook, an application object must first be created in Authentik.

I created a dedicated application called:

gitlab

together with an OAuth2/OpenID Provider configured for GitLab.

The redirect URI used by GitLab is:

https://gitlab.<your-domain>/users/auth/openid_connect/callback

After creating the provider, Authentik generates a Client ID and Client Secret which are later consumed by the Ansible automation. The credentials are stored securely inside the encrypted vault.yml file rather than being committed directly into the repository.

vault_gitlab_oidc_client_id: <secret>
vault_gitlab_oidc_client_secret: <secret>

This allows the full GitLab OIDC configuration to remain reproducible while keeping sensitive authentication data protected through Ansible Vault.

At this stage, the platform is ready for the final step: integrating GitLab with Authentik and validating the complete authentication flow through Microsoft Entra ID.

Summary

At this stage, the platform finally has a centralized identity layer.

Throughout this article, I:

• Federated Authentik with Microsoft Entra ID
• Configured Microsoft Graph permissions and identity synchronization
• Established a consistent authorization model based on Entra ID groups
• Integrated Argo CD with Authentik using OpenID Connect
• Implemented custom Property Mappings to expose only application-specific groups
• Integrated Rancher with Authentik and configured role-based access control
• Leveraged Rancher Projects and Namespaces to scope user access and generated kubeconfigs
• Deployed a new GitLab environment using Ansible
• Migrated GitLab to a dedicated architecture with separate PostgreSQL and application servers (remember to migrate the ArgoCD repository and change ArgoCD settings if you migrate GitLab instance like me. Also use a GitLab service account for this.)
• Enabled GitLab Container Registry
• Integrated GitLab with Authentik using OpenID Connect

Users authenticate using Microsoft Entra ID, Multi-Factor Authentication is enforced centrally, and application access is managed through groups synchronized into Authentik.

Perhaps more importantly, onboarding and offboarding users is now significantly simpler. Instead of managing accounts and permissions separately in every application, access can be controlled from a single location using Entra ID groups and Authentik.

However, there was still one component operating outside of this architecture.

Remote access to the platform was still provided through OpenVPN using LDAP authentication. While functional, it did not support the same Single Sign-On experience, centralized authorization model, or identity-based access controls available to the rest of the platform.

In the next article, I’ll replace OpenVPN with NetBird and extend the same identity model to the network layer.

We’ll deploy a self-hosted NetBird environment, integrate it with Authentik, synchronize users and groups from Microsoft Entra ID, and enable Single Sign-On for VPN access.

In the first article of this series, I focused on building the Kubernetes foundation. Starting with two Proxmox hosts, I designed the platform architecture and used Ansible to deploy a highly available RKE2 cluster running entirely on self-managed infrastructure.

In the second article, I introduced centralized secret management using Azure Key Vault and deployed External Secrets Operator and Argo CD. At that point, the cluster had all the core building blocks required for a modern platform: Kubernetes, GitOps, and centralized secret management.

However, installing Argo CD is only the beginning. The real challenge is defining how infrastructure, platform services, monitoring, and application workloads are organized, deployed, and managed over time. As the number of services grows, a GitOps repository can quickly become difficult to maintain without a clear structure and deployment strategy.

In this article, I’ll focus on the GitOps architecture that powers the platform. I’ll explain how I organize the repository using the Apps-of-Apps pattern, Argo CD Projects, and Sync Waves to create clear boundaries between infrastructure, platform services, monitoring, and applications. The repository is structured around dedicated workload layers such as 10-infra, 15-platform, 20-monitoring, and 30-apps, each managed by its own Argo CD root application.

Once the GitOps structure is in place, I’ll start deploying the first production workloads through Argo CD. This includes infrastructure services such as cert-manager, MetalLB, Traefik, ExternalDNS, Longhorn, Velero, Rancher, and Cloudflared, followed by platform services including Homepage, Authentik, Wiki.js, and GitLab Runner.

By the end of this article, the platform will be fully managed through GitOps. New infrastructure components, platform services, and applications can be deployed by committing changes to Git, with Argo CD continuously reconciling the desired state of the cluster.

The whole code for this article is avaliable here.

Connecting Argo CD to the GitOps Repository

In the previous article, I installed Argo CD and connected it to Azure Key Vault through External Secrets Operator, but I didn’t explain how to connect it to the GitLab instance. Before Argo CD can deploy anything it first needs access to the Git repository that contains the desired state of the platform.

The first step is creating a dedicated service account for Argo CD in GitLab.

After creating the service account, add it as a member of the GitOps repository with read access permissions.

I prefer using a dedicated account rather than personal credentials because repository access becomes independent of individual users and can be rotated without affecting platform administrators.

The service account only requires read access to the GitOps repository. After creating the account, I generated a Personal Access Token and stored both the username and token in Azure Key Vault:

argocd-gitlab-username
argocd-gitlab-token

Since I already knew that Authentik would become the central identity provider for the platform, I also prepared the future OIDC integration at this stage. The Authentik OIDC client credentials were generated and added to Azure Key Vault even though Authentik itself had not yet been deployed:

argocd-authentik-client-id
argocd-authentik-client-secret

The OIDC client will be later created inside Authentik, but I had already reserved the corresponding secret names in Azure Key Vault to avoid changing the Argo CD configuration later.

Finally, I stored the initial Argo CD administrator password hash. Unlike most applications where a plaintext password is provided and hashed internally, Argo CD expects the bcrypt hash itself inside the secret.

argocd-admin-password-bcrypthash

The resulting Azure Key Vault entries looked as follows:

argocd-admin-password-bcrypthash
argocd-gitlab-username
argocd-gitlab-token
argocd-authentik-client-id
argocd-authentik-client-secret

The following variables map Azure Key Vault secret names to the Argo CD Ansible deployment:

akv_argocd_admin_password_key: argocd-admin-password-bcrypthash
akv_gitlab_username_key: argocd-gitlab-username
akv_gitlab_token_key: argocd-gitlab-token
akv_authentik_client_id_key: argocd-authentik-client-id
akv_authentik_client_secret_key: argocd-authentik-client-secret

Instead of creating repository credentials manually through the Argo CD UI, the playbook (number 15 here) uses External Secrets Operator to retrieve the GitLab username and token from Azure Key Vault and generate a repository secret automatically:

- name: Create ExternalSecret for argocd-secret
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: external-secrets.io/v1
          kind: ExternalSecret
          metadata:
            name: argocd-secret
            namespace: "{{ argocd_namespace }}"
          spec:
            refreshInterval: 5m

            secretStoreRef:
              kind: ClusterSecretStore
              name: "{{ eso_cluster_secret_store_name }}"

            target:
              name: argocd-secret
              creationPolicy: Merge

            data:
              - secretKey: admin.password
                remoteRef:
                  key: "{{ akv_argocd_admin_password_key }}"

              - secretKey: dex.authentik.clientId
                remoteRef:
                  key: "{{ akv_authentik_client_id_key }}"

              - secretKey: dex.authentik.clientSecret
                remoteRef:
                  key: "{{ akv_authentik_client_secret_key }}"

At the same time, another ExternalSecret populates the main argocd-secret object. Besides the administrator password, it also provisions the future OIDC credentials that Argo CD will later use when Authentik is deployed:

data:
  - secretKey: admin.password
    remoteRef:
      key: "{{ akv_argocd_admin_password_key }}"
  - secretKey: dex.authentik.clientId
    remoteRef:
      key: "{{ akv_authentik_client_id_key }}"
  - secretKey: dex.authentik.clientSecret
    remoteRef:
      key: "{{ akv_authentik_client_secret_key }}"

Although Authentik was not available yet, this meant that the Argo CD deployment was already prepared for a future migration to OIDC-based authentication.

Once synchronized, External Secrets Operator generates the required Kubernetes Secrets automatically. Because the repository Secret is labeled as an Argo CD repository Secret, Argo CD immediately discovers it and registers the repository without requiring any manual configuration.

To verify that synchronization completed successfully:

kubectl -n argocd get externalsecret

You should also see the generated repository Secret:

kubectl -n argocd get secret gitops-repo

After synchronization finishes, the repository should appear in the Argo CD UI:

Settings
└── Repositories

The repository should be displayed with a successful connection status, confirming that Argo CD can authenticate and pull manifests directly from Git.

At this point Argo CD is fully connected to the GitOps repository and ready to manage the entire platform. The next step is designing the repository structure that will allow infrastructure, platform services, monitoring stacks, and application workloads to be deployed independently while remaining part of a single GitOps workflow.

We can log in with the bootstrap/break-glass account to the ArgoCD now.

Designing the GitOps Repository Structure

With the repository connected, Argo CD can finally start deploying workloads. Before creating any applications, however, I wanted to establish a repository structure that would remain maintainable as the platform grows.

One common mistake in GitOps repositories is placing everything into a single directory and allowing Argo CD to deploy every manifest it finds. While this works initially, it quickly becomes difficult to manage once infrastructure services, monitoring stacks, platform components, and application workloads start accumulating.

To avoid that problem, I separated the repository into several logical layers:

0-argocd/
├── 0-bootstrap/
├── 10-projects/
└── 20-applications/
10-infra/
15-platform/
20-monitoring/
30-apps/

Each directory represents a different responsibility within the platform:

- 0-argocd -Argo CD configuration and bootstrap

- 10-infra -Infrastructure services required by the cluster

- 15-platform -Shared platform services

- 20-monitoring -Monitoring and observability stack

- 30-apps -Business and application workloads

At the top level, I use the Apps-of-Apps pattern. Rather than creating every Argo CD Application manually, a small bootstrap application creates a set of root applications that are responsible for discovering workloads in each layer.

The deployment flow looks like this:

gitops-bootstrap
        │
        ▼
argocd-projects
        │
        ▼
argocd-root-applications
        │
        ├──────────┬────────────┬─────────┐
        ▼          ▼            ▼         ▼
      Infra     Platform    Monitoring   Apps

The first root application deploys all Argo CD Projects:

metadata:
  name: argocd-projects
spec:
  source:
    path: 0-argocd/10-projects

The second root application deploys all workload categories:

metadata:
  name: argocd-root-applications
spec:
  source:
    path: 0-argocd/20-applications

Each root application then recursively discovers individual application.yaml files within a dedicated directory.

For example, the infrastructure layer is configured as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: infra
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: infra
  source:
    repoURL: https://gitlab.<your-domain>/cloud-team/lab-infra/gitops.git
    targetRevision: main
    path: 10-infra
    directory:
      recurse: true  
      include: "*/application.yaml"
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

This means that any folder under 10-infra containing an application.yaml file automatically becomes part of the infrastructure layer managed by Argo CD.

As a result, onboarding a new service becomes extremely simple:

10-infra/
└── 150-new-service/
    ├── application.yaml
    ├── values.yaml
    └── config/

After committing the changes, Argo CD automatically discovers the new application and reconciles it without any additional manual configuration.

I also use dedicated Argo CD Projects to enforce separation between different workload categories:

infra
platform
monitoring
apps

This provides a clean security boundary between infrastructure administrators, platform operators, and application teams while still keeping everything managed through a single GitOps repository.

The final result is a repository structure that scales naturally. Infrastructure, platform services, monitoring stacks, and application workloads can evolve independently while still being deployed through a single GitOps workflow managed entirely by Argo CD.

The next step is defining deployment ordering so that infrastructure services are installed before platform workloads, monitoring components, and applications.

Controlling Deployment Order with Sync Waves

With the repository structure in place, the next challenge is deployment ordering.

Not all components of the platform can be deployed simultaneously. Certain services depend on others being available first. For example, certificates require cert-manager, ingress resources require Traefik, and workload namespaces often depend on infrastructure components already existing.

To solve this problem, I use Argo CD Sync Waves.

A Sync Wave is simply an annotation that tells Argo CD in which order resources should be synchronized. Applications with lower values are deployed first, while higher values are deployed later.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "20"

This allows the entire platform to be deployed automatically while still respecting dependencies between components.

The deployment flow in my environment looks as follows:

-500  Argo CD Projects
-300  Root Applications
  20  cert-manager
  25  ClusterIssuer
  30  MetalLB
  35  MetalLB Configuration
  40  Traefik
  45  ExternalDNS Secrets
  50  ExternalDNS
  55  Argo CD Ingress
  60  Rancher
  65  Longhorn
  70  Velero
  75  Cloudflared
 500+ Platform Services
1000+ Monitoring
1500+ Applications

The infrastructure layer is deployed first because nearly every other workload depends on it.

For example, cert-manager is installed before the Let’s Encrypt ClusterIssuer:

cert-manager
    ↓
ClusterIssuer
    ↓
Certificates

The same approach is used throughout the platform. Longhorn and Velero are deployed only after networking and ingress components are available, while platform services are deployed only after the infrastructure layer becomes healthy.

This ordering also simplifies troubleshooting. If a workload fails to deploy, Argo CD clearly shows the synchronization stage where the deployment stopped, making dependency-related issues significantly easier to identify.

An additional benefit of Sync Waves is that new services can be inserted into the deployment chain without modifying existing applications. For example, when I introduced Authentik, I simply assigned higher Sync Waves than the infrastructure components:

505  Homepage
505  Authentik PostgreSQL
510  Authentik
510  WikiJS PostgreSQL
515  WikiJS
540  GitLab Runner

This ensures that all required infrastructure components are already running before platform services begin deployment.

As a result, a fresh Argo CD bootstrap can deploy the entire platform from an empty cluster without any manual intervention. Argo CD automatically processes applications in the correct order, turning the Git repository into the single source of truth not only for configuration, but also for deployment orchestration.

The target outcome of our deployment is an architecture like this.

Infrastructure and Platform Workloads

With the GitOps repository structure, the Secrets added to the the Azure Key Vault (I’ve omitted this part, you can inspect the ExternalSecrets objects here to know which secrets to create) and deployment ordering in place, the platform can start deploying actual workloads.

I divided the environment into four logical layers:

10-infra      → Cluster infrastructure
15-platform   → Shared platform services
20-monitoring → Observability stack
30-apps       → Application workloads

The infrastructure layer contains all components required to operate the cluster itself.

cert-manager

The first infrastructure component is cert-manager. It is responsible for automatically issuing and renewing TLS certificates across the platform.

Certificates are issued through Let’s Encrypt using the DNS-01 challenge. A Cloudflare API token is stored in Azure Key Vault, synchronized into the cluster using External Secrets Operator, and then consumed by a ClusterIssuer used by cert-manager.

This allows certificates to be provisioned and renewed automatically without any manual intervention.

MetalLB

Since this platform runs on-premises rather than in a public cloud, there is no cloud provider load balancer available.

MetalLB solves this problem by providing LoadBalancer functionality inside the Kubernetes cluster. It is responsible for allocating IP addresses from predefined address pools and exposing services to the network.

Without MetalLB, services using:

type: LoadBalancer

would remain in a pending state because Kubernetes would have no mechanism for assigning external IP addresses.

The deployment is split into two parts:

MetalLB Core
        │
        ▼
MetalLB Configuration

This ensures that the controller is available before address pools and advertisements are applied.

Traefik

Traefik acts as the platform’s ingress controller.

It is responsible for:

• Terminating TLS connections
• Routing traffic to workloads
• Loading TLS certificates generated by cert-manager
• Processing IngressRoute resources
• Providing a single entry point for all HTTP and HTTPS traffic

Most platform services are exposed through Traefik.

A typical traffic flow looks like:

Internet
    │
    ▼ 
 Traefik
    │
    ▼
IngressRoute
    │
    ▼
 Service
    │
    ▼
   Pod

The below diagram shows how the above components cooperate to provision the IPs, routing and certificates with the help of Cloudflare (where I manage my domain).

ExternalDNS

Managing DNS records manually quickly becomes difficult as new services are deployed. ExternalDNS solves this problem by watching Kubernetes resources and automatically creating DNS records in Cloudflare.

The Cloudflare token is retrieved from Azure Key Vault through External Secrets Operator, meaning no DNS credentials are hardcoded inside Git repositories.

A typical flow looks as follows:

IngressRoute
        │
        ▼
ExternalDNS
        │
        ▼
Cloudflare DNS Record

As soon as a new application exposes a hostname, ExternalDNS automatically creates the corresponding DNS entry.

Cloudflared

The platform also integrates with Cloudflare using Cloudflare Tunnel.

Instead of exposing services directly to the internet through port forwarding, Cloudflared creates an outbound tunnel from the Kubernetes cluster to Cloudflare.

The resulting architecture looks like this:

Internet
    │
    ▼
Cloudflare
    │
    ▼
Cloudflare Tunnel
    │
    ▼
Cloudflared
    │
    ▼
 Traefik
    │
    ▼
Application

This approach provides several advantages:

• No inbound ports need to be opened
• Public services can remain behind Cloudflare
• Additional Cloudflare security controls can be applied
• Origin infrastructure remains hidden from the internet

To improve availability, Cloudflared runs multiple replicas.

Rancher

Rancher provides a centralized management interface for the Kubernetes environment.

While most platform operations are performed through GitOps and Argo CD, Rancher remains useful for:

• Cluster administration
• Node management
• Workload inspection
• Troubleshooting
• User access management (kubeconfigs)

The initial Rancher bootstrap password is also synchronized from Azure Key Vault using External Secrets Operator, ensuring that administrative credentials never need to be stored in Git.

Longhorn

Longhorn provides distributed block storage for the platform.

Persistent Volumes are automatically replicated across cluster nodes, allowing workloads to survive individual node failures.

Longhorn is used by multiple platform components that require persistent storage.

The deployment also creates recurring backup jobs which automatically export Longhorn backups to Azure Blob Storage according to a defined retention policy.

Remember to disable this setting in Longhorn, so that Longhorn replicas can be distributed across different zones and failure domains. In my case, the Kubernetes nodes that are hosted on a different bare metal Proxmox node have different zone labels (this configuration is part of Ansible scripts).

Velero

Velero complements Longhorn by providing Kubernetes-level backup and disaster recovery capabilities.

While Longhorn focuses on persistent volume data, Velero captures cluster resources such as:

• Deployments
• Services
• ConfigMaps
• Secrets
• Namespaces
• Persistent Volume metadata

Backups are automatically scheduled and stored in Azure Blob Storage (if you configure proper Azure Key Vault credentials for the Blob Storage).

This provides an additional recovery layer beyond volume-level backups and allows complete workload restoration if necessary.

With the infrastructure layer deployed, the cluster now has everything required to host higher-level services:

Certificates
Networking
Ingress
Load Balancing
DNS Automation
Cloudflare Integration
Persistent Storage
Backups
Cluster Management

The next layer builds on top of this foundation and introduces shared platform services such as Homepage, Authentik, Wiki.js, and GitLab Runner.

Deploying Platform Services

With the infrastructure layer fully operational, the cluster now provides everything required to host higher-level workloads.

At this point, the platform has:

• TLS certificate automation
• DNS automation
• Ingress routing
• Cloudflare integration
• Persistent storage
• Backup and recovery
• Cluster management

The next layer consists of platform services. These workloads are not strictly required for Kubernetes itself to function, but they provide capabilities that make the platform usable by administrators and application teams.

In my environment, the platform layer currently contains:

Homepage
Authentik
Wiki.js
GitLab Runner

Unlike the infrastructure layer, these workloads depend on services that were deployed previously. For example, they consume TLS certificates issued by cert-manager, use storage provisioned by Longhorn, and expose endpoints through Traefik and Cloudflare.

Homepage

Homepage acts as the landing page for the platform.

As the number of services increased, keeping track of URLs, dashboards, and administrative interfaces becomes increasingly difficult. Homepage provides a centralized dashboard where platform services can be organized and accessed from a single location.

The service itself is deployed through Argo CD as a standard Helm chart and follows the same GitOps workflow as every other workload in the environment.

Authentik

One of the primary goals of this platform is centralized identity and access management.

While Argo CD was already prepared for OIDC integration in the previous article, the identity provider itself had not yet been deployed.

To solve this problem, I chose Authentik.

Up to this point, most services could be accessed using local accounts, bootstrap credentials, or service-specific authentication mechanisms. While this approach works for a small number of services, it quickly becomes difficult to manage as the platform grows.

I wanted a single identity provider that could handle authentication for platform services such as Argo CD, Rancher, Grafana, NetBird, and future applications deployed to the cluster. After evaluating several options, I chose Authentik.

The main reasons were:

• Fully self-hosted
• Native OIDC and SAML support
• Flexible group-based authorization
• Straightforward Kubernetes deployment
• Excellent integration with cloud-native platforms

Since Argo CD had already been prepared for future OIDC integration in the previous section, introducing Authentik later would not require significant modifications to the GitOps structure.

Authentik PostgreSQL

Authentik requires a PostgreSQL database backend.

Rather than deploying the database manually, I created a dedicated Argo CD application responsible for PostgreSQL deployment.

The PostgreSQL deployment uses Longhorn-backed persistent storage, ensuring that database data survives pod restarts, node maintenance events, and cluster upgrades.

As with every other platform component, the deployment is managed entirely through Argo CD and GitOps.

Authentik Secrets from Azure Key Vault

One of the advantages of introducing Azure Key Vault and External Secrets Operator earlier in the series is that new workloads automatically inherit the same secret management model.

Authentik requires a secret key used internally for cryptographic operations. Rather than storing this value in Git or directly in Kubernetes, it is maintained in Azure Key Vault and synchronized through External Secrets Operator.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: authentik-secret
  namespace: authentik
spec:
  refreshInterval: 10m
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-kv
  target:
    name: authentik-secret
    creationPolicy: Owner
  data:
    - secretKey: AUTHENTIK_SECRET_KEY
      remoteRef:
        key: authentik-secret-key

Authentik Application

Like every other platform service, Authentik is deployed through its own Argo CD Application.

The application definition follows the same structure used throughout the repository:

15-platform/
└── authentik/
    ├── application.yaml
    ├── values.yaml
    └── config/

The Application references the Authentik Helm chart, applies platform-specific configuration, and enables automated synchronization.

Because it lives in the 15-platform layer, it is automatically discovered by the root Platform application without requiring any additional Argo CD configuration.

Adding a new platform service follows exactly the same workflow:

Create a folder
Commit changes
Push to Git
Argo CD deploys automatically

Authentik IngressRoute

Once Authentik is deployed, it needs to be exposed externally.

Like the rest of the platform, incoming traffic is routed through Traefik, but in this case we use cloudflared to expose it via the Cloudflare Tunnel.

The resulting request path looks like this:

Internet
     │
     ▼
Cloudflare
     │
     ▼
Cloudflare Tunnel
     │
     ▼
Cloudflared
     │
     ▼
Traefik
     │
     ▼
Authentik

The manifest should look like this:

# For cloudflare DNS record should not be created with extenal-dns, so here we remove external dns annotation
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: authentik-cloudflared
  namespace: authentik
spec:
  entryPoints:
    - cloudflared
  routes:
    - match: Host(`auth.<your_domain>`)
      kind: Rule
      services:
        - name: authentik-server
          port: 80
  # 
  # tls:
  #   secretName: authentik-tls

When using cloudflared the TLS will be provided by the Cloudflare, we don’t need to configure the cert-manager Certificate object for it. The entryPoint configuration corresponds to this Traefik configuration:

In the Cloudflare Tunnel options you need to point your domain you want to publish the Authentik instance with to the Traefik service like this:

This approach avoids exposing Kubernetes nodes directly to the internet while still allowing secure access to the Authentik portal.

The only public endpoint visible to users is the Authentik hostname:

https://auth.<your_domain>.com

However, after opening the link you should be prompted with registration of your first account. This is not safe, as you expose the installation to the Internet. What I did instead, I’ve disabled the published route in Cloudflare and I’ve installed it by exposing it to the VPN network and getting the certificate for a different domain (authentik.<your_domain>.com).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: authentik-tls
  namespace: authentik
spec:
  secretName: authentik-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    # auth.<your_domain> has cert managed by cloudflare tunnel
   # - auth.<your_domain>
   # this domain is only for internal VPN-based access. Using it for SSO will fail.
   - authentik.<your_domain>
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: authentik
  namespace: authentik
  annotations:
    external-dns.alpha.kubernetes.io/target: ingress.<your_domain>
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`authentik.<your_domain>`)
      kind: Rule
      services:
        - name: authentik-server
          port: 80
  tls:
    secretName: authentik-tls

After creating the initial account via the VPN network the instance should be operational and can be exposed via the cloudflared. For additional security layer you can configure Cloudflare WAF, but you should allowlist traffic originating from the cluster egress IP address as the ‘managed-challenges’ can break the OIDC integrations we will make in the future articles.

Future Integrations

At this stage, Authentik is deployed and operational, but it is not yet serving as the central identity provider for the platform.

The goal is to gradually migrate services toward centralized authentication using OIDC.

Several services have already been prepared for that transition. For example, Argo CD is configured to retrieve future Authentik OIDC client credentials from Azure Key Vault through External Secrets Operator. This allows OIDC integration to be enabled later without changing the overall GitOps architecture.

In future articles, I will integrate Authentik with:

• Argo CD
• Rancher
• Grafana
• NetBird

The end goal is to have a single authentication platform providing centralized identity management, group-based authorization, and single sign-on across the entire Kubernetes environment.

In the previous article, I focused on building the Kubernetes foundation itself. Starting with two bare-metal servers running Proxmox, I designed the cluster architecture, created the virtual machines, and used Ansible to automate the deployment of a highly available RKE2 cluster. By the end of that process, I had a working Kubernetes environment with three control-plane nodes and four worker nodes running entirely on infrastructure that I manage myself.

Getting the cluster running was only the first step though. While Kubernetes can store secrets natively, that approach quickly becomes difficult to manage as the platform grows. Database passwords, API keys, certificates, and other sensitive configuration end up scattered across multiple namespaces and applications. I wanted a single place to manage secrets while keeping them outside the cluster itself.

In this article, I’ll integrate the platform with Azure Key Vault.

Even though the Kubernetes cluster is running on-premises, Azure Key Vault provides a reliable and centralized way of storing secrets. The goal is to use Azure as the source of truth for sensitive data, while still allowing Kubernetes workloads to consume those secrets using standard Kubernetes mechanisms.

All of the code for this solution is available in this GitHub repository.

Azure Blob Storage for Terraform State

Before creating Azure Key Vault, I wanted to solve a different problem first: Terraform state management.

For small projects, storing the Terraform state file locally is usually sufficient. As the number of resources grows, however, relying on a local terraform.tfstate file becomes less practical. The state file contains information about all managed resources, and losing it can make future changes significantly more difficult.

To avoid that situation, I decided to store Terraform state in Azure Blob Storage. This gives me a central location for Terraform state while also making it easier to manage future Azure resources from multiple workstations if needed.

There is one small catch though: before Terraform can use Azure Blob Storage as a backend, the storage account itself must already exist. This creates a bootstrap problem because Terraform cannot store its state in a backend that hasn’t been created yet.

To solve this, I use a separate Terraform configuration with local state. Its sole purpose is to create the storage account and container that will later be used as the remote backend. Once the backend exists, all subsequent Terraform projects can switch to using Azure Blob Storage.

Creating a Service Principal for Terraform

I don’t want Terraform to authenticate using my personal Azure account. Instead, I create a dedicated Service Principal that Terraform will use whenever it interacts with Azure resources.

First, I select the correct subscription:

az account set  --subscription <SUBSCRIPTION-ID>
az account show

Next, I create a Service Principal with Contributor permissions scoped to the resource group that will contain the Kubernetes-related resources:

az ad sp create-for-rbac \
  --name <TERRAFORM_SP_NAME> \
  --role Contributor \
  --scopes \
  "/subscriptions/<SUBSCRIPTION-ID>/resourceGroups/<RESOURCE-GROUP-NAME>"

In the Azure CLI output, the password field represents the Service Principal client secret.

{
  "appId": "xxxxxxxxxxx",
  "displayName": "<TERRAFORM_SP_NAME>",
  "password": "xxxxxxxxxxx",
  "tenant": "xxxxxxxxxxxxxxxx"
}

These credentials should be treated as sensitive information and never committed to the source control system.

Granting Access to Azure Blob Storage

The Contributor role is not sufficient for managing Terraform state stored in Azure Storage. Terraform also needs permissions to read and write blobs.

To allow that, I assign the Storage Blob Data Contributor role to the Service Principal:

az role assignment create \
  --assignee "<APPLICATION_CLIENT_ID>" \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<SUBSCRIPTION-ID>/resourceGroups/<RESOURCE-GROUP-NAME>"

While setting up Azure Key Vault, I ran into a limitation that wasn’t immediately obvious. Even though the Terraform Service Principal had the Contributor role on the resource group, Terraform was unable to manage certain Key Vault access configurations.

The reason is that the Contributor role allows creating and modifying Azure resources, but it does not allow creating or modifying Azure RBAC role assignments. This becomes important when Terraform is expected to manage access permissions for Azure Key Vault.

To fix this, I granted the Terraform Service Principal the User Access Administrator role on the resource group:

az role assignment create \
  --assignee <TERRAFORM_SERVICE_PRINCIPAL_ID> \
  --role "User Access Administrator" \
  --scope "/subscriptions/<SUBSCRIPTION-ID>/resourceGroups/<RESOURCE-GROUP-NAME>"

This additional permission allows Terraform to create and manage Azure RBAC assignments, which is required when provisioning identities and assigning Key Vault permissions through Infrastructure-as-Code.

Bootstrapping the Backend Storage Account

The next step is creating the storage account that will hold Terraform state.

This Terraform configuration intentionally uses the default local backend because its only responsibility is creating the remote backend infrastructure itself.

The storage account is configured with several security controls:

• TLS 1.2 enforced
• Public blob access disabled
• Blob versioning enabled
• Soft-delete policies enabled
• Network restrictions allowing access only from approved IP ranges
• Private container for Terraform state files

The most important resource in the configuration is the storage account itself:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.2"
    }

    http = {
      source  = "hashicorp/http"
      version = "~> 3.0"
    }

  }
}

provider "azurerm" {
  features {}
}


data "http" "my_ip" {
  url = "https://api.ipify.org?format=json"
}


# ----------------------------
# Resource group
# ----------------------------
data "azurerm_resource_group" "rg" {
  name = "<your_domain>-k8s-cluster_RG"
}

# ----------------------------
# Locals (naming + IP)
# ----------------------------
locals {
  storage_account_name = "<your_domain>blobtfstate"
  my_ip                = "${jsondecode(data.http.my_ip.response_body).ip}/30"


  allowed_ips = [
    "insert-on-prem-cluster-IP/30",

  ]
  ip_rules = concat([local.my_ip], local.allowed_ips)
}

# ----------------------------
# Storage account (TF STATE)
# ----------------------------
resource "azurerm_storage_account" "tfstate" {
  name                = local.storage_account_name
  resource_group_name = data.azurerm_resource_group.rg.name
  location            = data.azurerm_resource_group.rg.location

  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"

  min_tls_version                 = "TLS1_2"
  allow_nested_items_to_be_public = false

  blob_properties {
    versioning_enabled = true

    delete_retention_policy {
      days = 30
    }

    container_delete_retention_policy {
      days = 7
    }

  }

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
    ip_rules       = local.ip_rules
  }

  tags = {
    purpose     = "terraform-state"
    environment = "<your_domain>-prd"
    managed_by  = "terraform"
  }
}

# ----------------------------
# Container
# ----------------------------
resource "azurerm_storage_container" "tfstate" {
  name                  = "tfstate"
  storage_account_id    = azurerm_storage_account.tfstate.id
  container_access_type = "private"
}

The variables.tf contents:

variable "subscription_id" {}
variable "client_id" {}
variable "client_secret" {
  sensitive = true
}
variable "tenant_id" {}

Authenticating Terraform

Let’s use environment variables for authentication to Azure Cloud:

export ARM_CLIENT_ID="..."
export ARM_CLIENT_SECRET="..."
export ARM_TENANT_ID="..."
export ARM_SUBSCRIPTION_ID="..."

Deploying the Backend

Once everything is configured, the bootstrap infrastructure can be deployed as any other Terraform project:

terraform init
terraform plan
terraform apply -auto-approve

After the deployment completes, the Azure Storage Account and the tfstate container become available. From this point onward, all future Terraform projects can use Azure Blob Storage as their remote backend instead of relying on local state files.

In the next step, I’ll use this newly created backend to provision and manage Azure Key Vault itself.

Initializing Azure Key Vault

In the previous section, I created the Azure Blob Storage account that will be used as the Terraform backend. The storage account was bootstrapped using local state because, at that point, the remote backend did not exist yet.

Now that the backend is available, I can create the actual Terraform project that will manage the Azure resources used by the Kubernetes platform. The code is available here. Instead of mixing everything into the bootstrap code, I prefer keeping the backend provisioning separate from the infrastructure it supports. The first project creates the Terraform backend, while the second project becomes the source of truth for platform-related Azure resources.

This project will manage Azure Key Vault and it will also contain additional components such as backup storage used by Longhorn and Velero.

Configuring the Remote Backend

Unlike the bootstrap project, this Terraform configuration stores its state in Azure Blob Storage.

The backend configuration points Terraform to the storage account and container created in the previous step:

00-provider-backend.tf

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.2"
    }

    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.0"
    }

    http = {
      source  = "hashicorp/http"
      version = "~> 3.0"
    }
  }

  backend "azurerm" {
    resource_group_name  = "<your_domain>-k8s-cluster_RG"
    storage_account_name = "<your_domain>blobtfstate"
    container_name       = "tfstate"
    key                  = "terraform.tfstate"
  }
}

provider "azurerm" {
  features {}
}
Execute 

terraform init

From this point onward, Terraform state is stored remotely inside Azure Blob Storage rather than on my workstation. This gives me versioned and centrally managed state while avoiding the risk of losing the state file after rebuilding my local machine.

Creating the Key Vault

The next step is provisioning Azure Key Vault itself and the blob storage for our future backups for our services (Velero, Longhorn and GitLab).

My goal is to use Azure Key Vault as the central source of truth for secrets consumed by Kubernetes. Instead of storing passwords, tokens, and API keys directly inside Kubernetes manifests, everything will be stored and managed in Azure.

The Key Vault itself is intentionally configured with several security controls enabled by default:

10-key-vault.tf

data "azurerm_resource_group" "rg" {
  name = var.resource_group_name
}

data "azurerm_client_config" "current" {}

data "http" "my_ip" {
  url = "https://api.ipify.org?format=json"
}

locals {
  current_ip = jsondecode(data.http.my_ip.response_body).ip

  tags = {
    environment = var.environment
    managed_by  = "terraform"
  }
}

resource "azurerm_key_vault" "main" {
  name                = var.key_vault_name
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name

  tenant_id = data.azurerm_client_config.current.tenant_id

  sku_name = "standard"

  enable_rbac_authorization = true

  purge_protection_enabled   = true
  soft_delete_retention_days = 90

  public_network_access_enabled = true

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"

    ip_rules = concat(
      [local.current_ip],
      var.allowed_ips
    )
  }

  tags = local.tags
}

A few settings are worth highlighting here:

• Soft Delete is enabled, allowing accidentally deleted secrets to be recovered.
• Purge Protection prevents permanent deletion of vault contents before the retention period expires.
• Network ACLs default to deny access and only allow explicitly defined IP ranges.
• Azure Services bypass is enabled so Azure-managed services can still communicate with the vault when required.

For this lab setup, I keep public access enabled while restricting access to specific IP ranges. In a larger production environment, I would likely replace this with private endpoints.

Creating Backup Storage for Longhorn, Velero and GitLab

20-blob-storage.tf

Rather than creating separate storage accounts for each service, I decided to provision a dedicated backup storage account and isolate the workloads using separate blob containers.

The storage account follows the same security principles as the Terraform backend:

• TLS 1.2 enforced
• Public blob access disabled
• Storage account firewall enabled
• Default deny network policy
• Soft-delete enabled
• Blob versioning enabled

To avoid accidentally locking myself out during deployments, Terraform automatically adds the public IP address of the workstation executing the deployment together with a list of predefined trusted IP ranges. This allows Terraform to manage the storage account while still keeping network access restricted.

locals {
  dynamic_runner_ip = "${jsondecode(data.http.my_ip.response_body).ip}/30"
  ip_rules = concat(
    [local.dynamic_runner_ip],
    var.allowed_ips
  )
}

# ----------------------------
# Backup storage account
# ----------------------------
resource "azurerm_storage_account" "backup" {
  name                = var.backup_storage_account_name
  resource_group_name = data.azurerm_resource_group.rg.name
  location            = data.azurerm_resource_group.rg.location

  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"

  min_tls_version                 = "TLS1_2"
  allow_nested_items_to_be_public = false

  blob_properties {
    versioning_enabled = true

    delete_retention_policy {
      days = var.blob_soft_delete_days
    }

    container_delete_retention_policy {
      days = var.container_soft_delete_days
    }
  }

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
    ip_rules       = local.ip_rules
  }

  tags = local.tags
}

# ----------------------------
# Containers
# ----------------------------
resource "azurerm_storage_container" "longhorn" {
  name                  = "longhorn"
  storage_account_id    = azurerm_storage_account.backup.id
  container_access_type = "private"
}

resource "azurerm_storage_container" "velero" {
  name                  = "velero"
  storage_account_id    = azurerm_storage_account.backup.id
  container_access_type = "private"
}


# ----------------------------
# Lifecycle / retention rules
# ----------------------------
resource "azurerm_storage_management_policy" "backup_retention" {
  storage_account_id = azurerm_storage_account.backup.id

  # ----------------------------
  # Longhorn
  # ----------------------------
  rule {
    name    = "longhorn-retention"
    enabled = true

    filters {
      prefix_match = ["longhorn/"]
      blob_types   = ["blockBlob"]
    }

    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than = var.tier_to_cool_after_days_since_modification_greater_than

        delete_after_days_since_modification_greater_than = var.longhorn_retention_days
      }

      snapshot {
        delete_after_days_since_creation_greater_than = var.longhorn_retention_days
      }

      version {
        delete_after_days_since_creation = var.longhorn_retention_days
      }
    }
  }

  # ----------------------------
  # Velero
  # ----------------------------
  rule {
    name    = "velero-retention"
    enabled = true

    filters {
      prefix_match = ["velero/"]
      blob_types   = ["blockBlob"]
    }

    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than = var.tier_to_cool_after_days_since_modification_greater_than

        delete_after_days_since_modification_greater_than = var.velero_retention_days
      }

      snapshot {
        delete_after_days_since_creation_greater_than = var.velero_retention_days
      }

      version {
        delete_after_days_since_creation = var.velero_retention_days
      }
    }
  }
  rule {
    name    = "gitlab-retention"
    enabled = true

    filters {
      prefix_match = ["gitlab/"]
      blob_types   = ["blockBlob"]
    }

    actions {
      base_blob {
        tier_to_cool_after_days_since_modification_greater_than = var.tier_to_cool_after_days_since_modification_greater_than

        delete_after_days_since_modification_greater_than = var.gitlab_retention_days
      }

      snapshot {
        delete_after_days_since_creation_greater_than = var.gitlab_retention_days
      }

      version {
        delete_after_days_since_creation = var.gitlab_retention_days
      }
    }
  }
}

resource "azurerm_storage_container" "gitlab" {
  name                  = "gitlab"
  storage_account_id    = azurerm_storage_account.backup.id
  container_access_type = "private"
}

To prevent backup storage from growing indefinitely, Azure Storage Management Policies are also configured. These lifecycle rules automatically move older backups to the Cool tier and delete them after a configurable retention period. This approach keeps storage costs predictable while ensuring that backup data is automatically cleaned up according to the retention requirements defined for each backup type.

Deploying the Azure resources

With the remote backend configured and the Key Vault definition in place, the deployment process is identical to any other Terraform project:

terraform init
terraform plan
terraform apply -auto-approve

After the deployment completes, Azure Key Vault becomes available and Terraform automatically stores its state in the Blob Storage backend created earlier.

By managing backup storage through Terraform, both the infrastructure and retention policies remain fully version-controlled and reproducible alongside the rest of the platform configuration.

Creating a Service Principal for Kubernetes

At this point, the Key Vault exists, but Kubernetes still has no way of accessing it. Since my cluster is running on-premises rather than in Azure, I can’t rely on Managed Identities. Instead, I’ll use a dedicated Azure Service Principal that the External Secrets Operator (ESO) will use to authenticate against Azure Key Vault.

The idea is simple: Kubernetes gets its own identity in Azure with only the permissions it needs. In this case, ESO only needs to read secrets from the vault, so there’s no reason to grant broader permissions.

To create the Service Principal, I ran:

az ad sp create-for-rbac \
  --name <KUBERNETES_SP_NAME> \
  --skip-assignment

The command returns a client ID, client secret, and tenant ID:

{
  "appId": "...",
  "displayName": "<KUBERNETES_SP_NAME>",
  "password": "...",
  "tenant": "..."
}

These credentials will later be stored securely using Ansible Vault and deployed into Kubernetes as a Secret consumed by External Secrets Operator.

Granting Access to Azure Key Vault

Next, I retrieved the Azure Resource ID of the Key Vault:

KV_ID=$(az keyvault show \
  --name <KEYVAULT_NAME> \
  --resource-group <YOUR_RESOURCE_GROUP>\
  --query id -o tsv)

With the vault ID available, I assigned the Key Vault Secrets User role to the newly created Service Principal:

az role assignment create \
  --assignee "<SERVICE_PRINCIPAL_ID>" \
  --role "Key Vault Secrets User" \
  --scope $KV_ID

Importantly, it does not grant permissions to:

• Create new secrets
• Modify existing secrets
• Delete secrets
• Manage Key Vault configuration

As a result, even if the credentials used by Kubernetes were somehow compromised, the attacker would still be limited to reading secrets already present in the vault.

This keeps the Kubernetes integration relatively simple while maintaining a clear separation between secret consumers and secret administrators.

Granting Administrative Access for Secret Management

While Kubernetes only needs read access, I still need a way to create and manage secrets in Azure Key Vault myself.

To retrieve my Azure user object ID, I used:

az ad signed-in-user show --query id -o tsv

Then I assigned myself the Key Vault Secrets Officer role:

az role assignment create \
  --assignee <YOUR_USER_ID> \
  --role "Key Vault Secrets Officer" \
  --scope $KV_ID

This role allows me to:

• Create secrets
• Update secret values
• Delete secrets
• Manage secret versions

At this stage, Azure is fully prepared. The Key Vault has been provisioned, access controls have been configured, and Kubernetes has a dedicated identity that can retrieve secrets. The next step is moving back to the cluster and installing External Secrets Operator, which will use these credentials to synchronize secrets from Azure Key Vault into Kubernetes.

Configuring External Secrets Operator with Ansible

Now that Azure Key Vault exists and the Kubernetes Service Principal has access to read secrets from it, I can move back to the Ansible configuration.

The goal here is to install External Secrets Operator into the RKE2 cluster and configure it to use Azure Key Vault as the external secret backend. I want to keep this consistent with the rest of the platform, so ESO will also be installed from Ansible rather than manually with helm install.

First, I added the ESO-related variables to group_vars/all.yml:

# External Secrets Operator

eso_chart_version: "2.7.0"
eso_namespace: "external-secrets-system"
azure_tenant_id: "xxx"
azure_keyvault_name: "xxx"

I pinned the chart version on purpose. I do not want the playbook to always install whatever happens to be the latest version at the time of execution. Version pinning makes the deployment more predictable and easier to troubleshoot later.

The namespace is also defined as a variable because I prefer keeping platform components separated from application workloads. In this case, ESO will be installed into its own namespace called external-secrets-system.

The Azure tenant ID and Key Vault name are not secrets by themselves, so I keep them in all.yml.

The actual sensitive values are stored in Ansible Vault:

ansible-vault edit group_vars/vault.yml

Inside the vault file, I store the sensitive values required by the following automation steps. The file is encrypted with Ansible Vault, so it can be committed to the repository safely as long as the Vault password itself is stored separately and never committed.

Depending on which integrations are enabled, this file contains credentials for External Secrets Operator, GitOps repository access, LDAP authentication, Argo CD administrator credentials, and optionally OIDC credentials for Authentik integration.

# Azure Key Vault authentication used by External Secrets Operator

azure_eso_client_id: "..."
azure_eso_client_secret: "..."

# LDAP integration

ldap_bind_dn: "..."
ldap_bind_password: "..."

# GitOps repository access (read only service account to your gitops repo)

gitlab_username: "..."
gitlab_token: "..."

# Argo CD administrator password (bcrypt hash)

argocd_admin_password_hash: "..."

# Optional OIDC / Authentik integration

authentik_client_id: "..."
authentik_client_secret: "..."

The complete playbook looks as follows:

- name: Install External Secrets Operator
  hosts: masters[0]
  become: true
  remote_user: cloud
  gather_facts: false

  vars_files:
    - ../group_vars/all.yml
    - ../group_vars/vault.yml

  tasks:

    ############################################################
    # Namespace
    ############################################################

    - name: Create ESO namespace
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: "{{ eso_namespace }}"

    ############################################################
    # Helm repo
    ############################################################

    - name: Add External Secrets Helm repo
      kubernetes.core.helm_repository:
        name: external-secrets
        repo_url: https://charts.external-secrets.io

    ############################################################
    # Install ESO
    ############################################################

    - name: Install External Secrets Operator
      kubernetes.core.helm:
        kubeconfig: "{{ rke2_kubeconfig }}"
        name: external-secrets
        chart_ref: external-secrets/external-secrets
        chart_version: "{{ eso_chart_version }}"
        release_namespace: "{{ eso_namespace }}"
        create_namespace: true
        wait: true
        update_repo_cache: true
        values:
          installCRDs: true

    ############################################################
    # Wait for CRDs
    ############################################################

    - name: Wait for ClusterSecretStore CRD
      command: >
        {{ rke2_kubectl }} get crd clustersecretstores.external-secrets.io
      register: crd_result
      retries: 30
      delay: 10
      changed_when: false
      until: crd_result.rc == 0

    - name: Wait for ClusterSecretStore API registration
      command: >
        {{ rke2_kubectl }} api-resources
      register: api_resources
      retries: 30
      delay: 5
      changed_when: false
      until: "'clustersecretstores' in api_resources.stdout"

    ############################################################
    # Azure SP bootstrap Secret
    ############################################################

    - name: Create Azure authentication secret
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: azure-kv-auth
            namespace: "{{ eso_namespace }}"
          type: Opaque
          stringData:
            ClientID: "{{ azure_eso_client_id }}"
            ClientSecret: "{{ azure_eso_client_secret }}"

    ############################################################
    # ClusterSecretStore
    ############################################################

    - name: Create Azure Key Vault ClusterSecretStore
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: external-secrets.io/v1
          kind: ClusterSecretStore
          metadata:
            name: azure-kv
          spec:
            provider:
              azurekv:
                tenantId: "{{ azure_tenant_id }}"
                vaultUrl: "https://{{ azure_keyvault_name }}.vault.azure.net/"

                authSecretRef:
                  clientId:
                    name: azure-kv-auth
                    namespace: "{{ eso_namespace }}"
                    key: ClientID

                  clientSecret:
                    name: azure-kv-auth
                    namespace: "{{ eso_namespace }}"
                    key: ClientSecret

    ############################################################
    # Wait for ESO deployment
    ############################################################

    - name: Wait for ESO deployment
      kubernetes.core.k8s_info:
        kubeconfig: "{{ rke2_kubeconfig }}"
        api_version: apps/v1
        kind: Deployment
        namespace: "{{ eso_namespace }}"
        name: external-secrets
      register: eso_status
      retries: 30
      delay: 10
      until: >
        eso_status.resources | length > 0 and
        eso_status.resources[0].status.availableReplicas | default(0) | int > 0

    ############################################################
    # Verify ClusterSecretStore
    ############################################################

    - name: Verify ClusterSecretStore
      command: >
        {{ rke2_kubectl }} get clustersecretstore azure-kv -o yaml
      register: css_verify
      changed_when: false

    - name: Display ClusterSecretStore
      debug:
        var: css_verify.stdout

    ############################################################
    # Show ESO version
    ############################################################

    - name: Get ESO deployment
      kubernetes.core.k8s_info:
        kubeconfig: "{{ rke2_kubeconfig }}"
        api_version: apps/v1
        kind: Deployment
        namespace: "{{ eso_namespace }}"
        name: external-secrets
      register: eso_deployment

    - name: Display ESO image version
      debug:
        msg: >
          ESO image:
          {{
            eso_deployment.resources[0].spec.template.spec.containers[0].image
          }}

The playbook installs External Secrets Operator, creates a dedicated namespace, deploys the required CRDs, creates a bootstrap Kubernetes Secret containing the Azure credentials, and configures a cluster-wide ClusterSecretStore pointing at Azure Key Vault.

One thing worth mentioning is the bootstrap secret. Since ESO needs credentials before it can access Azure Key Vault, the Service Principal credentials must initially exist as a regular Kubernetes Secret. This is the only Azure-related secret that needs to be created manually inside the cluster. Everything else can later be synchronized directly from Azure Key Vault.

After running the playbook, External Secrets Operator is installed and connected to Azure Key Vault. Kubernetes can now authenticate to Azure and retrieve secrets through the azure-kv ClusterSecretStore, which becomes the central integration point for all future workloads that require secret management.

Installing Argo CD with Azure Key Vault Integration

With Azure Key Vault and External Secrets Operator already deployed, the next step is installing Argo CD and bootstrapping the GitOps workflow.

To keep the platform fully reproducible, I install Argo CD using a dedicated Ansible playbook 15-install-argocd.yml.

Instead of manually configuring Argo CD through the UI, the playbook performs the entire deployment automatically.

- name: Install Argo CD with ESO-backed secrets, LDAP, Reloader and GitOps bootstrap
  hosts: masters[0]
  become: true
  gather_facts: false

  vars_files:
    - ../group_vars/all.yml

  vars:
    argocd_manifest_url: >-
      https://raw.githubusercontent.com/argoproj/argo-cd/{{ argocd_version }}/manifests/install.yaml

    # External Secrets
    eso_cluster_secret_store_name: azure-kv

    # Azure Key Vault secret names
    akv_argocd_admin_password_key: argocd-admin-password-bcrypthash

    akv_gitlab_username_key: argocd-gitlab-username
    akv_gitlab_token_key: argocd-gitlab-token

    akv_ldap_bind_dn_key: kubernetesldapacc-ldap-bind-dn
    akv_ldap_bind_password_key: kubernetesldapacc-ldap-bind-password

  tasks:

    ###########################################################################
    # Pre-flight checks
    ###########################################################################

    - name: Check ExternalSecret CRD exists
      command: kubectl get crd externalsecrets.external-secrets.io
      changed_when: false

    - name: Check ClusterSecretStore exists
      command: >
        kubectl get clustersecretstore.external-secrets.io
        {{ eso_cluster_secret_store_name }}
      changed_when: false

    ###########################################################################
    # Namespace
    ###########################################################################

    - name: Ensure Argo CD namespace exists
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: "{{ argocd_namespace }}"

    ###########################################################################
    # Install Argo CD upstream manifests
    ###########################################################################

    - name: Apply Argo CD upstream manifests
      command: >
        kubectl apply
        --server-side
        --force-conflicts
        -n {{ argocd_namespace }}
        -f {{ argocd_manifest_url }}

    - name: Wait for Argo CD deployments
      command: >
        kubectl -n {{ argocd_namespace }}
        wait deployment --all
        --for=condition=Available
        --timeout=300s
      changed_when: false

    ###########################################################################
    # Argo CD server parameters
    ###########################################################################

    - name: Configure Argo CD server insecure mode
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: argocd-cmd-params-cm
            namespace: "{{ argocd_namespace }}"
          data:
            server.insecure: "true"

    ###########################################################################
    # Argo CD service - ClusterIP only
    ###########################################################################

    - name: Configure argocd-server service as ClusterIP
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: Service
          metadata:
            name: argocd-server
            namespace: "{{ argocd_namespace }}"
          spec:
            type: ClusterIP
            selector:
              app.kubernetes.io/name: argocd-server
            ports:
              - name: http
                port: 80
                targetPort: 8080

    ###########################################################################
    # ExternalSecret: argocd-secret
    #
    # This merges values into the existing upstream argocd-secret.
    # Secrets come only from Azure Key Vault.
    ###########################################################################

    - name: Create ExternalSecret for argocd-secret
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: external-secrets.io/v1
          kind: ExternalSecret
          metadata:
            name: argocd-secret
            namespace: "{{ argocd_namespace }}"
          spec:
            refreshInterval: 1m

            secretStoreRef:
              kind: ClusterSecretStore
              name: "{{ eso_cluster_secret_store_name }}"

            target:
              name: argocd-secret
              creationPolicy: Merge

            data:
              - secretKey: dex.ldap.bindDN
                remoteRef:
                  key: "{{ akv_ldap_bind_dn_key }}"

              - secretKey: dex.ldap.bindPW
                remoteRef:
                  key: "{{ akv_ldap_bind_password_key }}"

              - secretKey: admin.password
                remoteRef:
                  key: "{{ akv_argocd_admin_password_key }}"


    ###########################################################################
    # ExternalSecret: GitOps repository secret
    #
    # Argo CD repository credential secret.
    # Username/password come only from Azure Key Vault.
    ###########################################################################

    - name: Create ExternalSecret for GitOps repository secret
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: external-secrets.io/v1
          kind: ExternalSecret
          metadata:
            name: gitops-repo
            namespace: "{{ argocd_namespace }}"
          spec:
            refreshInterval: 1m

            secretStoreRef:
              kind: ClusterSecretStore
              name: "{{ eso_cluster_secret_store_name }}"

            target:
              name: gitops-repo
              creationPolicy: Owner

              template:
                engineVersion: v2

                metadata:
                  labels:
                    argocd.argoproj.io/secret-type: repository

                data:
                  type: git
                  url: "{{ gitops_repo_url }}"
                  username: '{{ "{{ .username }}" }}'
                  password: '{{ "{{ .password }}" }}'

            data:
              - secretKey: username
                remoteRef:
                  key: "{{ akv_gitlab_username_key }}"

              - secretKey: password
                remoteRef:
                  key: "{{ akv_gitlab_token_key }}"

    ###########################################################################
    # Wait for ESO synchronization
    ###########################################################################

    - name: Wait for ExternalSecret argocd-secret to become Ready
      command: >
        kubectl -n {{ argocd_namespace }}
        wait externalsecret.external-secrets.io/argocd-secret
        --for=condition=Ready=True
        --timeout=300s
      changed_when: false

    - name: Wait for ExternalSecret gitops-repo to become Ready
      command: >
        kubectl -n {{ argocd_namespace }}
        wait externalsecret.external-secrets.io/gitops-repo
        --for=condition=Ready=True
        --timeout=300s
      changed_when: false

    - name: Verify argocd-secret exists
      command: kubectl -n {{ argocd_namespace }} get secret argocd-secret
      changed_when: false

    - name: Verify gitops-repo secret exists
      command: kubectl -n {{ argocd_namespace }} get secret gitops-repo
      changed_when: false

    ###########################################################################
    # Argo CD configuration with LDAP / Dex
    ###########################################################################

    - name: Configure Argo CD URL and Dex LDAP connector
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: argocd-cm
            namespace: "{{ argocd_namespace }}"
          data:
            url: "https://{{ argocd_hostname }}"
            admin.enabled: "true"
            sso.autoRedirect: "false"

            dex.config: |
              logger:
                level: info

              connectors:
              - type: ldap
                id: openldap
                name: OpenLDAP

                config:
                  host: "{{ ldap.host }}"
                  insecureNoSSL: true
                  insecureSkipVerify: true
                  startTLS: false

                  bindDN: "$dex.ldap.bindDN"
                  bindPW: "$dex.ldap.bindPW"

                  usernamePrompt: Username

                  userSearch:
                    baseDN: "{{ ldap.user_base_dn }}"
                    filter: "{{ ldap.user_filter }}"
                    username: "{{ ldap.user_attr }}"
                    idAttr: "{{ ldap.user_attr }}"
                    emailAttr: "{{ ldap.email_attr }}"
                    nameAttr: "{{ ldap.name_attr }}"

                  groupSearch:
                    baseDN: "{{ ldap.group_base_dn }}"
                    filter: "{{ ldap.group_filter }}"

                    userMatchers:
                    - userAttr: DN
                      groupAttr: "{{ ldap.group_attr }}"

                    nameAttr: cn

    ###########################################################################
    # Argo CD RBAC
    ###########################################################################

    - name: Configure Argo CD RBAC
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: argocd-rbac-cm
            namespace: "{{ argocd_namespace }}"
          data:
            policy.default: role:readonly
            scopes: '[groups,email]'
            policy.csv: |
              g, {{ argocd_admin_group }}, role:admin

    ###########################################################################
    # Reloader annotations
    #
    # These annotations are placed on Pod templates.
    # Patching Pod templates also triggers the initial rollout after config/secret changes.
    ###########################################################################

    - name: Add Reloader annotations to Argo CD deployments
      command: >
        kubectl -n {{ argocd_namespace }}
        patch deployment {{ item }}
        --type merge
        -p '{"spec":{"template":{"metadata":{"annotations":{"reloader.stakater.com/auto":"true","secret.reloader.stakater.com/reload":"argocd-secret,gitops-repo","configmap.reloader.stakater.com/reload":"argocd-cm,argocd-rbac-cm,argocd-cmd-params-cm"}}}}}'
      loop:
        - argocd-server
        - argocd-dex-server
        - argocd-repo-server
        - argocd-applicationset-controller
        - argocd-notifications-controller
      register: patch_deployments
      failed_when: false
      changed_when: "'patched' in patch_deployments.stdout or 'configured' in patch_deployments.stdout"

    - name: Add Reloader annotations to Argo CD application controller StatefulSet
      command: >
        kubectl -n {{ argocd_namespace }}
        patch statefulset argocd-application-controller
        --type merge
        -p '{"spec":{"template":{"metadata":{"annotations":{"reloader.stakater.com/auto":"true","secret.reloader.stakater.com/reload":"argocd-secret,gitops-repo","configmap.reloader.stakater.com/reload":"argocd-cm,argocd-rbac-cm,argocd-cmd-params-cm"}}}}}'
      register: patch_application_controller
      failed_when: false
      changed_when: "'patched' in patch_application_controller.stdout or 'configured' in patch_application_controller.stdout"

    ###########################################################################
    # Wait for Argo CD after rollout triggered by annotation/config changes
    ###########################################################################

    - name: Wait for argocd-server rollout
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout status deployment/argocd-server
        --timeout=300s
      changed_when: false

    - name: Wait for argocd-dex-server rollout
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout status deployment/argocd-dex-server
        --timeout=300s
      changed_when: false

    - name: Wait for argocd-repo-server rollout
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout status deployment/argocd-repo-server
        --timeout=300s
      changed_when: false

    ###########################################################################
    # Bootstrap GitOps
    #
    # This uses gitops-repo secret created by ESO.
    ###########################################################################

    - name: Bootstrap Argo CD from GitOps repository
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: argoproj.io/v1alpha1
          kind: Application
          metadata:
            name: gitops-bootstrap
            namespace: "{{ argocd_namespace }}"
          spec:
            project: default

            source:
              repoURL: "{{ gitops_repo_url }}"
              targetRevision: "{{ gitops_repo_revision | default('main') }}"
              path: 0-argocd/0-bootstrap
              directory:
                recurse: true

            destination:
              server: https://kubernetes.default.svc
              namespace: "{{ argocd_namespace }}"

            syncPolicy:
              automated:
                prune: true
                selfHeal: true

The playbook:

• Creates the Argo CD namespace
• Installs Argo CD from upstream manifests
• Configures Argo CD server settings
• Integrates Argo CD with Azure Key Vault through External Secrets Operator
• Creates the Git repository secret used by Argo CD
• Configures LDAP authentication through Dex (omit this step if you don’t want to authenticate with LDAP)
• Configures RBAC permissions
• Enables automatic reloads using Reloader
• Bootstraps GitOps applications from Gitlab

Note: Some older versions of ArgoCD may have a bug with LDAP password. If it contains $ sign then the LDAP integration will return an error. See this -> https://github.com/argoproj/argo-cd/issues/27803

To execute this script use:

ansible-playbook playbooks/15-install-argocd.yml \
  --ask-vault-pass

But you first have to fill in the variables and the vault.yml secrets that are used for this configuration.

Optional: Integrating Argo CD with Authentik via OIDC

While the current deployment uses either the built-in Argo CD administrator account or LDAP authentication through Dex, the platform is designed to support centralized authentication using Authentik in the future.

This optional playbook configures Argo CD to use Authentik as an external OpenID Connect (OIDC) provider. Client credentials are retrieved securely from Azure Key Vault through External Secrets Operator, and Dex is configured to trust Authentik for authentication.

The playbook:

• Creates additional External Secret mappings for Authentik OIDC credentials
• Retrieves the OIDC Client ID and Client Secret from Azure Key Vault
• Configures Dex to use Authentik as an identity provider
• Enables group synchronization using OIDC claims
• Maps Authentik groups directly to Argo CD roles through RBAC
• Restarts Dex and Argo CD to apply the new authentication configuration

At the time of writing, I continue using local administrator accounts and LDAP authentication while building the platform. Once Authentik itself is deployed and managed through Argo CD, I plan to switch all platform services to centralized OIDC authentication using Authentik as the single identity provider, but if you want to integrate with OIDC, use the below playbook.

- name: Install Argo CD with ESO-backed secrets, LDAP, Reloader and GitOps bootstrap
  hosts: masters[0]
  become: true
  gather_facts: false

  vars_files:
    - ../group_vars/all.yml

  vars:
    argocd_manifest_url: >-
      https://raw.githubusercontent.com/argoproj/argo-cd/{{ argocd_version }}/manifests/install.yaml

    # External Secrets
    eso_cluster_secret_store_name: azure-kv

    # Azure Key Vault secret names
    akv_argocd_admin_password_key: argocd-admin-password-bcrypthash

    akv_gitlab_username_key: argocd-gitlab-username
    akv_gitlab_token_key: argocd-gitlab-token

    akv_ldap_bind_dn_key: kubernetesldapacc-ldap-bind-dn
    akv_ldap_bind_password_key: kubernetesldapacc-ldap-bind-password
    akv_authentik_client_id_key: argocd-authentik-client-id
    akv_authentik_client_secret_key: argocd-authentik-client-secret

  tasks:

    ###########################################################################
    # Pre-flight checks
    ###########################################################################

    - name: Check ExternalSecret CRD exists
      command: kubectl get crd externalsecrets.external-secrets.io
      changed_when: false

    - name: Check ClusterSecretStore exists
      command: >
        kubectl get clustersecretstore.external-secrets.io
        {{ eso_cluster_secret_store_name }}
      changed_when: false

    ###########################################################################
    # Namespace
    ###########################################################################

    - name: Ensure Argo CD namespace exists
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: "{{ argocd_namespace }}"
    ###########################################################################
    # ExternalSecret: argocd-secret
    #
    # This merges values into the existing upstream argocd-secret.
    # Secrets come only from Azure Key Vault.
    ###########################################################################

    - name: Create ExternalSecret for argocd-secret
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: external-secrets.io/v1
          kind: ExternalSecret
          metadata:
            name: argocd-secret
            namespace: "{{ argocd_namespace }}"
          spec:
            refreshInterval: 5m

            secretStoreRef:
              kind: ClusterSecretStore
              name: "{{ eso_cluster_secret_store_name }}"

            target:
              name: argocd-secret
              creationPolicy: Merge

            data:
              - secretKey: dex.ldap.bindDN
                remoteRef:
                  key: "{{ akv_ldap_bind_dn_key }}"

              - secretKey: dex.ldap.bindPW
                remoteRef:
                  key: "{{ akv_ldap_bind_password_key }}"

              - secretKey: admin.password
                remoteRef:
                  key: "{{ akv_argocd_admin_password_key }}"

              - secretKey: dex.authentik.clientId
                remoteRef:
                  key: "{{ akv_authentik_client_id_key }}"

              - secretKey: dex.authentik.clientSecret
                remoteRef:
                  key: "{{ akv_authentik_client_secret_key }}"

    - name: Configure Argo CD URL and Dex LDAP connector
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: argocd-cm
            namespace: "{{ argocd_namespace }}"
          data:
            url: "https://{{ argocd_hostname }}"
            admin.enabled: "true"
            sso.autoRedirect: "false"
            dex.config: |
              connectors:
              - type: oidc

                id: authentik
                name: Authentik

                config:
                  issuer: https://auth.<your_domain>.pl/application/o/argocd-dex/

                  clientID: $dex.authentik.clientId
                  clientSecret: $dex.authentik.clientSecret

                  insecureEnableGroups: true

                  scopes:
                    - openid
                    - profile
                    - email
                    - groups

    - name: Configure Argo CD RBAC
      kubernetes.core.k8s:
        kubeconfig: "{{ rke2_kubeconfig }}"
        state: present
        definition:
          apiVersion: v1
          kind: ConfigMap
          metadata:
            name: argocd-rbac-cm
            namespace: "{{ argocd_namespace }}"
          data:
            policy.default: role:readonly

            scopes: '[groups,email]'


            policy.csv: |

              # ==========================================
              # Global
              # ==========================================
              g, argocd-root-admins, role:admin

              g, argocd-readonly, role:readonly

              # ==========================================
              # Applications
              # ==========================================

              g, argocd-apps-admins, role:apps-admin

              p, role:apps-admin, applications, get, apps/*, allow
              p, role:apps-admin, applications, sync, apps/*, allow
              p, role:apps-admin, applications, action/*, apps/*, allow
              p, role:apps-admin, logs, get, apps/*, allow

              # ==========================================
              # Platform
              # ==========================================

              g, argocd-platform-admins, role:platform-admin

              p, role:platform-admin, applications, get, platform/*, allow
              p, role:platform-admin, applications, sync, platform/*, allow
              p, role:platform-admin, applications, action/*, platform/*, allow
              p, role:platform-admin, logs, get, platform/*, allow

              # ==========================================
              # Infrastructure
              # ==========================================

              g, argocd-infra-admins, role:infra-admin

              p, role:infra-admin, applications, get, infra/*, allow
              p, role:infra-admin, applications, sync, infra/*, allow
              p, role:infra-admin, applications, action/*, infra/*, allow
              p, role:infra-admin, logs, get, infra/*, allow

              # ==========================================
              # Monitoring
              # ==========================================

              g, argocd-monitoring-admins, role:monitoring-admin

              p, role:monitoring-admin, applications, get, monitoring/*, allow
              p, role:monitoring-admin, applications, sync, monitoring/*, allow
              p, role:monitoring-admin, applications, action/*, monitoring/*, allow
              p, role:monitoring-admin, logs, get, monitoring/*, allow

              # ==========================================
              # Learning
              # ==========================================

              g, argocd-learning-admins, role:learning-admin

              p, role:learning-admin, applications, get, learning/*, allow
              p, role:learning-admin, applications, sync, learning/*, allow
              p, role:learning-admin, applications, action/*, learning/*, allow
              p, role:learning-admin, logs, get, learning/*, allow



    ###########################################################################
    # Restart ArgoCD after Dex/Auth changes
    ###########################################################################

    - name: Restart argocd-dex-server
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout restart deployment/argocd-dex-server
      changed_when: true

    - name: Wait for argocd-dex-server rollout
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout status deployment/argocd-dex-server
        --timeout=300s
      changed_when: false

    - name: Restart argocd-server
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout restart deployment/argocd-server
      changed_when: true

    - name: Wait for argocd-server rollout
      command: >
        kubectl -n {{ argocd_namespace }}
        rollout status deployment/argocd-server
        --timeout=300s
      changed_when: false

If you execute this script you should be able to log in with your OIDC provider (as long as you configured the proper secrets and links on the provider side). I’ll show the details of this in the next article.

Currently my architecture looks like this:

At this point, the platform has a solid foundation: a highly available Kubernetes cluster, centralized secret management with Azure Key Vault, automated secret synchronization through External Secrets Operator, and a fully bootstrapped GitOps workflow powered by Argo CD.

In the next article, I’ll continue building the platform by deploying Authentik as the central identity provider and integrating it with the rest of the environment. I’ll also start deploying additional workloads and platform services through GitOps to move closer to a fully self-managed Kubernetes platform.

Stay tuned. 🚀

What are we building?

Here’s what the final platform will include:

• High‑availability Kubernetes (RKE2 + kube‑vip)
• Storage (Longhorn + Velero for cluster state backups)
• Networking (Traefik + MetalLB + Cloudflare integration)
• Identity & access (Keycloak + LDAP)
• Observability (Prometheus, Grafana, Loki)

The goal is simple — to build something that feels like a real platform, not just a lab.

Prerequisites

To follow along, you’ll need:

• At least 2 bare‑metal servers
  (3 would be ideal — I’ll explain why later, but I’m working with 2)
• A Proxmox hypervisor installed on top of them, with disks configured for persistent storage
• Basic network connectivity, in my case exposed through OPNsense (or any equivalent edge router) with OpenVPN
• A domain name managed in a DNS provider (I am using Cloudflare in this case)
• Access to a public CA-compatible environment (for Let’s Encrypt auth-challenges to work)
• GitLab/GitHub account for the ArgoCD workloads
• Ansible and Git available on your Linux/Windows + WSL workstation

This series assumes you’re comfortable with basic Linux administration and Kubernetes concepts, but everything else will be explained as we go.

All the Ansible code used for this setup can be found here.

Architecture

At the bottom of the stack, everything starts with two physical machines. These are just regular bare-metal servers running Proxmox. There’s nothing particularly fancy here — no cloud, no managed services — just two hosts that provide the foundation for everything else.

On top of Proxmox, virtual machines are created to form the Kubernetes cluster. The setup includes three control-plane nodes and multiple worker nodes. The control-plane nodes are responsible for managing the cluster, while the worker nodes run the actual workloads. Even though there are three masters, they are spread across only two physical hosts, which is not perfect from a HA perspective.

All of these VMs are then joined into a Kubernetes cluster powered by RKE2. This layer acts as the orchestration engine — the “brain” of the whole system. It manages scheduling, networking, and the lifecycle of workloads. From the outside, this is the part people usually think of as “Kubernetes,” but in reality, it’s sitting on top of several other layers.

Above that, you have the actual Kubernetes objects — the things that make the cluster useful. In this diagram, that’s represented by simple pods running containers (for example, small Alpine containers). These are just placeholders, but they represent what the cluster is really meant to do: run application

To sum this up:

• The physical layer provides compute resources.
• The hypervisor (Proxmox) abstracts those resources into virtual machines.
• The VM layer defines your cluster topology — masters and workers.
• The Kubernetes layer (RKE2) turns those machines into a functioning cluster.
• And finally, the Kubernetes objects are where actual workloads live.

Let’s have a look at another diagram I’ve prepared.

Traffic enters the network through OPNsense, which acts as the edge router, and then gets forwarded into the internal network over the Proxmox bridges. From there, it reaches the Kubernetes cluster running on top of virtual machines.

On the control‑plane side, each master node runs the core components as static pods, along with a few supporting pieces deployed as DaemonSets. These are not vanilla Kubernetes installs, but come bundled and managed by RKE2, which simplifies how the cluster is bootstrapped and maintained.

Worker nodes are much simpler — they only run the data plane components (like networking) and the actual application workloads.

Overall, it’s the same setup as before, just showing more clearly how traffic flows through OPNsense into the cluster and how responsibilities are split between control plane and workers.

The Kubernetes cluster will be integrated with other services like CloudFlare for domain management, Let’s Encrypt for certificate rotation and an LDAP server that provides identity services. Let’s have a look at a diagram that shows these mechanisms from a different angle, although deploying most of these services will be covered in the next articles in this series

Traffic from the internet first hits OPNsense, which acts as the entry point and routes it into the internal network through Proxmox bridges. Inside the cluster, MetalLB assigns an IP to the Traefik service, making it reachable without relying on any cloud load balancer.

On top of that, TLS is fully automated. cert-manager requests certificates from Let’s Encrypt and uses Cloudflare DNS to prove domain ownership via DNS‑01 challenges. Once issued, certificates are stored inside the cluster and automatically picked up by Traefik.

Finally, there’s also an identity layer emerging in the background. A dedicated LDAP server provides a central source of identity for services that need authentication. This will later tie into tools like Keycloak/Authentik and other workloads, so authentication is handled consistently across the platform.

Overall, the setup keeps responsibilities clean:

OPNsense handles entry, MetalLB handles IPs, cert-manager handles certificates, and LDAP handles identity.

Design assumptions and trade-offs (and why they matter)

Using only 2 bare metals instead of 3

This setup runs on two physical machines, even though there are three control-plane nodes. In a perfect world, each master would sit on a separate host and each host in a separate zone of the provider to fully isolate failure domains. Here, I don’t have such capabilities.

In practice, this means the level of high availability depends on which of the bare metal hosts fails. If the host running two control-plane nodes (in my case this is proxmox-node0) goes down, the cluster loses quorum, since etcd requires a majority (n/2 + 1) to operate. At that point, the control plane becomes unavailable.

For this environment this trade-off is acceptable, but it’s important to be aware that:

multiple masters do not guarantee HA if they share the same underlying hardware.

Local drives instead of iSCSI

Storage is backed by local disks on each node, rather than a shared system like iSCSI or NFS. Disks are configured in RAID1 at the hypervisor level.

This is primarily a cost and simplicity trade-off:

• no external storage dependencies
• no network storage latency
• easier to manage in a small setup

The downside is tighter coupling between workloads and nodes, which will be later handled at the Kubernetes layer using Longhorn.

RKE2 instead of vanilla Kubernetes

The cluster is not built using kubeadm or a “from scratch” installation.

Instead, it uses RKE2, which bundles and manages core components internally. This reduces the amount of manual setup required and provides a consistent baseline, especially around security defaults and component lifecycle.

The trade-off is less visibility and control over individual components compared to a fully manual setup — but for this kind of platform build, it’s a pragmatic choice.

Separate identity layer (LDAP-first approach)

An external LDAP instance is used for providing the user’s identity. The LDAP server will be later integrated with Keycloak/Authentik so everything shares the same source of truth for users and access control.

No “production illusions” — but close enough when done right

This setup isn’t meant to pretend to be production-ready, but it’s also not far from it.

With a few adjustments, it could absolutely run production workloads. The two biggest improvements would be:

• distributing control-plane nodes across three independent physical hosts or availability zones
• implementing proper backup strategy (which we will do in the next articles)

Right now, the main limitation is the lack of physical separation between control-plane nodes. Fix that, and the cluster becomes much more resilient.

At that point, this stops being “just a lab” and starts looking a lot like a real production platform — just self‑managed instead of cloud‑managed.

Kubernetes Cluster Inventory

Let’s start by defining the below table with the names, IPs and the node parameters like vCPU, RAM and storage. We will also reserve IPs for some of the services. If you want to fill in your own table or copy from mine download file from here.

Creating Proxmox VMs

Let’s start by opening the Proxmox hypervisor and defining the templates that will be used for creating k8s nodes.

We’ll start with creating templates on each of the bare metals for both types of nodes (master and worker). For the node system I’ll use ubuntu-26–04, which needs to be downloaded within the Proxmox admin panel. Let’s click Create VM button in the top right corner and create the first template with parameters consistent with Figure 6 below.

Then, click PPM on the created object and convert it to the template.

Then we need to do it 3 more times (for both master and worker nodes, for each of the Proxmox machines). In the end we should see something like this:

Then we clone the templates to create the respective VMs.

It’s important to choose Full Clone and then enable the machine by Right-click -> Start.

In the end of the process we should see multiple VMs enabled on both bare metals.

VMs Ubuntu installation

After creating the VMs from templates, the next step is installing the Ubuntu Linux system on each node.
All machines should be configured consistently using the IP addresses and disk layouts matching what’s defined in Figure 4.

The installation itself is straightforward. Start by opening the web VNC console in Proxmox and connecting to the VM. From there, you can begin the Ubuntu installer and select the preferred system language.

Then we need to set up the network stack:

• subnet: 10.0.0.0/24 or according to your Proxmox virtual network configuration
• address: 10.0.0.x
• gateway: 10.0.0.1 or according to your Proxmox virtual network configuration
• nameservers: 8.8.8.8
• search domains: yourdomain.com

I also recommend disabling IPv6 if you don’t plan to use it.

Next, we need to configure the disk partitioning.

For worker nodes, it’s important to use the correct disks. The smaller disk should be used for the root (/) filesystem, while the larger disk should be left unmounted during installation. It will later be configured by the Ansible scripts and used by Longhorn.

For the master nodes, only a single disk is used, so the default configuration can be left unchanged.

Next, we configure the initial user and hostname for each machine.
The hostnames should match the naming defined in Figure 4, so everything stays consistent across the setup.

In my case, I use the username cloud. You can choose a different one if you prefer, but keep in mind that you’ll need to update the Ansible variables accordingly. Otherwise, the playbooks won’t be able to connect and execute properly.

At this stage, make sure to enable the OpenSSH server during installation.
You can do this by pressing Space when selecting optional components. This allows you to connect to the machine later using the username and password configured in the previous step.

Once the installation is complete, simply choose the option to reboot the system.

After the reboot, the system may prompt you to press ENTER to continue. This final step brings the machine to the login screen, where you can proceed with the configured credentials.

Repeat these steps for all machines.
Once everything is installed and configured, you should see all 7 VMs in the Proxmox panel, distributed across the two bare metal hosts as defined in Figure 4.

Once the base installation is done, we can move on to preparing the nodes for Kubernetes (RKE2).
Instead of configuring everything manually, I use Ansible to automate the process and keep the setup consistent across all machines.

Using Ansible for configuration

You can download the Ansible code that will be used for the RKE2 nodes configuration from here. You can clone the repository with git clone command and then move to the correct path with Ansible playbooks and configuration. To do it, run:

git clone https://github.com/heyimjustalex/medium.git
cd medium/Article_1/ansible_setup_script/

Prepare nodes

The hosts are defined in the inventory folder. If you used different IP addresses or hostnames than the ones shown in Figure 4, make sure to update them in the hosts.yml file.

all:
  children:
    masters:
      hosts:
        k8s-master01:
          ansible_host: 10.0.0.28
          k8s_zone: proxmox-node0
        k8s-master02:
          ansible_host: 10.0.0.29
          k8s_zone: proxmox-node0
        k8s-master03:
          ansible_host: 10.0.0.30
          k8s_zone: proxmox-node1
workers:
      hosts:
        k8s-worker01:
          ansible_host: 10.0.0.31
          k8s_zone: proxmox-node0
        k8s-worker02:
          ansible_host: 10.0.0.32
          k8s_zone: proxmox-node0
        k8s-worker03:
          ansible_host: 10.0.0.33
          k8s_zone: proxmox-node1
        k8s-worker04:
          ansible_host: 10.0.0.34
          k8s_zone: proxmox-node1

The k8s_zone variable is used later by Longhorn to distribute replicas across zones, which in this setup means across different bare metal hosts.

Before running the Ansible playbooks, make sure you can connect to all target nodes via SSH at least once using password authentication. This step is important because it adds each host’s fingerprint to your known_hosts file.

If you skip this, you’ll most likely run into an error like the one below:

fatal: [k8s-worker03]: FAILED! => {"msg": "Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this. Please add this host's fingerprint to your known_hosts file to manage this host."}

To avoid this, simply SSH into each node manually:

ssh cloud@10.0.0.x

For Ansible to execute tasks properly, it needs the ability to run commands with elevated privileges. This means configuring a user with passwordless sudo.

Log in to each server and add your user to the sudoers configuration. In this example, I’m using the cloud user:

echo "cloud ALL=(ALL) NOPASSWD: ALL" | sudo EDITOR='tee' visudo -f /etc/sudoers.d/99-cloud-ansible

This command:

• creates a new file in /etc/sudoers.d/
• adds a rule saying that user cloud can run any command as any user without being asked for a password

In practice, this means:

• Ansible can run sudo commands without breaking on password prompts
• no need to configure --ask-become-pass
• playbooks run fully unattended

In more restrictive environments, you might want to limit sudo privileges to specific commands instead of allowing full access. For simplicity and automation, I’m using full passwordless sudo here.

Generate SSH key

Next, we generate an SSH key pair to avoid using password-based authentication when running Ansible playbooks.

ssh-keygen -t ed25519 \
  -f ./keys/ansible_ed25519 \
  -C "ansible-managed-access"

This creates a new key pair that will be used by Ansible to securely connect to the target nodes.

Also we should change the permissions for the key

chmod 600 Article_1/ansible_setup_script/keys/ansible_ed25519

Generate RKE2 token

Before deploying the cluster, we need to generate a shared RKE2 token. This token is used when additional nodes join the cluster — both control plane and worker nodes use it to authenticate and connect to the initial server.

Because this token effectively grants access to the cluster, it should be treated as sensitive data and not stored in plain text. To keep things secure and still automation-friendly, I store it using Ansible Vault.

openssl rand -hex 32
ansible-vault create ./group_vars/vault.yml

Make sure to store the Ansible Vault password you created in a secure place, such as your password manager. You’ll need it whenever you run or modify the playbooks.

If you want to update the contents of the vault later, you can use:

ansible-vault edit ./group_vars/vault.yml

Deploy the RKE2 cluster using the Ansible scripts

Now that the base infrastructure is ready, we can move on to automating the cluster setup. The playbooks I use in this section can be found here.

Before running them, make sure to review the variables defined in group_vars/all.yml and adjust them to match your environment. At a minimum, you’ll likely want to update things like IPs, domains, or software versions depending on your setup.

1. The first playbook is used to bootstrap SSH access for Ansible across all nodes. Its purpose is to prepare the machines so that Ansible can connect securely using SSH keys instead of passwords.

It runs on all hosts and uses elevated privileges (become: true) to perform system-level changes.

To execute it, run:

ansible-playbook  playbooks/01-upload-ssh-keys.yml --ask-pass

2. The second script focuses on hardening the SSH configuration across all nodes. It applies a set of secure defaults aimed at reducing the attack surface of the system. Instead of leaving SSH in its default, more permissive state, the playbook enforces stricter access rules — such as disabling unnecessary features and tightening login behavior. It also validates the configuration before reloading the SSH service, which helps avoid accidental lockouts. To execute it, run:

ansible-playbook  playbooks/02-harden-ssh-config.yml

It’s worth noting that this step focuses on hardening the SSH service itself, but it does not restrict the privileges of the cloud user. In this setup, the user still has full sudo access without a password, which is convenient for automation but should be reconsidered in production environments.

3. The third script prepares all nodes so they meet the basic requirements expected by Kubernetes before installation.

It takes care of system-level configuration that Kubernetes relies on but that isn’t always correctly set by default. This includes adjusting how the OS handles memory and networking, loading required kernel modules, and applying sysctl settings so that cluster networking behaves properly. It also installs a small set of base packages needed for management and tooling, and ensures the environment is consistent across all machines.

Overall, this step removes a lot of the “it works on some nodes but not others” type of issues. To execute it, run:

ansible-playbook playbooks/03-prepare-nodes-for-k8s.yml

4. The fourth script bootstraps the first control plane node, effectively creating the Kubernetes cluster. It installs RKE2 on the initial master, applies the core configuration, and starts the API server. At this point, the cluster becomes accessible and ready to accept additional nodes.

The script also prepares kubectl access and performs basic validation to ensure the control plane is up and running before continuing with the rest of the deployment. To execute this step, run:

ansible-playbook playbooks/04-bootstrap-one-master-node.yml --ask-vault-pass

You can test whether the node has been initialized by logging into the VM and executing kubectl command (I use an alias k=kubectl).

5. The fifth script introduces kube-vip, which acts as a lightweight load balancer for the Kubernetes control plane.

In a bare-metal setup, there’s no cloud load balancer available to expose the API server behind a stable endpoint. kube-vip solves this by providing a virtual IP address that is shared across control plane nodes. At any point in time, one node “owns” the IP, and if that node goes down, the IP automatically moves to another one. This ensures that the Kubernetes API remains accessible.

This script installs kube-vip directly into the cluster as a DaemonSet and configures it to manage the control plane endpoint. From this point on, all nodes — and later also external tools — can communicate with the cluster using a single, stable IP address instead of pointing to a specific master node.

In practice, this is what enables a highly available control plane without relying on external load balancing components. To execute the playbook, run:

ansible-playbook  playbooks/05-upgrade-one-master-node-kube-vip.yml --ask-vault-pass

6. The sixth script brings the remaining control plane nodes into the cluster and turns the setup into a fully highly available control plane.

It installs RKE2 on the other master nodes and joins them to the cluster using the virtual IP provided by kube-vip, instead of connecting to a specific node. This is an important detail, because it ensures that all nodes rely on a stable entry point from the start.

It also propagates the kube-vip configuration to the new nodes, so each control plane node is capable of participating in managing the shared IP. After that, the services are restarted to apply everything consistently.

At this stage, the control plane is no longer a single-node setup. It becomes distributed, with multiple masters sharing responsibility and providing redundancy if one of them fails. The script also performs validation to ensure that the new nodes have successfully joined the cluster. Instead of just checking basic connectivity, it verifies that each node appears in the cluster and reaches a Ready state, confirming that it is fully operational.

ansible-playbook playbooks/06-install-and-join-other-master-nodes.yml --ask-vault-pass

We can also test manually whether the nodes joined the cluster, by logging in and executing relevant kubectl command:

kubectl get nodes

7. The seventh script is mainly about validating that kube-vip is working correctly and making sure the cluster consistently uses it as the control plane endpoint.

It starts by verifying that kube-vip is properly deployed across all control plane nodes and that it’s actually functional — including checking whether the Kubernetes API is reachable through the virtual IP. This confirms that the control plane load balancing is working as expected.

The second part of the script standardizes access to the cluster by updating all kubeconfig files on every master node. Instead of pointing to a specific node, they are rewritten to use the kube-vip address. This ensures that all interactions with the cluster always go through the highly available endpoint.

At this point, the setup is fully normalized — no matter which node you connect from, you always interact with the cluster through the same stable API endpoint. To execute the playbook, run:

ansible-playbook playbooks/07-validate-and-normalize-kube-vip.yml --ask-vault-pass

8. The eighth script installs RKE2 on the worker nodes and joins them to the cluster.

Similarly to the control plane nodes, workers connect to the cluster using the kube-vip address instead of a specific master node. This ensures that they rely on the same stable entry point and remain unaffected by individual control plane node failures.

Once installed, each worker registers itself with the cluster and becomes available to run workloads. The script also includes validation to confirm that the nodes have successfully joined and reached the Ready state.

At this point, the cluster becomes fully functional — with a distributed control plane and worker nodes ready to host applications. To execute the playbook, run:

ansible-playbook playbooks/08-install-rke-on-worker-nodes-and-join.yml --ask-vault-pass

9. The ninth script installs a set of additional utilities that make working with the cluster more practical on a daily basis.

It ensures that crictl is available and properly configured to interact with the container runtime used by RKE2. This is useful for low-level debugging and inspecting containers directly on the nodes. It also installs Helm on the control plane nodes, which is commonly used for deploying applications into the cluster.

At this point, the cluster is not only up and running, but also equipped with the basic tooling needed for troubleshooting, management, and application deployment. To execute this playbook, run:

ansible-playbook playbooks/09-install-other-utilities.yml

Before executing the next playbook it’s worth briefly explaining what Longhorn is.

Longhorn is a distributed block storage system designed for Kubernetes. It allows you to attach persistent volumes to workloads while replicating data across multiple nodes. In practice, it gives you highly available storage without relying on external SAN or cloud solutions, which makes it a very natural fit for a bare-metal setup like this one.

10. The tenth script prepares dedicated disks on the worker nodes so they can be used by Longhorn.

It ensures that each node has a properly configured storage device, formats it if needed, and mounts it under a consistent path. The configuration is made persistent, so the disks are automatically mounted after reboot.

The script also performs validation to confirm that the disk is available and correctly mounted before moving forward. This step ensures that all worker nodes provide reliable storage for Longhorn, which is later used to back Kubernetes volumes.

Note that this is why the larger disk was not used for the operating system on the worker nodes during the Proxmox setup, but instead reserved specifically for Longhorn storage.

To execute the playbook, run:

ansible-playbook playbooks/10-prepare-disks-for-longhorn.yml

11. The eleventh script introduces a simple but important concept — topology-aware storage using zones.

In this setup, the two Proxmox hosts act as separate failure domains. Each one is treated as its own “zone”, and nodes are labeled accordingly. This allows Longhorn to understand which nodes are physically separated and distribute replicas across them.

In practice, this means that when Longhorn creates a volume, it doesn’t store all replicas on nodes running on the same bare metal. Instead, it spreads them across both hosts. As a result, even if one entire Proxmox node goes down, the data is still available from the other one.

The script applies these topology labels to all Kubernetes nodes based on values defined in the inventory. Once applied, the cluster becomes aware of node placement, which enables Longhorn to make smarter decisions about data replication and improve overall resilience. To execute this playbook, run:

ansible-playbook playbooks/11-topology-labeling-nodes-for-longhorn.yml

12. The 12th playbook focuses on making DNS resolution and image pulls more predictable across the cluster.

It replaces systemd-resolved with a static DNS configuration, so all nodes use explicitly defined resolvers. It also updates CoreDNS, which handles both internal service discovery and external DNS queries inside the cluster, ensuring that external requests are forwarded reliably.

In addition, it configures containerd registry endpoints to make image pulls more consistent and prepares the system for pulling from different registries. Finally, it restarts the components and verifies that image downloads are working. To execute this playbook, run:

ansible-playbook playbooks/12-enforce-coredns-config.yml

13. The last, 13th playbook is essentially a resilience test for the whole cluster.

Instead of just checking whether the installation finished successfully, it verifies that the environment can survive controlled reboots without losing availability. It starts with a pre-flight validation to confirm that the cluster is healthy before any disruption is introduced. From there, it checks whether key configuration elements — especially Longhorn disk mounts and utility setup — are still present before rebooting anything.

The reboot itself is done in a rolling manner. Worker nodes are restarted first, because they do not affect etcd quorum, and each one is validated after coming back online. Only after that does the script move on to the control plane nodes, rebooting them one at a time to avoid breaking quorum. During this phase, it also verifies that kube-vip continues to expose the Kubernetes API through the virtual IP, which is critical for control plane availability.

Finally, the playbook performs a cluster-wide post-flight validation. It confirms that all nodes returned to the Ready state, kube-vip is healthy, the virtual IP is owned by exactly one master, and Longhorn storage is still mounted on the worker nodes. In other words, this is not just a reboot script — it is a practical validation that the cluster behaves like a highly available platform even after node restarts. To execute the validation playbook, run:

ansible-playbook playbooks/13-final-tests-reboot-and-validate.yml

Final step

As a final step, we copy the kubeconfig file from the control plane node to the local machine. This file contains the credentials and endpoint needed to interact with the cluster using kubectl.

scp cloud@10.0.0.28:/etc/rancher/rke2/rke2.yaml ~/.kube/config
chmod 600 ~/.kube/config

In this setup, the Ansible playbooks already configure the cluster to use the kube‑vip address for control plane communication, so no manual changes should be required.

You can still open the file and verify the API server endpoint if needed — for example, to confirm that it points to the kube‑vip address (in my case 10.0.0.41) rather than a specific node:

vim ~/.kube/config

After this, you should be able to manage the cluster directly from your local environment using kubectl (assuming you are connected to the cluster network, for example via VPN).

Summary

In this first part, we went from bare metal to a fully functional Kubernetes cluster built with RKE2 and automated entirely with Ansible.

Instead of treating Kubernetes as a single component, the focus here was on building a solid foundation — understanding how each layer fits together, from physical infrastructure and virtualization to control plane design, networking, and storage prerequisites.

Along the way, we introduced key concepts that make this setup feel much closer to a real platform than a simple lab:

• a highly available control plane using kube-vip instead of external load balancers
• automated provisioning with Ansible, ensuring consistency across all nodes
• topology-aware storage design that prepares the cluster for resilient Longhorn deployments
• and a validation approach that actually tests real-world behavior, like node reboots and failover

What we ended up with is not just a running cluster, but a platform-ready baseline that is predictable, reproducible, and aligned with real-world deployments.

In the next part, we’ll start building on top of this foundation by adding the key platform components: authentication, ArgoCD for GitOps, the Rancher management layer, and Longhorn for distributed storage.