In this lab — building directly on the work I did in Homelab #17 — I closed that gap by implementing a production-grade automated backup and disaster recovery pipeline using restic, AWS S3, and Terraform, and systemd-based automation. A backup you have never restored is not a backup. It is a hypothesis — and a very risky one to boot.

A quick overview of the tools involved on top of the previous lab’s used Proxmox, HAProxy, and Keepalived:

• restic is a modern backup tool that handles deduplication, compression, and client-side encryption in a single binary. Every snapshot is encrypted before it leaves your machine — AWS cannot read your data. Only you can, with the password you hold.
• AWS S3 is the offsite backup destination. With versioning, cross-region replication, and S3 Object Lock enforcing write-once immutability, even ransomware operating with stolen AWS credentials cannot delete a protected backup during its retention window.
• Terraform defines the entire AWS infrastructure — S3 buckets, IAM user, replication rules, bucket policies — as code. The whole setup is reproducible from scratch with a single terraform apply
• systemd timers handle backup scheduling. Unlike cron, they integrate with journalctl for persistent logging and handle missed runs automatically via Persistent=true

For those interested, here is a “brief” diagram of what this homelab was to look like:

Onwards, to glory!

Section I: Terraforming the AWS Infrastructure

As systems theorist and pediatrician John Gall put it in his 1975 classic Systemantics, “A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work.” Or as the legendary homelabber Nathan Pavlovsky put it, “How do you eat an elephant? One bite at a time.”

This is precisely what I needed to do with Terraform: to build things one part at a time. While I could have gone the “naive” route of having AI generate the code and just copying & pasting the entire setup and sloughing through debuggings, that would have done nothing for the learning process. So I went manually through the steps of creating the Terraform code, step by step. The steps were as follows:

Terraform Step #1: Initial File Setup

I got started by going into the AWS Terraform folder for Homelab #12 and copying and pasting the basic file structure of three Terraform files into my folder for this homelab. Specifically, I created the main.tf, output.tf, and variables.tf files. After reading the example files from Homelab #12 (which I was using as an example), I got to work by deleting unnecessary lines of text and making my own modifications. At this point in time, I had the following:

• variables.tf:

• output.tf:

• main.tf:

For the screenshot with the main.tf file, notice that I created a **second** AWS provider with an alias. This is necessary to be able to work with multiple regions simultaneously in AWS.

Wonderful. From there, I went through the standard Terraform deployment process with the terraform init, terraform plan, and terraform apply commands to confirm that the code was good at this time.

It was, and because there were no cloud resources specified, I didn’t create nor destroy any.

Terraform Step #2: Create Primary S3 Bucket

I then needed to create my initial S3 bucket for backup storage. For those who don’t know what it is, S3 is AWS’s object storage service that can store basically any type of digital file — as long as it is within a — welp — 50 TB limit.

I did this by adding the following lines to the Terraform files:

• variables.tf:

• main.tf:

Let’s digest what is going on. In variables.tf, I added in a variable that would name the Primary S3 bucket. In main.tf, I created and configured the bucket using the following Terraform resources:

• aws_s3_bucket — It reserves the bucket name in AWS and establishes the resource that everything else attaches to. Nothing is stored in it yet and none of the safety features are active; it is just an empty vessel
• aws_s3_bucket_versioning — Versioning tells S3 to keep every historical version of every object rather than overwriting it on each write. This is a critical prerequisite for Object Lock — which we will see in a later Terraform step — because it works by locking specific object versions, not object names. Without versioning enabled first, the Object Lock configuration will fail to apply.
• aws_s3_bucket_server_side_encryption_configuration — This tells AWS to automatically encrypt every object stored in the bucket at rest using AES-256. We will see later that restic already encrypts backup data client-side before it leaves a machine for a backup server, so this a second layer of encryption protecting against anyone who gains raw access to S3 storage infrastructure rather than going through the API.
• aws_s3_bucket_public_access_block — By default, S3 buckets can be made public if someone adds the wrong policy or ACL — a mistake that has exposed sensitive data for countless organizations. This resource sets four flags that collectively make it impossible for the bucket to ever be made public, even if someone later makes configuration or policy changes deliberately or accidentally

After doing some testing and tweaks with repeated rounds of terraform plan and terraform apply, I got the following error:

My Administrator user account is the identity Terraform ran as when it called the AWS API to build infrastructure — and the error indicates that it needs broad permissions to create and configure S3 resources. To prevent AWS from refusing every S3 API call that Terraform would make in this and future homelab sections, I got to work in the AWS Web Console by opening IAM >> Users >> Administrator and then going through the steps to grant it the AmazonS3FullAccess permissions.

Epic. A subsequent round of terraform plan and terraform apply resulted in the S3 bucket being created:

Wonderful.

Terraform Step #3: S3 Bucket Object Lock

I then needed to ensure that once a backup object is written, nothing can delete or overwrite it until the retention period expires, including ransomware operating with stolen AWS credentials or an accidental aws s3 rm command. For that to happen, I needed to create an S3 Bucket Object Lock configuration.

I went with Governance mode rather than Compliance mode because [per documentation] Governance still allows an administrator with the right IAM permission to override the lock in an emergency — for example, if I misconfigured a retention period and needed to clean up — while Compliance mode is truly permanent and irreversible. This minimizes the risk of costly mistakes very costly in the context of my homelab. For the retention period I chose 30 days — long enough that even if ransomware quietly encrypted my source data for weeks before being discovered, I would still have clean, uncompromised backups available to restore from.

It is also important to note that per AWS’ documentation, “Object Lock works only in buckets that have S3 Versioning enabled.” It was thus important to have my Terraform code use the depends_on keyword to ensure that Terraform waited until the aws_s3_bucket_versioning.primary resource was fully created before attempting to create the S3 Bucket Object Lock.

That was a lot of theory/explanation. I did this by adding the following lines to the Terraform files:

• variables.tf:

• main.tf:

A subsequent round of terraform plan and terraform apply confirmed that the code was working without any errors.

Terraform Step #4: Creating DR Bucket In Second Region

Next up was to create an S3 bucket for disaster recovery in a second region. In this case, it is to be in the us-east-2 region. The Terraform code update looks like the following

• variables.tf:

• main.tf:

Notice the red boxes highlighting the line provider = aws.Disaster_Recovery_region. This line is critical because it ensures that the S3 bucket gets created in the disaster recovery region. Otherwise, it would get created in the same region as the first S3 bucket.

A subsequent round of terraform plan and terraform apply confirmed that the secondary bucket was properly created:

Terraform Step #5: Cross-Region Replication

With the two buckets made, I needed to ensure that every object written to the primary S3 bucket gets automatically copied to the DR bucket asynchronously, without any script or chron job involved.
To make that happen, I based the next steps off a nice blog post’s Replication with Encryption section and added the following three resources to main.tf:

• aws_iam_role — S3 cannot replicate objects on its own; it needs an identity to act as when copying objects between buckets. This resource creates a dedicated IAM role that S3 is allowed to assume specifically for that purpose.
• aws_iam_role_policy — The role alone grants nothing. This policy, attached to the role above, gives S3 read permissions on the primary bucket and write permissions on the DR bucket — and nothing broader than that.
• aws_s3_bucket_replication_configuration — This is the actual replication rule, wired to the primary bucket. It tells S3 to use the role above and copy every object written to the primary bucket into the DR bucket.
  Notice that it has a depends_on statement for the two buckets’ versioning resources: versioning has to be active on both buckets before AWS will accept the configuration.

The code looked like this:

• main.tf

After that, I used the terraform plan and terraform apply commands to test it, and got an error

Looks like my Administrator IAM user was missing IAM permissions. Getting those added was a quick fix in the AWS console

Then I used terraform plan and terraform apply again — and confirmed that the buckets and the replica rules were made:

Terraform Step #7: Dedicated Account For Restic Backups

I then needed to take one last step in Terraform: creating a dedicated user account in AWS that restic would authenticate to when pushing encrypted backups to the S3 buckets. This is both per the Restic documentation’s tutorial (which does not use Terraform) and common sense — we need least privilege account access for security reasons. We do not want to use the AWS Administrator account’s credentials — if the credentials for the dedicated restic account were ever stolen or leaked, an attacker could only read and write to that one bucket and do nothing else in the AWS account. This is the difference between a compromised backup credential being a minor inconvenience versus a catastrophic account-wide breach.

The following section of the Restic documentation/tutorial provided the two permissions statements that I needed to create on the S3 bucket and the contents:

Time to plunge into Terraform. I made the following changes:

• output.tf

• main.tf

Notice the red-boxed text in the main.tf file. The two lines together cover two different levels of S3 permissions that AWS treats as separate things. aws_s3_bucket.primary.arn refers to the primary bucket itself — this is required for actions that operate on the primary bucket as a whole. “${aws_s3_bucket.primary.arn}/*” refers to every object inside it. It’s very important to have both because an S3 bucket and its contents are considered to be distinct resources in IAM’s permission model.

Another round of terraform plan and terraform apply resulted in the restic-backup-agent being created with the right permissions

With that, the Terraform section of this lab was completed!

Section II: restic Installation, Repository Init, & First Backup
I opened up the Proxmox setup from Homelab #17 and installed restic on the three boxes (haproxy1, haproxy2, web1) using the sudo apt install restic command. I then did the following the next steps off this and this restic documentation page.

First, on all three VMs, I created a new user called restic that is going to create the backups using the useradd — system — create-home — shell /sbin/nologin restic command

Screenshot From The web1 VM

Then I needed to give restic the ability to read every file on the system — including files owned by other users and root — **without** running as full root itself. By default, a non-root user like restic cannot read files it does not own, which would mean critical config files in /etc or application data in /var would simply be skipped during backup. Linux solves this through a feature called capabilities — a way of granting a binary one specific elevated privilege rather than handing it full root power.

Fortunately, there is a solution. Linux has a concept called capabilities. They are a security feature that break down the all-powerful root privilege into smaller, distinct units, allowing specific administrative permissions (like binding privileged ports or bypassing file read/write checks) to be granted to processes or binaries without blanket root access being granted.

Restic needs the cap_dac_read_search capability, which allows a file or account to read ANY file on the filesystem, which includes all files owned by root. However, Linux only applies a capability to the exact binary file it was set on, so instead of modifying the system-wide /usr/bin/restic binary (which would grant that capability to anyone on the system who runs it), I needed to create a private copy of the binary inside the restic user’s home directory and apply the capability there instead. This would be in line with the security principle of least privilege: only the restic user would be able to execute a capability-enabled copy of restic to back up the server’s files.

Screenshot From web1 VM

Notice that there was only one screenshot for brevity’s sake. These steps were derived from a restic documentation page and I also did the same on the haproxy1 and haproxy2 VMs too.

The next step was to make the password to encrypt the backups with. On each VM I first installed the apg Linux utility using the sudo apt install apg command, this would be the password for each of my VMs.

Screenshot From web1 VM

After that, I needed to create the environmental variables on each VM per the restic tutorial. Notice that these included the generated restic passwords, information about the primary backup region in AWS, and the AWS-related credentials. Because this information is sensitive, it was imperative that the command did not end up in the shell’s history file. To do that, I used the unset HISTFILE command. The variables were titled AWS_DEFAULT_REGION, RESTIC_REPOSITORY, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and RESTIC_PASSWORD.

Screenshot From web1 VM

Notice that this was done on the haproxy1 and haproxy2 VMs as well.
I then tried to create the repository with the restic init command — and immediately got a fatal error: client.PutObject: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters. This happened because the apt-installed version of restic (0.12.1 on Ubuntu) did not compute and send a checksum header when writing to S3, which AWS strictly requires when a bucket has Object Lock enabled, as documented in this long-standing GitHub issue that dates back to 2019. The fix of calculating content hashes during S3 uploads was introduced in PR #3246.

I thus needed to update restic to the latest version and then repeat the steps for copying the binary from /usr/bin/restic into the /home/restic/bin folder and configuring the appropriate permissions

I then used the restic init command on one machine to create the repository in the S3 primary bucket.

And then confirmed that all three machines could access it

Screenshot From haproxy1 VM

Screenshot From haproxy2 VM

Screenshot From web1 VM

The next step was to create a sample initial backup. I followed the steps in the tutorial and generated a test.bin binary file with the dd if=/dev/urandom bs=1M count=10 of=test.bin command and backed it up with the restic backup test.bin invocation.

Going into my primary bin in the AWS S3 console, we see the new folders called data/, index/, and snapshots/ were created on top of the previously-existing config file and keys/ folder — and that the first snapshot was made

I then did a little test where I deleted the test.bin file locally and then restored it with the restic restore c72a8b6f command (where c72a8b6f is the string that identified the snapshot) — and proved that the test.bin file was there with the ls command:

Section III: Making The Credentials More Long-Term

I have been working on this homelab over the course of multiple days. Being a pragmatist, I didn’t want to have to retype and troubleshoot the environmental variables that were critical to working with restic on each endpoint each time my VMs got rebooted for whatever reason.

The solution was to create a file where I could have all the necessary commands at the ready. Then I could use the source Linux command to take the file that I needed to make and to treat its contents as a set of commands to execute. These commands would prepare the environmental variables that I needed to get working.

To get started, I needed to first get the ID and password of the restic-backup-agent account that I created in Terraform when spinning up the buckets. I did this with the terraform output -raw iam_access_key and terraform output -raw iam_secret_key commands

Then I went into my haproxy1 box and created the restic-env file as **root** using the sudo nano /etc/restic-env command

I then needed to lock down the permissions to it. I did this with the sudo chown root:root /etc/restic-env command to set the file’s owner and group to root so that no other user account could accidentally stumble upon it, while sudo chmod 600 /etc/restic-env restricted permissions to read/write for root only.

This ensured that only I as the authorized root user (with the appropriate privilege escalation with sudo) would be able to access the file — and that no other account could.

I then wanted to save myself some time, so instead of doing the same steps manually on the haproxy2 and web1 VM, I used the sudo scp /etc/restic-env nathan@192.168.253.102:/tmp/restic-env and sudo scp /etc/restic-env nathan@192.168.253.103:/tmp/restic-env commands.

The reason the file was copied into /tmp first rather than directly into /etc is that scp runs as the regular nathan user, who does not have write access to /etc. /tmp serves as an authorized public staging area because Linux designates it as world-writable, meaning any user on the system can create files there.

A subsequent sudo mv then promoted the file into its final home in /etc with the correct ownership and permissions applied immediately after

On haproxy2

On web1

The main groundwork has thus been laid. Now each time I needed to run a restic command, I could lay the groundwork by invoking the source /etc/restic-env command to get the environment variables properly configured. See for example this command where I did this on the web1 VM.

Perfect.

Section IV: Cross Region Replication

I then opened up the secondary bucket in the AWS S3 website console and got a little unpleasant surprise. The data from the initial snapshot and all subsequent ones were not fully replicated to the secondary S3 bucket. This was a problem that needed to be fixed. Getting cross-region replication working required multiple rounds of debugging and iterative fixes.

I tried a few things in Terraform and then after consulting with a friend decided to create a temporary testing third S3 bucket and use “clickops” (i.e. manually creating the rules in the AWS website console) to create a replication rule. If that worked, then I could see what was up with regards to it and compare that to my current Terraform.

I did precisely that — and the replication worked! Below is the screenshot from my primary bucket on which I created the manual replication rule. In the red box is the test-replication-rule that I manually created in the GUI.

Notice that it had the Storage Class being Same as source instead of my other (Terraform-made replication rule) being Transition to Standard. It got me thinking that there was something with the Terraform code specifying the aws_s3_bucket_replication_configuration specifying the destination bucket being of a STANDARD storage class. I went in and started debugging and realized that the approach needed to be reworked

I then spoke back with my friend again and was told that it would be a better idea to just follow the Terraform documentation from the official Resource: aws_s3_bucket_replication_configuration configuration example for the one-way replication. I did exactly that with the Terraform AWS Provider v6 (and above) example and took advantage of the opportunity to update my AWS provider to version ~6 to keep it more up to date. I completely reworked the code for Step #5: Cross-Region Replication in line with the provided example. I should note that from the original example, I **removed** the aws_s3_bucket_acl resource which is no longer supported in modern AWS S3 buckets

Two of the most notable changes were for my primary_to_dr aws_s3_bucket_replication_configuration resource. These were as follows:

1. Inserting a marker enabling soft deletes (very important due to this cloud infrastructure acting as a backup system — we want accidental deletes to not be fatal)
2. Adding in a filter {} block that was previously missing and required to exist per documentation. This is a filter that allows for the rule to select a subset of objects to which the replication rule applies.

Once that was fixed, replication was still failing because the buckets had Object Lock enabled in GOVERNANCE mode, which requires the replication IAM role to carry additional permissions beyond basic read/write shown in the example.

What made this particularly confusing was that querying the object metadata via the AWS CLI was reporting a replication status of COMPLETED — despite the objects never actually appearing in the secondary bucket.

After progressively broadening the IAM policy using wildcard permissions (s3:Get*, s3:Put*, s3:List*) to confirm the root cause was permissions rather than a misconfiguration elsewhere, the policy was then locked back during multiple rounds of testing to only the permissions that the replication in this setup actually required. The final permissions looked like this

Testing with some sample restic backup <FILE_NAME> commands confirmed that the replication from the primary to secondary worked

Doing The Backup On haproxy1 VM

The Snapshot On The Primary S3 Bucket

The Snapshot On The Backup S3 Bucket

Notice the same hash corresponding to the same backed-up snapshot on both the primary and backup S3 buckets.

Section V: Retention Policy

With backups working, the next question is: how long do I keep them? Keeping every snapshot forever would balloon S3 costs over time. Removing some snapshots after a period also makes sense: there is no point in keeping older backups when newer ones are more than sufficient.

Because removing snapshots manually is tedious and error-prone, restic allows for a forget policy that automates this (check out the documentation).

Since all three machines shared the same repository, I only needed to run the command on one machine for it to affect all three VMs.

I opened up the console for the web1 VM and used the restic forget –keep-monthly 1 –prune command.

An explanation of the command’s flags:

• — keep-monthly 1 — This specifies that we keep the latest snapshot from each of the past 1 calendar month
• — prune — Immediately removes the underlying data blobs from the primary S3 backup bucket that are no longer referenced by any kept snapshot, freeing up storage rather than leaving orphaned data behind.

Section VI: Backup Hooks

Next up was to learn the concept of backup hooks. Without them, a backup tool like restic captures files exactly as they exist on disk at the moment the backup runs. Having hooks is especially important for databases since they are constantly writing to their data files.

A database is a collection of data files, index files, and transaction logs that must all be consistent with each other to be valid — and if restic snapshots these files at slightly different moments during an active write, the backup may capture some files mid-transaction. This can potentially leave the data files and transaction logs out of sync in a way that makes a backup unrestorable. Hooks resolve this problem by running a database dump command before restic starts, which results in a clean export file that represents a guaranteed consistent point in time state of the database.

I got started by opening up the web1 VM and installing SQLite. It’s a good tool for us to work with in this homelab section since it is a single file database — perfect for this stage of this exercise.

I quickly installed it using the sudo apt install sqlite3 -y command and then created a sample database with some test data per a SQLite tutorial I found online.

I then quickly installed resticprofile per the instructions on the documentation page and confirmed it was working with the resticprofile version command

The next thing I needed to do was to make a pre-backup hook script that would dump the SQLite database. To do this, I created a folder called scripts in which the hook script was to be stored and created a database_backups folder in which the backups would be placed. The final pre-backup.sh bash script looks like this

Great. I then needed to make a configuration file that would run this script before the restic backup. To get started, I made a resticprofile_config folder in which I placed a newly-created profiles.yaml file. It looks like this:

What it does is specify that the pre-backup.sh script will run to add in a new backup database file before the actual backup is performed. The backup will take all the files in the /home/nathan/database_backups folder — and back them up.

I then executed the source /etc/restic-env && resticprofile backup command to get the environment variables loaded in and to run the hook + restic backup combination.

It worked!

Then I needed to test the restoration command. To do this, I first deleted every single .db backup file in the database_backups/ folder, and then used the source /etc/restic-env && restic restore latest –target / –path “/home/nathan/database_backups” command

It worked. To briefly explain the command used, the –target / indicates that the restoration process will be based at the root of the file system while the –path “/home/nathan/database_backups” ensured that only the files in that specific directory were restored. The combination of / and “/home/nathan/database_backups” ensured that the restored .db files landed exactly where they originally were.

The last thing that I did for this section was open up the latest backup in sqlite3 and opened up the accounts table to see that the data was the same as it was originally.

It was. It should be noted that restic profile supports other hooks in addition to the one I used (run-before) — specifically run-after, run-after-fail, and run-finally. Run-after can be useful for something that needs to be done after a backup completes successfully, for example, cleaning up temporary dump files or triggering a downstream process that depends on a fresh backup existing. The run-after-fail hook will only fire when a backup fails and would be useful for alerting an administrator that a backup job did not complete as expected.

Section VII: Automation With Systemd Timers

Next up was to get my restic backup invocations done on an automated scheduling. To do that, I settled on using systemd instead of cron. Unlike cron, which simply fires a command and discards any record of what happened, systemd tracks each job run as a proper service with a full log captured by journald — meaning if a backup fails at 3AM you can run journalctl the next morning and see exactly what went wrong rather than wondering why your snapshots stopped appearing. (The archlinux Wiki page also had a nice section providing some additional benefits of using systemd over cron)

A systemd timer setup requires two files working together: a .service file that defines what to run, and a .timer file that defines when to run it. For brevity, I got to work only on web1 VM — and the same steps and logic would identically apply to haproxy1 and haproxy2

First, I created a restic-backup.sh script that would back up the files in the /home/nathan/database_backups and /home/nathan/sample_backups folders. Notice that it would also run the script for running the pre-backup hook and also apply the retention policy — the same steps that I did and explained in the previous sections. The script looked like this:

Then I created the /etc/systemd/system/restic-backup.service file. The screenshot of its text is below with explanatory comments. The actual information regarding how each field works had to be compiled through a series of websites, whose links are this, this, and this

After that, I created the /etc/systemd/system/restic-backup.timer file that would act as the scheduling mechanism for the backup script

The Persistent=true line deserves explanation — if a VM is powered off at the scheduled time, systemd will check on next boot whether the timer was missed and if so, run the backup within 10 minutes. Without this, a VM that was rebooted would simply skip that day’s backup entirely with no indication that anything was missed.

I then verified that the Service and Timer files that I created had no errors with the sudo systemd-analyze verify restic-backup.* command (I found the idea for this step from this website)

Then, I enabled and started the timer with the following commands listed on the following screenshot

Notice the usage of the daemon-reload command. It is required because systemd reads and caches all unit files into memory at boot, so after I created the .service and .timer files on the disk, I needed to tell the running systemd process to get those files in.

The last thing that I needed to do was that I wanted to trigger the service manually to confirm that everything works properly. I didn’t want to wait until 2 AM when the service would automatically fire up. To do this, I used the sudo systemctl start restic-backup.service, confirmed that it was running successfully with the with the sudo systemctl status restic-backup.service command, and then source /etc/restic-env && restic snapshots command confirmed that the new snapshot was made

The last thing I needed to do was to get the logs regarding the run. To do this, I used the journalctl command, which can filter logs by specific services with the -u flag.The command was journalctl -u restic-backup.service –no-pager | more (the | more was used to display the output one screen at a time)

Every future backup run will append to this log automatically, giving a complete audit trail of every backup that ran or failed. This is understood per the systemd-journald man page, which explains that “creates and maintains structured, indexed journals based on logging information that is received from… standard output and standard error of service units… and stores log data either persistently below /var/log/journal or in a volatile way below /run/log/journal/”

Section VIII: Alerting — Failure Notification

The next thing that I needed to implement is alerting for any backup failures. An unmonitored backup is not a backup. For example, if it silently fails for two weeks, you discover it only when you need to restore — and that is not good news.

For a lightweight implementation, I went with healthchecks.io, which is an open-source monitoring service that verifies if scheduled tasks — such as cron jobs, backups, or API calls — run on time and successfully. To get started, I got to work on registering my account

And it created an initial health check. I then needed to configure it, which I did per the following screenshots

Then per the documentation, I needed to insert in a curl request into the restic-backup.sh script

After that, I decided that it would be a good move to run it manually instead of waiting for the next 2 AM backup and confirm that I would get the ping coming in. I launched the ./restic-backup.sh script and lo and behold — it came in

To wrap up this section, I should note that “I initially considered self-hosting a full Prometheus and Alertmanager stack — where Prometheus would scrape metrics from the backup jobs and Alertmanager would route failure notifications to an alert destination such as email or Slack. However, my homelab workstation was configured in such a way to allow Proxmox to execute that I could not run Docker Desktop Engine. Therefore, I went with a quick implementation of healthchecks.io instead.

Section IX: File-Level and Directory Restore Testing

A backup that has never been restored is a hypothesis. Everything built up to this point — the S3 buckets, the cross-region replication, the retention policy, the systemd timer, the healthchecks.io alerting, and the resticprofile hooks — was infrastructure. This section is the proof that all of it actually works when it matters most.

The two restore tests below cover the most operationally relevant recovery scenarios for this setup — targeted single-file recovery and full directory recovery with application data verification. Since the retention policy was configured in Section V to keep only the latest monthly snapshot, both tests work against that single latest snapshot. In a production environment with daily snapshots and a longer retention window, an additional historical point-in-time restore test would verify that specific past states are recoverable.

Notice that in this screenshot I did the initial backup of the /home/nathan/database_backups and /home/nathan/sample_backups folders and then immediately afterwards used ls to display their contents so that you — my loyal readers — can get the sense of what exactly was in them.

Restore Test #1: Single File Restore With Hash Verification
In this round, I needed to restore a single .db file from the database backups folder and prove that what came back from the cloud-hosted restic snapshot matched what was originally backed up.

To prepare for this step, I first used the sha256sum command to get the hashes of all the .db backup files in the /home/nathan/database_backups. This was done with the sha256sum database_backups/*.db command

Excellent. I then created a /home/nathan/restore directory with the test1 subfolder and entered it

I then decided that out of all the files, I wanted to restore one called backup-20260322–020006.db — which I did with the restic restore 8e65ace0:/home/nathan/database_backups –target . –include * command

Notice that 8e65ace0 is the identifier for the restic snapshot that was taken. Next up was to use the sha256sum command to get its hash and to compare it with the original one — to confirm file integrity.

The two hashes were the same!

Restore Test #2: Directory Restore With Simulated Data Loss and Database Verification

The second test simulated a more serious scenario — the complete loss of both the database_backups and sample_backups directories — and verified that a full restore brings everything back intact, with the SQLite database confirmed queryable at the end. This is the test that proves the entire pipeline from the resticprofile pre-backup hook through S3 storage through restore actually recovers real application data in a usable state, not just files that are present on disk.

First, I recorded the hashes of all the files in both the database_backups and sample_backups folders with the sha256sum command, before simulating any loss.

Now for the next step: simulating catastrophic data loss by wiping both directories entirely. I did this using the rm -rf database_backups/* and rm -rf sample_backups/* commands

Then I used the source /etc/restic-env && restic restore 8e65ace0 –target / –include “/home/nathan/database_backups” && restic restore 8e65ace0 –target / –include “/home/nathan/sample_backups” to restore the files. Then, I used the sha256sum database_backups/* and sha256sum sample_backups/* command to confirm that the hashes were the same as originally

It worked! I then took a sample database snapshot and checked on the SQLite table inside of it to see if the values were still valid

They were!

I then did one last check by going into my AWS web console and looked in the S3 buckets in both my primary us-east-1 and DR us-east-2 regions to confirm that S3 successfully replicated the initial snapshot into the DR bucket in us-east-2

It was. The resticprofile pre-backup hook had correctly dumped the SQLite database before the backup ran, restic had encrypted and uploaded the dump to S3, S3 had replicated it to the DR bucket in us-east-2, and the restore brought it back in a fully usable state. The entire backup/disaster recovery pipeline was verified end to end.

Section X: Recovery Metrics

All this was nice and pretty — and very important, but management needs to answer the question of “How?” — how effective is and do we want our backup systems to be.

There are two key metrics that need to be accounted for when both planning a backup system and actually working with it: the RPO (Recovery Point Objective) and RTO (Recovery Time Objective).

• Recovery Time Objective (RTO): It is the maximum acceptable duration of downtime a business can tolerate for restoring systems & data after an unexpected outage. The units of measurement can be minutes or hours and answers the question of how quickly after an outage do we need a system to be available?
• Recovery Point Objective (RPO): It refers to how much data loss can your application tolerate — or restated — how old can this data be when the application is recovered. It is the maximum data loss that can happen/is tolerated.

Let’s apply this concept to this setup.

• Recovery Time Objective (RTO) — It was not formally measured during these tests with a stopwatch but based on the restore commands completing within a minute or two when invoked, a reasonable estimate would be < 5 minutes for a full directory restore.
  In a real-world environment with larger datasets, measuring RTO precisely with a stopwatch during a planned DR test would be mandatory before relying on the system
• Recovery Point Objective (RPO) — This is determined by the systemd timer schedule, with a backup running every day at 2 AM. In this case, it is 24 hours. In the worst case scenario a backup failure discovered the following morning means up to 24 hours of data could be lost

Section XI: Runbook

A backup system without a written recovery procedure is incomplete. The runbook is the document that someone — including a future version of myself under pressure at 3AM — can follow step by step to recover this system without needing to remember how any of it was configured.

I wrote RUNBOOK.md and pushed it to the homelab GitHub repository. It covers the four recovery scenarios tested in Section IX, the measured RPO and estimated RTO, credential setup instructions for a fresh recovery machine, how to switch to the DR bucket if the primary is unavailable, how to check backup logs via journalctl, and a restore test log table that gets updated every time a DR test is performed.

The restore test log is the most important part of the runbook — a recovery procedure that was written once and never tested again is almost as dangerous as having no procedure at all.

You can find it here.

Section XII: Cleanups

With the lab over, I needed to then tear the cloud infrastructure down with the terraform destroy command. Before I would be able to tear down the S3 buckets, I needed to ensure that they were empty.

Remember I had versioning enabled as a prerequisite for S3 Object Lock? That meant that I had multiple versions of the same data in S3. The AWS console for the two buckets having the Show Versions toggle indicates that very clearly

In the two screenshots I used the red curly bracket to show how the config file had multiple versions — as an illustrative example.

Because S3 Object Lock in GOVERNANCE mode prevents any object version from being deleted until its retention period expires, Terraform cannot remove the buckets on its own during terraform destroy. Fortunately I had the ability using my Administrator account to go into the AWS website consoles and delete all the data. After I did that, I quickly spun down the infrastructure using the terraform destroy command.

That was it for this lab. I can now answer one question confidently that if everything went wrong tonight, could I recover? After building a multi-region AWS backup pipeline from scratch with Terraform, instrumenting a VM with automated encrypted backups via restic and systemd, wiring up pre-backup database hooks with resticprofile, implementing failure alerting via healthchecks.io, and verifying the entire pipeline with hash-verified restore tests — the answer is yes.

All the collateral is on my GitHub page.

Before diving in, here’s a quick overview of the tools involved:

• Proxmox VE [Proxmox Virtual Environment] is an open-source server virtualization platform that allows you to run and manage virtual machines from a single web-based interface. Built on Debian Linux with KVM under the hood, it handles the provisioning, networking, and lifecycle of all the VMs in this lab — making it the foundation that everything else runs on top of.
• HAProxy [High Availability Proxy] is an open-source load balancer and proxy server that distributes incoming network traffic across multiple backend servers. Rather than a single server absorbing every request — and becoming a single point of failure — HAProxy sits in front of your infrastructure, intelligently routing traffic and continuously health-checking the backends behind it. If a backend goes down, HAProxy stops sending traffic to it automatically, without any manual intervention required.
• Keepalived is a Linux daemon that provides high availability at the network layer by managing a floating Virtual IP address (VIP) between two or more nodes. Think of it as an automatic traffic cop: it continuously monitors the health of a designated primary node, and the moment that node fails, it reassigns the VIP to a standby node so that clients connecting to that address never know anything has changed. This is the mechanism that turns two separate load balancer nodes into a single, resilient service.
• VRRP [Virtual Router Redundancy Protocol] is the underlying network protocol that Keepalived uses to coordinate which node currently “owns” the VIP. Nodes participating in a VRRP group constantly exchange heartbeat messages — a MASTER node broadcasts its presence, and BACKUP nodes listen. The moment the MASTER goes silent, the highest-priority BACKUP node promotes itself to MASTER and claims the VIP, completing the failover in a matter of seconds.

That’s enough theory. Let’s build it!

Section I: Proxmox Host Setup

First, I downloaded the Proxmox VE ISO from proxmox.com

Afterwards, I got to work on making the proxmox host in VMware Workstation Pro:

Notice that I enabled the “Virtualize Intel VT-x/EPT” in VMware Workstation’s processor settings to allow Proxmox — itself running as a VM — to use KVM hardware virtualization for the to-be-created Ubuntu VMs nested inside it. Without this flag, Proxmox cannot create and run VMs of its own. 120GB was chosen to comfortably accommodate the Proxmox OS, the planned three Ubuntu VM virtual disks (20GB + 20GB + 10GB), ISO storage, and snapshot headroom for destructive failover testing — without risking running out of space mid-lab at the worst possible moment.

I then tried to launch the VM and got the following message in a pop-up window: “Virtualized Intel VT-x/EPT is not supported on this platform. Continue without virtualized Intel VT-x/EPT?”

This message was bad news. I needed to get Proxmox — itself running as a VM — to use KVM hardware virtualization for the Ubuntu VMs nested inside it. By definition, this message had to go. Enabling nested virtualization required disabling Credential Guard, which resisted fixes including turning off Hyper-V via bcdedit, unchecking all Hyper-V-related Windows features, disabling Memory Integrity, and manually zeroing out the relevant registry keys under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceGuard and Lsa.

What finally worked was finding Microsoft’s Device Guard and Credential Guard hardware readiness tool, a PowerShell script that allowed me to disable Device and Credential Guard on my Homelab workstation.

I then successfully spun up the VM and went through the appropriate installer steps:

And then went to the https://192.168.253.136:8006/ address in my homelab workstation’s Chrome browser and logged in as root

Great. I then popped into the System >> Network tab and confirmed that I had a Linux Bridge titled vmbr0 for VM traffic. It is essentially a virtual network switch built into Proxmox that allows VMs created inside it to communicate with each other and with the outside network. Without it the VMs would be completely network-isolated with no way to pass traffic between them.

I then tested connectivity to external websites by opening up the Proxmox shell and using ping. This confirmed that the Proxmox VM’s network adapter, bridge configuration, and VMware NAT were all working together correctly end-to-end — because a Proxmox host that can’t reach broader internet can’t download packages, upload ISOs, or allow the VMs created inside it to communicate outside their local subnet.

I then tried to update the VM using the apt update && apt full-upgrade command — and got an error

The root cause behind the failure is that I did not have a Proxmox subscription. For the purposes of this homelab, I did not feel the need to get one — since per a Proxmox forum, “there is [effectively] no difference in functionality between an installation with a subscription and without.” To fix this, I went into the Updates >> Repositories window, disabled the Enterprise-related ones and added in the pve-no-subsciption ones as a replacement.

I then went back into the CLI and used the apt update && apt full-upgrade command to get the updates for this box underway

Wonderful. The web interface was working, and the VM was fully configured and installed. To wrap up this section, I took a snapshot of the Proxmox VE VM in VWare Workstation, so that I could have a reference point for any potential rollbacks if needed.

Section II: Ubuntu VM Creation and Cluster Infrastructure Setup
The first thing I did was to download the ISO for the Ubuntu Server — version 22.04.5 LTS (Jammy Jellyfish).

I then opened up Proxmox’s Datacenter >> pve >> local (pve) >> ISO Images tab, and used the Upload button

Great! I then quickly made the first Ubuntu Server VM named haproxy1, which was supposed to act as the primary load balancer. The specs were 2 cores, 4GB RAM, and a 20GB disk. To do this, I clicked the Create VM button in the top right-hand corner, and did the following:

Great. I then repeated the same series of steps to create two more Ubuntu Server VMs: haproxy2 (the secondary load balancer — same specs of 2 cores, 4GB RAM, and a 20GB disk) and web1 (the backend web server — the specs were 1 core, 2GB RAM, 10GB disk)

I then needed to fire up the VMs and to go through the installers for each one. I started with the one for haproxy1 — and assigned to it the static IP address of 192.168.1.101/24 and had the OpenSSH server installed on it:

This launched a reboot, and then I got the message: “Failed unmounting /cdrom”

This error appears because the Ubuntu installer leaves the ISO still attached to the virtual CD drive when it triggers the reboot. To resolve it, the CDROM drive needed to be detached from the VM in the Proxmox UI before the VM could boot cleanly from the installed virtual disk.

I did this by opening up the 100 (haproxy1) VM’s Hardware tab and edited the CD/DVD Drive’s settings to have it not use any media

I then booted the VM successfully, logged in, and confirmed with the ip a command that it had the 192.168.101/24 IP address working properly

Then I quickly did the sudo apt update && sudo apt upgrade command to ensure its packages were updated and installed properly

I walked through the steps for the creation and configuration of the haproxy1 VM. I then did the same for the haproxy2 VM (with its IP address = 192.168.102/24) and for the web1 VM (with its IP address = 192.168.1.103/24).

I then tried to ping www.google.com from each of these VMs — and got an error. Opening up the VMware application’s Virtual Network Editor, I saw that the NAT Network was of the 192.168.253.0/24 IP CIDR. This was the problem: I had assigned my VMs static IPs in the 192.168.1.x range, but VMware’s NAT gateway — the VMs’ only route to the internet — lives at 192.168.253.2. The VMs were trying to send traffic through a gateway that didn’t exist on their subnet, so every packet going outside their local network was dropped.

The fix was straightforward: to reassign all three VMs to the 192.168.253.x subnet so they could actually reach their gateway, and with it, the Internet. The hitch was that my minimal Ubuntu server installations didn’t have text editors like nano or vi installed, so I first had to flush the old IP address from the ens18 interface using sudo ip addr flush dev ens18, then install ping with sudo apt install iputils-ping -y and nano with sudo apt install nano -y. With those tools in place, I used nano to update /etc/netplan/00-installer-config.yaml on each VM — replacing the old 192.168.1.x addresses with the correct 192.168.253.x addresses, and swapping the deprecated gateway4 key for a proper routes block pointing to 192.168.253.2.

I was then able to ping www.google.com on each of the three VMs successfully! This meant that the connection to the broader internet was working.

I then used the sudo apt install nginx -y command to install the nginx web server onto the web1 VM, confirmed with the sudo systemctl status nginx command that it was running, and used the echo “<h1>Hello from web1</h1>” | sudo tee /var/www/html/index.html

The curl http://localhost:80 command then confirmed that this website was successfully up and running:

Great. I now had three VMs of the following IP addresses:

• haproxy1: 192.168.253.101
• haproxy2: 192.168.253.102
• web1: 192.168.253.103

And needed to test the connectivity and communication basics. I did this on each server by confirming that I could ping the other two servers, that the web1 server’s application could be accessed, and that I could perform an SSH connection between them. See the following screenshots for more details:

Testing On Server haproxy1:

Testing On Server haproxy2:

Testing On Server web1:

The last thing that I needed to do for this section was to take snapshots of each VM, now that they are fully configured and confirmed to be working well. I started with the haproxy1 server, opened up the Snapshots page and used the Take Snapshot button to take it

I then quickly did the same for the haproxy2 and web1 server.

Section III: Deploy HAProxy and Keepalived

The time has come for the heart of this lab. The first step was to get HAProxy installed. It is an open-source load balancer that sits in front of your backend servers and distributes incoming traffic across them, continuously health-checking each one so that if a backend goes down it automatically stops sending traffic there — and it needed to be installed on both haproxy1 and haproxy2 so that the load balancer itself had no single point of failure.

I did this by opening up the haproxy1 VM and used the sudo apt install haproxy command

The next step was to get the haproxy configured. I first used the cat /etc/haproxy/haproxy.cfg command on haproxy1 to see what the primary configuration file for the HAProxy load balancer software looks like. It defines the rules and settings for how HAProxy accepts incoming client connections and distributes them among a pool of backend servers to ensure high availability and optimal performance of web services

We see that the default HAProxy configuration file came by default with two sections:

• global, which sets system-level settings like logging, the stats socket, file paths, user/group permissions, and SSL cipher configurations
• defaults, which defines baseline behavior inherited by all frontends and backends — including HTTP mode, logging format, connection timeouts, and standard error page mappings

It is missing the frontend and backend sections, which are the parts that actually tell HAProxy what traffic to listen for and where to route it — without those, HAProxy has no idea what to do with incoming requests. I fixed this by inserting in the following:

Let’s digest what this does.

• frontend http_front is the entry point — it defines what HAProxy listens for. bind *:80 means it listens on port 80 on all network interfaces. When traffic arrives it gets forwarded to the default_backend called web_servers
• backend web_servers is where you define the actual servers that handle the requests. balance roundrobin distributes requests evenly across all available servers in rotation (yes, in this case we have only one — but it is the good pattern). option httpchk GET / tells HAProxy to health-check each server by making a real HTTP request to / rather than just checking if the port is open — more realistic because it confirms the web server is actually responding. server web1 192.168.253.103:80 check registers the web1 VM as a backend with health checking active.

Then I had everything run again with the sudo systemctl restart haproxy and sudo systemctl status haproxy commands

I then did the same on the haproxy2 machine.

Then I needed to get Keepalived installed. It is a Linux daemon that provides high availability at the network layer by managing a floating Virtual IP address (VIP) between two or more nodes by continuously monitoring the health of the primary node and automatically reassigning the VIP to a standby node the moment the primary fails. This ensures that clients always have a single, stable address to connect to regardless of which physical node is actually serving traffic.

I first used the sudo apt install keepalived on both haproxy servers, and then used the sudo nano /etc/keepalived/keepalivd.conf command to edit the configuration files. Please see the following screenshots with the explanatory comments regarding what each one does:

With the configuration files in place on both nodes, the final step was to enable and start both services together using systemctl enable — now haproxy keepalived on each node.

systemctl enable tells the system to automatically start the service on every boot — without this, HAProxy and Keepalived would need to be manually started every time the VM restarts, which defeats the purpose of building an automated HA setup. — now is a convenience flag that combines enable and start into a single command, so rather than running two separate commands (systemctl enable followed by systemctl start), this does both in one shot. Passing both haproxy and keepalived as arguments applies the same operation to both services simultaneously rather than running the command twice.

The result is that on both haproxy1 and haproxy2, HAProxy and Keepalived are now running and will automatically restart on reboot — with haproxy1 holding the VIP as MASTER and haproxy2 sitting in BACKUP state, ready to claim the VIP the moment haproxy1’s health check fails.

Epic. I then opened my Windows workstation and used the ping 192.168.253.200 command to confirm that I could reach that virtual IP address

Then the curl 192.168.253.200 command confirmed that I could retrieve the website from the web1 server

Wonderful. The setup was completed. There are good documentation/website resources regarding Haproxy works, including a DigitalOcean, this, this, and this. Likewise, good references regarding Keepalived are this, this, this, this.

Section IV: Finalizing & Troubleshooting Failovers

I started by shutting down the haproxy1 VM

And was able to successfully ping the Virtual IP address of 192.168.253.200 and to get its website:

Great. The Virtual IP successfully failed over to haproxy2 — and it was able to communicate to and get the website from the web1 backend web server (specified in its /etc/haproxy/haproxy.cfg file).

I then turned haproxy1 back on and used the sudo systemctl stop haproxy command to stop the haproxy service. I then also used the sudo systemctl status haproxy command to confirm that it was not running. This was a test of a more subtle failure: haproxy1 is still on but its haproxy service is not running on it.

I then tried to do the testing — and was not able to curl the website from the Virtual IP from my Windows workstation:

Interesting. It is important to note that then opened up the haproxy2 command line and was able to execute the curl 192.168.253.103 command successfully, so it wasn’t an issue stemming from the haproxy2 ←→ web1 connection

Issue 1: Missing keepalived_script User

Time to look at the logs. I used the sudo journalctl -u keepalived -n 20 logs to see what was going on:

Aha! The default user ‘keepalived_script’ for script execution does not exist! Per the man page for keepalived_script:

By default the scripts will be executed by user keepalived_script if that user exists, or if not by root, but for each script the user/group under which it is to be executed can be specified.

There are significant security implications if scripts are executed with root privileges, especially if the scripts themselves are modifiable or replaceable by a non root user. Consequently, security checks are made at startup to ensure that if a script is executed by root, then it cannot be modified or replaced by a non root user.

While the man page doesn’t explicitly specify what the “security implications” are, the basic idea is that it would violate the principle of least privilege — the security principle that any process should only run with the minimum permissions it needs to function. Creating a dedicated, non-login system user specifically for script execution ensures the health check runs with minimal privileges, limiting the potential damage if the script were ever compromised.

The fix was to create the keepalived_script user on both nodes using sudo useradd -r -s /sbin/nologin keepalived_script, followed by sudo systemctl restart keepalived.

Notice that the usage of the /sbin/nologin shell prevents anyone from ever logging into the system interactively as that user. Giving it a real shell like /bin/bash would create an unnecessary attack surface if the account were ever compromised.

Issue 2: killall Not Installed

Then, I tried again, and failed to curl the website from the 192.168.253.200 VIP. I took a look on haproxy2’s logs using the sudo journalctl -u keepalived -n 30 command, and got the following:

Aha. The culprit was that the killall Linux command was not on the haproxy1 nor on the haproxy2 machines, since they were originally installed as minimal Ubuntu web server installations! I confirmed that killall was not on the boxes with the which killall command showing nothing — and then installed it quickly with the sudo apt install psmisc command. The repeated invocation of which killall confirmed that it was properly installed.

I then on both boxes again did sudo systemctl restart keepalived and then tried to ping and curl from the VIP again. Remember that at this point in time my haproxy service remained stopped on the haproxy1 server and I still wanted to get the website from the web1 server by way of haproxy2. I then opened the logs — again — with the sudo journalctl -u keepalived -n 30 command

Issue 3: Full Path and Script Security Requirements

The logs indicated two issues:

1. I needed to specify the full path to the /usr/bin/killall binary explicitly. This is because Keepalived refuses to run scripts using bare command names like killall because it won’t search the PATH for security reasons — an attacker who could manipulate PATH could redirect the script to a malicious binary. By requiring the full path /usr/bin/killall you’re telling Keepalived exactly which binary to run with no ambiguity.
2. I needed to enable the script_security feature in Keepalived. This is because Keepalived detected that the keepalived.conf file defining the script was owned by root and refused to execute it as a security violation

I fixed this on both haproxy servers by updating the /etc/keepalived/keepalived.conf file to the following

I explicitly named the /usr/bin/killall binary and per this, this, and this Internet pages that I read, inserted a global_defs global definitions configuration block with enable_script_security to tell Keepalived that the health check script would run as the dedicated unprivileged keepalived_script user rather than root (super important per the man page), satisfying its security requirements and allowing the vrrp_script to execute cleanly. Then I did sudo systemctl restart keepalived again and looked at the logs with the sudo journalctl -u keepalived -n 30 command

Issue 4: killall Permission Errors on HAProxy Processes

Testing the line /usr/bin/killall -0 haproxy manually revealed that this line of the script was the problem.

I used some linux commands such as ps to dig into the haproxy processes of IDs = 117277 and 117279 and found the root cause. The original killall -0 haproxy check failed with permission errors and “no process found” because there are two HAProxy processes, running as the root and the haproxy user, while the script executed as a non-privileged user without permission to signal them. (See the fantastic explanation on a StackOverflow article regarding what is going on)

I then quickly rewrote the script in the /etc/keepalived/keepalived.conf file on both boxes to
use script “/usr/bin/systemctl is-active haproxy” to see if haproxy is up and running

Then I used sudo systemctl restart keepalived again and looked at the logs with the sudo journalctl -u keepalived -n 30 command. The logs showed that everything was working, and so did the ping 192.168.253.200 and curl 192.168.253.200 commands.

Excellent. I need to note that during the course of this extensive troubleshooting I brought the haproxy service back online on the haproxy1 machine.

I think that I was now ready to do the final testing. I also took two screenshots of the final configuration in the /etc/keepalived/keepalived.conf file and put it into the appropriate folder in my Homelab GitHub repository.

Section V: Final Failover Tests

With all the troubleshootings done, I needed to do the final failover testing from my Windows homelab workstation

Initial State: haproxy is running on both machines

Pass!

Test #1: Shut Down haproxy1 VM

Pass! I also took a look into the logs with the sudo journalctl -u keepalived -n 30 command and confirmed that haproxy2 entered the MASTER STATE as expected.

Test #2: Turned On haproxy1 VM and Stopped haproxy

Pass! I also took a look into the logs with the sudo journalctl -u keepalived -n 30 command and confirmed that haproxy2 entered the BACKUP STATE when I booted the haproxy1 VM on and then when I shut down that service, it reverted into MASTER STATE — as expected.

With all the tests complete, I quickly used the sudo systemctl start haproxy command on haproxy1 to get it back up and running.

Section VI: Automate HA Cluster With Ansible

To start, I needed to create a new VM in Proxmox for running Ansible with IP address = 192.168.253.104/24. The specs were as follows:

I created this VM to act as my Ansible control node and jump box. This is similar to how a real environment would handle automation — a dedicated machine would manage other nodes over SSH rather than running tooling directly on a Windows workstation.

Setting up the ansible-ctrl VM began with attempting to install Ansible directly using sudo apt install ansible -y. This failed immediately because the VM could not reach the package repositories.

The root cause was the same issue that I experienced with the other three VMs that I installed: the default netplan configuration that shipped with the minimal Ubuntu Server installation was missing a gateway entirely, which meant the VM had no route out of the subnet and could not resolve DNS or reach the internet despite having 8.8.8.8 configured as a nameserver. Making things harder, the minimal Ubuntu Server installation came with no text editors — neither nano nor vi were present — so the only way to write the correct netplan configuration was using the sudo tee /etc/netplan/50-cloud-init.yaml << EOF heredoc command to write the file directly from the terminal.

Once the corrected 50-cloud-init.yaml was applied with the VMware’s NAT gateway (the VMs’ only route to the Internet) at 192.168.253.2.properly specified, connectivity was restored and both iputils-ping and nano installed cleanly — after which Ansible installed without issue.

To enable passwordless SSH access from ansible-ctrl to the three managed nodes, I generated an SSH key pair on ansible-ctrl using the ssh-keygen -t ed25519 command and copied the public key to haproxy1, haproxy2, and web1 using ssh-copy-id. Notice that I selected the Ed25519 algorithm for generating the keys over the older RSA algorithm; it produces shorter keys that are faster to generate and verify, offers stronger security per bit, and is considered the modern standard for SSH authentication.

I then quickly tested that the passwordless SSH worked to all three target boxes before proceeding.

After that, I got to work on creating the inventory file (hosts.ini) on my ansible-ctrl machine

The syntax is fairly self-explanatory. The [haproxy] and [webservers] are the group definitions, while the [all:vars] is defining the variables for the [all] group — that is all the servers defined in the hosts.ini file. Per Ansible documentation and another section of the same documentation page:

Even if you do not define any groups in your inventory, Ansible creates two default groups: all and ungrouped. The all group contains every host. The ungrouped group contains all hosts that do not belong to any other group….

You can apply variables to parent groups (nested groups or groups of groups) as well as to child groups. The syntax is the same: :vars for INI format and vars: for YAML format

Excellent. Now it was time for the playbooks. As explained in the previous Ansible homelab, they ensure that everything is done consistently across environment and reduce risk of config drift — a common cause of outages in real clusters.

Playbook #1: Deploy Nginx And Website On Webserver
This one deployed nginx and an updated version of the website’s index.html file. This playbook was heavily based off some research and a blog post:

I then ran it, with the main change on the web1 server being that the VM got an updated index.html file installed on it, with the other steps of installing nginx and getting it running not being needed — all this was done before

And curling the website from my homelab workstation’s command prompt, I got this:

Playbook #2: Deploy Keepalived
When starting to work on this playbook, it was important to keep in mind that the configurations for Keepalived between both haproxy1 and haproxy2 are basically the same — except for the state information (MASTER versus BACKUP — there can be only one master and potentially multiple backup servers) and the priority number (the BACKUP server with the largest priority number will take over if the MASTER fails).

Because the playbook had to be extendible and flexible, I went with a template-based approach. The first thing that I needed to do was to create a host_vars directory. Per Red Hat’s explanation, “Ansible automatically loads variable files into playbooks in the host_vars and group_vars folders based on the host or group name and the associated file name.” Inside the host_vars folder, I then created two YAML files titled haproxy1.yml and haproxy2.yml with the desired values

Notice that the files were deliberately named haproxy1.yml and haproxy2.yml — Ansible would automatically parse the file names out and map them to the hosts of the same name.

Then I needed to make my configuration template that the playbook would read and work with. Ansible’s ansible.builtin.template module uses and processes files structured in the Jinja2 templating language (with the .j2 extension). To get started, I quickly made the templates subdirectory folder (which Ansible automatically searches for by default) and then created the keepalived.conf.j2 file.

Then I created the keepalived.yml Ansible playbook

And executed it successfully

Beautiful.

Playbook #3: Configure and Deploy HAProxy

This playbook would leverage the same template-based approach, explained in details in the section for Playbook #2. I quickly took a look at the /etc/haproxy/haproxy.cfg files on both the haproxy1 and haproxy2 servers and confirmed that they were the same. Therefore, the template that I needed to make would not require variables to be injected — for this time.

Off to work. I went into the templates subdirectory folder and created an haproxy.cfg.j2 file

Then I created the haproxy.yml Ansible playbook

And executed it successfully

What started as a straightforward goal — eliminate the single point of failure from a load balancer setup — turned into a comprehensive exercise in building, breaking, and systematically debugging a production-like high-availability cluster from the ground up. The troubleshooting gauntlet across Section IV was the real education: every issue with Keepalived’s script execution, permission model, and process checking was a lesson that no tutorial would have taught as effectively as hitting it firsthand. The last section with the Ansible playbooks built on Ansible experience from past homelabs and taught me how templating with variables works — which allows for flexible deployment among multiple systems in one swoop.

Comprehensive screenshots detailing the configuration settings and Ansible playbooks can be found on my Homelab GitHub repository.

DevSecOps is the practice of integrating security checks directly into the development and deployment pipeline rather than treating it as a separate gate at the end. Think of it as “shifting security left” — catching vulnerabilities, misconfigurations, and compliance issues early in the development process when they’re cheap and easy to fix, not after code reaches production. This automated approach ensures that every change is vetted for security issues before it can be deployed, turning security from a manual bottleneck into an automatic quality gate that runs at machine speed.

GitOps is a declarative approach to infrastructure and application deployment where Git becomes your single source of truth. Instead of running manual kubectl commands or clicking through dashboards to deploy changes, one simply pushes code to a Git repository. My GitOps tool (ArgoCD) continuously watched that repository and automatically synced any changes to my Kubernetes cluster. This gives powerful benefits: automatic rollbacks, complete audit trails (every change was a Git commit), and disaster recovery became as simple as pointing a new cluster at my repository.

ArgoCD [Argo Continuous Delivery] is the GitOps controller that makes magic happen. It’s a Kubernetes-native continuous delivery tool that monitored my Git repositories and ensured my cluster matched what I defined there. When a change is committed — say, bumping an application from 2 to 3 replicas — ArgoCD detects it within minutes and applies them automatically. It provides a beautiful web UI where one can visualize the applications, see what is in sync or out of sync, and manually trigger syncs when needed. ArgoCD is particularly elegant due to its “pull” model: it pulls updates from Git.

HashiCorp Vault is the industry-standard secrets management solution that securely stores, accesses, and manages sensitive data. Think of it as a hardened database specifically designed for secrets — it encrypts everything at rest, provides fine-grained access controls, maintains detailed audit logs of who accessed what and when, and even supports dynamic secrets that expire automatically.

Kubernetes External Secrets Operator (ESO) is a Kubernetes operator that synchronizes secrets from external secret management systems into your cluster. Think of it as a bridge between your secrets provider and Kubernetes — it automatically fetches secrets from external APIs like HashiCorp Vault or AWS Secrets Manager, creates native Kubernetes Secret objects from them, and continuously reconciles to keep everything in sync when the upstream secrets change.

At its core, the External Secrets Operator consists of three key components: the ExternalSecret custom resource (where you define which external secrets to sync and how), the SecretStore or ClusterSecretStore (which configures the connection to your external provider like Vault, including authentication and path details), and the controller (which runs in the cluster, watches for changes in these custom resources, fetches the actual secrets from the provider, and creates/updates the corresponding Kubernetes Secrets automatically).

Having it properly work and configured is also important for another reason. We need to be able to ensure the “separation of concerns” in DevOps — where one can be able to migrate between different secrets managers — such as AWS Parameter Store, Azure Secrets, Google Cloud Secrets, Hashicorp Vault, etc — and not need to go and modify 50+ projects.

That was a lot of theory. Now for the action!

Section I: Kubernetes (K3s) Setup
I needed a Kubernetes cluster to run my stuff, but spinning up a full production-grade Kubernetes installation would have consumed massive resources on my homelab workstation. That’s where K3s came in — it’s a lightweight, certified Kubernetes distribution designed specifically for resource-constrained environments like edge devices, IoT, and yes, homelabs.

This is different from the Minikube that I used in the Kubernetes homelab. K3s runs as a native Linux service with a stripped-down architecture perfect for edge computing, IoT environments, and other resource-constrained environments, while Minikube is designed for local development and testing with VM-based isolation that adds overhead and is typically torn down after each session.

K3s are designed to run on Linux platforms. Given that my homelab workstation is a Windows machine, I started by getting the Windows Subsystem for Linux installed and configured. For those who don’t know what it is, it is a Windows feature that allows you to run Linux on my box without installing a separate VM or doing dual booting. This was done per the following screenshot:

Then, I needed to get K3s installed. I did this by launching the installation script from the Quick Start guide with the following command: curl -sfL https://get.k3s.io | sh -

Next up was to make kubectl work easily from my regular user account (without needing sudo every time) by copying the admin kubeconfig file from /etc/rancher/k3s/k3s.yaml (root-protected) to the standard Kubernetes location at ~/.kube/config in my home directory, with proper ownership.

First, the mkdir -p ~/.kube command created the hidden .kube directory in my home folder. Then, the sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config command copied the powerful original admin kubeconfig file into the ~/.kube/config directory. Then, the sudo chown $USER:$USER ~/.kube/config command changed the ownership of the file from root to my username — ensuring that I could read the file and run Kubernetes commands seamlessly without the sudo keyword.

This allowed me to then execute the kubectl get nodes command (without the sudo keyword) — to verify that everything was working. Not needing to invoke the sudo keyword aligns with the principle of least privilege (kubectl doesn’t need root on your host OS), and honestly, typing sudo every time gets old fast.

Section II: Relevant Theory

In my previous homelabs, I deployed applications to Kubernetes by hand-writing YAML files for each resource. That worked fine for simple setups — a Deployment here, a Service there. But real-world applications like Vault are far more complex. They require dozens of interconnected resources: Deployments, Services, ConfigMaps, Secrets, RBAC roles, and more — all carefully configured to work together. Writing and maintaining all of that by hand would be error-prone and time-consuming.

That’s where Helm comes in. Helm is essentially a package manager for Kubernetes, similar to how apt manages software on Ubuntu. Helm Charts bundle all of those resources into a single, pre-configured, reusable package. Instead of writing dozens of YAML files myself, a Helm Chart allows me to deploy Vault with one command.

I dug around for some additional explanations regarding what Helm Charts are, and found this fantastic Medium writeup.

I then took advantage of an opportunity to refresh myself on how YAML manifests work in Kubernetes after finding a fantastic Medium blog post that broke down the structure of Kubernetes YAML files in a refreshing and intuitive way. I need to give a big big thank you to Ayoub Adjour for taking the time to do this — I previously used to hate Kubernetes and now feel a lot better about how it works. If you ever work with Kubernetes, please save this article. It will help you.

Section III: Helm Chart Installation
I then installed Helm in my WSL Ubuntu instance using the steps described in a DevOpsCube tutorial (specifically those outlined in Method #1)

Section IV: Approach For Setups & Kubernetes Secrets Operator
The goal of any Cloud or DevOps engineer is to set up infrastructure in such a way that it is (1) simple to do, (2) reduces cognitive load, and (3) is idempotent [i.e. is repeatable and unchanged in final result state each time we got things working].

To do this, I created a folder in my homelab GitHub repository, and created a Makefile. This Makefile would first confirm that the required dependencies exist, and then kick off a series of shell scripts for getting everything set up.

The goal is to just have the make init command kicked off and successfully run everything, so that 6+ months down the line, someone can open up the repository, read the README.md explanation text, and then get everything set up in one fell swoop.

I then started with the initial steps for the Kubernetes Secrets operator. A friend of mine pointed me in the direction of the following documentation, so that’s what I referenced for the following steps.

First, for my initial Makefile. It looked like this:

This Makefile starts with .PHONY to declare that init and check-deps are command targets rather than actual files, preventing conflicts if files with those names exist. It then automates the setup of External Secrets Operator by first verifying that all required tools (helm, kubectl) are installed and that the K3s cluster is accessible, then running the installation script. The check-deps target acts as a safety gate — if any dependency is missing or the cluster is unreachable, it fails with a clear error message before attempting the actual setup of the external secrets operator in setup_external_secrets.sh.

Preview For The Future: The approach is to have separate .sh script files for each aspect of the initialization/configuration of the infrastructure: one for setting up the Kubernetes external secrets operator, one for setting up the Hashicorp Vault, one for setting up the ArgoCD application, etc and etc. Additionally, there would be separate .sh script files to be able to tear everything down completely that was previously created and installed — with nothing left over.

With the reference to the ./setup_external_secrets.sh script ready, I then got to work by creating the file for setting up the External Secrets Operator. This script was based on the steps enumerated in the corresponding documentation. It looks like this

Notice the two lines commented out at the very bottom. Those would be addressed at a later time in the script once there is a Vault up and running with the appropriate secrets for the External Secrets Operator to pull information from. I then quickly did a little test of what I had by executing make init in the command-line and confirming that things are running with the kubectl get pods -n external-secrets instruction

I then took advantage of the opportunity to get an uninstallation capability added into the Makefile and made a separate uinstall_external_secrets.sh shell script file — since I wanted my Makefile to be able to both create and destroy in one fell swoop my entire GitOps environment. Here is what both looked like now

Wonderful. The scripts’ instructions seemed pretty self-explanatory to me, so I won’t bother expanding on them with explanations further. Let’s test it

Looks delicious. Notice that I deliberately did not finalize everything in this step. This is because I needed to get some other pieces finalized properly before returning back to this one.

Section V: Setting Up The HashiCorp Vault

Next up was to make a separate shell script for installing and uninstalling the HashiCorp Vault, and then to integrate it into the Makefile. Notice that for this round, I did not introduce any secrets into the Vault at this time. My goal in this section was to just be able to spin it up and spin it down — and ensure that it could be interacted with in the UI.

I did this by first specifying the following YAML file for the configuration of the HashiCorp Vault. Notice that it was heavily based off a documentation YAML file on the HashiCorp website.

Then I configured the setup_vault.sh and uinstall_vault.sh scripts to be able to spin it up and down whenever needed.

setup_vault.sh:

uinstall_vault.sh:

And fed them into the Makefile appropriately — see why I had separate scripts for configuring things?

Notice that the screenshots of the files are what was done after a pile of troubleshooting steps and tweaks that I had to take to get the scripts and the Makefile polished off and working. Finally, after I got things working, I used the make init command to spin up the infrastructure — and got the following

And was able to open Vault at http://localhost:8200 — and then logged in using the default token of value “root” (which was specified in the YAML file)

Epic. Then I quickly used the make uninstall command to spin everything down

And a quick browser check confirmed that I could not find anything about the http://localhost:8200 address.

Side Note
When I first set up HashiCorp Vault using the official Helm chart in my GitOps homelab cluster (running via Docker Desktop’s Kubernetes on WSL2), the Vault UI service was correctly exposed as a NodePort on 8200 — but nothing loaded at http://localhost:8200 in the browser. This turned out to be a classic WSL2 networking quirk: NodePort bindings don’t automatically forward to the Windows host’s localhost loopback interface, even though curl worked fine from inside WSL.

Incorporating the kubectl port-forward svc/vault-ui 8200:8200 & command into my setup_vault.sh script bridged the gap instantly, letting me access the UI reliably at http://localhost:8200 from my Chrome browser on the Windows homelab workstation. This was a lesson learned the hard way about WSL2 + Docker Desktop Kubernetes networking gotchas!

Section VI: Sample “Hello World” — Style Application

I needed to get a sample “Hello World” — style application spun up in this section of the lab and successfully deployed and decommissioned with the Makefile commands. The reason for this step is that I needed a sample application for usage in this lab’s next section, which would involve setting up and testing the secret injection using the Kubernetes External Secrets Operator.

I fortunately had prior experience in a past CI/CD homelab with making a sample Flask application, so I quickly opened up its Section II and followed the steps to make an application. This is why I like writing my work up on Medium — these writeups are a good go-to resource in times of need!

The final index.html file looked like this:

Great. Now I needed to create a setup script that builds the Docker container for usage and launches the application in Kubernetes, as well as an uninstall script that would take the Kubernetes-run application down when needed. This took a lot of time to get working and debugged, but I finally did it.

The install script built and pushed the Flask container image to Docker Hub, then deployed the application to Kubernetes in the app namespace using a Deployment and NodePort Service. It waited for the pods to become ready and started a local port-forward so the app was immediately accessible at http://localhost:5000 (remember the WSL networking quirk we discussed earlier?). Two YAML files handled the Kubernetes side: the Deployment kept three replicated pods running with the container image (pulling from Docker Hub and exposing port 5000 internally), while the Service provided a stable external entry point via NodePort so the app could be reached from outside the cluster.

The companion uninstall script reversed everything: it removed the image and repository from Docker Hub, uninstalled Docker Engine, deleted the Kubernetes Deployment and Service, terminated any lingering port-forward processes, and force-deleted the app namespace — even cleaning up hidden Kubernetes objects like events or EndpointSlices that sometimes prevent normal deletion.

Once everything was up and running, the application could be opened in the browser at http://localhost:5000 — and looked like this:

Having the ability to cleanly and completely create and tear down this application is very important both for this lab and in general when working with DevOps. Feel free to check out the GitHub repository for this lab to see exactly how the code and Kubernetes manifests work.

Section VII: Getting Secrets Manually Entered Into The Vault

With the sample Hello World application now cleanly deployable and removable via Makefile commands, the next logical step was to prepare for testing real secret injection into it using the External Secrets Operator — the core focus of this homelab.

To do this, I modified the Vault setup script to pause after Vault was running and prompt the user (in colorful, step-by-step console messages) to manually log into the Vault UI at http://localhost:8200, enable a KV secrets engine, and enter a small set of realistic demo secrets: an API key, database password, Stripe secret key, app environment flag, and a feature toggle boolean. These secrets — stored under paths like secret/apps/hello-world — would later be read by the External Secrets Operator and injected into the application.

Here is what I got in the console now after running the make init command:

I then went into the Vault and tested the process of manually entering them per the prompt. Here is what I got:

Now the stage has been set for doing what this lab is really about.

Section VIII: Setting Up Secret Injection

This is where I started to glue all the fun together. In the previous sections, I made a sample application, a prompt to the user creating the environment to go into the Vault and manually enter a predefined series of secrets, and the initial steps for getting the Kubernetes External Secrets Operator. Now, I needed to get the setup and usage of the Kubernetes External Secrets Operator over the finish line.

The following diagram shows how data flows from external providers like Vault through the External Secrets Operator and on into the final application:

To copy the wise words of a fantastic GitHub-based writeup: “The ExternalSecret specifies what to fetch, [and] the SecretStore specifies how to access [it].”

The first thing I did get moving in this direction was to add in and test some “guardrail” code to the setup_external_secrets.sh shell script file to verify and ensure that Vault was installed, running, and ready before continuing to install and finalize the External Secrets. This is because properly configuring and connecting the External Secrets Operator to the Vault naturally requires a Vault instance to be up and running beforehand.

And it is precisely for this reason why my Makefile for the make init command had setup_vault.sh being invoked before the setup_external_secrets.sh script. Order matters.

With Vault confirmed healthy, the script then creates the app namespace, installs the External Secrets Operator (ESO) via Helm with the crucial — set installCRDs=true flag, and waits patiently for everything to settle. After that, it proceeded to configure Vault authentication, writing a read-only demo-policy, creating the demo-role bound to the default service account in the app namespace, and finally applied the SecretStore (connection details) and ExternalSecret (sync instructions).

And then my automated shell script did some final checks to confirm that the Secret Store and ExternalSecret were successfully set up

Side Note: It is important to briefly touch upon CRDs (Custom Resource Definitions). They are Kubernetes extensions that teach the cluster about entirely new resource types — in this case SecretStore and ExternalSecret — that don’t exist in vanilla Kubernetes. Enabling them during installation is essential because without those definitions the API server wouldn’t recognize or let you create the YAML files that tell ESO how to connect to Vault and which secrets to sync. Skipping or delaying the CRDs would cause immediate kubectl apply failures with messages like “no matches for kind SecretStore,” so the script explicitly waits for them to be fully established.

The next big task was to get these secrets fed into the Flask application’s container as environment variables per the appropriate documentation page. To lay the groundwork for this step, I first went into the helloworld-deployment.yaml page which specified the helloworld-app container’s deployment and per a Kubernetes documentation page first created five sample environment variables for the container with some hard-coded text values.

The goal for this step was to ensure that my tools are working inside the container, so that I am not forced later to chase my tail debugging toolset-side issues later on when I am working on polishing off the final secret injection. This is what my helloworld-deployment.yaml file looked like:

Delicious. I then finalized the Makefile as having all the scripts for setting up the vault, external secrets, and the Hello World app — in that exact order — and the scripts for tearing them down in the reverse uncommented. Then, I used the make init command to get everything set up. The kubectl get pods -A command then successfully showed all the pods that Kubernetes was running

I then took the names for all the three pods that Kubernetes ran for this app and used the kubectl describe pod -n app <POD_NAME> to see the environment variables inside them. While I successfully did this over all three pods, the following screenshot will show the output for only one kubectl describe pod -n app hello-world-app-deployment-5df7d46d89-pqmkw

Notice that it was very important to have the -n app flag in the command to get the desired results. This is because Kubernetes namespaces are like separate workspaces, and kubectl commands default to searching in the default namespace unless you tell it otherwise. With no application pod in the default namespace, I would get error or a non-existent result for the command attempts.

Excellent. We cleared the first hurdle. Now it was time to get the secrets fed into the container from the ExternalSecret resource. To do this, I spun down the infrastructure that I previously created with the make uninstall command and carefully and slowly read the documentation page to see how the secrets are to be passed into a Kubernetes-managed container, and got to work.

I made a mild tweak to the ExternalSecret resource YAML file to make its name be hello-world-external-secret — different from the name of the target/name: hello-world-secrets — since I didn’t want the two names to be exactly the same. The ExternalSecret resource YAML now looked like this:

And then I updated the helloworld-deployment.yaml file in accordance with the documentation. The secrets were to be piped into the container’s environment variables, so I modified the containers/env section — as seen per the following screenshot

Notice that I explicitly referred to five secrets being fed in from the Kubernetes-run hello-world-secrets by name. The documentation had an interesting note at the bottom of the page that if “you have a large number of secrets and you want to pass all of them as environment variables” you can basically import them in one fell swoop. Because I only had five secrets and not a significantly larger number — and that the bulk of the documentation relied on the “name the secret explicitly” approach — I went with the approach of explicitly naming each one that I wanted to feed into the container.

Then I used my Makefile with the make init command-invoked scripts to launch the infrastructure setup, entered the script-specified five secrets into the Vault, and had it spin up the application pods. After everything was finalized, I used the kubectl get pods -A command to see what pods were up and running

Three application pods. Just as expected. I then used the kubectl exec -n app <POD_NAME> — env command on each of the three to see the environment variables inside them. While the following screenshot is for one specific pod, all three had the same result of my five secrets being successfully piped in:

The last step was to have my application reference the environment variables that I fed into the container — and to present them onto the screen. To do that, I quickly added some application version and secret-presenting HTML code into my index.html file

And then opening up the browser, I got the final application presented

Epic. It should be noted that in the real world, we would never even consider injecting secrets into a Web server since those are publicly accessible and liable to be pwned. This was only done here for homelab demonstration purposes.

This was the most involved section of any homelab of mine to date, but at the same time, it was the most beneficial. Secrets need to be handled properly and I’m told many shops do not encapsulate and modularize the complexity with secrets management well. While this text-writeup of what I did was relatively brief, the code itself was significantly longer. For those interested, you can find it on my GitHub.

Section IX: CI Pipeline with DevSecOps Quality Gates

Up until now I had been building Docker images of the application and pushing them to Docker Hub in a section of my Makefile-invoked setup_hello_world_app.sh shell script. This approach works in the context of a 1-man homelab but definitely isn’t how real teams ship dozens of times a day.

That’s why the next step was to build a proper CI Pipeline. The big idea of Continuous Integration is that every single code change gets automatically tested, built, scanned for syntax and security issues, and validated before it has any chance of reaching production. The goal is to answer the question each time code is pushed: “Is this change safe and good enough to move forward?”

For actually running this CI pipeline I chose GitHub Actions. It’s the built-in CI/CD system that lives directly inside every GitHub repository — which made it the fastest and most natural choice. Because everything (code + manifests + pipeline definition) already lives in the same GitHub repo, I get zero extra accounts, no new webhooks to configure, and the pipeline can easily read my code, build the image, push it to Docker Hub, and even commit specific updates back to the same repo as needed!

To start, I needed to securely connect my CI pipeline to both Docker Hub and GitHub. To do this, I opened up GitHub → Settings → Secrets and variables → Actions

In a separate Window, I followed the steps outlined in the GitHub page for managing personal access tokens to create a personal access token scoped on the HomelabSeriesGithubRepo with read and write permissions, so that a workflow could safely commit specific updates back into the repository.

After that, I quickly made the following Repository Secrets: DOCKER_USERNAME with my Docker Hub username, DOCKER_PASSWORD using a Docker Hub access token (instead of my account password for better security), and PERSONAL_ACCESS_TOKEN — the text string of that personal access token that I just made.

Then I created a .github folder for the repository. It is a specialized directory used to store project-specific configurations, such as automation workflows (GitHub Actions), issue templates, and community health files.

Epic. I then created inside it the workflows sub-folder. This sub-folder is **strictly required** for Workflow definitions. GitHub Workflows are the YAML configuration files that define automated CI and CD pipeline jobs. In this case, this is where I needed to place instructions for how my CI pipeline would work and operate.

With the groundwork laid, I created the ci.yml file inside the .github/workflows/ folder to define my entire CI pipeline as code. This workflow automatically triggers on code changes, runs a series of quality and security gates (specifically YAML linting, CodeQL security scanning, Docker container building, and Trivy scanning the container for vulnerabilities). Then, if everything passes — it updates the Kubernetes manifest with the new image SHA and commits it back to the repo, creating a fully automated path from code push to deployment-ready artifact. This is DevSecOps in practice: automated security checks are baked into the pipeline, not bolted on afterward. Code changes deliberately should not be able to bypass them.

You can see the CI Yaml file on the GitHub page with the actual steps. Notice that it was carefully configured to trigger only upon changes to the Homelab16 — GitOps and Secrets Management sub-folder, since this GitHub repository contains multiple homelab projects and I only wanted CI to trigger for changes specific to this project.

Excellent. After getting the CI YAML file pushed into the GitHub repository, I then committed and pushed a small test change to the app.py Flask application file to test it. As expected, the CI pipeline got activated — and I got the following for the first round of CI pipeline execution:

The YamlLint did its job and provided 10 errors and 9 warnings regarding the way the YAML files were written and formatted. That is the entire point of having a linter! Notice that the CI job halted execution since this stage of the pipeline failed.

After going in through the findings and fixing them, I pushed the changes to the GitHub repository again — which triggered the pipeline. This time, I got through all but the third-from-last stage of the CI pipeline, with AquaSecurity’s Trivy Container scanner catching 3 High Severity vulnerabilities in Python packages.

Houston, we have a big problem. Per the red boxes of the middle screenshot, we see that the three vulnerabilities were in the Python packages within the usr/local/lib/python3.10/site-packages/ folder. Looks like the Docker container’s Python needed to be updated to a non-vulnerable version. We currently had the following at the container’s top:

I needed to get a more recent (and hopefully non-vulnerable) Python image for my container, so that it would have newer versions of these packages by default. I went onto the Docker Hub’s page for Python images, found that the latest version is 3.14-slim, rewrote the Dockerfile’s first line to be FROM python:3.14-slim, and then pushed the changes into the repository. This naturally triggered the CI pipeline again — and now we got this

Mixture of good news and bad news. The good news is that the fixing of the Dockerfile allowed us to get through the Trivy container-scanning quality gate. The bad news is that there was a permissions error that did not allow us to get through the final stage of this CI pipeline. The permissions error occurred because my Personal Access Token was missing the “Contents: Read and write” permission required for the workflow to push commits back to the repository. I regenerated the token through GitHub’s Developer Settings, ensuring it had both Contents (Read and write) and Workflows permissions, then updated the PERSONAL_ACCESS_TOKEN secret in the HomelabSeriesGithubRepo repository settings.

After pushing another test change, the CI pipeline executed successfully end-to-end — validating the code, building and scanning the container, updating the manifest with the new image SHA, deploying the container into DockerHub, and committing the change back to the repository automatically. However, this triggered a secondary issue: the bot’s commit created a line in the deployment manifest that exceeded yamllint’s default 80-character limit (due to the long commit SHA), causing subsequent CI runs to fail. I resolved this by configuring YamlLint to allow up to 120 characters per line (via a .yamllint config file placed in the HomelabSeriesGithubRepo root folder), which accommodated the naturally long Docker image references while still catching genuinely problematic formatting.

Screenshots of Successful CI Pipeline Execution:

Screenshot of the Github Action’s Commit — It automatically updated the helloworld-deployment.yaml file to ensure that the container image the application would pull in was of the right version — represented by the hash

Screenshot of the Image In DockerHub

One More Note Regarding What Was Done: Originally, when I started on the project, my helloworld-deployment.yaml file had the line “image: nathanpavl/sample_hello_world_flask_app:latest” to tell Kubernetes to pull the latest image. I ensured that Kubernetes would pull in a specific image represented by a hash, after it got built, passed the CI pipeline, and uploaded to DockerHub successfully. This takes advantage of the idea that everything in DevOps should be a hashable artifact — and is important from a security perspective: if an attacker got into Docker Hub and put in a new image, they would not be able to tamper with the application to make it execute a non-approved version (unlike the case of having the :latest tag in the Kubernetes manifest).

We went through a lot here. In hashing we trust — except MD5!

Section X: Decommissioning Manual Application Deployment Scripts

Remember I in the previous section created bash scripts for setting up the vault, secrets, and setting up the application — in that order? I also created their idempotent counterparts for cleanly and completely undoing the changes, and had them executed in the reverse order.

Due to the fact that I made the Continuous Integration (CI) pipeline, there was no further need for the setup_hello_world_app.sh and uinstall_hello_world_app.sh scripts anymore. The CI pipeline already cleanly [and far more securely] built the container image and pushed it into Docker Hub than anything the setup_hello_world_app.sh script could have done. Additionally, it is ArgoCD which will handle the Continuous Deployment and the actual execution and deployment of the CI-pipeline-built container image.

I put in a lot of effort and care into these two scripts, so I didn’t want to delete them. I instead created a scripts/old_scripts folder, moved those two into it, and removed the references to those in the Makefile.

Section XI: Continuous Deployment Pipeline With ArgoCD

In the previous sections, I was able to build the application with the Continuous Integration pipeline running on GitHub Actions. The last part remaining was building a Continuous Deployment (CD) pipeline with ArgoCD. A Continuous Deployment pipeline automates the deployment of validated code changes to production environments — ensuring that every commit that passes CI automatically flows into the live system without manual intervention. I chose ArgoCD because it implements GitOps, where Git becomes the single source of truth: ArgoCD continuously monitors my repository for manifest changes and automatically syncs them to the Kubernetes cluster, giving me declarative infrastructure management with built-in rollback capabilities and a complete audit trail of every deployment.

First, I read the ArgoCD website’s “Getting Started” documentation article about how ArgoCD is created and deployed, and wrote a setup_argocd.sh and uninstall_argocd.sh bash scripts that were completely and fully idempotent. [And yes, I heavily tested the idempotency — that the uninstall_argocd.sh script completely reversed everything that the setup_argocd.sh script did.] The setup_argocd.sh script installs ArgoCD into the Kubernetes cluster via kubectl, waits for all components to be ready, sets up port forwarding to expose the UI at https://localhost:8080, and retrieves the auto-generated admin password for first-time login.

Launching it, we get:

And opening the website and entering the script-outputted credentials, we get:

Great. I then needed to create an ArgoCD application manifest using a — you guessed it — another YAML file! To this end, I first made an argocd sub-folder, and then used the Declarative Setup, the argocd-cm.yaml, and Application Specification documentation pages to make it. It looked like this:

Notice that it keeps track of changes made to the Kubernetes manifests in the Homelab16 — GitOps and Secrets Management/app/kubernetes_manifests folder. When the CI pipeline finishes executing and updates the tag in the helloworld_deployment.yaml file, ArgoCD will detect that and automatically deploy the newly-generated container fresh into production!

I then needed to create another (yes, another) shell script that would apply the ArgoCD application manifest once the ArgoCD application was spun up. The reason why I needed to do it separately was an enterprise best practice of separating concerns: first we spin up ArgoCD, then we can deploy any application of our choosing into it, and can swap the specific application in and out in a modular way. I thus created the setup_app_in_argocd.sh and uninstall_app_in_argocd.sh scripts and added them to the Makefile.

Then, I used the make init command to launch the application and see if I could get everything working. As per the previous tests, I was able to get ArgoCD opened — but the application was not successfully deployed.

I then used the kubectl describe application hello-world-app -n argocd command to get information about what was going on — and got the following

The key message was “Application referencing project Homelab16 which does not exist” — gotcha. I needed to create a project-specific YAML file since I wanted to create a project called “Homelab16” — not a project of the default name default. I then created a homelab16-project.yaml file in the argocd/ folder to define the ArgoCD Project, which provides logical grouping and security boundaries for applications. However, after resolving this and re-running the pipeline, I encountered another issue: the pods were failing with CreateContainerConfigError.

The root cause became clear after examining what ArgoCD was deploying — it was only syncing the helloworld-deployment.yaml and helloworld-service.yaml files, but not the secretstore.yaml and externalsecret.yaml configurations that External Secrets Operator needed to inject secrets into the containers from the Vault. These ESO configuration files were sitting in scripts/setup_scripts/configuration_yamls/, which my shell scripts manually applied via kubectl apply — but ArgoCD was watching app/kubernetes_manifests/ and had no visibility into that separate directory.

This violated the core GitOps principle: Git should be the single source of truth, with ArgoCD deploying everything declaratively from a single location. To fix this, I moved secretstore.yaml and externalsecret.yaml into app/kubernetes_manifests/ alongside the application manifests, refactored setup_external_secrets.sh and uninstall_external_secrets.sh to remove the manual kubectl apply commands (archiving the old versions in an old_scripts/ folder for reference), and let ArgoCD handle deploying all four manifest files automatically. I also moved the vault-config.yml file into a dedicated vault-related folder, given that the vault is not the app and more of a “cluster service.”

This approach aligned the entire stack with proper GitOps architecture: infrastructure scripts (setup_vault.sh, setup_external_secrets.sh, setup_argocd.sh) install the platforms, while ArgoCD manages application deployment from Git.

I then ran the pipeline for getting my infrastructure spun up using the make init command and opened up ArgoCD to this

That looked good. What about the application at http://localhost:5000?

It was up and running. Yay!

Section XII: Testing The Pipeline End-to-End

The last thing that I wanted to do is to test the CI/CD pipeline end-to-end. I did this by opening up the app.py file and changing the version number from 1.0 to 2.0

I pushed to the repository, and the changes successfully passed the CI’s checks

And then GitHub Actions successfully updated the helloworld-deployment.yaml file for the right container image’s hash

Then looking into ArgoCD, I saw that it saw the changes that the GitHub Actions’ commit did to the helloworld-deployment.yaml file — and automatically triggered an update

Opening up the website-shown application and refreshing the browser window, I got the expected change displayed

Wonderful!

Section XIII: Spinning Down The Infrastructure

With this exercise being over, I needed to spin down the entire infrastructure that was created. I did this by typing the make uninstall command, which per the Makefile invoked the following scripts in order:

./scripts/uninstall_scripts/uninstall_app_in_argocd.sh

./scripts/uninstall_scripts/uninstall_argocd.sh

./scripts/uninstall_scripts/uninstall_external_secrets.sh

./scripts/uninstall_scripts/uninstall_vault.sh

It is important for me to re-iterate — each script was designed to idempotently “undo” every single action/installation that its installation counterpart did.

Then I used the kubectl get pods -A and kubectl get namespaces command to confirm that there were no pods or namespaces left over

All the code and files can be found on my homelab GitHub page.

What started as curiosity about how enterprise teams deploy securely dozens of times per day ended with me building exactly that — a fully automated GitOps pipeline where code changes flow through security gates, become immutable artifacts, and deploy to production with a simple git push and zero manual intervention. This homelab proved that enterprise-grade DevSecOps practices are accessible and implementable. Once you experience declarative infrastructure with security baked in at every step, manual deployments truly feel like ancient history. Thanks for reading!

Think of a landing zone as building a well-planned neighborhood before anyone moves in: it has organized streets (hub-and-spoke networking for centralized connectivity and security), building rules everyone follows (policies and guardrails like IAM roles and SCPs to enforce standards), shared utilities (logging with CloudTrail, monitoring via CloudWatch, and cost alerts), and separate lots for different houses (multiple accounts or VPCs to isolate workloads). This acts as a strong foundation that ensures new projects can “land” quickly and safely without chaos, security holes, or surprise bills. It is very important to get this right.

Microsoft’s Azure pioneered the detailed “landing zone” concept, but AWS has its own mature equivalent, which provides the same core ideas: multi-account isolation, governance, and scalability.

I chose AWS here because it was a good alternative cloud provider — without the account signups and eligibility issues that I experienced when attempting the initial lab using the Microsoft accounts. Onwards!

Section I: Initial Preparations

I first opened up the PowerShell console with Administrative privileges on my Windows homelab machine and confirmed with the aws sts get-caller-identity command that my AWS account credentials from past homelabs were still configured and on the machine

I also had Terraform installed on my workstation from past homelabs. I also made a folder dedicated to this homelab in my HomelabSeriesGithubRepository for the code and all the related files to go in.

Section II: Terraform

I initially took some sample code from my Homelab #12 and made the following initial Terraform files and their related configurations:

After that, I read a little bit about how these landing zones are made and found that I need to make an AWS Organization. AWS Organizations lets you manage multiple AWS accounts from one main account. It groups accounts into organizational units (OUs) and applies shared rules like security policies (SCPs), consolidated billing, and features such as CloudTrail logging or GuardDuty across all accounts.This keeps everything organized, secure, and cost-controlled as your cloud setup grows. I then found an aws_organizations_organization Terraform Registry page, and copied the appropriate code snippet into the main.tf file:

And then inserted the following code of lines into the output.tf file:

Next, I added Organizational Units (OUs) under the root of the AWS Organization.

OUs act like folders that group related AWS accounts together — specifically one for Security and one for Sandbox testing. They allow you to apply different security rules or policies to each section instead of treating everything the same. This grouping is essential for scalable governance and is required before you can properly deploy Control Tower or enforce targeted controls.

This was done with the following lines in the main.tf file:

Having Organizational Units (OUs) is going to be essential for the later usage of an AWS Control Tower instance. Per the involved documentation, an AWS Control Tower resource instance requires the Amazon Resource Number of organizational unit(s) as the target_identifier parameter when created. They automatically organize and place the accounts that are in these OUs into the right groups, so that consistent security rules, logging, and guardrails can be applied and enforced.

I deployed an AWS Control Tower landing zone using Terraform’s aws_controltower_landing_zone resource to create a governed, multi-account foundation aligned with NIST SP 800–53 Rev. 5 principles. The manifest_json configuration field defined a Security and Sandbox OU structure, mandatory accounts (Management, Log Archive, and Audit), and enabled centralized logging with 365-day retention in dedicated buckets — supporting key NIST controls like AU-2 (audit events) and AU-11 (retention) by automatically provisioning organization-wide CloudTrail trails and isolating audit data.

To add deeper NIST compliance checks, I then found and decided to use the official AWS Config conformance pack (aws_config_conformance_pack) using the public “Operational Best Practices for NIST 800–53 rev 5” template, which I found on a GitHub page. This pack activates dozens of automated Config rules that are to continuously evaluate resources against NIST standards (e.g., MFA on root accounts, encrypted volumes, restricted security groups, and proper IAM policies), providing ongoing monitoring for critical families like Access Control (AC), Configuration Management (CM), and System & Information Integrity (SI). Think of it as an automated compliance auditor running 24/7.

That same webpage has plenty of other packs for other compliance standards, including CMMC, FedRamp, PCI-DSS, and HIPAA as well. It’s a very useful repository.

The Terraform code that I added into the main.tf file was as follows:

Then I used the terraform plan -out tfplan command and got it to work successfully, with the tfplan file being generated

Then I used the terraform apply tfplan command and got the following two errors

Notice the first one related to the creation of the ControlTower Landing Zone and not having the AWSControlTowerAdmin permissions.

I spent an entire afternoon troubleshooting — creating IAM roles, adjusting trust policies, and adding wait times for IAM propagation. However, after consulting AWS documentation and best practices, it became clear that AWS Control Tower’s bootstrap process is simply too complex for full automation from scratch. The IAM Identity Center prerequisites, service-linked role dependencies, and circular service integrations are intentionally designed to be handled through a one-time manual setup. This wasn’t a failure of Infrastructure-as-Code; it’s an intentional design choice by AWS.

During this process, I also attempted to use Terraform to deploy a NIST 800–53 conformance pack, and wrangled IAM role assumption errors, duplicate OU conflicts, import workflows, and permission denials — and decided that Terraform was also not worth it.

When Plan A fails, one shifts to Plan B.

Section III: Manual Landing Zone Creation

Rather than continue fighting against the platform’s intended workflow, I opted for the pragmatic approach of doing everything manually. Switching to manual configuration allowed me to focus on the core concepts of landing zones, NIST alignment, and security measures without getting bogged down in automation hurdles. This pragmatic choice reinforced an important lesson: sometimes the fastest way to learn and demonstrate the value of a governed cloud foundation is to work directly with the platform’s intended workflow.

So, I got to work to make the Landing Zone manually. I opened up my AWS account, and went through the steps of creating it

And here is what I got:

Section IV: Automating NIST 800–53 With AWS Config

Next up was to get the NIST 800–53 conformance pack deployed. I opened up the AWS Config → left menu → Conformance Packs page, and went through the deployment steps

After waiting a bit, the deployment was completed

Opening up the panel, one could see both the compliance score and the rules down below, as well as the status as to whether we were compliant or not

For example, we saw that the root account did not have MFA enabled. Enabling it would allow for the rule’s requirements to be met and the compliance score to appropriately go up.

Then, I noticed that I initially forgot to enable the AWS Config recorder (required for the conformance pack to actually evaluate resources), so I quickly fixed it manually in the console to activate continuous monitoring and see the NIST rules in action.

The AWS Config recorder is the engine that continuously captures and tracks the configuration state of all your AWS resources — such as EC2 instances, S3 buckets, IAM policies, and security groups — creating a historical timeline of changes so you can audit, troubleshoot, detect drift, and run compliance checks like the NIST conformance pack.

Without it, AWS Config has nothing to evaluate, so the conformance pack rules stay silent; enabling the recorder turns on the “always-on monitoring” that makes a governed landing zone truly effective for ongoing compliance and security visibility.

Section V: GuardDuty For Security Measures (Threat Detection — SI-4 Compliance)

I took a look at some of the non-compliant rules, and saw that there was a rule checking if Amazon GuardDuty was enabled in my AWS account and region.

For those who don’t know, AWS GuardDuty is a continuous security monitoring and threat detection service that automatically analyzes your AWS account logs, network traffic, and resource activity — to catch malicious IP calls, unauthorized access attempts, or unusual behavior. It uses machine learning and threat intelligence to identify potential security issues in real time, alerting you so you can respond before damage occurs — making it a key layer of protection in a governed landing zone.

Time to get it enabled. I did this by opening the Amazon GuardDuty page and clicking on the Get Started button:

And the Enable GuardDuty

And it currently is enabled and didn’t show anything abnormal or malicious due to the fact that my landing zone had no resources or instances deployed in it

Then I needed to get the GuardDuty delegated administrator configured. The Aggregator account is one of the mandatory AWS Control Tower automatically created during landing zone setup, specifically designated to centralize and store security-related data like audit logs and findings from across the organization.

I had to designate it as the GuardDuty delegated administrator so that GuardDuty could collect threat detection data from every account in the organization into this single, secure location — ensuring centralized monitoring and compliance with NIST SI-4 requirements for continuous system monitoring and incident response.

And then reopening the rule pack up, I got the following change in compliance status

It is compliant now!

Section VI: Creating and Attaching SCPs for Governance

Next, I worked on adding some Service Control Policies (SCP) to enforce governance across the organization.

SCPs are powerful guardrails in AWS Organizations that act like organization-wide rules. They define what actions users and roles are allowed (or denied) to perform in any account, no matter who is logged in or what permissions they have. They are used to enforce security boundaries, prevent risky actions (like root user operations), restrict regions, or require tagging — making them essential for maintaining least-privilege access and compliance in a multi-account landing zone like this one.

The Service Control Policies page can be found by opening the AWS Organizations >> Policies >> Service Control Policies page

I went ahead to create four SCPs with the repeated usage of the Create Policy button:

• SCP #1: Block Accounts From Leaving the Organization (CM-2)

We want to be in full control of accounts in this enterprise, so we want to block them from having the ability to leave the organization. The AWS documentation has some nice pages regarding the LeaveOrganization action, which is performed by an account that wants to leave.

I did this with the following step. If there are any questions regarding the SCP policy syntax, they can be addressed by looking at the pertinent documentation page.

This prevents accidental or malicious removal of accounts from the Landing-Zone maintained environment and adheres to the CM-2 — baseline configuration compliance standard

• SCP #2: Deny All Regions Except For The One Currently Used:

In this manual deployment of an AWS landing zone, my work has been entirely confined to the us-east-2 region. I needed to prevent the accidental use of all other regions — and wanted to ensure that nothing would slip under the nose. (There was a nice Medium page that had a rule that I used and tweaked)

This is in good adherence with the NIST 800–53’s SC-7 Boundary Protection and CM-6 Configuration Baseline compliance standards

• SCP #3: Require Tags on All Resources:

Remember tags from Terraform, with code that looks like this?

That was the first time we worked with tags over the course of this homelab series. It should be noted that such tags can also be made manually as well.

I wanted to ensure that resources are always created with tags — which are great for ensuring inventory enumeration and cost tracking. Notice that per documentation, we can use the aws:ResourceTag/<KEY_NAME> field to specify what resource keys/tags we needed.

Here is what I did:

Notice the usage of the following condition statement:

It denies resource creation if either the environment or owner tag is missing

This SCP helps us meet the NIST CM-8 (System Component Inventory) compliance standard

• SCP #4: Cost Optimization: I came across this SCP when going through the AWS GitHub page of Service Control Policy Examples and found this one rather cool. It is less driven by the need to comply with the NIST 800–53 standard and more to ensure that we minimize the risk of cost overruns.

It ensures only free-tier eligible instances (t2.micro, t3.micro) can be launched, preventing costly instance types

This was fun. I learned a lot about what SCPs are and what they do. Now I see why having such landing zones are very important — they ensure that anything that gets spun up later is only done in a proper way.

Here is what I had in terms of SCP policies at the end of this section

Section VII: Enabling and Configuring AWS Security Hub

Next, I enabled AWS Security Hub to bring together all security-related findings from GuardDuty, the NIST conformance pack, and other services into one centralized dashboard. This step was essential for gaining a unified view of the landing zone’s security posture, including an overall security score and prioritized alerts, and to be able to turn scattered data into actionable insights for ongoing monitoring and compliance.

I did this by opening up the Security Hub page in the web console

Then I clicked the Get Started button and accepted the default configurations

When enabling AWS Security Hub, I designated a delegated administrator account (the Aggregator account from my Control Tower setup) to centralize security findings across the entire organization. This delegation allows Security Hub to collect, aggregate, and prioritize alerts from GuardDuty, AWS Config conformance rules, and other services into one unified view within the designated admin account, ensuring consistent monitoring and faster incident response across all accounts in the landing zone.

Then, I selected the checkbox for Enable Security Hub For My Account to kick things off

Nice. The AWS Security Hub was activated. It should be noted that this is actually the older name of the AWS service — the updated name is AWS Security Hub Cloud Security Posture Management (AWS Security Hub CSPM).
While the AWS Config conformance pack that I worked with in this writeup’s Section IV provided detailed, resource-level rule checks against NIST standards, enabling the NIST standard in AWS Security Hub CSPM was also necessary because it aggregated those results (along with GuardDuty findings and other sources) into a broader, organization-wide perspective with scoring and prioritization — ensuring both granular compliance validation and high-level security oversight in one tool.

I got this quickly enabled by opening up the Security Hub CSPM >> Security Standards page, finding the NIST Special Publication 800–53 Revision 5 page and clicking the Enable Standard button

I also popped into the left-side Integrations menu and confirmed that the Amazon GuardDuty and AWS Config integrations have their findings being fed into/accepted by the Security Hub CSPM

Everything is good now. Notice that I could open the Findings tab to see what was up:

This page was definitely interesting to read. I was very happy with what I had done in this section so far, so I decided to move on to the next round.

Section VIII: Implement Cost Controls

Now, for some cost controls! And no, we can’t abbreviate it as “cost co” — that’s a retail chain (see what I did there?)

Cloud spend is finicky and sensitive — and all those credits that AWS gave me for learning aren’t unlimited. So off I went to get the appropriate budgets created. I opened up the AWS Budgets page and clicked the Create a Budget button

And then decided to create a ten-dollar budget because you know, ten dollars is ten dollars, and pushed that over the finish line

Section IX: Testing WIth A Sample EC2 Resource

The very last action that I needed to take with regards to this landing zone was to create a sample EC2 instance to test it. I did this by opening the EC2 page in the AWS website console, and clicking the Launch Instance button.

Then, I selected the configurations, which are shown per the following screenshot

Notice that I had the Amazon Machine Image (AMI) be of the t3.micro free tier — and included the tags that were required by the SCP rule that I previously made. Specifically, the tags’ key / value pairs were Environment / Testing and Owner / Nathan. I also selected the option to have the AMI be configured without a key pair. I didn’t care too much about keys for this lab.

I then got the instance made and waited about 2–3 minutes for it to be initialized — and then opened it up.

Cool. I then opened the AWS Config >> Rules page to see what whether it would be flagged per the NIST 800–53 conformance pack, and got the following

I didn’t see it appear at all in the AWS Config rules list — instead, the dashboard showed only a handful of non-compliant items like unencrypted CloudTrail logs, missing S3 data events, EBS encryption by default disabled, IAM password policy gaps, and root account MFA not enabled.

I then also checked GuardDuty. Per the screenshot, the EC2 instance was also not flagged.

Launching this test EC2 instance and observing the conformance pack and GuardDuty behaviors — even if no immediate flags appeared for the instance itself — highlighted how AWS Config and security services focus on ongoing, configuration-driven monitoring rather than instant detection of every resource, reinforcing that a true landing zone’s strength lies in continuous compliance and threat visibility over time rather than one-time checks.

Section X: Teardown

The main bulk of the homelab has been completed. I now needed to clean everything up so as to leave nothing remaining and minimize the risk of any potential cloud spend costs piling up:

• AWS Control Tower: I opened up the AWS Control Tower >> Landing Zone Settings page, selected the Decommission tab, and clicked the Decommission Landing Zone button

• SCPs: I opened up the AWS Organizations >> Policies >> Service Control Policies page, and then selected all the SCPs that I previously made (they were of the “Customer Managed Policy” kind)

• GuardDuty: I opened up the GuardDuty >> Settings page and clicked the Disable button

• SecurityHub: I opened up the SecurityHub >> General page and clicked the Disable button

• Conformance Packs: I opened up the AWS Config >> Conformance Packs page and selected the Delete action for the NIST 800–53 Conformance Pack

• AWS Config Recorder: I opened up the AWS Config >> Settings page and turned the recorder off
• Security Hub CSPM: I opened up the Security Hub CSPM >> Security Standards page and then disabled the NIST 800–53 standard for the sake of being thorough

I also selected the Integrations tab and clicked the Stop Accepting Findings for AWS Config and GuardDuty

All these steps were undertaken to ensure no lingering resources or potential costs remained after the lab. I kept the ten dollar budget in force to ensure that I would get notified in case something *did* happen to remain to continue piling up the costs.

Section XI: Conclusion

Looking back on Homelab #15, building this AWS multi-account landing zone — even with its share of Terraform struggles, permission hurdles, and manual pivots — gave me a much deeper appreciation for why companies invest so heavily in governed foundations. Control Tower provided the structure, the NIST conformance pack added automated compliance checks, GuardDuty brought threat detection, SCPs enforced boundaries, Security Hub gave centralized visibility, and cost controls kept things realistic. While I started with full IaC ambitions, the real lesson was learning when to adapt and work with the platform’s intended flow. Most importantly, I now understand that a true landing zone isn’t just about accounts and tools — it’s about creating a secure, observable, scalable environment where teams can build confidently without constant firefighting. That’s knowledge I’ll carry into every future cloud project.

Welcome back to this round of homelabs. In this episode, I wanted to continue learning more about networking in order to continue growing the skills. The thing with Networking is that there is always more to learn, so you can’t be just finished with a quick byte (see what I did there?)

In this round, I built a small business network in Cisco Packet Tracer from scratch, with the following network architecture:

Onwards!

Section I: Network Foundations

To start, I opened up Cisco Packet Tracer and put down the core components of my [simulated] network that I wanted to get fully configured and connected. Specifically, these were 1 router, 1 switch, and 6 PCs (two of each belonging to the Sales, IT, and Guest users — and which will be placed on their own VLANs). The network looked like this:

Next up was to get everything connected. Remember from Homelab #3 that straight-through cables connect systems of different types? This is the same case here — with straight through cables being applied to connect the PCs to the Switch and the Switch to the Router.

Next up was to configure the switch. Specifically, I needed to get all the VLANs created on it, the appropriate interfaces configured, and the trunking port for configuring the trunk port that will carry the traffic from the switch to the router. To do this, I opened up the Switch’s CLI and got to work with the enable and configure terminal commands.

I first had to create the three VLANs: VLAN 10 (Sales), VLAN 20 (IT), VLAN 30 (Guest) with the vlan <VLAN_ID> commands:

As of now, here is what I had per the show vlan command:

Then I needed to get them assigned to the appropriate ports. This was done in the configure mode with the following commands.

• VLAN 10 (Sales): The interface range f0/1 — f0/2 command was used to get access to the FastEthernet Ports #1 and #2, to which the two PCs are connected. Then I used the switchport mode access command to set it to operate in access mode, meaning it carries traffic for only one specific VLAN — and then the switchport access vlan 10 command set the two computers attached to them as being part of VLAN #10 (sales)

• VLAN 20 (IT):

• VLAN 30 (Guest):

The logic behind the commands used for configuring the ethernet interfaces for the Sales VLAN was exactly the same as for the commands used for the IT and Guest VLANs too, so I didn’t bother to rewrite extensive explanations for the obvious.

Next up was to configure the Trunk port for communicating from the switch to the central router. For those who don’t remember, a Trunk port carries traffic for multiple VLANs over a single physical link. It uses tags per the 802.1Q protocol to identify each packet’s VLAN.

I did this by using the interface gi0/1 command to enter the appropriate GigabitEthernet 0/1 interface, and then used the switchport trunk allowed vlan 10,20,30 instructions to set it to allow traffic from VLANs 10, 20, and 30.

Then a quick saving operation with the end and write memory commands quickly got those changes “locked and loaded.” Then a quick typing of show vlan brief showed what we ended up with the final configurations on the core switch on:

I then got to work with configuring the router. First I used the hostname “Central_Router” command to set its name

I then needed to configure the switch-facing interface appropriately.

First, I needed to bring the interface up with the interface GigabitEthernet0/0/0 command, and then used the no shutdown instruction to bring it up. By default, most router and switch interfaces are disabled, so using this command turns it on for usage.

Then I needed to get the sub-interfaces configured for each of the three VLANs. This was done for VLAN 10 with the following commands:

interface gi0/0/0.10

Explanation: this command sets the sub-interface interface of Gi0/0 (which faces the switch per the network diagram) to VLAN 10

encapsulation dot1Q 10

Explanation: This command configures the router’s interface to use the IEEE 802.1Q standard to tag traffic for VLAN 20. The router does it by adding the 802.1Q tags to Ethernet frames — and these will be set to the value of 10, corresponding to the VLAN 20.

ip address 192.168.10.1 255.255.255.0

Explanation: We set the IP address ranges for this VLAN

no shutdown

Explanation: We ensure that this sub-interface remains up

Great. I then did similar commands for the other VLANs as well.

• VLAN 20 (IT):

• VLAN 30 (Guest):

And locked the changes with the exit and write memory instructions.

Notice that we now have green triangles appear on the Cisco Packet Tracer application indicating a successful Layer 2 connection has been formed.

The next step was to do what is typically done only on home and small office networks: to have the router also function as the DHCP server. Larger enterprises typically have their own separate DHCP servers.

I entered the configuration mode with the enable and configure terminal commands and then first entered in the following commands to exclude the first ten IP addresses for the Sales, IT, and Guest departments’ VLANs.

This was done to reserve them for static assignments like the default gateway, servers, printers, or network infrastructure devices that require predictable, unchanging IP addresses.

Next up was to configure DHCP pools. Each VLAN requires its own DHCP pool configured on the router, which provides clients with an IP address, subnet mask, default gateway, and DNS server.

Notice that the default-router parameter is essential because without it, devices can only communicate with other hosts on their immediate subnet — they have no way to reach other VLANs or external networks. When a device needs to send traffic outside its subnet (for example, a Sales PC reaching an IT server or accessing the internet), it forwards that traffic to the default gateway, which is the router’s subinterface IP for that VLAN. This is why the default-router value (192.168.10.1, 192.168.20.1, 192.168.30.1) must exactly match the IP address we configured on each corresponding subinterface — the router must be reachable at that address to route traffic.

The commands used were as follows:

I then opened up each of the PC-related panels, and set the Gateway/DNS IPv4 option to DHCP — and all the PCs got the right IP addresses corresponding to the VLANs they were in!

Sample Screenshot For SalesPC1:

Then on the central router, I used the show ip dhcp binding command to see what IP addresses were assigned to what systems (identified by their MAC addresses):

I then opened the command prompt on the SalesPC1 and then pinged all the IP addresses listed from the previous screenshot — except for the 192.168.10.11 IP address, which belonged to the SalesPC1 itself. All the pings worked!

I then also opened up the command prompt on GuestPC1 and did the same for all the IP addresses corresponding to the hosts on this network.

They worked!

Section II: Security With ACLs

Remember the pings I was able to make from SalesPC1 and GuestPC1 to all the other computers on the other VLANs? This is indicative of a security problem: Guest users could access the Sales and IT networks, and there was no traffic segmentation beyond simple subnet separation!

Access Control Lists (ACLs) solve this by providing granular traffic filtering at the network layer — they’re essentially firewall rules applied directly on routers and switches. In enterprise environments, ACLs complement dedicated firewalls rather than replace them: firewalls typically protect the network perimeter (blocking external threats), while ACLs handle internal segmentation (controlling traffic between VLANs and departments). ACLs are already built into the routing infrastructure, making them ideal for this use case — no additional hardware required, minimal performance impact, and they operate at wire speed on the router’s existing interfaces.

The next thing that I needed to introduce is the concept of ACL identification. To make a long story short, there are different ACL types, with the two primary and common ones being the Standard and Extended ACL. All ACL types can be identified either by numerical identifiers being within certain numerical ranges — or by name.

While in this lab I decided to go with names for ease of readability and description, others can and have gone with numerical identifiers. The following table shows which numerical ranges correspond to specific ACL types

While ACLs are a universal networking concept (every vendor implements them), the syntax and configuration I used for this section is Cisco-specific, particularly the distinction between standard ACLs (filtering by source IP only) and extended ACLs (filtering by source, destination, protocol, and port).

Off to work! First, I needed to block the Guest VLAN from accessing internal networks — and to be only able to access the internet. I opened up the Router’s configuration terminal and did the following

Here I blocked ALL IP traffic from the Guest VLAN (192.168.30.0/24) to the Sales and IT VLANs (192.168.10.0/24 and 192.168.20.0/24). The 0.0.0.255 wildcard mask specifies that any host in the 192.168.30.x range is blocked from reaching any hosts in the IP ranges of the two VLANs). Together, these two deny statements ensure that Guest users cannot access any corporate resources on either internal network.

Notice the permit ip any any line at the last screenshot’s bottom. This one is critical and often confuses beginners. Without it, the ACL’s implicit deny-all rule at the end of all statements would block all traffic from the Guest VLAN, including broader Internet access. ACLs process rules top-to-bottom and stop at the first match, so this permit statement ensures that after blocking Guest access to Sales and IT networks (lines 1–2), all other traffic from Guest devices (such as internet-bound traffic) is allowed. This is a fundamental ACL principle: be specific about what you deny, then explicitly permit everything else, or the invisible ‘deny ip any any’ at the end will block it.

Then I did the next set of commands to apply it to the interface

Notice that this command applies the ACL named BLOCK-GUEST-ACCESS to the g0/0/0.30 subinterface on the router for VLAN 30 in the inbound direction. “Inbound” means the ACL examines packets as they arrive at this interface from Guest devices. When a Guest PC tries to send traffic through the router, the ACL checks it against our rules before routing it — if it’s destined for Sales or IT, it’s dropped. Otherwise, it’s forwarded.

I also forgot to add in a remark line, which is basically the equivalent of a documentation comment for an ACL rule, so I quickly popped open the ACL by name again and added it

Then I saved the changes with the exit and write memory commands.

I then needed to create another ACL to allow the Sales staff to only access IT web services only. These web services would be hosted by the IT staff on their VLAN on ports 80 and 443 for the HTTP and HTTPS services respectively. See the screenshots below for the commands used:

Carefully consider the rules that I wrote. Remember that ACL rules are evaluated from top to bottom for any conditions that are true, and that there is an implicit ‘deny ip any any’ rule at the bottom. Here, I:

• Explicitly allowed the ports #80 and #443 for the HTTP and HTTPS services
• Permitted ICMP from any to any address on the entire network, which enables basic network troubleshooting with the ping and traceroute commands
  NOTE: In real-world hardened environments, ICMP is often hardened because of the threats that having a blanket allowal of ICMP traffic poses. I only allowed myself to do this in the context of a limited homelab exercise
• Denied all other traffic from the Sales to IT VLANs and their respective IP address ranges
• Allowed all other traffic that didn’t match earlier rules, such as Sales communicating with the Guest VLAN, Internet, etc. This was done with the permit ip any any line at the end, which was super important to have because otherwise the implicit deny ip any any rule would kick in and prevent the Sales team from opening their workstations and being able to access the broader Internet

Then I tested out the two ACLs that I deployed on the router with the write memory commands with pings.

I opened up the GuestPC1 command prompt and pinged the 192.168.10.11, 192.168.10.12, 192.168.20.11, and 192.168.20.12 addresses — and as expected got the Destination Host Unreachable results. As expected, the GuestPC1 could not connect to/communicate with the hosts in the Sales and IT VLANs.

After that, I opened the command prompt for the SalesPC1 computer and pinged the two computers in the IT VLAN:

That worked as expected. Then I pinged the two computers in the Guest VLAN — and didn’t get a response back

Don’t worry — this is exactly the behavior we expected. Take a look at the ACL highlighted in the red box below.

This rule explicitly denies all traffic originating from the Guest VLAN (192.168.30.0/24) destined for the Sales VLAN (192.168.10.0/24). Because Cisco ACLs are stateless, this deny also applies to return traffic.

When SalesPC1 sent an ICMP Echo Request to a host in the Guest VLAN, the request itself was permitted and left the Sales network successfully. However, when the Guest hosts attempted to send the ICMP Echo Reply back to SalesPC1 , that reply packet matched the deny rule in the ACL and was dropped.

As a result, the ping failed — not because the request couldn’t be sent, but because the reply was explicitly blocked by design. This behavior enforces proper isolation of the Guest VLAN and confirms that the ACL is functioning as intended.

Section III: Security With Port Security
I spent this section of the exercise learning port security. Port security addresses a vulnerability that ACLs cannot: physical network access. Even with perfect ACL configurations controlling traffic between VLANs, an attacker who gains physical access to your building could simply unplug an authorized device and connect their own laptop to the network. Port security solves this by allowing network administrators to restrict which devices can connect to each switch port, limit the number of devices per port, and automatically disable ports when violations occur. This is particularly critical in environments with publicly accessible network jacks (conference rooms, lobbies, common areas) or when you need to prevent employees from connecting unauthorized devices like personal routers, wireless access points, or additional switches that could compromise network integrity.

I got started with the following commands shown in the following screenshot (using the following linked page as a guide)

To simply explain, I enabled port security on the FastEthernet0 ports #1–6, allowed only one device per port, and made a port shutdown automatically if there was any security violation. While Cisco offers other violation modes like restrict (drops packets but keeps port up) or protect (silently drops packets), the shutdown mode is the most secure because it immediately disables the port and forces manual administrator review before network access is restored, ensuring that security incidents are investigated rather than ignored.

Additionally, the usage of the switchport port-security mac-address sticky command had the switch automatically dynamically/automatically learn the MAC addresses of the connected MAC devices currently connected to it. This ensured that the switch would only allow traffic from devices containing these MAC addresses, and would shut down the ports if an unauthorized device tried to connect.

After that, I needed to shut down the remaining FastEthernet ports on this switch, #’s 7 through 24 to prevent any unnecessary connections. This was done with the following commands

To wrap this section up, I took a look at the status of port security on the FastEthernet0/1 interface with the show port-security interface f0/1 command and the show ip interface brief command

Section IV: Internet Access (NAT) & Hardening

Next up, I took a “Cloud” device in Cisco Packet Tracer. The cloud icon (PT_Cloud) represents a generic connection to an external WAN network.

Then I used the straight-through cable to connect the router to the cloud device. Remember that straight-through cables connect systems of different types — in this case a router to the WAN “cloud” device. I then needed to bring the interface online for the connection to be formed, and to get the IP address assigned

Notice the ip address dhcp line. This command on interface G0/1 made the router act as a DHCP client on its WAN interface, simulating how a real router connects to an ISP. Instead of manually configuring a public IP address (which we wouldn’t know in advance), the router automatically requests an IP from the Internet cloud’s DHCP server — just like your home router gets its public IP from Comcast or AT&T. This is the same router that’s acting as a DHCP server for our internal VLANs on the LAN side, demonstrating how routers play different roles on different interfaces: DHCP client on the WAN, DHCP server on the LAN.

I then got to work configuring the Port Address translation (PAT), also known as “NAT overload” — to allow all devices on my private VLANs (Sales, IT, and Guest) to share a single public IP address when accessing the internet. PAT is the most common form of NAT used in homes and small offices because it conserves public IPv4 addresses (which are scarce) by mapping multiple private internal IPs to one external IP, distinguishing connections using unique TCP/UDP port numbers.

The commands I used were as follows:

That was a very long screenshot. Let’s digest this in detail. I first created a standard access list to identify the internal traffic that should be translated: access-list 1 permit 192.168.10.0 0.0.0.255 (and similarly for .20.0 and .30.0).

Then, I enabled PAT with ip nat inside source list 1 interface GigabitEthernet0/0/1 overload, which told the router to overload PAT translations using the dynamically assigned IP on the WAN interface (G0/0/1). Ii then marked the interfaces on the “inside” with the ip nat inside command for the G0/0/0 and the G0/0/0.10 G0/0/0.20, and G0/0/0.30 and the ip nat outside command for the “outside” G0/0/1 interface. I then added a default route with the ip route 0.0.0.0 0.0.0.0 g0/0/1 to ensure that there was a default pathway to the g0/0/1 interface and the broader Internet for traffic coming onto the router.

Next up was to do some basic hardening on the router. First, I used the enable secret Cisco123! command to create a password for the router’s privileged configuration mode.

Then the command username admin privilege 15 secret Admin123! created a local admin account named ‘admin’ on the router, giving it full privileged access (privilege level 15, equivalent to enable mode) right after login, while securely storing the password ‘Admin123!’ using strong hashing instead of weak or clear-text storage. In Cisco IOS, privilege levels range from 0 to 15: level 1 is the basic user mode with limited read-only commands, while level 15 grants complete administrative control, allowing configuration changes, debugging, and system management — most admins log in at level 15.

Then I needed to get SSH set up. This was done with the commands in the following screenshot:

To explain simply, I created a hostname and domain name of the hostname.domainname format (this being lab.local) because Cisco IOS requires domain-name to be set before SSH keys can be generated. From there, I generated the RSA keys of size 2048 bits that SSH would use, and used the line vty 0 4, login local, and transport input ssh commands to specify that up to five simultaneous virtual (i.e. remote) terminal connections are allowed using the SSH protocol only (Telnet is not allowed).

Next up to configure the console port — that is the physical serial connection you use when directly plugging in with a console cable. The console port has a line number of 0, thus ‘line console 0’. I did the following commands to require password authentication, and then made a banner to have a sample warning message saying “Authorized Access Only”

These steps were nice and well-explained for the Cisco Router. I then opened up the switch and quickly did the same.

Section V: Network Troubleshooting Reference Section

This section serves as my personal cheat sheet for troubleshooting Cisco switch/router issues, based on lab experience. Key commands that proved most useful:

• Layer 1: show interfaces (or show interfaces [int])
• Layer 2: show vlan brief, show interfaces trunk, show port-security, show mac address-table
  For Layer 2 Loops: show spanning-tree
  For Neighbor Discovery: show cdp neighbors
• Layer 3: show ip interface brief, show ip route, show ip dhcp binding, show ip arp
• Security: show access-lists, show ip nat translations
• General: show running-config, ping, traceroute, show ip interface [int], show version

Troubleshooting Steps

Use a bottom-up OSI method for most connectivity problems (start at Layer 1)

Layer 1 — Physical
Check cables, power, and link lights first. Use show ip interface brief — “administratively down” needs no shutdown; “down/down” indicates physical or Layer 2 issues.

Run show interfaces [int] and look for:

• Input/output errors, CRC errors, giants/runts, collisions, late collisions
• Speed/duplex mismatches (common cause of errors) Swap cables/ports to isolate hardware faults.

Layer 2 — Data Link / Switching

Confirm VLAN membership with show vlan brief.

Check trunking with show interfaces trunk — missing VLANs on trunks are frequent problems.

Additional checks:

• show mac address-table — verify MAC learning/port mapping
• show spanning-tree (or summary/vlan) — look for blocking ports or loops
• show cdp neighbors — confirm physical connectivity to neighbors. For port security violations: show port-security interface [int] — “err-disabled” ports need shutdown then no shutdown

Layer 3 — Network / Routing & Addressing

Verify IP assignment: show ip dhcp binding (or verify that the IPs were assigned with static config)

Test gateway reachability: ping <gateway IP>

Check routing table: show ip route — ensure a default route exists (gateway of last resort)

Use show ip arp for ARP issues, extended ping (source interface, size for MTU), and traceroute to find drops

For internet access: show ip nat translations — no active translations when testing external sites indicates NAT/ACL misconfiguration

Higher Layers — DNS, Transport, Application

Ping external IP (e.g., 8.8.8.8) succeeds but domain names (google.com) fail → DNS problem (check client/relay config).

For Layer 4/7 issues: test ports (limited telnet in IOS), review app logs, browser errors, or try alternate clients. Higher-layer failures often appear after lower layers test clean.

Security & ACLs

One-way or service-specific blocks: show access-lists and show ip interface [int] (check applied direction).

ACLs match top-down, implicit deny at end — order and placement matter.

General Best Practices

• show logging for events/errors
• show processes cpu / memory for resource issues
• Compare with show running-config against backups/docs. Work methodically: change one thing at a time, verify after each step, document your logic. Avoid random fixes that create new problems

That’s all for this round. I’ll see you in the next one!

I chose VyOS — an open-source Linux-based network operating system that is a unified platform for routing, firewalling, and VPN — as my primary routing platform. It’s a powerful, enterprise-grade network OS that’s completely free and runs efficiently in VMs. This allowed me to get hands-on experience with the same routing concepts used in Cisco and Juniper environments, but without the hardware costs or licensing headaches. By combining VyOS with a fresh pfSense firewall and keeping the setup to just four VMs, I created a realistic multi-site network topology that demonstrates real-world troubleshooting skills while remaining lightweight enough to run on my homelab workstation’s modest hardware.

The final network looked like this:

Let’s get started!

Section I: Preparing the pfSense Firewall

I downloaded the pfSense firewall image from https://atxfiles.netgate.com/mirror/downloads/ (version 2.6.0). With regards to my using the older pfSense firewall installer, I went with the legacy pfSense CE 2.6.0 full offline ISO to sidestep potential VMware NAT connectivity glitches that could affect a modern Netgate installer when getting pfSense installed and configured in a homelab virtualized network environment that does not communicate with the broader Internet.

Then I opened up VMware Workstation as my Virtual Machine editor. I wanted to get some practice with it since I am **very** familiar with VirtualBox. First up was to use the Virtual Network Editor by using the Edit >> Virtual Network Editor button. Notice that the following screenshot is what I had from before getting started on this homelab. — from other homelabs

Notice that VMnet0, VMnet1, VMnet8 were all taken and are not part of this homelab exercise. I then got to work on making the virtual networks.

• LAN Trunk — Trunk network that will carry VLANs 10, 20, 30 between the different VMs

Notice the Subnet Address was of value 0.0.0.0. In this Custom configuration, the DHCP service was disabled, no explicit subnet IP or subnet mask was specified, and there was no host connection or NAT functionality enabled. As a result, no IP subnet was configured on the host side, and VMware displays 0.0.0.0 to indicate that there is no assigned subnet. This is exactly what we want for a pure Layer 2 network that only forwards VLAN-tagged Ethernet frames without any Layer 3 routing intelligence.

• Core-to-Edge: This network was the direct transit link between VyOS-Core and VyOS-Edge VMs

Notice that in this case, I used the value of Subnet Mask = 255.255.255.248 , which is equivalent to /29 in prefix length. This (per a very handy subnet sheet) resulted in 8 total addresses on this subnet, and only 6 hosts.

There is a reason for what I did. I tried to use a Subnet Mask = 255.255.255.252 with a /30 prefix length value, which would result in four total addresses and two hosts, since the first and last addresses are the Network ID and Broadcast addresses respectively. I intended to do such since I only planned to have two hosts connected to this Core-to-Edge network. Doing such resulted in the following error, so I had to make do with a slightly larger subnet:

• WAN-Link: This network was to be the “simulated Internet” between the VyOS-Edge, VyOS-Core, and pfSense-New internal nodes

• pfSense-LAN: This is a management network intended to allow me to access the pfSense web GUI from the host machine during initial setup and configuration. It essentially was my “out-of-band” access path so I could configure pfSense’s interfaces, VLANs, and firewall rules through a web browser before the routing and trunk networks are fully operational.

Notice that I enabled the “Connect a host virtual adapter to this network” option for the pfSense-LAN network because it creates a virtual network interface on my Windows host machine, allowing me to directly access the pfSense web GUI from my browser for initial configuration and ongoing management. This would be important later.

Wonderful. Next up was to make the VMs. I got started by making the pfSense Virtual Machine that would act as a firewall and my simulated “Internet gateway” from scratch. The desired specs were to have 2 CPUs, 4 GB of RAM, and a 20 GB disk:

I then needed to create the 3 network interfaces for this VM. Remember the network diagram from the introduction to this homelab? The pfSense VM was to connect to three networks; the WAN-Link network, LAN-Trunk, and pfSense-LAN management networks,. I thus went back into the Virtual Machine network page, added three network interfaces, and connected them to the three networks in order.

I then quickly fired up the VM and went through the installation walkthrough and arrived at this menu screen:

Next up was to get the Network Interface configurations down. Below is a screenshot of the presented 3 network interfaces that were seen, along with my annotation of the specific network that they correspond to.

Then I configured them appropriately

The em2 interface, to which I assigned the pfSense LAN interface, corresponds to the pfSense-LAN virtual network. Because the pfSense firewall serves its web GUI on the LAN interface and my homelab workstation is connected to the pfSense-LAN network, the em2 interface was the right one to configure for management access.

OK, I then needed to be able to access the pfSense Firewall’s GUI from my homelab workstation’s browser. Remember that my homelab workstation is to be considered a part of the pfSense-LAN management/setup network? To do this, I selected option 2) Set Interface(s) IP Address and decided to configure the LAN (em2) interface corresponding to the pfSense LAN Network, and then entered in a temporary 192.168.1.100/24 IP address

And then configured the DHCP server and the starting and ending IP address range to be 192.168.1.100–192.168.1.200, entered n for the setting of whether we wanted to revert to HTTP as the webConfigurator protocol, and got the changes pushed through

Wonderful. Next up was to configure some settings on my Homelab workstation VM. Remember I created the Homelab13_PFSENSE_LAN virtual network, whose information I am re-showing below?

Remember that I enabled the “Connect a host virtual adapter to this network” option for the pfSense-LAN network because it creates a virtual network interface on my Windows host machine? Now came the time to take advantage of it. This virtual network was originally named VMNet12 and renamed to Homelab13_PFSENSE_LAN.

Opening up my Windows homelab workstation’s Control Panel, I found the adapter that was created for this network.

Great. This allowed me to successfully open up the pfSense Firewall’s login page.

And then logged in with the default admin username and pfsense password.

Just as a reminder, I already configured the pfSense’s firewall’s LAN interface for the pfSense-LAN network as being of the static 192.168.1.100/24 IP address.

I then needed to configure the other two interfaces: the WAN interface for the WAN-Link simulated Internet network and the OPT1 interface for the LAN-Trunk network. [See the network topology image at the top of this writeup’s page for a visual understanding of how the final planned network topology was to look]

Before getting started with configuring these two, I quickly renamed my LAN Interface in pfSense as being pfSense-LAN-Interface. I did this in order to prevent any potential confusion with the originally-named OPT1 interface which I was to use for the LAN-Trunk Network. I then clicked the Save and Apply Changes button to make things go through.

Perfect. I started with configuring the WAN interface for the WAN-Link simulated Internet network. I named it WAN-Link Interface, set the IPv4 Configuration Type to DHCP, and set the Alias IPv4 Address field to 203.0.113.0/24.

I clicked the Save and Apply Changes button to make things go through.

Next up was to prepare the VLANs that were to be in the LAN-Trunk network so that I could go and configure the LAN-Trunk Network interface appropriately. Per the network concept diagram at the top of this Homelab writeup, there were supposed to be three VLANs made: VLAN 10 (Admin Network), VLAN 20 (Guest Network), VLAN 30 (IoT Network). I quickly went into the Interfaces >> Interface Assignments page

Then selected the VLANs button and arrived at the VLAN Interfaces menu

I then clicked the +Add button three times and proceeded to add the three VLANs in order, with OPT1 being the Parent Interface. See the screenshots for the configuration information

Great. I then needed to go into the Interface Assignments Page where we had the information regarding the three network interfaces on this pfSense VM.

And added in the interfaces for those three VLANs:

Then I went into each one and quickly configured them by checking the Enable Interface button, and setting the appropriate IP address ranges for each:

• VLAN 10 (Admin Network): 10.0.10.0/24

• VLAN 20 (Guest Network): 10.0.20.0/24

• VLAN 30 (IoT Network): 10.0.30.0/24

Excellent. Next up was to create some firewall rules for each interface. Per documentation, pfSense applies firewall rules to traffic entering a specific interface (e.g. VLAN 30 LAN Trunk Network Interface rules filter traffic flowing from that interface into pfSense). Additionally, pfSense has an implicit deny rule on any traffic that is not explicitly allowed.

Due to the fact that I had six interfaces to deal with from the three initial VM ones and the three additional ones for the VLANs #10, 20, and 30, I am going to put a brief description of the rules that I made, along with the respective screenshots.

• pfSense-LAN Network Interface: I added in three rules at the bottom of the pfSense-provided-by-default ones, which allowed any host on the pfSense-LAN network to reach out to any of the hosts on the VLAN 10, 20, and 30 networks

• WAN-Link Network Interface: I left the WAN interface with default deny rules, blocking all unsolicited inbound traffic from the simulated internet — following the security principle of allowing outbound traffic while blocking unsolicited inbound connections. The outbound allow rules were (as you will see) later configured on each VLAN interface

• LAN-Trunk Network Interface: I left it as is — with no rules. This is because the LAN-Trunk interface operates purely at Layer 2, carrying VLAN-tagged frames between pfSense and the VyOS routers. All routing logic and firewall rule enforcement occurs at the Layer 3 VLAN subinterfaces (VLAN 10, 20, 30), which is why no firewall rules are configured directly on the trunk interface itself.

• VLAN 10 (Admin Network): I configured VLAN 10 (Admin network) with permissive firewall rules allowing administrators full access to all networks, including unrestricted internet access via the WAN-Link interface and web GUI management access to pfSense on both HTTP and HTTPS ports. This reflects the privileged nature of the admin network, where trusted users require broad connectivity for system management and troubleshooting across the entire lab environment.

• VLAN 20 (Guest Network): I made rules allowing the Guest VLAN access to the broader internet only and blocked it from connecting to the Admin and IoT VLANs. I also blocked it from connecting to the pfSense firewall’s GUI at the HTTP and HTTPS ports of 80 and 443 respectively.

• VLAN 30 (IoT Network): Because IoT devices generally rely on DNS, HTTP/HTTPS, and NTP, I made rules allowing devices on the IoT network to reach out to the broader Internet using those ports. All other traffic was implicitly denied.

After configuring all these firewall rules, I can confidently say my network segmentation skills are on fire… wall. 🔥🧱

Section II: Deploying VyOS VMs

With pfSense configured and my VLANs properly segmented, it was time to tackle the heart of this homelab: dynamic routing with VyOS. While pfSense excels at firewalling and security, VyOS is purpose-built for enterprise-grade routing protocols — and that’s exactly what I needed to simulate a realistic multi-site network topology.

I chose VyOS for several compelling reasons. First, it’s a fully-featured, open-source network operating system that supports the same routing protocols you’d find in Cisco, Juniper, and Arista equipment — OSPF, BGP, EIGRP, static routing, and more — without the licensing costs or hardware requirements. Second, VyOS runs efficiently as a virtual machine, making it perfect for homelab environments where you want enterprise capabilities without enterprise budgets. Finally, the skills I’d gain configuring VyOS translate directly to other network platforms, since the underlying routing concepts (LSAs, AS numbers, neighbor relationships, route advertisements) are universal across vendors.

This lab involves my deploying two VyOS routers: VyOS-Core as my central hub running OSPF Area 0 and BGP AS 65000, and VyOS-Edge as my branch office router running OSPF Area 1 and BGP AS 65001. Together, they would form the backbone of my multi-site network, exchanging routes dynamically and providing the foundation for site-to-site VPN connectivity. This is where the real networking magic happens — moving beyond static routes and firewall rules into the world of dynamic routing protocols that power the internet itself.

Introduction is over. Time to get into action. I got started by going to the https://vyos.net/get/nightly-builds/ website to get VyOS’s latest rolling release. For VyOS, a rolling release is a continuously updated version of the open-source network OS, incorporating the latest code from developers and contributors daily.

I then got to work on creating my first VM: the VyOS-Core VM. Since I have previously demonstrated in this and other past homelab writeups how I create virtual machines, I will refrain from showing on a screenshot-by-screenshot basis the individual steps Instead, I will say that I created the new VyOS-Core VM with the following specs: 2 vCPUs, 2 GB RAM, 10 GB memory disk, and three network interface adapters connected to the Core-to-Edge, LAN-Trunk, and LAN-Trunk networks. VyOS-Core will serve as the central router in this lab, handling traffic between all three VLANs (Admin, Guest, and IoT) while learning and sharing routes dynamically with VyOS-Edge using OSPF and BGP — essentially acting as the main site router that connects the internal networks and exchanges routing information with the edge router that sits closer to the simulated internet. [See the network diagram towards the top of this writeup for a visual illustration of how it fits into the overall network that I have been working to make]

I then also went and created the VyOS-Edge VM with the following specs: 2 vCPUs, 2 GB RAM, 10 GB memory disk, and two network interface adapters connected to the WAN-Link and Core-to-Edge networks.

Then, I got to work, booted up the VyOS-Core VM, and arrived at this screen

I quickly logged in with the vyos / vyos username and password combination. Then, I used the install image command to deploy VyOS to my permanent storage.

When I first booted the VyOS VM from the ISO, it was running entirely in RAM as a live environment — meaning any configuration changes would be lost on reboot. Running the install image command permanently installs VyOS to the virtual hard disk, ensuring the operating system and any future configurations persist across reboots, making it a proper permanent router rather than a temporary live session.

There was a series of presented installation configuration options, I just blitzed through those and selected the default options. After installation, I got the prompt to reboot now

I initiated the reboot with the reboot command. Then I powered off the VM, removed the VyOS ISO disk from the configuration, turned the VM back on again, and logged in. I then used the show system image command to confirm that my VyOS-Core VM had the permanent image installed and running

Cool. I then repeated the same series of steps on the VyOS-Edge VM. I ran the install image command, rebooted the VM, powered it off, removed the VyOS ISO disk from the configuration, turned it back on again, logged in, and used the show system image command to confirm that it also had the permanent image installed and running.

Section III: Configuring VyOS VMs’ Interfaces

I then pulled out the VyOS documentation page for ethernet interfaces and used the show interfaces command to get a sense of the VyOS-Core and VyOS-Edge VM’s interfaces

Interfaces on VyOS-Core VM:

Interfaces on VyOS-Edge VM:

All this was pretty informative. I then saw the Description field and after looking at the documentation page decided to fill that field in with the set interfaces ethernet <interface> description <description> command. I first needed to get into the configuration mode with the configure command:

This I did in short order, then I used the commit and save commands to make the changes active in the current session and permanently persistent across all sessions respectively

Now I had the following. I used the show interfaces command to see what was done

Updated Interface Descriptions on VyOS-Core VM:

Updated Interface Descriptions on VyOS-Edge VM:

These descriptions were then useful when working on the next steps to configure them. The next step was to assign IP address values to each interface. In each VM, I went into the configuration mode using the configure command, and then used the set interfaces ethernet <interface> address <address> command to assign the IP addresses before “locking” the changes in with the commit and save commands.

Notice that for the VLANs 10, 20, and 30 connecting to the VyOS Core VM, I also used the set interfaces ethernet <interface> vif <vlan-id> address <address> command. Per the VyOS documentation, 802.1q VLAN interfaces are represented as virtual subinterfaces using the vif <vlan-id> command.

On the VyOS-Core VM:

On the VyOS-Edge VM:

Section IV: Configuring Dynamic Routing (OSP & BGP)

This section is the real heart of this homelab exercise. In this section, I configured the two critical enterprise routing protocols on the virtual network that I have created: OSPF and BGP.

OSPF (Open Shortest Path First): OSPF is a link-state routing protocol used within organizations to automatically discover and share routes between routers by having each router build a complete map of the network topology. When a router learns about a network change (like a link going down), it floods that information to all other routers, which then independently calculate the best path to every destination using Dijkstra’s shortest-path algorithm. OSPF is widely deployed in enterprise networks because it converges quickly during failures, supports intelligent load balancing across multiple paths, and scales efficiently by dividing large networks into hierarchical areas.

BGP (Border Gateway Protocol): BGP is the routing protocol that powers the internet, designed to exchange routes between different autonomous systems (AS) — independent networks operated by different organizations, ISPs, or companies, each identified by a unique AS number. Unlike OSPF which focuses on finding the fastest path within a network, BGP makes routing decisions based on complex policies like business relationships, network performance, and path attributes, allowing organizations to control how traffic enters and exits their networks. BGP is considered the “glue” of the internet because it enables thousands of independent networks worldwide to interconnect and share reachability information, ensuring that when you visit a website, packets can navigate across multiple organizations’ networks to reach their destination.

In this exercise, I configured VyOS-Core to run OSPF Area 0 (the backbone area) and advertise the three internal VLAN networks along with the Core-to-Edge transit link that connects it to VyOS-Edge. VyOS-Edge runs OSPF Area 1, which represents a separate site or location that connects back to the backbone through the Core-to-Edge link. OSPF uses “areas” to divide large networks into smaller, more manageable sections — and here, Area 0 always serves as the central backbone that all other areas must connect through. This OSPF configuration ensures that both routers automatically learn about all internal networks and can route traffic between them dynamically.

For BGP, I configured VyOS-Core as AS 65000 and VyOS-Edge as AS 65001 to simulate two independent organizations or network domains peering over the WAN-Link network. An autonomous system (AS) is essentially a collection of networks under a single administrative control with its own routing policies — think of it like a company’s entire network infrastructure identified by a unique number. This eBGP (external BGP) peering allows the two “autonomous systems” to exchange routing information across what we’re treating as the simulated internet, demonstrating how separate networks establish trust relationships and share reachability information.

By running both OSPF internally within each site and BGP externally between sites, I simulated a realistic enterprise architecture where internal networks use fast-converging link-state protocols while external connections use policy-based BGP for inter-organizational routing.

That was a lot of words. Now for the actual steps.

I started by opening up the VyOS Core router’s configurations option using the configure command and (per the appropriate documentation) configured the VLAN 10, 20, and 30 address spaces to be for OSPF with the set protocols ospf area 0 network <VLAN_CIDR> command. Then I saved the changes with the commit and save commands.

Then I opened up the VyOS-Edge router using the configure command and used the set protocols ospf area 1 network 172.16.0.0/30 command to set the Core-to-Edge network into Area 1

Now it was time to work on BGP. VyOS-Core (AS 65000) represents the internal enterprise network, while VyOS-Edge (AS 65001) represents an ISP-style edge that owns and advertises the WAN-Link “Simulated Ineternet” network.

Internal point-to-point and transit links (such as the Core-to-Edge) are handled by OSPF and intentionally not advertised via BGP, since they are infrastructure-only and have no meaning outside the AS. Instead, BGP is used solely to exchange meaningful destination prefixes across the AS boundary, keeping internal plumbing abstracted and clean.

Now, for the commands. I first went onto the VyOS-Core VM and entered the configuration mode with the configuration command. First I needed to have the VyOS-Core VM say that it is of the AS 65000 and to establish a formal peering session with the VyOS-Edge router. I used the set protocols bgp system-as 65000 command to set the Autonomous System number on the VyOS-Core router to 65000 and then the set protocols bgp 65000 neighbor 203.0.113.2 remote-as 65001 command for this.

The IP address 203.0.113.2 served as the specific ‘digital front door’ for the VyOS-Edge router on the simulated WAN. By pointing VyOS-Core to this exact host address rather than the broader network ID, I enabled a targeted TCP connection that allowed the two routers to securely exchange routing information.

Then I needed to advertise the LAN-Trunk VLAN 10, 20, and 30 networks, so that they could be accessible to the WAN-Link “Simulated Internet” network. I did this using the set protocols bpg address-family ipv4-unicast network <IP_CIDR> command

I tried to use the commit message and got an error “WARNING: BGP neighbor ‘203.0.113.2’ requires address-family!” I quickly fixed this by executing the set protocols bgp neighbor 203.0.113.2 address-family ipv4-unicast command. Then I entered commit and save to lock the changes in.

Then I exited the configuration console and used the show ip bgp neighbors command to see what the configuration looked like.

Nice. The next step was to go onto the VyOS-Edge VM, enter the configure mode with the configure command. Then I used the set protocols bgp system-as 65001 command to set its Autonomous System number and the set protocols bgp neighbor 203.0.113.1 remote-as 65000 to establish a formal peering session with the VyOS-Core router.

Then I needed to advertise the “Simulated Internet’s” WAN address space. I did this with the set protocols bgp address-family ipv4-unicast network 203.0.113.0/24 command. Then I Iset up the address family, just like with the previous WARNING message with the set protocols bgp neighbor 203.0.113.0/24 address-family ipv4-unicast command.

After entering commit and save to lock the changes in, I exited the configuration console and used the show ip bgp neighbors to see what it looked like.

Looked right. Astute readers may have noticed that I did not perform any advertisements of the Core-to-Edge transit link (172.16.0.0/30 IP addresses). This is because the 172.16.0.0/30 subnet exists purely as infrastructure glue — it’s a point-to-point link that lets Core and Edge talk to each other directly. Both routers already had this as a connected route in their local tables and could already reach each other’s peering IP without any help from BGP.

Bottom Line: doing BGP routing on point-to-point links is inefficient and would just clutter the routing table with unnecessary paths that provide no value to actual end-user traffic.

Section V: Site-to-Site WireGuard VPN

Next up was to configure a site-to-site WireGuard VPN tunnel between VyOS-Core and VyOS-Edge to simulate secure connectivity between two geographically separated locations over the untrusted WAN-Link “Internet”. While OSPF and BGP handle the routing, the VPN ensures that sensitive data traversing between sites remains confidential and tamper-proof.

It is important to note that VyOS images come with Wireguard installed and that good documentation also exists for it. I also found a good WireGuard Quick Start installation tutorial and an additional blog post that was useful for this section. Onwards!

First up was to generate the required base64-encoded public and private keys. Per the WireGuard Quick Start installation tutorial, this was done on both the VyOS-Core and VyOS-Edge VMs with the wg genkey | tee privatekey | wg pubkey > publickey command

On the VyOS-Core VM:

On the VyOS-Edge VM:

After that, it was time to create the actual tunnel. I am going to attach the screenshots of the steps performed on each VM and then will provide the appropriate explanation

On the VyOS-Core VM:

On the VyOS-Edge VM:

These steps are mostly straight forward. On each of the VyOS VMs, I created a special interface for the VPN, assigned an IP address to it (either 10.99.99.1 or 10.99.99.2), added the private key for decrypting incoming traffic and encrypting outgoing traffic, and set the port where it will listen for incoming WireGuard VPN packets. Then, I created a reference to the peer VyOS node’s interface for the Wireguard VPN, provided its public key and specified the physical IP address and port where the peer could be found.

After that, I used the quick show interfaces wireguard <wg01 or wg02> summary command on the core and edge VMs (with the corresponding wg01 and wg02 interfaces to see if the connection was fully formed.

On the VyOS-Core VM:

On the VyOS-Edge VM:

[Side Note: the sudo wg show <wg01 or wg02> command is also similar in terms of the nature of its output — it can be useful in the future]

Notice the fact that they both sent out 3 KB but did not receive anything. This means that a handshake connection was not fully finished and no connection was made. Time to troubleshoot.

Troubleshooting Log: WireGuard Tunnel Establishment

• The Issue: Initial tunnel status showed Sent data but 0 Received bytes, indicating a one-way handshake failure.
• The Investigation: I performed a deep dive into the configuration of both VyOS-Core and VyOS-Edge. By comparing the generated public keys against the peer configurations, I identified a public key mismatch on the Edge router.
• The Fix: I entered configuration mode on VyOS-Edge and corrected the peer key using the following command: set interfaces wireguard wg01 peer to-wg01 public-key ‘<CORRECT-Core-public-key>
  I then performed a clean reboot of both VMs to ensure all interface states were reset.
• The Validation: To “wake up” the tunnel (since WireGuard stays silent until traffic is present), I initiated cross-pings:

** From VyOS-Core: ping 10.99.99.2 count 5
** From VyOS-Edge: ping 10.99.99.1 count 5

• Result: The handshake was successfully completed. Running the show interfaces wireguard <wg01 / wg02> summary command on both VMs confirmed bidirectional traffic with bytes being sent and received. See the screenshots below:

On the VyOS-Core VM:

On the VyOS-Edge VM:

To Recap: The WireGuard VPN was necessary because it acted as the secure ‘virtual bridge’ across my simulated internet (the WAN-Link). Without it, the Core and Edge routers would be completely isolated, unable to exchange private traffic or routing information securely across an untrusted network. By establishing this tunnel, I created a protected path that allows me to simulate a real-world multi-site corporate network where data stays encrypted as it travels between physical locations.

Section VI: Inter-Site Reachability & Post-Tunnel Routing

A common pitfall in VPN setups is assuming that a successful handshake equals full connectivity. In reality, VyOS-Core knows its local VLANs, but VyOS-Edge is ‘blind’ to them until they are explicitly routed. This is why I had to advertise the internal subnets across the newly established WireGuard tunnel.

The tunnel itself was a mandatory first step. Because the Admin VLAN uses private IP space (specifically the Private IP Class A CIDR of 10.0.0.0/8), it cannot traverse the ‘public’ WAN-Link without being encapsulated and encrypted. The VPN acts as a secure escort for this sensitive management traffic, protecting it from exposure. However, even with a secure pipe in place, without a routing entry — either static or dynamic — the Edge router would receive traffic from the internal VLANs and have no routing entry to send an appropriate response back, effectively killing the connection. Connections are a two-way-street. By adding such a static routing entry, I transformed a simple peer-to-peer connection into a fully reachable site-to-site network.

While I could have advertised all three VLANs (Admin, Guest, and IoT), I chose to focus on the Admin VLAN first to practice the principle of Least Privilege. In a production environment, you don’t always want your IoT or Guest traffic traversing a secure site-to-site VPN. By selectively routing only the management network, I demonstrated the ability to control traffic flow and preserve tunnel resources for the most critical data.

Now for the actual work. On the VyOS-Edge VM, I entered the configuration mode with the configure command and used the set protocols static route 10.0.10.0/24 next-hop 10.99.99.1 command to add the route in

Afterwards, I used the commit and save commands to lock the changes in, and used the show ip route command on the VyOS-Edge VM to see what the routing table looked like now

Looks good.

Section VII: Zone-Based Firewall Security Policies

Because VyOS-Core is directly connected to both the WireGuard tunnel and the LAN trunk, it functions as the immediate gateway between these two worlds. While this direct connection simplifies routing, it also creates a significant security risk: without a policy in place, any traffic exiting the tunnel has an unrestricted ‘all-access pass’ to my private VLANs.

Since this internal traffic flows directly between the VyOS interfaces, it never touches the pfSense firewall; consequently, pfSense remains ‘blind’ to these movements and cannot enforce any rules. Implementing VyOS Zone-Policy firewalls was therefore a mandatory step to establish a checkpoint at this internal junction, ensuring that my network segmentation remains intact even for traffic already inside the perimeter.

A Zone-Based Firewall (ZBF) replaces traditional interface-based rules with a more scalable model where security policies are applied to traffic moving between defined Zones. By grouping my WireGuard tunnel into a ‘VPN Zone’ and my management network into an ‘Admin Zone,’ I could easily enforce granular security boundaries based on the direction of traffic flow rather than just its entry point.

Now for the actual steps on the VyOS-Core VM. First, I created the zones with the set firewall zone <ZONE_NAME> member interface <INTERFACE> command:

Then I needed to create the appropriate Firewall rulesets to strictly permit only established sessions, pings, and SSH management traffic while dropping everything else by default. By enforcing a ‘Least Privilege’ approach, the configuration ensures that even if the VPN is accessed, lateral movement into the sensitive Admin VLAN is restricted to only the specific protocols I’ve explicitly authorized.

Notice the last line of the first screenshot with the command set firewall ipv4 name VPN_TO_ADMIN rule 10 state related. In the context of this Firewall, the Related state allows the policy to be “context-aware” by permitting new connections that are technically separate but logically tied to an already authorized session, such as the signaling or error messages required for complex protocols to function correctly. Essentially, this rule ensures that while I am explicitly allowing primary traffic like SSH or ICMP, the firewall is smart enough to also permit the ‘helper’ packets — like network error notifications or secondary data channels — that these protocols need to successfully complete their tasks.

Then I applied the final “glue” to tell VyOS “When traffic moves from the VPN zone to the ADMIN zone, follow the rules I just wrote.”

Then I used the show firewall ipv4 name VPN_TO_ADMIN command to see what the final rules looked like

And the show firewall zone command provided a nice summary of what I did for this section

Section VIII: Deploying Test-Client and Appropriate Network Interfaces

In this section, I deployed an Ubuntu Desktop VM and performed the appropriate testing. I got the image for Ubuntu 22.04.5 LTS (Jammy Jellyfish) from https://releases.ubuntu.com/jammy/, and then created a new VM with the 1 vCPU, 2 GB RAM, a 20 GB hard drive, and the network interface card being temporarily NAT [after finalizing setup and configurations, it would be connected to the virtual Homelab13_LAN_TRUNK network]. When going through the initial installer configuration, I also selected the Minimal Installation option so as to minimize the unnecessary bloat installed.

After creating the Ubuntu Test-Client VM with a NAT network adapter, I encountered connectivity issues where ping 8.8.8.8 failed and DHCP requests timed out. Diagnosing the problem revealed that while the VM had an IP address, all routes showed linkdown status, indicating VMware’s virtual network infrastructure wasn’t functioning. The root cause was that the VMware NAT Service on my Windows host wasn’t running. After manually starting it in the Windows Services Manager Console (also known as services.msc), the VM immediately obtained proper network connectivity and could successfully ping external hosts. This experience reinforced that VMware’s virtual networking depends on host-side Windows services, and verifying these services are running should be the first troubleshooting step for VM network issues.

After this, I got to work preparing the VM for the final tests. I needed the VM to be able to support the 802.1Q for ethernet frames to be tagged with vlan information. I took some of the information from an Ubuntu help page to do this — with the following commands:

sudo apt-get install vlan — This command installs the user-space VLAN configuration package on Ubuntu, which provides the necessary utilities and ensures all dependencies are installed for managing VLAN interfaces from the command line.

sudo modprobe 8021q — This command loads the **kernel module** that provides the low-level driver support for processing 802.1Q VLAN-tagged Ethernet frames at the network stack level.

Because it is a kernel level module (unlike vlan being a permanently installed user-space package), I needed to also explicitly add it to the /etc/modules file. This would ensure that this setup is permanent by telling the system to load this kernel module each time Ubuntu boots.

Why Both vlan and 8021q Are Needed: The 8021q kernel module handles the actual VLAN tagging/untagging at the driver level (how the operating system processes tagged frames), while the vlan package provides the user-space tools and ensures proper system integration so you can create and manage VLAN interfaces using commands like ip link add with proper VLAN support.

Up until now, I had my Ubuntu VM being on a single NAT interface. That was great up to this point because I could run the sudo apt-get install vlan command and get the appropriate installation packages from the broader Internet and finalize VM setup. Now I needed to move it onto the LAN-Trunk network so that it could become a part of the virtualized network that I built with the pfSense and VyOS VMs. I did this by powering the VM off, changing the network adapter settings from NAT to the Homelab13_LAN_TRUNK, and powering it back on.

The following screenshots show the commands I used for the three VLANs:

• VLAN 10 (Admin Network):

• VLAN 20 (Guest Network):

• VLAN 30 (IoT Network):

Notice that I didn’t add default routes for VLAN 20 and 30 yet — only one default route can be active at a time as a routing “catch-all” case. Finally, a quick invocation of the ip addr show command confirmed that the VLAN subinterfaces were fully created and running.

Section IX: Project Conclusion

On the Ubuntu machine, I had the default route for VLAN 10 to go out to the pfSense interface on 10.0.10.1/24 interface.

To wrap up the lab, I performed a connectivity test from the Ubuntu Test-Client. I successfully established a ping to the pfSense gateway (10.0.10.1). This successfully demonstrated the previous section’s primary technical objective: the correct implementation of 802.1Q VLAN trunking and sub-interface configuration.

With all this work done over the course of this homelab exercise and burnout setting in, I chose to conclude this exercise. While there could have been additional troubleshootings that could have been done to verify and refine the connectivity further and steps taken to improve the network architecture (the main one would be to remove the connection from the VyOS-Core to the LAN-Trunk so that the traffic would flow from the Ubuntu VM through only through the pfSense firewall to the simulated Internet), I achieved my core goals of learning VyOS, BGP, OSPF, VLANs subinterfaces, and other networking related concepts — and needed to move on to future homelabs.

A CI/CD pipeline — which is in short for Continuous Integration/Continuous Delivery (or Deployment) pipeline — is an automated workflow that streamlines how code changes are built, tested, and released. It starts with Continuous Integration, where developers regularly merge code into a shared repository, triggering automatic builds and tests to catch issues early, followed by Continuous Delivery, which prepares passing code for seamless deployment to staging or production environments, with the option for fully automatic deployments. CI/CD minimizes manual errors, accelerates feedback loops, ensures consistent quality, and enables rapid, reliable iterations — essential for modern cloud-native applications and infrastructure management.

Enter Jenkins: a powerful, open-source automation server that’s perfect for orchestrating these pipelines, offering extensible plugins to integrate with Git, Docker for image builds, Terraform for IaC (Infrastructure as Code), and cloud providers like AWS or Azure — all self-hostable and highly customizable. Let’s get started!

Section I: Setup

The first thing that I did was go onto Oracle’s website, downloaded the appropriate Java installer for the architecture of my Windows homelab workstation, executed it, and got Java installed

I then went onto the Jenkins website, downloaded the installer, and went through the appropriate steps to get Jenkins installed

When asked what the logon type is, I selected the Run service as local or domain user option. This is because (per Jenkins’ documentation), “it is recommended to install and run Jenkins as an independent Windows service using a local or domain user, as it is much safer than running Jenkins using LocalSystem(Windows equivalent of root) which will grant Jenkins full access to your machine and services.”

And I selected the default port of 8080.

I then hit a message saying that my originally installed version of Java (25-something) did not match the Jenkins version — since “Only Java 17 and 21 are supported by Jenkins.” I quickly uninstalled the Java package that I had installed using the Get-Package ‘Java*” | Uninstall-Package command in PowerShell and then downloaded the installer for version 21 off Oracle’s website. Then I continued with the installation and finished it.

Jenkins then auto-started, and I opened it up in my browser at the http://localhost:8080

I then opened up the appropriate file, got the administrator password, put it in, and arrived at the Customize Jenkins window

I then clicked the Install Suggested Plugins button and got everything under way

Next up was to create my first admin user

I kept the Jenkins URL as http://localhost:8080 and the setup was complete! I then opened it and logged in to confirm access

The next thing that I needed to install was Infracost. It is an open-source tool that provides cloud cost estimates for infrastructure-as-code changes, primarily in Terraform, by integrating into CI/CD workflows and pull requests to show the financial impact before deployment. It is important because it shifts FinOps left, enabling engineers and teams to proactively prevent cost overruns, enforce best practices, and optimize resources early in the development process — avoiding expensive surprises and rework after infrastructure is provisioned.

I manually downloaded the .zip file of the latest release from Infracost’s Get Started website, created a sub-folder called “Infracost” at the C:\Program Files\ overall folder, and then updated my Windows environment PATH variable to include that.

Opening up a new PowerShell window and entering infracost into the command line confirmed that it was installed.

Wonderful. I then needed to use the infracost auth login command to open up a browser window that would authenticate my CLI for usage.

Wonderful. This resulted in the API key being appropriately saved to my computer automatically

I already had Docker Desktop, Terraform, git, AWS CLI, and Azure CLI installed from past homelabs, so we were good on the installations. The last thing that I needed to do as part of this setup process was to create a main project directory, and to create the subfolders.

The folders made were as follows:

• app and app/templates
• keys
• scripts
• terraform, with the aws and azure sub-folders

Section II: Creating a Flask Application

I got to work by creating a simple Flask application in app\app.py:

Next up was to create a Python virtual environment. It is an isolated directory that contains its own Python interpreter and packages, letting me install libraries specific to just this project without impacting other projects or the overall Python setup on my computer. I needed everything to stay project-specific, in order to avoid version conflicts from my system’s global packages and to keep my setups reproducible and portable.

To create it, I went into the folder in which I had the files for this project and used the python -m venv . command, followed by the .\Scripts\activate command to activate the virtual environment

Then, I checked with the pip list command to see what packages the Python package manager currently had installed. As expected for this virtual environment, it was just pip

Then I quickly installed flask using the pip install flask command, and then afterwards saved the information regarding the required packages using the pip freeze > requirements.txt file

The requirements.txt file is a Python convention that lists all the Python packages and their specific versions needed to run your application, allowing anyone (or any automated system like Jenkins) to recreate the exact same environment with a single pip install -r requirements.txt command. Looking at it, we see the following, which is **exactly** what was expected for this virtual environment:

Next up was to create the index.html file (in this project’s app\templates sub-folder) as the to-be-presented by the Flask app HTML code. I also installed the Live Preview extension for my Visual Code Studio code editor that allowed me to take a look at how the final HTML code I was making would look like:

Next up was to make the Dockerfile. It looked like this:

I need to now explain two points about what this Dockerfile is doing. The first is regarding the usage of the –no-cache-dir flag for the RUN pip install command. The Python package manager (pip) downloads packages to speed up future installations on your local machine, but in Docker containers this cache is useless (since each build starts fresh) and just bloats the image size. Accordingly, it is best practice to have –no-cache-dir in a Dockerfile, to prevent unnecessarily cached data from bloating the image from megabytes to potentially hundreds of megabytes larger.

The second is the initiation of the FLASK_APP environmental variable inside the container. It tells Flask which Python file contains the application and needs to be executed (app.py). In this case the FLASK_APP environment variable is technically optional since python app.py is an explicit command in the Dockerfile’s CMD instruction, instead of using flask run — which does rely on it.

The last step was to create the .dockerignore file, to tell Docker which files and folders to ignore when building the container image. A well-crafted .dockerignore file is important to keep the container image smaller, help it build faster, and avoid accidentally including sensitive data or files that do not belong in production.

And while I was at it, I made the .gitignore file too

Section III: AWS Infrastructure With Terraform

Next up was preparing the AWS infrastructure. The goal of this effort is to create an EC2 instance using Terraform that Jenkins can later connect to and manage. Jenkins acts as an automation orchestrator — it will SSH into this VM to deploy, update, and manage the Dockerized Flask application throughout the CI/CD pipeline.

First up was to generate an appropriate SSH key pair for AWS on my local machine. I used the chdir keys command to go into the project repository’s keys directory and then got to work on generating the appropriate SSH key pair. This was done using the ssh-keygen -t rsa -b 4096 -f “aws-key” command

This generated both the public and private key pairs that I wanted to use. Then I needed to get the public key fed into the AWS console. To do this, I logged into the AWS Management Console, then went into the EC2 >> Key Pairs page

Then after that, I clicked the Actions >> Import Key Pair button and using the Browse button selected the generated aws-key.pub file.

And then imported the key successfully.

Next up was to create the Terraform for the EC2 instance. Remember how in a previous homelab, I typed every single line of Terraform manually for my first AWS EC2 instance and took hours? Now I got the opportunity to atone for this mistake!

I went online and found a tutorial for creating an EC2 instance whose code I could copy, paste and tweak. I heavily based my approach off of it and tweaked the code appropriately. The biggest tweak was finding a Red Hat Enterprise Linux 10 (HVM) AMI in a t3.micro tier that I wanted to use.

For those who want to see what I wrote, the files are located in the appropriate folder in my Homelab Series GitHub repository. Don’t forget that there is an AWS-specific sub-folder underneath the folder for this current project.

It was then time to test the Terraform code appropriately. I did this by entering the directory for the AWS component of this project, and then got started with the terraform init command

Then I used the terraform validate command to see that the configuration works and is valid.

It was. Then I used the terraform plan command to see what will be created

Cool. This Terraform plan looked suitable for making what we needed it to make. Notice that at this point in time, I did not take any further steps. Jenkins will perform the terraform apply command appropriately to provision the instance when time comes.

Section IV: Azure Infrastructure With Terraform

We did this with AWS, the next step was to do the same with Azure!

I started by using the ssh-keygen -t rsa -b 4069 -f ..\..\keys\azure-key command to generate an SSH key pair for Azure

Then I used the az login command in my console to quickly log in — and it also got me my subscription ID

The next step was to create a service principal in Azure — essentially a dedicated, non-human identity (like a service account) to allow Jenkins to securely authenticate and interact with Azure resources without using my personal user credentials. Since Jenkins runs outside of Azure (like on my homelab workstation), it couldn’t use a managed identity. Creating a service principal provided a secure way for automated pipelines to log in via the Azure CLI (using the client ID, secret, and tenant ID), perform actions like deploying resources, and follow least-privilege principles, while avoiding risks like credential expiration or sharing personal accounts.

I did this with the following command: az ad sp create-for-rbac — name “jenkins-sp” — role contributor — scopes /subscriptions/<REDACTED>

I then saved the presented output for later.

Next up was to create the Terraform for the VM that was to be spun up in Azure. I took a page out of the Homelab #7 writeup and reused the same tutorial as a starting point from which to work with. Notice that the tutorial’s code was only a starting point. I had to tweak some things, and refactor some of the lines of code. For example, I did not want the name of a resource group to be generated using the random_pet resource, and I moved some “magic string” configuration variables out into the variables.tf file. I also wanted to use the SSH key that I previously made by hand with the ssh-keygen function and appropriately removed the Microsoft-tutorial provided lines that randomly generated it on my behalf.

The next step that I had to take was to set up the credentials for authenticating. Per the appropriate documentation, I needed to set up the credentials as environment variables, so I did exactly that. This is where the saved information from my creation of the service principal name came in!

With all these preparations done, the next step was to test the Terraform directly. This step is exactly just like the previously-done testing for AWS. First up was terraform init

Then I used the terraform validate command to see that the configuration works and is valid.

OK. The configuration is valid. The last command was terraform plan — and I got errors

There were two minor issues that had to be fixed. I made a mistake and wrote in the variables.tf file the file containing the path to the public key for working with Azure as being azure_key.pub instead of azure-key.pub (using an underscore instead of a dash), and in the outputs.tf file, I needed to add in .ip_address in two lines referencing the azurerm_public_ip.my_terraform_public_ip so I could get its IP address value, out of the ~21 available object properties. Minor fixes.

Rerunning terraform plan resulted in no further errors

We were finished with Azure, for now.

Section IV: Configuring Jenkins

Next up was to get Jenkins configured. I opened it up at http://localhost:8080, and clicked the gear-looking Manage Jenkins button in the top right-hand corner, and then selected the plugins menu option.

I then clicked the Available Plugins and Installed Plugins settings and ensured that I got all of the following plugins fully installed:

• Docker Pipeline
• Docker plugin
• Terraform
• Git plugin (should already be installed)
• Pipeline
• SSH Agent
• Credentials Binding
• PowerShell plugin

These plugins provide the essential capabilities for the CI/CD pipeline: Docker Pipeline and Docker plugin enable building and pushing container images, Terraform plugin allows infrastructure provisioning, Git plugin pulls code from GitHub, Pipeline plugin enables the Jenkinsfile scripted workflow, SSH Agent facilitates secure connections to cloud VMs for deployment, Credentials Binding securely injects stored secrets (AWS keys, Azure credentials, Docker Hub passwords) into the pipeline without exposing them, and PowerShell plugin allows Jenkins to execute Windows-native commands on the workstation.

I then restarted Jenkins appropriately to have the final changes locked in.

After the post-restart reboot, the next step was to configure credentials. I clicked the Manage Jenkins >> Credentials button and arrived at this page.

I then clicked the (global) button and arrived at this page

Wonderful. It was now time to go on a credential-addition spree. To not bore you with the details, I listed the accounts whose credentials I added below:

• Docker Hub: The username and password for my Docker Hub account
• SSH Keys: I added the the private key for SSHing into the AWS EC2 instance (with the username being ec2-user) and into the Azure Virtual Machine (with the username being azureadmin)
• AWS Credentials: I used the notepad $env:USERPROFILE\.aws\credentials command in my PowerShell window to quickly open up the file where I had the API keys that Terraform needs to authenticate and provision infrastructure in my AWS account. Notice that these are different from the SSH key used to authenticate and connect to EC2 instances after they are created
• Azure Credentials: These include the IDs of the specific subscription and tenant that I am working in and the service principal’s username (a.k.a. as the appId) and password
• GitHub Credentials: This was a username + password pair, where I used my GitHub username and generated a token on my GitHub account as a password

The final state of the fed-in-credentials looked like the following:

Great. Next up was to start configuring Jenkins’ integrations with external tools.

First up was to configure Git, which is used for version control. I went to the Manage Jenkins >> Tools panels and then scrolled down to the Git Installations section and set the path to the Git executable as being C:\Program Files\Git\bin\git.exe. I kept the original value of the Name field (literally Default) as is — and then clicked the Save button

Next up was Docker. I configured Jenkins to use my existing Docker Desktop installation by leaving the installation root empty and unchecking “Install automatically,” which tells Jenkins to simply use the docker command already available in the Windows PATH that Docker Desktop configured during its installation. This approach avoids redundant Docker installations and allows Jenkins to seamlessly execute Docker commands — building and pushing images — through the Docker engine already running on my workstation.

I then also clicked the Save button to get the changes applied.

Section V: Preparing The GitHub Repository

In this section, I quickly finalized my configurations for the GitHub repository for this homelab. Remember that I have been working with an overall GitHub repository for all the homelabs of this series, with specific folders dedicated to each specific one.
I went into my file explorer window on my homelab workstation, and reviewed the contents to confirm that I was happy with the project structure — specifically the sub-folders and files stored in it.

It looked good to me. I also verified that the .gitignore file excluded everything that I did not want there — especially the keys for authenticating to the AWS and Azure-hosted EC2 and VM instances.

I then used the git status, git add . , and git commit commands to push the sub-folder that I created for this homelab, along with all the relevant files into the repository.

I then took a look at all the files that got added to my GitHub repository, and confirmed that everything that I wanted to be there was there — as expected.

Section VI: Create Jenkins Pipeline

The next step was to build the Jenkins pipeline. A Jenkins pipeline is essentially a scripted sequence of automated steps that guide your code from commit to deployment — like building, testing, and releasing software — all defined in a file called a Jenkinsfile. For those wanting to learn more, the Jenkins project has good documentation on its website.

Think of a pipeline as an assembly line for software projects, where developed and committed updates get automatically tested and deployed into the appropriate environments!

The Jenkinsfile itself follows the following general format (this screenshot is lifted from the Jenkins documentation on its website)

I then got to work with creating separate Jenkinsfiles (Jenkinsfile.aws and Jenkinsfile.azure). The reason why I created two separate ones rather than a single combined pipeline was to maintain independent deployment control over each cloud provider. Such allowed me to deploy to AWS without affecting Azure (and vice versa), enabled parallel pipeline execution for faster deployments, allowed for simplified debugging when issues are cloud-specific, and demonstrated a professional multi-cloud architecture pattern. Each Jenkinsfile was to handle its cloud provider’s specific authentication requirements and deployment steps, making the code more maintainable. Such an approach also had the added benefit of intending to make the Jenkins dashboard more informative — I could see at a glance which cloud deployments succeeded or failed and their individual execution times.

First up was to create the Jenkinsfile.aws file. I typed everything out by hand, and you can access it in the following file, along with the copious comments that I have made

You can open it and read the comments to learn more about how things were done.

Then I created another Jenksinfile, this time titled Jenkinsfile.azure for the planned usage of the Azure cloud provider. It was heavily based on the previous Jenkinsfile.aws file, but appropriate tweaks were made. You can see its text here:

HomelabSeriesGithubRepo/Homelab12 - CI CD Pipeline at main · secnate/HomelabSeriesGithubRepo

I then quickly committed and pushed the two Jenkins files into the homelab GitHub repository

Section V: Create AWS Jenkins Pipeline & Job

After creating the Jenkinsfiles that define the pipeline logic, I needed to create a Jenkins job, which is Jenkins’ way of executing those pipelines. A Jenkins job acts as a configured instance that tells Jenkins where to find the code (GitHub repository), which branch to monitor, when to trigger builds (on code changes), and which Jenkinsfile to execute, essentially bridging the gap between the automation instructions I wrote and Jenkins’ ability to run them.

First up was to create the AWS pipeline. I did this by opening up Jenkins at the http://localhost:8080 URL, then clicked the + New Item button and proceeded with configuring the pipeline, and then clicking the Save button appropriately. All the configuration details will be available in the screenshots shown below

Notice the configuration of the schedule to Poll SCM. The schedule H/5 * * * * tells Jenkins to check GitHub for changes every 5 minutes, with the H (hash) symbol being a special Jenkins feature that distributes the polling time across the 5-minute window to prevent all jobs from checking simultaneously. For example, instead of every job polling at exactly :00, :05, :10, etc., Jenkins uses a hash of the job name to spread checks across the interval (like :02, :07, :12 for one job and :03, :08, :13 for another), which prevents server overload when you have many jobs and ensures more efficient resource usage.

Next was to test it. I went to the Jenkins dashboard

And clicked on the Build Now green play button. The build failed, so I went into the Build History information panel page

And then clicked the Terminal-looking icon on the right side of the page to the console output and error messages:

And started to work on fixing the Jenkinsfile and the issues raised so that the pipeline could execute smoothly & without a hitch. This was a beast to get working — with multiple errors, starting with the usage of a misplaced } curly bracket. Other issues that I fixed included: removing an -e flag that I unnecessarily included in the echo statements of the Jenkinsfile’s last section, replacing the sh commands with the bat commands since my homelab machine is a Windows one, replacing the ${ <VARIABLE> } syntax for environmental variable interpolation with the Windows %VAR% syntax, needing to regenerate the AWS RSA keys to fix a mismatch error, and a race condition going on in the created EC2 instance involving docker getting spun up and created.

There was also an issue regarding the Docker, Terraform, and Infracost commands being under my local user account and not being seen by Jenkins, which ran under the Windows SYSTEM account. For those who don’t know, the Windows SYSTEM account is a powerful “super” account that does not have visibility into user instances by default. I also on my Windows workstation had the SSH, SCP, and other related utilities in the C:\Windows\System32\OpenSSH folder, which were not seen by the executing pipeline, nor in my CMD and PowerShell Windows when debugging.

To remediate this, I quickly updated the Path variable on my Windows computer too to have those seen for my own personal needs for the SSH and SCP-related utilities, and then ensured my Jenkinsfile also had them referenced

These workarounds were suitable for my Homelab needs as they derived from the fact that I needed to get Jenkins running on my Windows machine. It is my understanding that Jenkins is typically run and used on Linux, and there are many pain points that people in the industry typically experience when trying to get it to run on Windows (per a conversation I had with some people on a tech Slack chat). Due to time constraints, and the need to continue the momentum on the homelab series, I did not dig around into the nuances regarding getting a Jenkins pipeline deployed on a Linux machine or another alternative option.

The hardest part wasn’t provisioning cloud infrastructure — it was convincing Windows OpenSSH that Jenkins was allowed to read its SSH keys. On Windows, OpenSSH will silently reject private keys unless both the key file and its parent directory have tightly restricted ACLs. Because Jenkins typically runs as the SYSTEM account on Windows, the SSH private key must be readable by SYSTEM (and only SYSTEM), or OpenSSH will refuse to load it before any network authentication occurs. I did this using the following lines in the stage(‘Deploy Application to EC2’) stage of the Jenkinsfile:

bat ”””

icacls “%SSH_KEY_FILE%” /reset

icacls “%SSH_KEY_FILE%” /grant:r “NT AUTHORITY\\SYSTEM:(F)”

icacls “%SSH_KEY_FILE%” /inheritance:r

”””

As a final note: all the testing for getting the pipelines in the AWS Jenkinsfile working was done on a stage-by-stage basis, starting by commenting out all the stages but the first one, and then adding each successive stage as its predecessor was tested, clicking the green play button to launch a build, and then looking at the console output to confirm it was working.

If I could redo this lab all over again, I’d have installed and run Jenkins on a Linux machine, not my Windows homelab workstation. Lesson painfully learned.

After pulling teeth for quite a few evenings, I finally got the Jenkinsfile working, clicked the build button

And opened up the console output to keep track of what I was executing in Jenkins

Took a look at the **very** interesting Infracost estimate

And scrolled down to the bottom to confirm successful pipeline completion

Looked nice. I then opened up the http://3.21.44.173:5000/ path in the browser, and look at what I got!

Whew. Again, learn from my pain, do not run Jenkins on Windows. I could have pulled this off in half the time if it was running on a Linux system or container — and this is industry best practice too!

Section VI: Create Azure Jenkins Pipeline & Job

Next up was to get another Jenkins pipeline and job working for Azure. As a first step to try and save me a lot of the sweat and tears in the AWS pipeline-related work, I opened up the jenkinsfile.aws and jenkinsfile.azure files and copied and pasted the updates/fixes for the previous issues.

Then I got to work with making my new pipeline. First I clicked the + New Item button in the Jenkins page

And went through the following configuration steps. Notice that this pipeline is similar to the AWS one, so I selected for the first step the Copy From Multi-Cloud-AWS-Deploy option

This conveniently presented the configuration window, with all the settings from the Multi-Cloud-AWS-Deploy pipeline conveniently there. I then tweaked them to the following values, and saved + applied the changes to get the Azure pipeline underway

And then I also did repeated rounds of testing to get this over the finish line. Notice that I temporarily disabled the Multi-Cloud-AWS-Deploy pipeline so that I could focus only on the Azure pipeline without needing to deal with the automatic AWS-based deployment of EC2 and VPC instance. Just like in the previous Terraform homelab, changed the region from a restricted location that was blocked to eastus, upgraded the Public IP from Basic SKU (which was retired September 30, 2025 per Microsoft’s official announcement) to a Standard SKU with Static allocation. Some other adjustments in the Terraform, some additional testing, and that did the trick.

Finally!

Section VII: Testing Automated Builds (Git Push Trigger)

Next up was to test automated builds. This is because the CI/CD pipelines that I created were supposed to catch any changes with the GitHub webhook trigger within a five minute period, and trigger both the AWS and Azure pipelines without manual intervention.

I did this by making a small change in the app.py file by changing the version number from 1.0 to 2.0. This change in version number would also be presented in the presented HTML outputs in the Flask applications for both of them.

Then I committed the changes and pushed it out to my GitHub repository, and waited for five minutes to see what would transpire.

Sure enough, the two pipelines got triggered, and the builds got underway

And then after that, I took a look at the pipeline-made resources, and voila, just as expected — the Version 2.0 appeared!

For AWS:

For Azure:

Section VIII: Cost Management and Monitoring

As we all know, cloud spend isn’t free. Let’s take a look at the costs for the two cloud providers.

First, in AWS, I opened the console data of the last Jenkins pipeline, and took a look at the cost estimate generated by Infracost.

So here we saw that this project used 7 cloud resources, one was estimated to cost money, and 6 were free. The total monthly cost was about $8 (or $8.39). Not too shabby for a homelab exercise.

Then I logged into the AWS Billing and Cost Management >> Cost Explorer, set the date range to the time period I was working on this homelab exercise, and took a look at how much I generated in total costs.

About $0.20. Not bad. My free credits that AWS gave to students handily covered all that — and some.

Next up was to check on the Azure-related costs. I also opened up the console data of the last executed Jenkins pipeline, and took a look at the Infracost-provided data estimate:

Well well well. Infracost indicated that to deploy the same Flask application into Azure, it takes 9 cloud resources [instead of AWS’ 7], and that 6 were free and 3 involved expenses. The total estimated monthly cost was about $16.04. If this was a real Flask application and I was looking to optimize cloud spend, I would naturally lean towards Azure for an approximately ~$8 worth of total monthly savings.

Now to check on the total cloud spend in Azure. I went into the Cost Management portal in Azure, and set the date ranges for the time period that I worked on this homelab

Five cents in total. Pretty cheap. My Azure credits handily covered that as well.

The last thing that I did for this section was to use the azure-cost-cli tool, which I found online. The Azure Cost CLI is a cross-platform command-line tool that queries Azure’s Cost Management API to display cost breakdowns, budgets, and spending forecasts directly in the terminal, providing quick visibility into Azure resource expenses without needing to navigate the Azure Portal. I included it in the homelab as a bonus to demonstrate cost-awareness practices in DevOps — enabling developers and operators to monitor infrastructure spending as part of their daily workflow and catch budget overruns early. Cloud spend is a very finicky thing and can accumulate quickly if left unchecked.

I had dotnet installed on my homelab workstation, so I used the dotnet tool install –global azure-cost-cli command to install the tool

And then I used the azure-cost -s “<REDACTED_SUBSCRIPTION_ID>” to take a look at the total costs involved. As expected, it shows about five cents in total.

Then I used the azure-cost costByResource -s “<REDACTED_SUBSCRIPTION_ID>” command to see a breakdown by resources

So I spent about 5 cents in total on the Azure component of this homelab, with four cents going to the Standard IPv4 Static Public IP and 1 cent going to a storage disk for the VM instance’s configuration and data (i.e. acting like its physical hard drive).

Section VIII: Cleaning Up Resources

To avoid unnecessary cloud costs, I made sure to terminate all provisioned resources once the homelab experiments were complete.

The infrastructure was originally deployed through the Jenkins pipeline, which managed Terraform initialization and credential handling. Attempting to run the terraform destroy command locally in the configuration directories on my homelab workstation didn’t work, as it couldn’t properly access the remote state or the same execution context used by Jenkins.

Rather than spending additional time modifying the Jenkinsfile to add a dedicated destroy stage (which would have been ideal for full automation), I opted for a pragmatic approach: I manually terminated the resources via the AWS and Azure consoles.

This involved:

• Identifying and deleting the resource groups in Azure
• Terminating EC2 instances, deleting VPCs, and other resources in AWS

All high-cost resources (e.g., running VMs, NAT gateways, managed disks) were shut down and removed first, followed by the remaining supporting infrastructure. This ensured no ongoing spend while wrapping up the project quickly.

While not as elegant as a fully automated terraform destroy through the pipeline, this manual cleanup was sufficient for a short-lived homelab and allowed me to responsibly decommission everything without leaving stray costs behind.

Lesson Learned: For future homelabs or similar work, I’ll need to add a destroy parameter to the Jenkins pipeline from the start to make teardown as seamless as creation, and have all pipelines take it into account. Such a parameter would look something like this in the Jenkins configuration console:

For me this was quite the journey. I certainly learned a lot, and I hope you did too. See you in the next round!

Observability is the ability to understand your system’s internal states from its external outputs — primarily metrics, logs, and traces — allowing you to detect, diagnose, and fix issues without guesswork.

In modern IT infrastructure, it’s essential for maintaining reliable services: spotting high resource usage before outages, identifying bottlenecks or leaks, and troubleshooting performance issues quickly — shifting from reactive firefighting to proactive management and ensuring scalability, availability, and efficiency.

Prometheus excels at metrics collection with a pull-based architecture — the server periodically scrapes time-series data from instrumented targets (via exporters or client libraries), stores it efficiently in a local TSDB, and supports powerful querying with PromQL, alerting via Alertmanager, and service discovery.

Pair it with Grafana for stunning, interactive dashboards that visualize Prometheus data through graphs, alerts, and templates that make complex metrics intuitive at a glance — and you can see how easy keeping track of what is going on in your infrastructure can become.

Let’s get started.

Section I: Setup

I needed to start by preparing for the actual work. From the past Docker container homelab, I already had Docker installed on my computer. I quickly opened PowerShell and verified that Docker is running with the docker –version and docker ps command

I then made a folder on my Windows homelab computer’s desktop for this homelab and made sub-folders titled prometheus and grafana. These were to be used later.

Great. Next up is to encode the configuration of the Prometheus server in — you guessed it — another YAML file! I wrote up the following YAML, heavily based on the example prometheus.yml file on the official Prometheus GitHub repository:

The scrape_configs section in Prometheus’s configuration file defined what systems to monitor and where to find them. I configured Prometheus as a data collector that regularly visited different addresses to gather metrics — a process called “scraping.” Here, I configured four distinct targets, each running on its own port:

1. Prometheus itself on localhost:9090 — I had Prometheus monitor its own performance and health through meta-metrics, allowing me to track how well the monitoring system itself was performing.
2. Windows Exporter on host.docker.internal:9182 — Windows Exporter is a specialized Windows service that translates Windows performance counters (CPU usage, memory stats, disk I/O, running services) into Prometheus-compatible metrics, essentially acting as a bridge between Windows’s native monitoring capabilities and Prometheus’s data format. I used the special Docker hostname host.docker.internal because Prometheus ran inside a container and needed to reach out to the Windows host machine where this service was installed.
3. Node Exporter on an Ubuntu VM — Node Exporter is the Linux equivalent of Windows Exporter. This service exposes hardware and kernel-level metrics like CPU load, memory pressure, filesystem usage, and network statistics by reading from /proc and /sys filesystems and presenting them in a format Prometheus could understand.
4. cAdvisor on cadvisor:8080 — cAdvisor (Container Advisor), a Google-developed tool, provides resource isolation metrics, historical usage data, and network statistics for a container. This gave me visibility into how the Prometheus, Grafana, and cAdvisor containers themselves were performing.

Each of these exporters — Windows Exporter, Node Exporter, and cAdvisor — acted as a translator, converting system-specific metrics into the standardized Prometheus format. Every 15 seconds (my configured scrape interval), Prometheus visited each of these endpoints, pulled the latest metrics, and stored them in its time-series database, ready for visualization in Grafana or evaluation against alert rules.

Cool. Next up was to make preparations for the alerting rules in Prometheus.

A quick intro to how alerting works in Prometheus: It transforms passive monitoring into active incident detection by continuously evaluating rules against collected metrics and firing alerts when conditions are met. Unlike traditional monitoring systems that might only check conditions periodically, Prometheus evaluates alert rules at a regular interval (I set mine to 30 seconds) against every scrape of fresh data, ensuring near real-time detection of issues.

The way alerts work follows a simple but powerful pattern: you define a condition using PromQL (Prometheus’s query language), specify how long that condition must be true before firing, and attach metadata to help responders understand what’s happening. For example, my high CPU alert uses the expression 100 — (avg by(instance) (rate(windows_cpu_time_total{mode=”idle”}[5m])) * 100) > 80, which calculates the average non-idle CPU time over the past 5 minutes and triggers when it exceeds 80%. The for: 2m parameter means this condition must remain true for 2 consecutive minutes before the alert actually fires — this prevents false alarms from brief, harmless spikes.

Each alert moves through three states in its lifecycle: inactive (condition is not met), pending (condition is met but hasn’t exceeded the for duration yet), and firing (condition has been true long enough and the alert is active). This state machine approach prevents alert fatigue by filtering out transient issues while still catching sustained problems that need attention. I configured six different alerts covering CPU usage, memory pressure, disk space, service availability, and container resource consumption — each with carefully chosen thresholds and durations based on what would actually constitute a problem worth investigating.

The alerts themselves are defined in a separate YAML file (alert_rules.yml) organized into groups, with each alert containing the PromQL expression, the duration threshold, severity labels (like warning or critical), and human-readable annotations that explain what’s wrong and provide context. When an alert fires, Prometheus can forward it to Alertmanager (a separate component) for routing to notification channels like email, Slack, or PagerDuty, though in my homelab setup I focused on viewing active alerts directly in the Prometheus web UI to understand the alerting mechanism itself. This separation between metric collection (Prometheus), alert evaluation (Prometheus), and alert routing (Alertmanager) creates a flexible system where you can define sophisticated rules regarding what constitutes a problem and how to respond to them.

Theory lesson is over. I made the alerts_rules.yml

Notice the system_alerts group at the top of the file. In Prometheus, groups are organizational containers that let you bundle related alert rules together and control how often they’re evaluated as a unit. Here, I used the system_alerts group to evaluate all my infrastructure alerts every 30 seconds, but in production environments one might create separate groups like critical_alerts (evaluated every 10 seconds) and warning_alerts (evaluated every 60 seconds) to prioritize urgent issues while reducing the load of checking less critical conditions. Groups are purely organizational — they don’t affect how alerts fire, they just determine when Prometheus checks the rules and help you logically separate concerns like “database alerts” versus “network alerts” in larger monitoring setups.

Next up was to create a Docker Compose file for spinning up the Prometheus, Grafana, and cadvisor containers. Docker Compose is a tool that uses a YAML file to define and run multi-container applications as a single orchestrated stack, allowing you to start, stop, and manage related containers together with one command (docker-compose up -d) instead of running separate docker run commands for each service. In my previous Docker homelab (Homelab #9), I didn’t use Docker Compose because the goal was to learn the fundamentals of Docker by manually running individual containers with docker run, understanding networking, volumes, and container management at a granular level. However, for this observability lab, Docker Compose made sense because Prometheus, Grafana, and cAdvisor needed to work together as an integrated monitoring stack with shared networking and coordinated lifecycle management.

Below is the screenshot of what I wrote for the docker-compose.yaml files, as well as the explanatory comments to guide the reader through what each line means:

Additional note regarding the command field. Per the documentation, the command field in a Docker Compose file overrides the default command declared by the container’s image. In other words, it controls what the container runs when it starts. Notice that it keeps the image’s entrypoint intact. This is often very useful when dealing with “starting point” containers from Docker Hub or another shared repository that you need to get working for your own needs.

To illustrate, I pulled the latest Prometheus Docker container using the docker pull prom/prometheus:latest command on my Homelab machine, and then used the docker inspect prom/prometheus:latest command to see what the default command was — as well as the other out-of-box configuration and state information. Here is what I got:

That was educational. You can now see how the command that I specified for the Prometheus image in the Docker Compose file differs from what they provided out of box!

With the Docker Compose file made, I went into the directory where the project was located and used the docker-compose up -d command to start up all the services.

OK, now to check that the containers we wanted to launch are actually running

Looks good. Then I quickly used the docker logs prometheus, docker logs grafana, and docker logs cadvisor commands to look at their respective logs

OK, we got some logs coming in. Now to see how things look in the browsers. I went into my homelab workstation’s browser and confirmed access to

• Prometheus — at http://localhost:9090

• Grafana — at http://localhost:3000

• cAdvisor — at http://localhost:8080

Great. Now it’s time to configure things on more of the boxes themselves.

First up was to install Windows Exporter onto my workstation. Windows Exporter is a Prometheus-compatible monitoring agent that runs as a Windows service and exposes system performance metrics (CPU, memory, disk, network, services) through an HTTP endpoint on port 9182. It acts as a translator between Windows Performance Counters (built-in Windows metrics that track system resource usage and are viewable in the Performance Monitor application) and Prometheus’s metrics format, allowing Prometheus to scrape and monitor Windows-specific system statistics.

I went onto the GitHub page for Windows Exporter and downloaded the .exe executable from the https://github.com/prometheus-community/windows_exporter/releases page and put it into a homelab-specific folder.

Then I fired it up by double-clicking the file and got it running

Then, it was time to access Windows Exporter in my browser. It serves its metrics through a built-in HTTP web server on port 9182, allowing Prometheus (or any web browser) to access the metrics by simply making HTTP GET requests to http://localhost:9182/metrics, which returns all the Windows performance data in plain text format that Prometheus can parse and store.

And we can see that the process is running appropriately

OK. Next up was to install Node Exporter on the Ubuntu VM that I quickly made for this lab. Node Exporter is the Linux equivalent of Windows Exporter — a lightweight Prometheus agent that exposes hardware and kernel-level metrics by reading from Linux’s /proc and /sys filesystems and serving them on port 9100. I installed Node Exporter on my Ubuntu VM to provide Prometheus with visibility into the VM’s CPU usage, memory consumption, disk space, and network traffic, creating a multi-platform monitoring setup where I could track both my Windows host and Linux VM from a single Prometheus instance and compare system performance across different operating systems.

Notice that this newly-created VM was configured with a host-only network adapter, and I confirmed that it could be pinged from my homelab workstation. The exact steps of creating an Ubuntu VM, configuring its network interfaces, and setting static IP addresses were covered multiple times in past homelab writeups, so I will not cover them for this round.

In this newly-created VM, I followed the steps in the Prometheus Documentation page to download the node_exporter application, version 1.10.2 (the releases can be found on the GitHub page).

First I used the wget https://github.com /prometheus/node_exporter/releases/download/v1.10.2/node_exporter-1.10.2.darwin-amd64.tar.gz command to download the initial .tar.gz file.

Then I extracted its contents appropriately with the tar xvfz node_exporter-*.*-amd64.tar.gz command

And then opening the folder using the cd node_exporter -*.*-amd64 command, we see the actual executable

OK so this is just a starting point. We had to make a new user, specifically for Node Exporter! Why? We learned a lot in security classes about the concept of least privilege — and this is a prime illustration of such a case in point. If we run Node Exporter under the root user, an attacker could exploit a potential vulnerability and get access to the entire system. That’s not good. Meanwhile, running it under a personal account — say “nathanuser” — is also a bad idea. A personal user might have sudo privileges, or an attacker might get access to their personal files, SSH keys, and etc.

So that’s why I made a dedicated user under which Node Explorer will run under a dedicated systemd-managed service. I did this with the sudo useradd –no-create-home –shell /bin/false node_exporter command

Let’s discuss briefly why this command was used.

The –no-create-home flag told the useradd command to not create a /home/node_exporter/ directory. This is because this is just a background service running, it doesn’t need to store any personal files, there is less clutter in the /home/ folder, and also…. There is no place for an attacker to work in and access files/download files if Node Exporter gets compromised.

The –shell /bin/false flag was used to specify a program that immediately exits with a failure code — which prevents any console from appearing. This is intended to prevent interactive logins. If someone tries to log in as node_exporter, they will not be successful — and if an attacker compromises Node Exporter or gets credentials somehow, they won’t be able to do the following, which they could do with a regular /bin/bash shell:

So now we see why hardening systems and being deliberate about what kinds of accounts are created (and the configurations thereof) are super important!

Next up was to set up Node Exporter as a systemd service, where it will run in the background and automatically start at boot. The following steps are derived from a KodeKloud tutorial.

First, I used the sudo nano /etc/systemd/system/node_exporter.service command to open up the text editor

And then wrote the following service file:

To explain what some of its lines mean: The Wants=network-online.target field establishes a soft dependency on the network being available (meaning Node Exporter prefers the network to be ready but will still attempt to start even if it isn’t), while the After=network-online.target field controls the startup order by ensuring Node Exporter waits for the network to come online before launching. These directives together prevent Node Exporter from starting before it can bind to port 9100 and serve metrics. The Type=simple setting tells systemd that Node Exporter runs as a straightforward foreground process that stays directly under systemd’s control, rather than spawning a copy of itself to run in the background (called “forking”) or detaching from the terminal (called “daemonizing”). The ExecStart=/usr/local/bin/node_exporter line specifies the exact binary path to execute when starting the service. The WantedBy=multi-user.target field tells systemd “when you run systemctl enable node_exporter, create a link that makes this service start automatically whenever the system boots into multi-user mode.” Essentially, it’s like adding Node Exporter to the system’s startup programs list so it runs every time the server boots up.

Neato burrito. Next up was to start the service.

To do this, I ran sudo systemctl daemon-reload to tell systemd to reload its configuration and recognize the new node_exporter.service file I just created. Then sudo systemctl start node_exporter was to launch the Node Exporter service, sudo systemctl enable node_exporter was to configure it to start automatically on every boot, and finally sudo systemctl status node_exporter to verify that the service was running correctly.

Welp. That didn’t work. Scrolling a bit to the right revealed the red line error message: node_exporter.service: Failed at step EXEC spawning /user/local/bin/node_exporter: No such file or directory

Ah. That was because I forgot to copy over the node exporter executable that I downloaded and extracted from that .tar.gz repository. Doable fix. I quickly went into the folder in which I had the node_exporter executable downloaded and used the sudo cp node_exporter /usr/local/bin/node_exporter command to quickly copy things over.

Then I repeated the previously performed systemctl-related commands again — and got an Exec format error:

This is an error related to architecture incompatibility problems. I used the uname -m command to see my VM’s architecture and used the file /usr/local/bin/node_exporter command to see the node_exporter executable’s architecture.

Aha. This was the problem! The downloaded node_exporter binary was a Mach-0 executable! A Mach-O (Mach object) file is the native executable file format used by Apple’s operating systems, including macOS, iOS, and others based on the Mach kernel. It functions similarly to PE (Portable Executable) files in Windows and ELF (Executable and Linkable Format) files in Linux. Definitely a platform issue. My mistake.

This was not the end of the world and could be quickly fixed. I used the wget https://github.com /prometheus/node_exporter/releases/download/v1.10.2/node_exporter-1.10.2.linux-amd64.tar.gz command to quickly download the Linux-version of an amd64 architecture version of node exporter. Then I used the tar xvfz node_exporter-*.*-amd64.tar.gz command to quickly extract it. Afterwards, I opened the extracted folder, used the sudo cp node_exporter /usr/local/bin/node_exporter to copy it over, and then ran the systemd-related commands all over again.

Finally, I reached the promised land, the Shangri-La, the state of nirvana. The node_exporter service was active and running!

I then needed to verify that the metrics were being collected. I did this by opening my homelab workstation’s browser and going to http://192.168.100.253:9100/metrics

Beautiful. I then needed to make a tweak to my prometheus.yml file to ensure that the IP address is of the right value and corresponded to my actual Ubuntu VM address.

Wonderful. I then needed to lock and load the changes. This was done with the docker restart prometheus command, followed by the docker logs prometheus command to see the play-by-play

Then to check up on whether Prometheus had all the targets up and running, I went into my homelab workstation’s browser and entered the http://localhost:9090/targets IP address

Perfect. We have been on quite the setup journey. Onwards to the next sections!

Section II: Configure Grafana Dashboard

After getting the data being fed into Prometheus, I turned my attention to Grafana. As a reminder, it is the graphical display of information from Prometheus (and other) data sources — and can be conveniently used as a single pane of glass.

I started by opening Grafana in my homelab workstation’s browser at the http://localhost:3000 address

Then I logged in using the default admin/admin credential, changed the password as prompted, and ended up on the homepage.

Then I opened up the Connections >> Data Sources page. As expected, there were no data sources currently defined

Then I clicked the Add Data Source button and selected Prometheus

And arrived at this beautiful configuration page

I quickly entered the http://prometheus:9090 url to configure the data source, scrolled down to the bottom, and clicked the Save and Test button. I got a green checkmark showing that Prometheus API has been successfully queried

Great. Now for the actual data. I went back to the Grafana’s Homepage and clicked the Dashboards >> New Dashboard button. I clicked the +Add Visualization button and selected Prometheus as my data source.

So to explain (per the Grafana documentation), a Grafana dashboard is a set of one or more panels — visual representation of data composed of a query and a visualization — which are neatly organized and arranged into one or more rows. So that’s why I got presented with the page for creating a brand new panel

Obviously, it was time to make the panels. Due to the number of panels involved, I will present a bulleted list of each with their properties followed by an explanatory screenshot:

• Panel 1: Windows CPU Usage

Query: 100 — (avg by(instance) (rate(windows_cpu_time_total{mode=”idle”}[5m])) * 100)
Legend: {{instance}} CPU
Title: Windows CPU Usage (%)
Unit: Percent (0–100)
Thresholds: 80 (orange), 90 (red)

• Panel 2: Windows Memory Usage

Query: 100 — ((windows_memory_physical_free_bytes / windows_memory_physical_total_bytes) * 100)
Legend: {{instance}} Memory
Title: Windows Memory Usage (%)
Unit: Percent (0–100)

• Panel 3: Windows Disk Space

Query: (windows_logical_disk_free_bytes{volume=~”C:”} / windows_logical_disk_size_bytes{volume=~”C:”}) * 100
Title: Windows C: Drive Free Space (%)
Legend: {{instance}} Disk Space
Unit: Percent (0–100)

• Panel 4: Ubuntu VM CPU

Query: 100 — (avg by(instance) (rate(node_cpu_seconds_total{instance=”ubuntu-vm”,mode=”idle”}[5m])) * 100)
Title: Ubuntu VM CPU Usage (%)
Legend: {{instance}} CPU Usage
Unit: Percent (0–100)

• Panel 5: Ubuntu VM Memory

Query: (1 — (node_memory_MemAvailable_bytes{instance=”ubuntu-vm”} / node_memory_MemTotal_bytes{instance=”ubuntu-vm”})) * 100
Title: Ubuntu VM Memory Usage (%)
Legend: {{instance}} Memory Usage
Unit: Percent (0–100)

• Panel 6: Network Traffic

Query A: rate(windows_net_bytes_received_total[5m])
Query B: rate(windows_net_bytes_sent_total[5m])
Title: Windows Network Traffic
Unit: bytes/sec

• Panel 7: Container CPU Usage

Query: sum(rate(container_cpu_usage_seconds_total[5m])) by (container)
Title: Container CPU Usage (cores)

• Panel 8: Container Memory

Query: container_memory_usage_bytes{name!=””} / 1024 / 1024
Title: Container Memory Usage (MB)
Legend: {{ id }}

• Panel 9: System Uptime (hours)

Query (Windows): (time() — windows_system_boot_time_timestamp)/3600
Query (Linux): (node_time_seconds — node_boot_time_seconds)/3600
Title: System Uptime (hours)

• Panel 10: Active Alerts

Query: ALERTS{alertstate=”firing”}
Title: Active Alerts
Visualization: Stat (show count)
Legend: “{{job}}” (volume: {{volume}})

Wonderful. With the panels made, I had to then rearrange the panels into a logical layout on one single screen. I rearranged them into a nice dashboard and had the panels with similar themes (memory usage %, CPU usage %, etc). I also used the Add Row button to name the row corresponding to these panels as Homelab Monitoring — Windows & Linux

I then used the Save dashboard button in the top right-hand corner and named what I created the “System Monitoring Dashboard.”

Also notice that Grafana has a cool ability to export/share dashboard-related information, specifically using JSON. I went into the settings page and saw the JSON Model

There was no need for me to do anything with it at the time, but it was good to know that it was available if I needed it for whatever reason.

Section III: Testing & Alert Simulation

The next section of the homelab involved stress testing the systems involved and monitoring the generated Prometheus alerts and the Grafana dashboard. This section will have sub-sections related to each task performed.

Task #1: Test Windows CPU Alert

The first thing I did was create a testing and alert simulation folder inside the repository where I kept the files related to this homelab. I then created a cpu-stress.ps1 PowerShell script

Then, I ran it

I then set the Grafana dashboard from the previous section to cover the data collected from the last 30 minutes, and took a look at the Windows CPU Usage (%) graph. As expected, there was a spike

Notice that the graph did not cross the 80% threshold. I then went into Prometheus and saw — as expected — that the HighCPUUsage alert was in an Inactive state — i.e. — it did not fire.

I also out of curiosity pulled up Prometheus’ graphing capabilities, put in the PromQL query corresponding to the alert, and took a look at the data more granularly.

Everything was working as expected.

Task #2: Test Memory Alert (Ubuntu VM)

I opened up the Ubuntu VM that I had up and running and installed some updates using the sudo apt-get update command. Then I used the sudo apt-get install -y stress-ng command to install stress-ng. To explain simply what it is, stress-ng is a workload generator for Linux and other Unix-like systems and helps test their ability to maintain a good level of efficiency under unfavorable conditions. It features 370+ stress tests for CPU (integer, float, matrix ops), memory (cache, virtual memory), I/O, and kernel interfaces.

I then on the Ubuntu VM generated high memory usage with the following command: stress-ng –vm 1 –vm-bytes 90% –timeout 5m. Per the documentation, the –vm 1 flag told stress-ng to start one worker that continually writes to allocated/mapped memory, the –vm-bytes 90% flag tells the worker to work with 90% of the total available memory, and the –timeout flag told stress-ng to end the stress tests after a 5 minute period

After this was completed, I took a look into Grafana. Notice that the memory usage briefly crossed the 90% memory usage threshold, as expected.

Then, I looked at Prometheus to see if any alerts fired on the VM-related activity. An alert did not fire.

This was expected. As explained per the arrows pointing to two keywords in the above screenshot, the system_alerts for High CPU usage and High Memory usage were for Windows system only.

Task #3: Test Service Down Alert

On the Ubuntu VM, I used the sudo systemctl stop node_exporter command to stop node_exporter. As a reminder, it acts as a metrics/data collector on this Ubuntu VM endpoint for Prometheus to then be able to scrape.

I then waited for a few minutes, and just as expected, Prometheus had the TargetDown alert fire

And additionally, the Grafana panel showing Active Alerts showed that there was an alert on the Ubuntu VM

Everything was operating just as expected. Having achieved the goal of this test, I quickly restarted the node_exporter service back up on the Ubuntu VM and confirmed it was back up and running

Task #4: Test Container Stress

I needed to do one more stress test. Per an explanation by Ravi Patel, stress testing with the simulation of heavy loads and stress conditions can help identify potential bottlenecks, vulnerabilities, and performance issues before they impact users. We previously did that in the second task on the Ubuntu VM. This time, I needed to do it in the context of a Docker container.

To do this, I used the polinux/stress Docker container image. First, I pulled the image using the docker pull polinux/stress command. Then I used the docker run -ti — rm polinux/stress stress — cpu 2 — io 1 — vm 1 — timeout 300s — verbose command to launch a stress test. The –rm flag ensured automatic cleanup after the exit, there were two workers performing continuous calculations, and one Input/Output worker generated sync operations.

In Grafana, the graph for Container CPU Usage showed an expected spike

I didn’t see any alerts fire in Prometheus. This was expected given that my Container High Memory alert rule specifically checks for usage as a percentage of defined container memory limits (container_spec_memory_limit_bytes), and my stress test container ran without any resource limits configured. In a production environment, containers would typically have CPU and memory limits defined in their Docker Compose or Kubernetes configurations, making these percentage-based alerts appropriate. For this homelab, I confirmed that the alert rules were syntactically correct, validating my understanding of alert rule structure even though the specific metrics required to trigger these container alerts weren’t present in my test scenario.

Section IV: Cleanups

With this homelab exercise over, I shut down my Ubuntu VM in VirtualBox, and then used the docker-compose down command to end the execution of the grafana, cadvisor, and the prometheus containers

Then I also closed the Window in which the Windows Exporter application was running too.

That’s all for this round. I certainly learned a lot and now understand how Prometheus, Grafana, and PromQL work. There certainly was a bit of setup to go through, but once everything is set up, it’s nice to sit back and visually take in a lot of data regarding your infrastructure’s functioning on a single screen, and be reassured that alerts will fire if any issues are detected.

See you all in the next round!

Kubernetes (often shortened to K8s) is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications across clusters of machines. Originally developed by Google and now maintained by the Cloud Native Computing Foundation, Kubernetes has become the industry standard for managing containerized workloads in production environments. It’s critical because it solves the challenges of running containers at scale — automatically handling failures, distributing traffic, managing storage, and ensuring applications stay available even as infrastructure changes beneath them.

Let’s get started.

Section I: Initial Prep

Before starting with this round, I quickly confirmed that I had AWS Free Tier alerts and CloudWatch billing alerts enabled in my AWS account. I also created a $10 budget in the Azure portal and set an alert to fire at the 80% threshold.

Then I quickly installed Docker Desktop on my Windows homelab laptop (yes, outside of any VMs this time). It is the desktop application of Docker which simplifies the development and container management process. Then I installed

• kubectl, a command line tool that lets you manage Kubernetes
• minikube, a command line tool that lets you run Kubernetes on your local machine, primarily for testing and development purposes

A quick note on the distinction between these two tools: While both kubectl and minikube are command-line tools used in my Kubernetes workflow, they serve fundamentally different purposes. minikube is specifically for managing the Kubernetes cluster itself — it handles cluster lifecycle operations like starting, stopping, deleting the cluster, and managing cluster-level features through addons. Think of it as the tool that builds and maintains the infrastructure.

kubectl, on the other hand, is the universal Kubernetes client that works with any Kubernetes cluster (whether it’s Minikube, EKS, AKS, or a production cluster) and is used to manage the applications and resources running inside the cluster — deployments, pods, services, and so on.

In simple terms: minikube manages the stage on your local computer, kubectl manages the actors performing on it.

Section II: Local Kubernetes With Minikube

I started by opening up a PowerShell window with administrative privileges and launching my first Kubernetes cluster. This was done with the minikube start –driver=docker –cpus=2 –memory=4096 command

Let’s briefly discuss what I just did. In Kubernetes, a node is essentially a worker machine (virtual or physical) in the cluster that runs your applications inside pods (groups of one or more containers).

When using Minikube (a tool for running a local single-node or multi-node Kubernetes cluster for development/testing), the behavior depends on the driver.

• With most drivers (e.g., virtualbox, hyperkit), Minikube creates a virtual machine (VM), and the Kubernetes node runs inside that VM.
• With the — driver=docker flag (as in my command), Minikube runs the Kubernetes cluster directly as Docker containers on the host machine, without a separate VM. Specifically, for a single-node cluster (the default), the entire Kubernetes node (control plane + worker components like kubelet, kube-proxy, etc.) runs inside one main Docker container.
  This container is often referred to informally as the “node container” or “node image” because it’s built from a pre-configured base container image that Minikube pulls and uses to bootstrap the node.

Now, regarding the minikube start –driver=docker –cpus=2 –memory=4096 command. This command started a local Kubernetes cluster using the Docker driver:

• Minikube first pulled a base container image (this is the “node image”) from a registry.
• It then created and ran a Docker container from that image on my workstation’s Docker daemon.
• This container became my Kubernetes node, allocated 2 CPUs and 4096 MB (4 GB) of memory as specified.
• Inside this container, Minikube set up the full Kubernetes stack (API server, etcd, scheduler, etc.), and the pods ran as nested containers within it.

Then I used the kubectl cluster-info command to check the cluster’s status. Here is what I got

And then I got information regarding the cluster’s nodes using the kubectl get nodes command. It is a simple way to list all the worker machines (nodes) in a Kubernetes cluster and to get information about them at a glance.

And then I used the kubectl get nodes -o wide command to get some additional details, such as the internal and external IP addresses.

Now to introduce the notion of namespaces in Kubernetes. Namespaces are like virtual partitions that allow you to divide a single cluster into multiple isolated environments, making it easier to manage resources for different teams, projects, or applications without conflicts.

The kubectl get all — all-namespaces command lists all major resources (such as pods, services, deployments, and replicasets) across every namespace in your cluster, giving you a comprehensive overview of what’s running everywhere in one go. I executed it here:

Let’s take a look further at what we saw. The two namespaces listed in the output’s first column are

• default: It is automatically created when you set up a cluster and serves as the catch-all space for resources (like pods, services, and deployments) that you create without specifying any other namespace. It is usually used in simple setups, testing, or small teams
• kube-system: It is a reserved, system-created space dedicated to hosting core Kubernetes control plane components (like kube-apiserver, kube-scheduler, and kube-controller-manager) and essential add-ons (such as CoreDNS, kube-proxy, and networking plugins).
  You should never deploy your own applications here! It is strictly for cluster-internal resources. Access to it is typically restricted to prevent accidental disruptions to the cluster’s operation.

From the screenshot, we see that all but one of the resources listed are from the kube-system namespace. These include coredns (the cluster’s DNS service for resolving internal domain names and service discovery), minikube (a Minikube-specific service often tied to addon management or internal cluster utilities), kube-proxy (handles network proxying and load balancing for services across nodes), kube-scheduler-minikube (the scheduler component responsible for assigning pods to nodes), and storage-provisioner (dynamically provisions persistent storage volumes for pods).

Now it’s time to learn a bit about Kubernetes architecture. Per a fantastic article explaining it, it is divided around two core components: the control plane and the worker nodes.

Some of the components that we saw from that screenshot are as follows:

Control Plane Components (in kube-system namespace):

• etcd: Database storing all cluster data
• kube-apiserver: Front-end for Kubernetes control plane, handles all API requests
• kube-scheduler: Watches for new pods and assigns them to nodes
• kube-controller-manager: Runs controller processes (node, replication, endpoints, service account)

Node Components:

• kubelet: Agent running on each node, ensures containers are running
• kube-proxy: Network proxy maintaining network rules
• Container runtime: Docker (in our case) runs the containers

Coolio. From there, I used the minikube addons enable dashboard command to enable the Kubernetes dashboard and the minikube dashboard to start it and open it up in the browser. It is exactly as it sounds, a graphical user interface shown in the browser for managing Kubernetes resources.

Clicking around, I learned more about it. For example, clicking “Cluster” → “Nodes” , I saw my minikube node

Section II: First Deployment

Time to deploy Nginx. It is a high-performance web server and reverse proxy that’s widely used in production environments to serve web content, load balance traffic, and handle thousands of concurrent connections efficiently. I started to deploy it in this Kubernetes lab because it’s a lightweight, real-world application that perfectly demonstrates container orchestration concepts like scaling, service discovery, and self-healing — skills that directly translate to managing production workloads in cloud-native environments.

Remember YAML files from Ansible? Turns out they are the primary means of defining, creating, and managing objects and resources declaratively in Kubernetes. They serve as blueprints for an application’s desired state, which Kubernetes then works to maintain. A nice piece by the Cloud Native Computing Foundation and a Medium post had some useful information regarding how YAML files work for Kubernetes.

Time to get to work. I started by creating a nginx-deployment.yaml file, heavily based on a sample one found in a Kuberentes documentation page.

Then I applied the deployment with the following commands:

• kubectl apply -f .\nginx-deployment.yaml →
• kubectl get deployments
• kubectl get pods

Now it’s time to create the NodePort service. To explain simply, a NodePort service exposes your Kubernetes application on a specific port on every node in the cluster, making it accessible from outside the cluster without needing a cloud load balancer. I started by creating the nginx-service.yaml file

Then I applied and tested the service using the following commands:

• kubectl apply -f nginx-service.yaml
• kubectl get services
• minikube service nginx-service –url

Then I went into my homelab computer’s browser and saw this:

It worked.

Section III: Pod Failover

To slightly rehash what was mentioned previously regarding Pods — and to go over concepts in greater detail with a referencing of Kubernetes’ official documentation — a “Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. Those resources include:

• Shared storage, as Volumes
• Networking, as a unique cluster IP address
• Information about how to run each container, such as the container image version or specific ports to use”

Now for the concept of Pod failovers. Kubernetes pod failover is a self-healing automatic process by which Kubernetes detects a failed pod and replaces it with a new, identical pod on a healthy node. This ensures continuous application availability without manual intervention.

Let’s start. First, I started by watching the pods in real-time using the kubectl get pods -w command.

SIDE NOTE: The kubectl help get pods command outputted a pile of information regarding how the kubectl get pods command works and explained what the usage of the -w flag is. The -w flag tells kubectl to watch for changes after listing/getting the requested pod objects

Then I opened another terminal window and deleted a pod with the kubectl delete pod nginx-deployment-6f9664446b-7txt4 command

And then going back into the same terminal where I used the kubectl get pods -w command, I saw the following

See what happened here? The pod got recreated! That is exactly what we expected.

Section IV: Information Regarding Pods

I then wanted to learn a bit more about what happened on the newly-created pod, so I used the kubectl logs nginx-deployment-6f9664446b-nshnj to see what happened with the pod that I deleted

Then I wanted to learn a bit more about the nginx-deployment and nginx-service resources that I created. The kubectl help <command_name> is the Kubernetes equivalent of the Linux man pages and was useful for learning what the kubectl describe command does.

Basically, the kubectl describe command prints a detailed description of a given resource. The first parameter is the resource’s type and the second parameter is the resource’s name. I used the kubectl describe deployment nginx-deployment and the kubectl describe service nginx-service commands appropriately

Remember why I created two different Kubernetes objects: nginx-deployment manages the actual application (creating and maintaining 3 Nginx pods, handling restarts and scaling), while nginx-service handles the networking layer (exposing those pods to external traffic via NodePort so you can actually access the web server from your browser). This separation of concerns is a core Kubernetes design pattern — deployments manage the “what runs” and services manage the “how to reach it.”

OK. Now for some additional fun. First, I used the kubectl get all command to display all the main Kubernetes resources in the current namespace in one view, including pods, services, deployments, and replica sets.It’s the go-to command for getting a snapshot of what’s running in your cluster without having to run separate kubectl get pods, kubectl get services, etc. commands individually.

Remember how I in the previous docker homelab was able to execute commands inside a container using the docker container exec command? Kubernetes has a similar one for pods!

I used the kubectl exec -it pod/nginx-deployment-6f9664446b-8ft5k — /bin/bash command to open up a shell on the pod and to execute some commands

That was nice. Then I wanted to get some more metrics information regarding the Kubernetes pods.

To do that, I enabled the metrics server in minikube using the minikube addons enable metrics-server command and used the kubectl top pods and kubectl top nodes commands to see information regarding resource usage

Pretty informative.

Section V: AWS EKS Setup

The time had come to work with Amazon Elastic Kubernetes Service (EKS). I already had the AWS CLI installed on my homelab workstation from a past homelab. I quickly installed eksctl, a simple CLI tool for creating clusters on EKS — Amazon’s new managed Kubernetes service for EC2. I fortunately had AWS credentials already configured on my homelab workstation from a past exercise, so I didn’t need to configure those with the aws configure command.

With everything set up, I got to work on creating an EKS cluster in the free tier with the following command: eksctl create cluster — name my-eks-cluster — region us-east-1 — nodegroup-name standard-workers — node-type t3.micro — nodes 2 — nodes-min 1 — nodes-max 2 — managed

Notice that the command has the –managed flag. It tells eksctl to create managed node groups, which means that it is AWS which handles updates, patches, scaling, and replacement of unhealthy nodes. Without this flag, cloud engineers have self-managed node groups where they are responsible for manually updating the AMI, handling node upgrades, and managing the EC2 instances — a total hassle. Managed node groups are generally best practice because they significantly reduce operational overhead while maintaining the desired Kubernetes functionality.

The AWS management website also confirms the AWS EKS cluster being made:

Then I used the kubectl config current-context command to confirm the cluster’s state. Notice that a Kubernetes context is a configuration that tells kubectl which cluster to talk to, which user credentials to use, and which namespace to work in. It basically is your “connection profile” that lets you easily switch between different clusters like Minikube, EKS, and AKS.

Then I used the kubectl get nodes to list all the nodes in the Kubernetes cluster and to get a summary of their status and information

Now for the wrinkle

Creating the EKS cluster only provisions the infrastructure (the Kubernetes control plane and worker nodes) and it’s an empty cluster with no applications running yet. That’s why I needed to then run kubectl apply to actually deploy the Nginx application into this new EKS cluster, just like I previously did with Minikube. This is because the kubectl apply command is the main tool for declaratively managing Kubernetes resources — and it is used to create or update resources in a cluster to match a desired state defined in configuration files (YAML or JSON). The YAML files tell Kubernetes what containers to run and how to expose them, which doesn’t happen automatically when the cluster is created.

This is because Kubernetes follows a declarative model where the cluster infrastructure (nodes, networking, control plane) is completely separate from the applications that run on it.

When you create an EKS cluster, AWS provisions the “stage” (the Kubernetes platform), but you as the developer/operator must explicitly tell Kubernetes what “actors” (applications) to put on that stage and how they should behave. This separation gives full control over what runs where and allows for applications to be managed independently of infrastructure.

We got the theory out of the way, and now it was time to apply the same manifests. First, I did that with the kubectl apply -f .\nginx-service.yaml command

And the kubectl apply -f nginx-deployment.yaml command

Now to see what services were up and running:

Wonderful. Everything was properly working in AWS.

Section VI: Azure AKS Setup

I fortunately had Azure CLI installed on my homelab workstation from a past homelab and a prior Azure subscription. I quickly logged in using the az login command and selected my subscription.

Now it was time to do the same thing in Azure that I previously did on AWS. After drumrolling hard for 5 seconds on my desk, I got straight to work.

First, I created a resource group using the az group create — name myAKSResourceGroup — location eastus2 command. I needed to create a resource group first because Azure requires every resource to belong to an organizational container that logically groups all related components (the AKS cluster, VMs, disks, networking). Having such a resource group makes it easy to manage costs and delete everything together (such as when the lab is finished — for example).

And then I created an Azure Kubernetes Service cluster using the following command: az aks create — resource-group myAKSResourceGroup — name myAKSCluster — node-count 2 — node-vm-size Standard_B2s — enable-addons monitoring — generate-ssh-keys

The — node-vm-size Standard_B2s flag was used to specify smaller, budget-friendly VMs that kept costs low while still providing enough resources for my testing and learning needs. I included — enable-addons monitoring to get Azure’s built-in monitoring dashboards that gave visibility into my cluster’s health and performance metrics — without having to manually configure additional monitoring tools.

I then needed to download the access credentials for my Azure Kubernetes Service cluster. This is because creating the AKS cluster only operated in Azure. Downloading the credentials and automatically adding them to my computer’s local Kubernetes configuration file ensured that kubectl would know how to connect to and authenticate with my new AKS cluster.

To do this, I used the az aks get-credentials — resource-group myAKSResourceGroup — name myAKSCluster

You may wonder why I didn’t take such a step with the AWS Kubernetes cluster I made. The answer is that the eksctl did the same thing and downloaded the credentials automatically for me as part of the cluster creation process (per this and this page of AWS documentation). It was just that Azure required an extra manual step for this to be done.

Now to see what services were up and running, I used the kubectl config current-context and kubectl get nodes command:

Looks about right. Now it was time to deploy the nginx-deployment.yaml and the nginx-service.yaml files into Azure. I did this using the kubectl apply -f nginx-deployment.yaml and kubectl apply -f nginx-service.yaml commands:

Remember that the nginx-deployment.yaml defined how to run the application (managing the Nginx pod replicas, handling automatic restarts, and maintaining desired state), while nginx-service.yaml defined how to access the application over the network (by creating the NodePort networking layer that exposed those pods to external traffic). This is deliberately done per Kubernetes’ design philosophy of decoupling application management from networking concerns.

The kubectl get all command then gave information regarding the nature of the deployment

Looks nice.

Section VII: Multi-Cloud Management

I then used the kubectl config get-contexts command to get a sense of the multiple contexts that I have running on Kubernetes across Minikube, Azure’s AKS, and AWS’s EKS

I then used the kubectl config use-context minikube command to switch over to the minikube context

And then got some information regarding what was going on in it using the kubectl get nodes , kubectl config current-context , and kubectl cluster-info commands:

Then I did the same for the Azure AKS service by switching into the myAKSCluster context with the kubectl config use-context myAKSCluster command and using the same commands that I previously used to get some more information regarding the context

And I did the same for the AWS’ EKS service by switching into the appropriate context with the kubectl config use-context Administrator@my-eks-cluster.us-east-1.eksctl.io command — and using the same commands that I previously used to get some more information regarding the context

Next up, to do some scripting to show how this could be done in a more automated way! Since I was working with Kubernetes in the PowerShell command line windows, I created a new multicloud-deployment.ps1 script file

And then executed it

Looks nice.

So let’s rehash. I kept Minikube running as my local development/testing environment alongside the cloud cluster. Minikube lets cloud engineers quickly test changes, experiment without incurred cloud-computing costs, and compare how the same Nginx deployment behaves across different platforms (local vs AWS vs Azure). Now you see that Kubernetes is portable and one can manage multiple clusters simultaneously by switching contexts with kubectl config use-context and with nice scripts — which is a critical skill for real-world multi-cloud or hybrid-cloud environments.

Section VIII: Python Scripting

And now it’s time for some non-monty Python! (See what I did there?)

I used the python -m pip install kubernetes command to install the Python Kubernetes library. It is an official Python SDK that allows you to programmatically interact with any Kubernetes cluster using Python code instead of kubectl commands, enabling automation of cluster management tasks like listing pods, creating deployments, or monitoring resources through standard Python scripts.

Now let’s type some if else. I made a list_pods.py Python file and then executed the script

And looking at the output, we see that the desired Cluster, Namespace, Pod Name, Status, IP, Node, Created data was properly outputted

Section IX: Cleanups

I did all the stuff that I needed to do. Now it was time to delete the Kubernetes clusters from execution. This is especially important to do in the cloud so that we don’t incur charges from leftover cluster instances running.

First, on my local machine, I used the minikube delete command to halt the cluster on my local machine

Next up, for AWS! I used the eksctl delete cluster — name my-eks-cluster — region us-east-1 command to do this. For Azure, I used the az aks delete — resource-group myAKSResourceGroup — name myAKSCluster — yes — no-wait and az group delete — name myAKSResourceGroup — yes — no-wait commands to delete the Kubernetes clusters in those clouds as well.

This was quite the journey. I’ll see you in the next round!

Think of Docker containers like shipping containers for software — they package your application and everything it needs into a standardized box that can run anywhere, whether on your laptop, a server, or the cloud. This solves the age-old problem where code works perfectly on a developer’s machine but breaks in production, and makes deploying and scaling applications dramatically simpler and faster.

Section I: Initial Preparation

For this round, I picked up with the initially-made control and managed note Ubuntu virtual machines from Homelab #4 where I learned and practiced Ansible. Before digging too deep into the new project, remember that we had the website deployed on the managed node, and could confirm that it was up and running on the control node with the curl http://192.168.56.214 command?

I wanted to check in Ansible whether my control node had the nginx server up and running, to take it down if such were the case (to minimize the risk of the website it is running accidentally interfering with the container we want to get deployed), and then to proceed onwards with the actual homelab.

To do this, this is where the notion of an Ansible Ad-Hoc command comes in. To put it simply, it is a one-liner that administrators can use to make tweaks or check statuses in situations that are one-offs or so rare that they do not require a formal playbook to be written, tested, and used. More information regarding Ansible Ad-Hoc commands can be found on the Introduction to Ad Hoc Commands documentation page.

Here, I wanted to check the status, so I quickly used an ad-hoc command in Ansible to check

It was up and running. Time for another ad-hoc command to stop it:

OK now it’s time for actual docker and containerization. Remember how in a previous homelab I spent hours writing Terraform code from scratch, only to be told by the pros on a tech slack chat that Terraform is treated like CSS in the industry: find good stuff, copy, paste, and tweak. I wanted to get through this homelab as efficiently as possible, so I asked them whether Ansible was treated the same way, and if so, where they go to get good Ansible snippets to work with.

They responded that Ansible is also treated that way, and referred me primarily to Ansible Galaxy, Jeff Geerling’s Ansible content (which is very popular and also hosted on Ansible Galaxy), and Cloud Posse’s GitHub page. Ansible Galaxy is the official community hub (galaxy.ansible.com) where people publish and download ready-made Ansible automation content so you don’t have to write everything from scratch. These include roles and collections:

• A role is a standalone, reusable package that does one specific job really well (e.g., install and configure Nginx, Docker, or PostgreSQL) — think of it as a single Lego brick you drop into your playbooks.
• A collection is the newer, more powerful format: a namespaced bundle that can contain multiple roles (those “lego bricks”), custom modules, plugins, and more, all versioned together (e.g., community.docker or ansible.windows).

They exist so homelabbers and pros alike can set up complex servers in minutes instead of hours, keep configurations consistent, and share or reuse the same battle-tested code across projects.

Now that we prepared appropriately both on the VM and in terms of learning a bit more about Ansible management theory, let’s move into the actual implementation steps.

Section II: Preparing By Installing Appropriate Roles & Collections

On my control VM, I first used the mkdir ~/docker-homelab command. Then I used the ansible-galaxy role install geerlingguy.docker command to install the geerlingguy.docker role. Per the website, it is a role — i.e. a pre-prepared “lego” of Ansible-specified action — for installing Docker on a Linux system. Digging into the GitHub repository confirms that this is a pretty robust role, which works very well across different Linux distribution types.

The documentation also provided a friendly reminder that I needed to install the geerlingguy.pip role since

Many users of this role wish to also use Ansible to then build Docker images and manage Docker containers on the server where Docker is installed. In this case, you can easily add in the docker Python library using the geerlingguy.pip role

I thus installed the geerlingguy.pip role with the ansible-galaxy role install geerlingguy.pip command. This role installs the pip package manager on Linux systems:

Wonderful. I also took advantage of the opportunity to verify with the ansible-galaxy collection list | less that the community.docker collection for working with docker is installed. The usage of the “| less” in the command was to ensure that I could see all the lines outputted to the screen — one screen at a time.

Looks nice. Let’s get moving. I started by copying over the same inventory file from the ansible-homelab folder to the docker-homelab folder on the control VM. This is because this homelab exercise uses the same setup and managed node as the previously-completed ansible Homelab #4, so the same inventory file should suffice.

I then created a files and playbooks subfolders with the mkdir command:

Section III: Installing Docker With Ansible

Time to install Docker on the managed VM. To do this, I went into the playbooks folder on the control VM and used the nano docker-setup.yml command to kick things off. I then got to work and wrote up the YAML playbook appropriately. I wrote a lot of explanatory comments for each line as I went through this learning process, so for this screenshot, I used the grep -v ‘#’ docker-setup.yml command to show only the non-commented lines which make an impact when executed:

I then troubleshot the playbook, dealt with some syntax errors, indentation issues, and then executed the playbook successfully.

And confirmed on the managed vm that the two were installed and that docker is up and running:

Excellent.

Section IV: Node.JS Application With Docker

So I deployed Docker on the Ubuntu VM with ansible. It works. Afterwards, I switched to working hands-on with it on the server because that was the real learning goal for me right now. Docker is a tool that lets you package your app (code + everything it needs to run) into a tiny, portable box called a container.

This container works exactly the same on your laptop, a friend’s computer, or any cloud server — no more “it works on my machine” problems.

Now I needed to make a sample Node.JS application. On the control node, I created a folder for this homelab and then put inside it another folder which I would put the sample Node.JS application.

I then went into the folder and created a new server.js with a sample Node.JS application, which I found on the Node.JS website. The code looks like this:

Wonderful. Then I needed to make a package.json file, which I did by using the npm init command.

Then I used npm install to create a package-lock.json file.

package.json is the manifest file that declares your Node.js project’s metadata, dependencies, scripts, and configuration; package-lock.json is an auto-generated file that locks the exact versions of every installed package (including nested dependencies) to ensure consistent, reproducible installs across machines. Together, they make your project portable and prevent “it works on my machine” issues.

I then created a Docker file per a sample tutorial from Docker’s website.

A Dockerfile is a simple text file that contains a list of instructions telling Docker how to build your app into a ready-to-run container image — no manual setup needed, and it works the same way on your laptop, a server, or in the cloud.

Then I made a .dockerignore file, which is a file placed at the root folder of a Docker project which specifies which files and directories to exclude from the build context.

Excellent. Then I used the command sudo docker build -t my-node-app . to build the container

Then I used the sudo docker run -d -p 3000:3000 –name my-node-app-container my-node-app to launch the application

There were some troubleshooting fixings that I had to do to get to this point, including fixing a line or two in the dockerfile, but bottom-line, I got it working!

Section V: Flask Application With Docker

Time to make a Flask application into a Docker container. To put it simply, Flask is a popular, lightweight web application development framework in Python. I started by making a flask_app folder, installing Flask using the pip install Flask command, and then updating the PATH variable to ensure that the flask folder installed is seen.

Then I created hello.py and filled it out per the text of a nice Medium tutorial I found.

Then I created a requirements.txt file. It is a file which Python projects use to list its dependencies and versions. For this case, I kept things really simple with two lines:

Then I built the dockerfile, the container, and launched execution:

During my building process I noticed an error. I put down app.py into the dockerfile originally, which led to a container execution error. I caught the mistake and rewrote the line in the dockerfile’s command to run the Flask application at the hello.py file.

Section VI: Working With Docker Networking

The next step is to try out Docker Networking. This allows containers to communicate with each other and the outside world — including non-Docker network services. There is a good website providing an introduction to Docker networking but the concepts are very similar to managing networking in VMs. For example, a few network types include:

• Bridge (default) — Containers on same host can communicate
• Host — Container uses host’s network directly (no isolation)
• None — No networking
• Custom Bridge — Like bridge, but with more control

Let’s start. First, I wanted to see the default networks with the sudo docker network ls command:

We got the following default networks:

• bridge: The default network driver, which creates a private internal network for containers to communicate with each other and the host.
• host: This driver removes network isolation between the container and the host, allowing the container to use the host’s network stack directly.
• none: This driver provides complete network isolation, meaning the container has no network interfaces.

After that, I used the docker network inspect bridge command to learn more regarding the default bridge network

Per the screenshot, we see that the my-flask-app-container is attached to the bridge network for communicating with the VM machine that the container is running on.

Next up was to create a custom bridge network. Google tells me that there are multiple uses for such. The main one in my opinion though is that custom networks provide better isolation than the default bridge network, preventing containers from different projects or applications from interfering with each other’s network traffic. I used the sudo docker network create –driver bridge –subnet 172.20.0.0/16 –gateway 172.20.0.1 homelab-network

Then I verified and inspected the newly created network with the sudo docker network ls and sudo docker network inspect homelab-network commands

Then I launched the flask app connected to the custom homelab-network using the sudo docker run -d –name flask-networked –network homelab-network -p 5001:5000 –restart unless-stopped my-flask-app command. Notice the usage of the -p 5001:5000 command, which maps the host port #5001 to the container’s port #5000. I then confirmed Flask is up and running:

Then I used the sudo docker run -d –name nginx-proxy –network homelab-network -p 8080:80 –restart unless-stopped nginx:alpine command to run Nginx as a reverse proxy.Then I used the sudo docker inspect nginx-proxy | head -n 25 command to get information regarding its state:

Now I needed to configure the Nginx container to proxy requests to flaskcat. Per the Nginx documentation, I needed to create a nginx.conf file and then pass it on into the Nginx container to get it working properly. To explain simply, a nginx.conf file is used to specify how NGINX should handle requests for different locations within a server.

I quickly made it on the VM using the nano nginx.conf command. You can see what it looks like here.

Next up, to get it into the actual container. The following documentation articles here, here, and here indicate that configuring the Nginx server is an art with multiple layers of blocks and contexts. In this case, we just wanted to affect the server context, simply… so I dug around on the container to see its internal file structure. The docker container exec command allows one to execute commands within the container — including commands to see information regarding a file’s structure. I checked a few things and found the container’s original value of the file I need to replace the server context with — using the sudo docker exec nginx-proxy cat /etc/nginx/conf.d/default.conf command:

Let’s get the actual file that I made into the container. This was done using the sudo docker cp nginx.conf nginx-proxy:/etc/nginx/conf.d/default.conf command

Then I used the sudo docker restart nginx-proxy command to restart the container — and then used the curl http://localhost:8080 command to see if I could access the Flask app through Nginx.

It worked!

This is not all. Let’s demonstrate network isolation for Docker containers. What I did was to create a second network using the sudo docker network create –driver bridge –subnet 172.21.0.0/16 isolated-network command

And then ran the nginx container on it using the sudo docker run -d –name isolated-app –network isolated-network nginx:alpine command

Wonderful. Now for the demonstration. I used the docker in-container execution capabilities to do this test. I originally tried to use the ping and then the curl commands from inside the flask-networked container to reach out to the isolated-app but they both were not found as executables inside the container. Research indicated that the nginx — and other official Docker container images — are deliberately lean and exclude “non-essential” tools and configurations both for size and security reasons. Finally, I came across a solution to do this demonstration: using the sudo docker exec flask-networked python -c “import socket; socket.gethostbyname(‘isolated-app’)” command.

As expected, we got an error in name resolution!

Last step. I connected the flask-networked container to the isolated-network using the sudo docker network connect isolated-network flask-networked command and then used the sudo docker exec flask-networked python -c “import socket; print(socket.gethostbyname(‘isolated-app’));” command to confirm connectivity. Notice this time it worked!

Time to do some cleanups! First I stopped the containers and deleted them

And then after that, I used the sudo docker network rm command to tear down the networks appropriately. Then I confirmed that everything is restored to the original clean slate using the sudo docker network ls and sudo docker ps -a commands.

We’ve been on quite a networking journey. Let’s briefly rehash a portion of the networking concepts learned before we proceed into the next round. The following list is not all-inclusive:

• DNS Resolution: Containers on custom networks can reach each other by name, with Docker providing automatic DNS resolution. For example flask-networked → 172.20.0.2
  Per Docker documentation, default bridge networks do not have automatic DNS resolution. Containers on default bridge networks can only access each other through IP addresses.
• Network Isolation: Containers on different networks can’t communicate.
  Also, user-defined bridge networks provide better isolation by creating separate, scoped network segments for groups of containers and preventing unrelated services from communicating by default. This reduces security risks, improves control, and prevents network congestion. This is unlike the default bridge network — which connects all containers.
• IP Ranges: Custom networks also have the bonus of allowing for custom IP ranges. Meanwhile the default Docker bridge network automatically assigns IP addresses from a pre-defined range (typically 172.17.0.0/16).
• Multi-Network Containers: A container can be on multiple networks, just like a regular physical host can be connected to multiple ethernet networks via cable!
• Port Mapping: Port mapping (or port forwarding) in Docker exposes a container’s internal port to a port on the host machine, allowing outside traffic to reach services running inside the container. It creates a bridge between the container’s private network and the host’s public-facing network.

And finally, for some of the reasons mentioned — using custom networks in Docker is generally considered to be best practice!

Section VI: Pushing Container Image to Docker Hub

In this section, I learned about Docker Hub. Docker Hub is the world’s largest container registry — a public repository where developers share pre-built images like nginx, python, or postgres, letting you pull production-ready software in seconds instead of spending hours installing and configuring from scratch. It’s like GitHub for containers: you can use official images maintained by companies, share your own images publicly, or store private images for your team.

I created an account in Docker Hub, and then logged into it on my machine.

Then I used the sudo docker build -t my-flask-app . command to make the container image.

And then after that, used the sudo docker tag my-flask-app nathanpavl/my-flask-app:v1.0 to tag the image. Think of it as labeling the container image as being of such and such version or release information. Then I pushed it to docker using the sudo docker push nathanpavl/my-flask-app:v1.0 command

Opening the Hub page on my docker website, it’s confirmed: I pushed the container to Docker successfully!

Section VII: Docker Image Optimization & Multi-Stage Builds

When we build Docker containers, we don’t start from zeroes and ones — we start from a base container image of… something… that we then use as a starting point to build our final application container image from. First you get a python image, then you do some configurations, then you add package this and package that. Think of it as a multi-layer cake that you will build on top of a provided cake base board.

There is a nice StackOverflow post that I found with a fantastic (its main) explanation by a user by the name of David Castillo but basically, every RUN, COPY, or FROM in your Dockerfile creates a new layer. Each layer is extra memory and consumption — and is cached separately. If something changes — only the relevant layer gets updated. Docker also has a nice piece on its website as well explaining this. Bottom-line — the more layers a container has, the larger the final container image memory size will be.

You can for example see all the layers of my-flask-app using the sudo docker history my-flask-app command.

The layering mechanism is definitely useful but needs to be thoughtfully considered. Containers managed unwisely will result in unnecessary packages that bloat the image, security risks (more packages = more vulns to handle & manage), slower build times, extended download and launch times. It… doesn’t take the cake. (See what I did there?)

Time to demonstrate multi-stage docker building, so that we can see how this works. Think of multi-docker stage building as being in a kitchen, you whip the final product up on a messy countertop with the ingredients, and then pass the desired outcome to the final customer — without any of the leftover ingredients and soiled utensils.

I created a new docker file called Dockerfile.multistage. The following screenshot’s comments explain how the multi-stage process works

Then I built it using the sudo docker build -f Dockerfile.multistage -t my-slim-flask-app . command and got it running using the sudo docker run -d -p 5000:5000 –name my-slim-flask-app-container my-slim-flask-app command

And confirmed it works:

The sudo docker history my-slim-flask-app command revealed that it has 16 layers instead of the original container’s 21 layers. This is precisely what I was going for

I now reached an interesting wrinkle: the sudo docker images <container_name>:latest –format “Size: {{ .Size }}” command showed… that both had a size of 195 MB

This is actually expected and illustrates an important point. Multi-stage builds help with size savings when you (1) install build tools like the gcc compiler, make, etc. (2) compile code or (3) download large dependencies temporarily. Remember the cake making analogy? These are intermediary ingredients for making the final executing program that we do not want in the final container. Since Flask and Python are pure script packages with no compilation needed, there was nothing to leave behind!

Let’s do one more illustrative example to wrap up this multi-stage build section and the overall homelab. I created a new sub-directory under the homelab_9 folder titled bulk_multistage_build and got to work. In this case we will be building another Flask app, but we will have some build tools that we want to install. This is actually realistic for the case of a scripted Python application since some — yes — Python libraries like NumPy are heavily C-based and need those compiler tools for making the final image. I started by making the requirements.txt file:

Then I created the Dockerfile.heavy file, where I build the cake in one fell swoop:

Notice that here we install build tools that could be used for compiling NumPy or another such C-based library. Then we create a Dockerfile.lighter file where the cake is made in steps:

Then I quickly made the app.py file

I then quickly built the containers for the heavy and lighter docker files

And checked the history of the two:

And they were up and running!

Notice that the heavier container has 20 layers and the lighter one has only 16 — there is a difference. And the size of the overall containers matters too: 1.75 GB versus 274 MB. That is an approximately 84% reduction — pretty sizable!

So now you see why multi-stage builds are important!

This was quite the journey, and a very involved homelab. I learned a lot about how containers and Docker work, hopefully it also taught you something too. Thank you for reading — and I will see you in the next one!