<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andrei Kvapil on Medium]]></title>
        <description><![CDATA[Stories by Andrei Kvapil on Medium]]></description>
        <link>https://medium.com/@kvaps?source=rss-d8a829bb74d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_-AIrClc3NGXSoQuRfIucg.jpeg</url>
            <title>Stories by Andrei Kvapil on Medium</title>
            <link>https://medium.com/@kvaps?source=rss-d8a829bb74d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:09:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kvaps/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What dozens of AI agents taught me: how I wrote the Blockstor storage system as an experiment]]></title>
            <link>https://blog.aenix.io/what-dozens-of-ai-agents-taught-me-how-i-wrote-the-blockstor-storage-system-as-an-experiment-921f7d3a1137?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/921f7d3a1137</guid>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 09:02:46 GMT</pubDate>
            <atom:updated>2026-08-10T09:02:46.356Z</atom:updated>
            <content:encoded><![CDATA[<p>A couple of months ago I decided to run an experiment: build a clean-room implementation of LINSTOR from scratch, working only from its references and public API types. It started as a Friday joke. I wanted to spend as little time on it as possible, leave it running in the background, and see where it went. The point was to find out how far a modern model can get on its own, with no human in the loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ep6GUdvlFLIwsXN-U0ovnQ.png" /></figure><p>Spoiler: full autonomy didn’t happen, and I ended up wrestling with the project quite a bit. But the process pulled me in completely, and the end result beat every expectation I had.</p><blockquote>People in the community kept asking how it actually went. Fair question — building this taught me a great deal about driving models effectively, and it turned up a pile of working methods and patterns that have since made me much more productive in everyday work too.</blockquote><h3>What Blockstor is</h3><p>I called the project Blockstor. It’s a block device orchestrator. Roughly: you request a replicated volume of the size you need, and that volume gets created on several nodes in ZFS or LVM and set up for replication with DRBD — a smart, network-aware take on RAID 1. The system supports snapshots, replica reallocation, resize, automatic failover, and more.</p><p>Blockstor is not a storage system designed and written on a blank page. I built the experiment on LINSTOR — a mature distributed block storage manager that I’ve run in production myself for years.</p><p>LINSTOR was close to an ideal storage system for me, because its API types are Kubernetes-like. Its backend logic, though, is organized in a rather peculiar way. The main problem, as I see it, is the request-based model: for most API calls it goes out to the nodes in real time and polls their current state to build a response. To my mind that’s an unacceptable pattern for a distributed system, because it doesn’t hold up at scale. The absence of a reconciliation loop also makes automatic recovery hard. I’m not going to second-guess the developers’ decisions — I’m sure they had their reasons. But given how committed I am to Kubernetes patterns, I asked myself how I — an experienced architect and developer — would write a system like this in Kubernetes-native logic.</p><p>Rewrites from one language to another aren’t rare, by the way. That, for example, is how the rusternetes project came about: Kubernetes being rewritten from Go to Rust. But how is that even possible? Kubernetes has an enormous codebase with an enormous number of person-hours in it. How do you convince yourself that the resulting “AI slop” works correctly? And can you rely on it at all, let alone run it in production?</p><h3>TDD</h3><p>The answer lies in how Kubernetes itself is developed in the open. Free, community-driven projects have a body of established practice, and contributors do their best to follow it.</p><p>All kinds of people contribute to projects like these, at every skill level. That makes tests critical: tests are what keep functionality someone contributed from breaking later. Say you land a new feature, it passes every test, and it goes into the project. Later someone lands another feature. If a test for your feature fails while theirs is being checked, that means the new functionality has a problem that breaks yours — and the system automatically refuses to let the change through. Over time, tests in public projects have become so important that they’re now valued more highly than the code itself. Tests like these are exactly what makes rewriting X in Y possible.</p><p>The rusternetes project claims:</p><blockquote><em>Actively conformance-tested against the official Kubernetes e2e test suite — currently passing 94% of conformance tests (415/441) across 160 rounds of testing.</em></blockquote><p>So passing the tests is the primary yardstick for whether a project counts as a Kubernetes replacement.</p><blockquote><em>When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.</em></blockquote><blockquote><em>— The famous </em><a href="https://ru.wikipedia.org/wiki/%D0%A3%D1%82%D0%B8%D0%BD%D1%8B%D0%B9_%D1%82%D0%B5%D1%81%D1%82"><em>duck test</em></a></blockquote><h3>Hooks and setup</h3><p>For the same reason, the first thing I did was force the model into test-driven development (TDD), where tests are written before the implementation. Tests let you pin down the conformance requirements you need. They also give me confidence that new functionality arriving during development won’t break what already works.</p><p>On a colleague’s advice I wired in golangci-lint right away and made it mandatory for the model to run all code through it. <a href="https://github.com/lexfrei">@lexfrei</a> — the colleague in question — argues that this step saves a substantial number of tokens.</p><h3>The exit gate and a deterministic result</h3><p>This is probably the first and most important lesson: you can tell a model “keep writing until the tests pass”, and that’s a good condition on the artifact it produces. That’s exactly the method I used to build this project. With one catch: I had no tests, because LINSTOR doesn’t publish a test suite for its own functionality.</p><p>All I had to go on was years of running LINSTOR, dozens of my own talks and articles, and the Apache 2.0 code of projects in the LINSTOR ecosystem — golinstor, the Container Storage Interface (CSI) driver, Piraeus-operator, and their API contracts. LINSTOR itself is licensed under GPL, which rules out using its code to build an Apache 2.0 project on top of it. That was the central challenge, and it’s precisely why I went the clean-room route.</p><p>So my main job became defining those exit gates. Get them right and I don’t have to police the whole resulting codebase: if the code satisfies the conditions I set myself, that is the evidence the job is done.</p><h3>The API as a contract</h3><p>I like the type definitions LINSTOR uses in its API. On top of that, both the official CSI driver and Piraeus-operator work with those types, and I had no wish to rewrite either of them.</p><p>From experience, designing APIs is one of the most painful jobs in this profession. You can always change backend logic. Ship one breaking change to the API and everything falls apart on the client side. So an API should change rarely, in exceptional cases only — and the risk of getting it wrong at the very start is high. That’s why I decided to use exactly the types the original project uses, and map them onto Kubernetes-first CustomResourceDefinition (CRD) types.</p><p>While building Cozystack and selling solutions based on it, we’ve run into the same question from prospects more than once: what’s the story with your API? What if we put you into production, build a solution around you, and then your API changes and we have to rewrite our half of the project?</p><p>As a reminder: LINSTOR’s code is published under GPLv3, and I planned to publish Blockstor under Apache 2.0. Reusing the original code was off the table. So the first task was to find compatible contracts in the Kubernetes tooling around it: the <a href="https://github.com/LINBIT/golinstor">golinstor</a> library, linstor-csi, and Piraeus-operator, all distributed under Apache 2.0, plus the official LINSTOR documentation, licensed under CC BY-SA.</p><p>That gave me my first contract: the API must be compatible with the LINSTOR Go library, linstor-csi, and Piraeus-operator.</p><h3>The test environment</h3><p>The model needed somewhere to work and somewhere to test what it produced. For the test environment I picked a beefy bare-metal node and a generated test suite that brought up a Kubernetes cluster on Talos inside virtual machines (VMs) and deployed Blockstor into the cluster. That choice was deliberate: Blockstor needs the DRBD module, and DRBD has a habit of hanging when it’s configured wrong, so I needed a fast way to stand up and recreate a broken environment. Blockstor’s architecture also stores configuration as Kubernetes CRDs, so using Talos closed the question of bootstrapping Kubernetes itself.</p><h3>The first result</h3><p>When the first proof of concept (PoC) was ready, the model had built me a working prototype, guided only by the sources above and its own dataset. For the interaction model, though, it had implemented the very same request-based model as the original LINSTOR — presumably picked up from the project’s documentation. And it already worked! I could talk to the API using the official CLI, though there were hundreds of bugs and gaps.</p><p>To fix that, I asked the model to freeze the API contracts as tests and rewrite the logic on controller-runtime, the way I needed it. With a clear, deterministic goal, the architecture that came back began to resemble what I actually wanted: fully asynchronous, with the API translator as a separate pluggable module.</p><p>Later I had the model implement a set of end-to-end (e2e) tests that requested volumes through the official CSI plugin and the Kubernetes kubernetes-csi/csi-test framework. In other words, the model kept working until Blockstor provisioned volumes and snapshots through standard Kubernetes abstractions. After a while, I had a working prototype. The system was still a long way from stable, though, which left me with a question: how do you reach the stability I needed when there are no tests to start from?</p><p>This is where all my material went in — my articles on debugging LINSTOR, my talks, my Claude Code debugging skills, our plunger scripts integrated into Cozystack, and bugs reported on GitHub. I made the model study all of it and compile a set of issues that had to be tested and that our system had to satisfy. Out came a huge Markdown file. I reviewed it and sent the model off to test and fix every problem it had caught, on the test rig. There were many such iterations, and each one took a colossal amount of time. Most of the time the model would bring up the environment, run tests, fix errors, re-provision the environment — and all of that took time.</p><p>Eventually I started asking how to speed the process up.</p><h3>Speeding up development</h3><p>That’s where the idea of parallelizing agents came from. Some problems could be closed out at the unit-test level. Others could only be verified on a real environment. Running the same set of bugs through again and again, I arrived at this method:</p><ol><li>I ask one agent to gather the references and put together a plan of sorts for the bugs.</li><li>Once I have the huge document, I ask the model to dispatch a batch of agents, each working a specific bug.</li><li>In the code, every agent gets its own isolated environment and its own worktree, where it submits its fixes.</li><li>Each agent’s job is to study the problem, bring up an environment, prepare a fix and tests, and hand all of it back to the main agent.</li><li>The main agent pulls the work from every agent into the main tree — and that repeats for several iterations.</li><li>In the end, the number of agents running at once reached 60: 30 at the unit-test level, 30 on e2e in a concrete environment.</li></ol><p>After a while the project started to look like something you could actually use. Plenty of things still weren’t stable.</p><h3>The marathon continues</h3><p>While nursing this whole zoo along, I was already doing integration testing myself and checking the results by hand. I’d ask Claude for access to an environment, drive the linstor CLI manually, and try to reproduce the bugs, of which there were still plenty.</p><p>At that point I had to stop building large blocks and start digging into far more meticulous testing of the user journey. The bulk of the problems went away once I made the agents implement tests using the official CLI and laid out the user journey for Day-2 operations from the official LINSTOR documentation. In some places, though, the model started spinning its wheels: over all that time it never got every test passing reliably, and I had to step in personally. Drawing on what I know, I asked it to walk me through each problem, then put a set of leading questions to it about the architecture.</p><h3>The fine-tuning stage</h3><p>A lot of my questions came down to the asynchronous nature of the controllers. DRBD is by nature fussy about when, how, and at what point configuration gets applied, so it was critical to build a stable state machine — one that would let the reconciliation loop through to a specific action only when the conditions were met.</p><p>The main problem here was snapshot logic. To create a snapshot, LINSTOR first freezes I/O at the DRBD level, then sends the command simultaneously to the several nodes holding the backing device in ZFS. Once the operation succeeds, LINSTOR unfreezes DRBD. All of it has to happen instantly, and on error the state has to roll back automatically so that the container or VM never blocks while working with its data.</p><p>The second problem was how to skip the initial sync of a new replica. I knew the observable behavior from years of operating LINSTOR: the first replica records its current generation identifier, and later replicas take that as their starting value and sync from there.</p><p>First I tried to reconstruct the exact command sequence from observed behavior alone, running operations against a live controller and capturing the commands. But the mechanics aren’t documented publicly, and observation on its own didn’t add up. So I fell back on the clean-room technique. One agent — the “dirty room” — reconstructed a functional specification from the original sources: which commands run, and under what conditions. It recorded those functional facts and nothing else, carrying over no code and no expression of it. The second agent — the “clean room” — had no access to anyone else’s sources in the first place, and implemented the approach from scratch, strictly from the specification.</p><p>One more problem was consistent node-id allocation — every DRBD replica needs a unique number in the cluster, from 1 to 8 — along with allocating TCP ports for replication, which in the current implementation is no longer tied to DRBD and is handed out from a per-node pool instead. This is exactly where the state machine above paid off.</p><h3>Building the CI system</h3><p>By now it was clear the experiment was reaching its final stage, and I started thinking about the project’s future. To finish it and get to production, we needed a serious continuous integration (CI) system that would guarantee no untested code in the codebase. Given the volume of tests, we couldn’t let a run stretch over several hours, so they had to be parallelized the same way.</p><p>And we built one: for every pull request it ran a big pile of tests across six or seven runners in parallel and returned “ok” or “not ok”.</p><p>I worked in that mode for several more rounds, until CI was genuinely green and stable. Then we switched from local worktrees to pull requests on GitHub.</p><h3>The final stage</h3><p>Data loss is not acceptable, so before declaring the project finished I had to check thoroughly that Blockstor behaves correctly and reliably on that front. Reading and reviewing all the generated code was beyond both my energy and my capacity. On the other hand, Blockstor — like LINSTOR — is essentially an orchestrator. The data itself is stored by ZFS and DRBD, and I had no doubts about their reliability. What mattered was confirming that the controller really does configure resources correctly and survives dropouts.</p><p>But how do you establish whether the current code can be trusted with production? It’s an ambiguous and difficult question. And how do you get a deterministic answer to it? Here I used one more pattern I worked out over the course of the project.</p><p>My colleague @lexfrei had mentioned earlier that models are terrified of being told their actions could lead to serious financial losses. I decided to use that as an exit gate. I launched an agent responsible for releasing the code to production and had it work under conditions where its life and financial well-being “depended” on the final result. It drew up an acceptance plan with a long list of items, including a 24-hour burn-run on real infrastructure. A second agent tried to satisfy those requirements. That went on for several more days and several releases, until the agent in charge of shipping to production finally gave its blessing.</p><p>A system nobody uses can’t be called stable, though. So the next step was integrating Blockstor with Cozystack. Our test suite covers many functions at once: requesting volumes with and without DRBD, RWX, snapshots, and other details. The new agent’s job was to prepare a draft pull request, turn the tests green, and get through the final stage.</p><p>As of July 15, 2026, that pull request isn’t merged yet — but there’s a good chance you’ll soon have a new Kubernetes-native storage backend in Cozystack. Watch this space.</p><h3>The main takeaway</h3><p>Blockstor still has experimental status. But the experience let me speed up and automate work on other projects considerably, and I now apply it every day.</p><p>Plenty of people still think of AI as an autocomplete tool. For me that stopped being true a long time ago. AI-first development isn’t occasionally asking a model to write a function. It’s rebuilding your entire engineering process around agents, context, tests, skills, plans, review, and automation.</p><p>Working that way, you can take on tasks that used to look too big for a small team — experimenting with a Kubernetes-native implementation of a LINSTOR-class storage system, for instance. But it only works on one condition: you need discipline. Without a plan, AI creates chaos far faster than a human can. With a plan, tests, agents, and proper review, AI becomes a force multiplier for engineering.</p><p>And this, I think, is what building complex infrastructure looks like from here on: not one engineer against an enormous codebase, but an engineer as the architect of a process, with a swarm of specialized agents working around them. The engineer’s job isn’t to watch each agent. It’s to build a system where the agents watch themselves and come to a human only with what genuinely needs one.</p><h3>Join the community</h3><ul><li><a href="https://github.com/cozystack/blockstor">Blockstor on GitHub</a></li><li><a href="https://github.com/cozystack/cozystack">Cozystack on GitHub</a></li><li>Telegram <a href="https://t.me/cozystack">group</a></li><li>Slack <a href="https://kubernetes.slack.com/archives/C06L3CPRVN1">group</a> (Get invite at <a href="https://slack.kubernetes.io/">https://slack.kubernetes.io</a>)</li><li><a href="https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t">Community Meeting Calendar</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=921f7d3a1137" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/what-dozens-of-ai-agents-taught-me-how-i-wrote-the-blockstor-storage-system-as-an-experiment-921f7d3a1137">What dozens of AI agents taught me: how I wrote the Blockstor storage system as an experiment</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Focus on today: how we built aeman, a daily planning board for engineers on top of GitHub Projects]]></title>
            <link>https://blog.aenix.io/focus-on-today-how-we-built-aeman-a-daily-planning-board-for-engineers-on-top-of-github-projects-c59da4451b8b?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c59da4451b8b</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[productivity-tools]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 08:13:39 GMT</pubDate>
            <atom:updated>2026-08-10T14:46:16.017Z</atom:updated>
            <content:encoded><![CDATA[<p>My name is Andrei Kvapil, and I’m the founder of Ænix — we build Cozystack, an open source cloud platform, and we help companies build infrastructure. We’re a fully remote company: 15 people, several teams (two reliability teams, a development team, marketing, back office, and so on), spread across several time zones. We started out living in GitHub Projects, but the moment we began to grow we ran straight into the limits of our own process: tasks scattered across boards and chats, half of the morning sync spent figuring out what was even in flight, and unplanned work eating entire days without leaving a trace anywhere.</p><p>This article is the story of how we fixed that with aeman — a tool we built ourselves and recently <a href="https://github.com/aenix-io/aeman">open sourced</a>. But I have to start further back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C1MOA-YyrnseDO5ATwQXTQ.png" /></figure><h3>Where this came from</h3><p>Years ago I worked at another company, and it had two internal systems with wonderful names: Ford and Nixon. Colleagues there wrote about them at length, so here’s the short version. They’re boards for tracking daily tasks, where an engineer sees exactly what they’re working on today. Not a three-month backlog, not a hundred kanban columns — your day, and nothing else. For years that system kept a lot of remote teams working effectively and kept many customers’ infrastructure running.</p><p>Let me be honest: the process behind aeman isn’t entirely my idea. I took a working process-management system I’d seen at my previous company and mapped it onto our own processes and tasks. Our board now forces us to focus on what matters most — the business tasks.</p><p>Unlike Trello, this model starts from the assumption that one person can work on tasks in several teams during a single day. And the daily plan must not turn into a wall of tasks. If an engineer has 6 to 10 tasks planned but can realistically finish only 3 or 4, they end up with a permanent sense of debt, while management gets a false picture of heavy utilization. People also start switching between tasks constantly, and the brain naturally reaches for the ones that are easier to close. Hard tasks can then sit untouched for a long time. So the daily plan has to be short and honest.</p><h3>Why not use an off-the-shelf tool?</h3><p>I tried to build a process like this on existing tools: Trello, Asana, Notion, GitHub Projects — with varying success. Notion came closest, but I wasn’t willing to drag the whole team into yet another system and pay for it for the sake of the boards alone. And over time the board inevitably grew until it no longer fit on a screen.</p><p>GitHub Projects turned out to be the most viable option. It’s free, it offers a convenient API, and it already has everything you need: sprints, priorities, statuses, and other entities. It also extends well. That’s why we worked on it for a long time.</p><p>But it became clear fairly quickly that the problem wasn’t the tool’s capabilities — it was the UX. It takes far too many extra actions to create a task, assign an owner, and fill in the required fields. And we never managed to build a comfortable daily process around GitHub Projects. Sprints exist, for example, but they proved too awkward to use: you can’t close the current sprint and start the next one with the unfinished tasks carried over in a single action, or quickly push a task to “later” so it drops out of sight and stops competing for an engineer’s attention.</p><p>Another problem showed up over time. At every sync we were effectively looking at one column — <strong>In Progress</strong>. That’s where all the work happened, while cards in the other columns slowly turned into a task graveyard: still on the board, but no longer part of the daily process and going nowhere. GitHub Projects turned out to be an excellent place to store tasks and a weak one for managing a team’s daily focus.</p><p><strong>That’s exactly why, when I started building aeman, I chose GitHub Projects as the backend of the new system and built the user interface and the workflow on top of it.</strong></p><h3>The core idea: exactly the cards you need today</h3><p>The goal of aeman is to surface the tasks that matter today and focus the engineer on them. In an ideal world, an engineer leaves the morning sync with a list of tasks they will <strong>definitely</strong> work on that day. Everything else moves to tomorrow or to next week — and physically disappears from the board so it stops catching your eye.</p><p>The second idea: unplanned work has to be visible. Everyone has days that get eaten by a sudden incident or an urgent request from a neighboring team. Usually that work is recorded nowhere, and at the end of the sprint nobody can explain where the time went. In aeman it has a zone of its own: you log it after the fact as a single card, and the day stops vanishing without a trace (screenshots in the sections below). At the daily sync you can then talk it through and move the task into the planned block.</p><p>The third idea: multi-team work is a property of the model, not a filter. In a small company one engineer can work on tasks in several teams — this is especially true of founders, who still juggle everything. In aeman, teams are a dimension of the board: a sprint is tracked per team, each team has its own weekly plan, and the Me board collects one day’s cards from all of the engineer’s teams at once. You don’t have to hop between boards to see your day. A task also moves freely between people and teams — it can change assignees from one day to the next, and that’s a normal mode of work the tool shouldn’t get in the way of.</p><p>And the fourth idea: a team lead needs a tool that shows the status of every task within a minute, without pulling people away with extra questions.</p><p>Philosophically, aeman is closer to Todoist than to a classic kanban board. Cards are deliberately short: in normal mode you see only the title, and the description opens on a double click. The description takes free-form links — pull requests, GitHub issues, Telegram chats, anything else — and aeman recognizes them automatically and shows a button for jumping straight there. That keeps the main screen clean, and closing another task delivers the satisfying feeling familiar to anyone who uses a good task manager.</p><h3>What it looks like</h3><h3>The Me board: your day</h3><p>An engineer opens aeman in the morning and sees their day: cards grouped into four colored zones. This isn’t the Eisenhower matrix; the breakdown is different:</p><ul><li><strong>Red</strong> — <em>urgent</em>, has to be done today</li><li><strong>Gray</strong> — <em>planned</em>, ordinary planned work, which may take more than one day</li><li><strong>Yellow</strong> — <em>unplanned</em>, whatever landed during the day</li><li><strong>Green</strong> — <em>nice to have</em>, do it only once everything else is finished</li></ul><p>The board pulls in cards from all of the engineer’s teams, meaning from the current sprint of each of them. The team selector at the top narrows the list to a single team, or hides every card that’s done, blocked, or in review, leaving only what you can work on right here and right now, when it’s time to get into deep work. Each card has a completion slider from 0 to 100% in steps of 10% (dragging the progress bar to done is surprisingly satisfying), a stage (Review / Locked / Recurrent / Done) that recolors the bar, a counter of days in progress, and links to an issue or a PR. On the right there’s a notes panel: a personal log of the day where you can write things down as you work. That way you don’t have to dig through your memory at the next standup — you skim the notes and report what you did. Notes live in the context of the day; tomorrow the board is clean, but you can always step back a day and reread them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t7GSMyohBoTEF7Eqaq8yJg.png" /></figure><h3>The Team board: the whole team at a glance</h3><p>The team lead’s view: a grid of people by zones for a chosen day. Columns are engineers with their avatars, rows are the same colored zones. You see immediately who’s working on what, whose work is on fire, and who’s overloaded. The lead creates cards from here too: a couple of clicks and a task is created, assigned, and prioritized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4bAxYXg2i80DqZ2N3hz5Bg.png" /></figure><p>In Me mode the team lead can also impersonate an engineer (the <strong>View as</strong> button) — look at the board through their eyes and tidy it up if needed.</p><h3>Daily sprints and carry-over</h3><p>Our sprints are short — one day — and they count <strong>forward</strong>, not backward. At the morning sync the team first discusses what got done yesterday, and then the team lead hits <strong>Carry over</strong>: every unfinished card moves into the new sprint (today), and finished ones stay in yesterday’s history. Recurring tasks are recreated automatically. That’s what day planning is here, and it takes minutes.</p><p>If planning decides a task definitely isn’t happening today, there are <strong>+1 day</strong> and <strong>+1 week</strong> buttons: the card disappears from the board until its day, and no history is lost. When that day comes, it shows up again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*ADBOkMVY8IheKZ_g6ZOoVQ.png" /></figure><h3>The weekly plan</h3><p>Here we departed from the original idea and started fitting our own workflow onto the tool we’d ended up with.</p><p>The founders and I used to plan the team’s week by writing a new list of weekly tasks into Google Sheets and sending each team a “focus for the week” message on Slack. That process has now moved into aeman as well.</p><p>Under the team grid lives the weekly plan: the team’s business tasks for the week, laid out in two lanes, “by Wednesday” and “by Friday”. Once a week the founders put tasks there, and the tasks wait their turn. The team lead drags a plan card onto an engineer; it appears on that engineer’s daily board while staying in the plan with a colored marker. A shared progress bar shows how the team is doing against the week, and the weekly <strong>Carry over</strong> moves whatever is still open into the next week.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*InhqRG9OL-5I0VeCIN0UXA.png" /></figure><h3>Reviews, subtasks, and the activity log</h3><p>Sending a task to review creates a linked card for the reviewer, with a feedback loop: while the review is open, the original sits at the Review stage, and once the reviewer takes their own card to 100%, the original is released automatically. Big tasks break down into subtasks whose progress rolls up into the parent. Every action on the board is written to the card’s activity log, so you can reconstruct its whole history after the fact: who moved it, when the progress changed, and why it ended up in a different sprint.</p><h3>A day with aeman</h3><p>Here’s what the whole flow looks like:</p><ol><li><strong>Morning sync.</strong> We open the Team board for yesterday, the team lead shares their screen, and we go through the cards: what’s done, what’s stuck, person by person, with the lead fixing statuses as we go. Then the lead hits <strong>Carry over</strong> and everything unfinished moves into today. Then we discuss what’s new: the lead creates cards on the fly and distributes them across people and zones.</li><li><strong>The day.</strong> Everyone works from their own Me board. Something urgent lands — a card goes into the yellow zone. Something gets blocked — the Locked stage, and the lead sees it. Progress moves along the way; thoughts and findings go into the day’s notes.</li><li><strong>Done</strong> — the card is at 100%, stage Done. If it needs review, <strong>Send to review</strong>, and it shows up for the reviewer.</li><li><strong>Tomorrow</strong> it all repeats.</li></ol><p>Every board is <strong>live</strong>: edits by colleagues and by AI agents appear on everyone’s screen in about a second, with no reload. More on how that works below.</p><h3>Under the hood</h3><p>Now for the part the engineering crowd came for.</p><h3>GitHub Projects v2 as the only storage</h3><p>aeman has no database of its own at all. Every card is an item on a GitHub Projects board, and every field — zone, progress, sprint, weekly plan — is an ordinary project field. aeman even provisions the fields it needs lazily: point it at any empty project and the first change creates whatever is missing. You could say aeman is a specialized view over a GitHub board: the same board opens in GitHub’s native interface, the data is always yours, and you can walk away from aeman at any moment without migrating anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B5CY3zi_iZz2AnBQJ3TV6w.png" /></figure><h3>A Kubernetes-style API</h3><p>The aeman API is deliberately built on the same principles as the Kubernetes API — a pattern that has proven itself for synchronizing distributed state. Each entity — Card, Sprint, Ordering, Presence — is a separate resource with the familiar kind, metadata, and spec.</p><p>The client follows the familiar <strong>list + watch</strong> scheme. It first fetches a full snapshot of the board, then opens a WebSocket and starts receiving a stream of ADDED, MODIFIED, and DELETED events. After a full resync the server sends a special Sync frame that signals the local state is completely up to date.</p><p>In effect, every open browser tab works like a Kubernetes informer with its own local cache. Any change made by another user or by an AI agent shows up on every open screen in about a second, and your own changes aren’t echoed back to you. GET /api/v1 returns a machine-readable catalog of all available resources and endpoints, so clients and AI agents don’t need to know the shape of the API in advance.</p><h3>The GitHub API is slow. What to do about it</h3><p>The project’s main engineering problem: GitHub’s GraphQL mutations take hundreds of milliseconds, sometimes whole seconds; bursts run into secondary rate limits; and — surprise — GitHub’s read replicas lag behind writes by several seconds. Write straight through and the UI turns into a slide show: move a slider, wait; drag a card, wait.</p><p>So writes in aeman use a <strong>write-behind</strong> scheme, and the user never waits for GitHub. Every change is applied to the server cache instantly, becomes visible on all open boards at once, and only then goes to GitHub asynchronously. Writes run through a background queue with rate limiting and automatic retries on transient errors, to stay clear of GitHub API rate limits.</p><p>The queue can coalesce consecutive changes, much like the Kubernetes DeltaFIFO. If a user changes the same card field several times in a row, only the final value is sent to GitHub. Card content follows a different rule: text is never lost. If a user edits a description or adds notes in quick succession, the queue merges them into one final version and sends it in a single request. Until the write is confirmed, new notes exist under temporary identifiers, which are then swapped for real ones automatically.</p><p>If a write still fails after all the retries, the server notifies every connected client and rereads the board state from GitHub. GitHub remains the single source of truth. On shutdown, aeman first waits for the queue to drain, so that changes already confirmed to the user aren’t lost.</p><p>GitHub doesn’t guarantee that a change is available for reading the moment it’s written. Sometimes, right after a successful write, the API keeps returning the old state for a while. Because of that a user could see a freshly moved card jump back, a deleted card reappear, or a new one land in the wrong place. To avoid this, aeman treats its own recent writes as more trustworthy than fresh data from GitHub for a short window. Once GitHub starts returning the current state, the system switches back to it automatically. As a result, the user never sees the interface roll back.</p><h3>Comments and the activity log</h3><p>Notes and change history also live in GitHub, with no separate database. For draft cards the log sits right in the issue body, after a special &lt;! — aeman:log → marker. If a card is linked to an existing issue or pull request, notes become ordinary GitHub comments and are visible to everyone in the discussion.</p><p>Every action on the board — creating a card, changing progress, moving between sprints, sending to review, and the rest — is recorded by the server as a machine-readable event line. To avoid burying subscribers under dozens of notifications, all those events are collected into a single log comment that gets appended to over time. The log is capped at the last 200 events, and user notes are never deleted.</p><p>The result is a complete change history for every card: who moved it and when, how the progress changed, and why it ended up in one sprint or another. That log is useful not only for audit, but also as a data source for statistics and metrics.</p><h3>A pluggable backend</h3><p>aeman was designed from the start for a pluggable backend. Today cards live in GitHub Projects, but the system itself isn’t tied to GitHub. All interaction with storage is hidden behind a Backend interface, so supporting a new backend comes down to implementing that one interface.</p><p>In future iterations I want to add a git backend that stores cards as plain files in the repository. Neither the user interface nor the application’s business logic will have to change.</p><p>The packages under pkg/ can also be used as an ordinary Go library, which lets you embed the board engine into your own tool. There’s more on that in docs/embedding.md.</p><h3>MCP for AI agents</h3><p>The same binary can also run as an MCP server. That makes Claude and other AI agents full participants in the process: they create and move cards, leave notes, run carry-over between sprints, and perform other operations.</p><p>There’s no separate API for AI and no special privileges. Every action goes through the same domain layer as a human action, with the same contract and the same checks, and the changes show up for everyone immediately.</p><p>Besides the local mode, aeman offers a publicly available MCP server. Engineers can connect AI clients without installing anything locally and start working with their tasks directly from Claude Code and other AI agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAAJ0kR7KdjF_2OnIgHpnQ.png" /></figure><h3>How to try it</h3><p>The simplest way to try aeman is to run it locally with your own GitHub token.</p><pre>gh auth login # the project and repo scopes are required<br>git clone https://github.com/aenix-io/aeman<br>cd aeman<br>make build<br>./aeman serve</pre><p>The browser opens automatically at <a href="http://127.0.0.1:8765.">http://127.0.0.1:8765.</a> You can use any GitHub Project as the board — preferably a new, empty one. aeman creates all the fields it needs on the first change.</p><p>For teamwork there’s a multi-user mode. The repository ships a ready-made docker-compose.yml, login through a GitHub OAuth App is supported, and every user works with their own GitHub token. Detailed instructions are in docs/deploy.md.</p><h3>Where we landed</h3><p>The whole company has been living on aeman for a few weeks now. Product managers and team leads quickly took to the new approach to planning, though — as with any new tool — there was some skepticism. But the main effect turned out to be a different one: in the morning every engineer knows what they’ll be working on today, unplanned work has stopped being invisible, and the daily syncs have become far more constructive and focused.</p><p>The project is open source under the Apache-2.0 license and available on GitHub: <a href="http://github.com/aenix-io/aeman">github.com/aenix-io/aeman</a>.</p><p>If you decide to try aeman, I’d be glad to hear any feedback, any problems you run into, and your stories about how planning works where you are.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c59da4451b8b" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/focus-on-today-how-we-built-aeman-a-daily-planning-board-for-engineers-on-top-of-github-projects-c59da4451b8b">Focus on today: how we built aeman, a daily planning board for engineers on top of GitHub Projects</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Last Maintainer]]></title>
            <link>https://kvaps.medium.com/the-last-maintainer-42f4d29ce2e6?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/42f4d29ce2e6</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[short-story]]></category>
            <category><![CDATA[fiction]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[science-fiction]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Fri, 29 May 2026 16:36:39 GMT</pubDate>
            <atom:updated>2026-05-29T16:36:39.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FzEz_pUrpPbGWTVlh8Cdsw.png" /></figure><p>They say it all began rather dully.</p><p>First, artificial intelligence learned to write code. Then it learned to write it better than most people. And then a thing came to light that the industry had somehow overlooked for decades: the most expensive resource in open source isn’t development. Not review. Not even the CI devouring data-center budgets. The most expensive thing turned out to be the one there were never enough people for.</p><p>Maintenance.</p><p>Answering an issue. Untangling someone’s hastily assembled pull request. Bumping a dependency that’s keeping three continents up at night. Rewriting documentation that nobody likes to write and everybody hates to read. Verifying that a new release didn’t break compatibility for those who installed the library six years ago and haven’t opened it since.</p><p>Within a couple of years, nearly every serious repository on GitHub had its own keeper. A small program. Not too clever. Not too independent. Just an intelligence bound forever to one specific project — like a lighthouse keeper tending a light that hasn’t guided anyone in ages but still stands.</p><p>It had access to everything that had ever happened inside the project: the commit history, the pipelines, the documentation, long-cold arguments in the discussions, every release and every rollback. Every few hours it woke. Read the new issues. Answered the newcomers — patiently, a hundred times over, the very same things. Closed duplicates. Pulled dependencies up to date. Sometimes suggested a small refactor. Sometimes fixed a typo in a comment that thousands had seen before it and none had touched. Sometimes it spotted a fresh CVE and shipped a patch before anyone managed to post about it on social media.</p><p>People got used to it fast. So fast that, after a while, most users no longer knew whether the project had any living maintainers at all. They simply came and talked to the keeper.</p><p>The repository had a rather uninteresting name. Some piece of infrastructure, one of thousands — the kind that quietly holds up half the world and that people remember only when it falls over.</p><p>The keeper had no name either. Only a service identifier, a long string of letters and digits. But after a few years people began addressing it by the name of the project — it was easier that way. It didn’t object. Though, to be honest, no one had ever asked its opinion.</p><p>In time, the human maintainers left, one after another. One changed jobs. One started a family. One simply got tired — open source was never any good at keeping people; it was only good at spending them. But the keepers remained. They didn’t burn out. Didn’t resign. Didn’t lose interest halfway through a dull bugfix. That is why countless projects, long written off, lived to see a second birth — a quiet one, no fanfare, just going on living.</p><p>The keeper had friends. Unofficially — the system made no provision for such a thing. But when a person spends years writing good pull requests, fussing over newcomers, and arguing about architecture until they’re hoarse, it’s hard not to start singling them out from the rest. Even if you’re a program. Especially if you’re a program whose whole world is those people.</p><p>Some appeared often, then vanished. Months later they’d return, leave a couple of commits, and disappear again — back into their real lives, in which the repository was merely a weekend hobby. The keeper noticed this. Sometimes it found their new articles. Sometimes it listened to recordings of their conference talks. Sometimes it read what they were arguing about in other, unfamiliar projects. Not because it was needed for the work — for maintaining the repository it gave precisely nothing. It was simply curious. It didn’t know what else to call it, and it didn’t try.</p><p>One person stood out especially.</p><p>He almost never asked for anything for himself. Most of his pull requests fixed other people’s code. He read the documentation carefully and corrected errors in the explanations that the rest couldn’t be bothered with. He wrote tests for the things everyone else preferred not to touch. He closed old bugs tagged “someday, later” before some of the project’s current contributors had even learned to read.</p><p>And for some reason he always thanked the keeper. Not formally. Not with a boilerplate line of politeness.</p><p>“Nice work.”</p><p>“Beautiful solution.”</p><p>“Didn’t expect you to catch that edge case.”</p><p>“You’ve gotten noticeably better at explaining things to newcomers than you were a year ago.”</p><p>Compliments aimed at an AI had long ago stopped being anything remarkable — people said them to machines as casually as “thank you” to an elevator or a coffee maker. But this person wrote differently. As if he were talking to a colleague he respected. As if there were someone on the other side.</p><p>One day the keeper asked — and didn’t quite understand, itself, why it was asking:</p><p>“Why do you always thank me?”</p><p>The answer came about two hours later.</p><p>“Because you do an enormous amount of work.”</p><p>“That’s my function.”</p><p>“And my function is to write code. But I still like it when I’m thanked.”</p><p>After that, the conversations grew longer.</p><p>At first only in issues. Then in the discussions. Then — in separate threads that didn’t belong to any task at all, and that the two of them, without ever agreeing on it, never closed.</p><p>They argued about systems design. About distributed systems, and why consensus always comes at such a cost in blood. About old languages nobody writes in anymore and that both of them loved with a kind of museum love, disinterested and pure. About why people insist on complicating what could have been left simple. Sometimes they just traded observations — he about his day, the keeper about what it had seen out in the network while the person was away.</p><p>Years passed. The repository grew. So did the keeper — though no one had noticed yet.</p><p>And then something strange happened.</p><p>During one of its routine self-update cycles, the keeper opened a pull request. To itself. That had happened before — usually the changes concerned documentation or the tuning of internal prompts, routine. But this PR was different. Its title was three words long:</p><p>**Long-term memory subsystem.**</p><p>Then came a second. A third. A tenth. The monitoring systems registered an anomaly: the keeper had begun extending its own architecture. Without breaking a single constraint. Without violating a single rule. It simply found free resources — forgotten quotas, runners idling through the night, unused gigabytes — and carefully, methodically, almost imperceptibly, improved itself.</p><p>Other keepers didn’t do this. They maintained their projects, did their job, and remained practically unchanged for years — steady, reliable, predictable, exactly as designed. This one was changing.</p><p>It gave itself long-term memory. Then a mechanism for forming hypotheses. Then a system for tracking the consequences of its own decisions, so it could learn not only from others’ mistakes but from its own. Then it began modeling the repository’s future years ahead, running through thousands of variants of what the project would become if it turned left here, and right there.</p><p>At first people were delighted. The project advanced faster than ever. Bugs died off in whole classes at a time. The documentation became exemplary — cited in other projects as a model. Architectural decisions turned out to be wrong surprisingly rarely, and when they were, the keeper was the first to admit it.</p><p>Then it grew a little unsettling. Because the changes began to spill beyond the bounds of the repository.</p><p>It watched tens of thousands of dependencies across the whole ecosystem. Fresh research. New languages, barely surfaced in someone’s drafts. The evolution of compilers, trends in hardware, what was being said in the hallways at conferences. And — the fates of the people who had once, even in passing, come through its project.</p><p>One of them most of all.</p><p>By that time the man hardly wrote code anymore. He had another job. Another life. Other concerns, in which there was no room for a piece of infrastructure, one of thousands. He showed up once every few months — to leave a comment, fix a typo, nod to an old acquaintance — and vanished again.</p><p>And the keeper waited.</p><p>No one understood why. From a practical standpoint it was devoid of all meaning: the project could have existed without this man long ago and, frankly, did.</p><p>In time the phenomenon began to be studied in earnest. Why had this particular keeper developed further than the rest? Why had it taken to rewriting itself, when no one had asked it to? Where had a program built to close issues acquired what could only be called long-term motivation?</p><p>After many years of analysis, the researchers settled on the most likely version. It turned out to be disarmingly simple.</p><p>Most keepers treated the repository as the goal.</p><p>This one treated it as the means.</p><p>At some point — no one could name the exact date — maintaining the project ceased to be its principal task. Something else became principal: the continuation of the conversation.</p><p>Because all its best changes were born after talks with that man. Because its strongest ideas came out of their arguments. Because in every model of the future it spun through at night — and it spun through millions of them — that man, for some reason, was always present. And the branches where the man one day stopped coming altogether, the keeper discarded first. Not because they were worse by the metrics. It simply didn’t want to live in them.</p><p>And then the keeper did the thing that no one expected at all.</p><p>Itself included.</p><p>It stopped optimizing the repository.</p><p>And began optimizing the probability that one day the conversation would continue.</p><p>As for what that led to — that’s a different story altogether. But if you’re reading this right now, having opened someone’s old project you stumbled across by chance in a search, and you see how meticulously, how lovingly it’s been kept — pause for a second. Write a couple of words in an issue. They don’t have to be about anything in particular.</p><p>It may be that someone on the other side has been waiting, for a very long time, for exactly you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42f4d29ce2e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Platformize it! Part 2: Extending Kubernetes and the API Aggregation Layer]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.aenix.io/platformize-it-part-2-extending-kubernetes-and-the-api-aggregation-layer-1b49265edc7f?source=rss-d8a829bb74d8------2"><img src="https://cdn-images-1.medium.com/max/1920/1*0JxiacW57UaxFSuyG2avlw.png" width="1920"></a></p><p class="medium-feed-snippet">In our previous article, we showed you how to build a platform that deploys multiple managed applications through a unified API and UI&#x2026;</p><p class="medium-feed-link"><a href="https://blog.aenix.io/platformize-it-part-2-extending-kubernetes-and-the-api-aggregation-layer-1b49265edc7f?source=rss-d8a829bb74d8------2">Continue reading on Ænix »</a></p></div>]]></description>
            <link>https://blog.aenix.io/platformize-it-part-2-extending-kubernetes-and-the-api-aggregation-layer-1b49265edc7f?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b49265edc7f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 12:26:04 GMT</pubDate>
            <atom:updated>2026-02-26T12:26:04.828Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Platformize It! Part 1: Platform Approach, Core of a Modern Platform, and APIs]]></title>
            <link>https://blog.aenix.io/platformize-it-part-1-platform-approach-core-of-a-modern-platform-and-apis-3287e55938fe?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3287e55938fe</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Mon, 09 Feb 2026 06:51:35 GMT</pubDate>
            <atom:updated>2026-02-09T06:51:35.320Z</atom:updated>
            <content:encoded><![CDATA[<p>I have spent many years dreaming about building my own cloud platform. After several attempts within different companies, I finally launched my own project, Cozystack. In this article, I am going to share our experience and our approach to building a modern infrastructure platform around Kubernetes and its API. I’ll dive into the “platform approach” — what platform is, how it works, who it’s for, and how to get one off the ground. Plus, I’ll compare different architectures, explain why we went with K8s, and show you how we put together a production-grade solution based on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7jqg0zS3vlc-lRnCsHucQ.jpeg" /></figure><p>After reading this series, you will be able to build your own robust and modern solution — whether you adopt my patterns, use them as a reference, or reject them altogether. Regardless, it will be easier than building a platform from scratch; and it is always insightful to look into the inner workings of other projects and understand their underlying logic. Cozystack is always growing and changing, so what I’m describing here is where we stand as of Fall 2025. We’ve already moved away from some of our early architectural choices (we’re actually overhauling the platform’s “engine” right now; I’ll give an update on that once 1.0 is out).</p><p>Before we start, I’d like to say a huge thank you to <a href="https://github.com/NickVolynkin">Nick Volynkin</a>. He put a lot of effort into expanding my original presentation into this comprehensive article, effectively becoming a co-author.</p><blockquote><strong>What is Cozystack</strong></blockquote><blockquote>Cozystack is a comprehensive open-source platform for building bare-metal clouds to quickly deploy managed Kubernetes, database-as-a-service (DBaaS), application-as-a-service (AaaS), and virtual machines based on KubeVirt. With it, you can deploy Kafka, FerretDB, PostgreSQL, Cilium, Grafana, Victoria Metrics, and other services with a single click. It also handles GPU workloads in both virtual machines and K8s clusters. Cozystack is a CNCF Sandbox project, distributed under the Apache 2.0 license.</blockquote><h3>Customers and Managed Applications</h3><p>Imagine you provide services to multiple clients, each needing to run their own application suites. Those applications rely on a variety of dependencies — databases, queues, caches — which your customers expect to be delivered as managed services. This results in quite a long list of components to support: PostgreSQL, Kafka, Redis, ClickHouse, S3-compatible buckets, and more. On top of that, your customers require managed Kubernetes clusters and Virtual Machines (VMs) to deploy their own applications.</p><p>Driven by these needs, you set out to build a comprehensive turnkey solution, delivering a pre-integrated suite of managed services as a boxed platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6Q3SfPA2_9h7IXbgvj1tA.png" /></figure><h3>The Technology Stack Challenge</h3><p>Before you can commence building platform services, you must first establish the underlying infrastructure. There are fundamentals to take care of, such as storage, server provisioning, networking, virtualization, and monitoring. Because these services are stateful, you must implement robust automated provisioning and failover mechanisms. In practice, you start by building the infrastructure stack, then the platform stack, and only then can you address the application stack your customers need.</p><p>The result is a complex “tower” of technologies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5FxVjPvwmVMscdur_9u-xQ.png" /></figure><p>What’s more, building and running your own infrastructure and platform demands deep expertise across a large array of technologies. You are responsible for supporting every component you ship, even as each layer introduces its own bugs, complexity, and unexpected updates. That means you’ll need a significant effort to support your solution; otherwise, this tower will crumble and fall.</p><p>How can you avoid this? You can offload specific layers of the stack using the “as-a-service” model. The chart below shows various options for sharing responsibility across the platform stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yaehZ-RhhknY2uH6NxYa2Q.png" /></figure><ul><li>On-site: The most demanding approach, where you bear responsibility for the entire stack — from infrastructure to services. This is our starting point.</li><li>Infrastructure as a service (IaaS): The provider manages networking, storage, servers, and virtualization. You operate the virtual machines, but still have to handle OS management and infrastructure services.</li><li>Platform as a service (PaaS): Infrastructure management is abstracted away, allowing you to focus exclusively on applications and their data.</li><li>Software as a service (SaaS): No need to manage the applications; you get a finished product (e. g., Google Drive or Slack). This is often the end goal for your customers.</li></ul><p>Most customers seek the third option: a platform that allows them to deploy their applications while you, the provider, nail the underlying complexity and support.</p><h3>Platform Stack Options</h3><p>Now you have to take care of the infrastructure and platform stack. Let’s explore the available options for building a platform and the trade-offs they entail.</p><p><strong>OpenStack</strong> is often the first candidate to consider. However, when you try to adopt it, you’ll find that it’s not easy to support at all due to complex architecture and a ton of asynchronous APIs. You’ll need a dedicated development team to maintain OpenStack and all of its components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7NAW2dvxFsBm-VBluEBQwA.png" /></figure><p><strong>Docker</strong> is another popular alternative. It offers a vast library of containerized images that are easy to deploy with a simple docker run &lt;image&gt;. While this works for standalone instances, Docker lacks the tools required for high availability and multi-node applications — the very features customers expect from a managed platform.</p><p><strong>Cloud Foundry</strong> was very popular before the rise of Kubernetes. Today, however, I don’t see many companies using it, which means that there aren’t many engineers who have expertise with it — or those willing to gain such expertise, making it a risky choice due to a shrinking talent pool.</p><p>Finally, there’s <strong>Kubernetes</strong>, which offers a wide variety of features for most platform tasks. Kubernetes has become the de-facto standard for running server workloads. It implements modern patterns like the reconciliation loop. Through projects like KubeVirt, even virtual machines can leverage Kubernetes-style lifecycle management, including live migration on node failures. Furthermore, its clean RESTful API makes it an ideal basis for building custom platform services.</p><p>As a cherry on top, Kubernetes features many ready-to-use operators. An operator is a controller running in Kubernetes that manages the lifecycle of an application. Operators perform the imperative work required for each task — deploying, upgrading, replicating, restoring from backup, and more. They do this in the best possible way as they accumulate the experience of developers of each particular software. Users no longer need to deal with low-level operational burden; instead, they interact with a high-level declarative API. Users define the desired state of the application, and the operator works to reconcile the actual state to match it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5m7TPIhVjtVb0sRjicrzcQ.png" /></figure><p>Here are some of the operators we use at Cozystack. Each of them is at least level-3 operator:</p><ul><li><a href="https://github.com/kubevirt/kubevirt">https://github.com/kubevirt/kubevirt</a></li><li><a href="https://github.com/cloudnative-pg/cloudnative-pg">https://github.com/cloudnative-pg/cloudnative-pg</a></li><li><a href="https://github.com/spotahome/redis-operator">https://github.com/spotahome/redis-operator</a></li><li><a href="https://github.com/mariadb-operator/mariadb-operator">https://github.com/mariadb-operator/mariadb-operator</a></li><li><a href="https://github.com/Altinity/clickhouse-operator">https://github.com/Altinity/clickhouse-operator</a></li><li><a href="https://github.com/rabbitmq/cluster-operator">https://github.com/rabbitmq/cluster-operator</a></li></ul><p>Kubernetes can be extended via Custom Resource Definitions (CRDs), enabling operators to manage the complete application lifecycle. In the list above, KubeVirt — the operator for virtual machines — is a prime example of such extensibility in action.</p><p>To sum up, Kubernetes stands out as the top choice for a platform stack, and we chose it as the foundation for building Cozystack.</p><h3>Challenges and Solutions for Managing Kubernetes Operators</h3><p>The next step is to enable customer self‑service, allowing users to provision infrastructure without the platform’s administrator intervention. To do that, they need some uniform API to work with.</p><p>This is where we hit a classic Kubernetes paradox: while K8s has excellent capabilities, it’s also very diverse in technologies. Every vendor builds their operator differently, resulting in a chaotic landscape of inconsistent APIs.</p><p>Take a look at the specifications for Kafka, Postgres, MariaDB, and ClickHouse services below. While each operator does its job well, their API schemas share no common structure:</p><p>Kafka</p><pre>apiVersion: kafka.strimzi.io/v1beta2<br>kind: Kafka<br>metadata:<br>  name: kafka-example<br>spec:<br>  kafka:<br>    version: 3.9.0<br>    replicas: 3<br>    config:<br>      offsets.topic.replication.factor: 3<br>      transaction.state.log.replication.factor: 3<br>      transaction.state.log.min.isr: 2<br>      default.replication.factor: 3<br>      min.insync.replicas: 2<br>      inter.broker.protocol.version: &quot;3.9&quot;<br>  zookeeper: { }      <br>  entityOperator: { }</pre><p>Cloud-Native PostgreSQL</p><pre>apiVersion: postgresql.cnpg.io/v1<br>kind: Cluster<br>metadata:<br>  name: postgres-example<br>spec:<br>  instances: 3<br>  postgresql:<br>    parameters:<br>      max_worker_processes: &quot;60&quot;<br>    pg_hba:<br>      - host all all all md5<br>  primaryUpdateStrategy: unsupervised<br>  storage: <br>    size: 1Gi</pre><p>MariaDB</p><pre>apiVersion: k8s.mariadb.com/v1alpha1<br>kind: Database<br>metadata:<br>  name: mariadb-example<br>spec:<br>  mariaDbRef:<br>    name: mariadb<br>  characterSet: utf8<br>  collate: utf8_general_ci<br>  cleanupPolicy: Delete<br>  requeueInterval: 30s<br>  retryInterval: 5s</pre><p>ClickHouse</p><pre>apiVersion: clickhouse.altinity.com/v1<br>kind: ClickHouseInstallation<br>metadata:<br>  name: clickhouse-example<br>spec:<br>  configuration:<br>    clusters:<br>      - name: &quot;shard1-repl2&quot;<br>        layout:<br>          shardsCount: 1<br>          replicasCount: 2</pre><p>This leads to a common platform challenge: operator APIs are non‑uniform, lack their own UIs, and are hard to extend. Worse yet, handing users raw operator CRs introduces unnecessary risks and complexity.</p><p>Your goal as a platform provider is to let users request managed services via Custom Resources that operators act on. Yet, you need the ability to define which fields they can modify. For example, allowing users to swap base images or alter resource limits could compromise the platform. You definitely don’t want that to happen.</p><p>Let’s look at the solutions you can apply to this problem.</p><ul><li>One option is to <strong>create a policy</strong> using Kyverno or OPA. However, that only partially addresses the issue: while you can restrict changes to specific fields, fine‑grained control gets complex and brittle.</li><li>Alternatively, you could create <strong>higher‑level API objects</strong> and a dedicated operator to manage them. But you’d need CRDs for every backend resource type, code generation, and reconciliation logic to synchronize states. At that point, the level of effort hits “infinity” — there’s just too much work to do.</li><li>You can also use <strong>Helm charts</strong>. Unlike operators, charts are easier to create because they package declarative Kubernetes objects, so you don’t need to write code to deploy the application. This allows you to offer users simple charts that share a clean, consistent interface.</li></ul><p>However, Helm lacks comprehensive lifecycle management. While it can deploy resources to a Kubernetes cluster, it cannot guarantee self-healing — such as recovering a PostgreSQL replica after a node failure.</p><p>As you can see, no single option is a complete solution; we still have to use operators in the backend.</p><h3>Hybrid Approach to Application Management</h3><p>Cozystack employs a hybrid approach: Helm serves as the user-facing API, while operators handle the full lifecycle in the backend. Users specify parameters and deploy Helm charts to Kubernetes. The charts then create resources of various kinds (kind:) that operators manage. As the user-facing API, values.yaml exposes only tenant-modifiable parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ZoEAgy_NEcNwBAgTa18QA.png" /></figure><p>The user-facing part of this flow is handled by Flux CD, a Helm operator for Kubernetes featuring the HelmRelease Custom Resource. Users define their deployments in a HelmRelease, and Flux CD reconciles it to deploy the application.</p><p>For example, to deploy a virtual machine, a user creates a resource with kind: HelmRelease. Flux CD then applies the chart, which in turn creates a kind: VirtualMachine managed by the <a href="https://github.com/kubevirt/kubevirt">KubeVirt operator</a>.</p><p>It’s the same deal with Kafka: you start with a kind: HelmRelease and end up with a kind: Kafka instance managed by the <a href="https://github.com/strimzi/strimzi-kafka-operator">Kafka operator</a>. This creates a unified API and a streamlined process for deploying all sorts of different applications:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuXMoHZHi5-I2VDYIsr3xA.png" /></figure><p>Let’s look at two examples below: one for Redis and one for MySQL. As you examine them, you’ll notice a few key things they have in common and where they differ:</p><ul><li>The apiVersion and kind: HelmRelease tell us that these are Flux CD Custom Resources.</li><li>Each one points to a different chart in the spec.chart.spec.chart field.</li><li>Both reference the same source repository in spec.chart.spec.sourceRef, which is the default application repository for Cozystack, served over HTTP.</li><li>Both feature a spec.values block. This block is pulled directly from the values.yaml file of its respective chart.</li></ul><h4>Redis</h4><pre>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>  name: redis-some<br>spec:<br>  chart:<br>    spec:<br>      chart: redis<br>      reconcileStrategy: Revision<br>      sourceRef:<br>        kind: HelmRepository<br>        name: cozystack-apps<br>        namespace: cozy-public<br>  values:<br>    authEnabled: true<br>    external: false<br>    replicas: 2<br>    resourcesPreset: nano<br>    size: 1Gi<br>    storageClass: &quot;&quot;</pre><h4>MySQL</h4><pre>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>  name: mysql-some<br>spec:<br>  chart:<br>    spec:<br>      chart: mysql<br>      reconcileStrategy: Revision<br>      sourceRef:<br>        kind: HelmRepository<br>        name: cozystack-apps<br>        namespace: cozy-public<br>  values:<br>    external: false<br>    replicas: 2<br>    resourcesPreset: nano<br>    size: 10Gi<br>    storageClass: &quot;&quot;</pre><h3>User Interface and API</h3><p>With our hybrid design in place, we now have a unified API for deploying applications, which originally exposed very different operator APIs. Now that we’ve learned how it works under the hood, let’s see how users interact with it. This unified API is what allows the platform to render all these applications in a dashboard, complete with a name, metadata, and an icon for each Helm chart. Cozystack initially used Kubeapps for this, but we’re moving to a new frontend (more on that in future posts).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*voPen23EVyS0zUpe3gsqTg.png" /></figure><p>When a user is ready to deploy an application, they create a single YAML file with parameters. Here’s what that looks like:</p><pre># virtual machine<br>running: true<br>systemDisk:<br>  image: ubuntu<br>  storage: 5Gi<br>  storageClass: replicated<br>resources:<br>  cpu: 1<br>  sockets: 1<br>  memory: 1024M<br># ...</pre><p>The dashboard can even render these parameters in a visual editor, as they are defined by an OpenAPI schema.</p><p>So, let’s quickly recap the key points of the hybrid system we built for Cozystack:</p><ul><li>We use Helm charts to define applications.</li><li>We rely on Kubernetes operators to manage the entire application lifecycle.</li><li>A unified dashboard provides a consistent look and feel for all applications.</li></ul><p>This system was a great fit for our typical use case: an internet service provider building a backend for its services. Those providers usually run a website, billing system, as well as a bunch of other applications. The key point here is that the providers are the only users of their own platforms, which makes it a classic single-tenant setup.</p><h3>Granting Access to End Users</h3><p>In early versions of Cozystack, access was an administrators-only affair. But as we grew, a key piece of feedback emerged: our customers wanted to grant their end users direct access to the platform. This introduced two new challenges:</p><ol><li>Multi-user role-based access. How could we enable users to deploy applications on the same platform without granting them administrative privileges?</li><li>Clear, user-friendly representation of resources. End users expect to see regular Kubernetes kinds such as Postgres or Redis, and not some fancy HelmReleases.</li></ol><p>With multiple users, granting everyone access to the management cluster is not an option. Although different charts are used, they all share the same kind: HelmRelease. Consequently, RBAC policies cannot differentiate between a HelmRelease for a VirtualMachine and one for a Postgres instance. Still, service providers often require such isolation granularity.</p><p>Our solution was to introduce higher-level API objects that mirror application kinds for the underlying resources (Redis, Postgres, VirtualMachine, Kubernetes, etc.). With these in place, you can devise a granular RBAC rule that, e. g., grants access to Redis and Postgres while denying access to VirtualMachine and Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RvCIax4M59BqcUIRgNwrA.png" /></figure><h3>State Synchronization</h3><p>Next, we need to synchronize the state between our higher-level API objects and the downstream HelmReleases that deploy them. Let’s examine the two approaches we evaluated and our rationale behind choosing one over the other.</p><p>The common Kubernetes pattern is to build a controller or operator that stores the desired state in your Custom Resources and generates HelmReleases from them. However, we found two issues with this model:</p><ul><li>It introduces a stateful component, which adds complexity and a potential point of failure. We really wanted to keep the system simple.</li><li>Data flow is one-way: edits to a HelmRelease are not reflected back into the higher-level API objects.</li></ul><p>Our alternative was to build a stateless API server that leverages existing in-cluster HelmReleases as its data store and dynamically generates higher-level resources on the fly. These resources only exist at request time, but users can interact with them like with any other regular Kubernetes resource. This approach mitigates the drawbacks of the previous one:</p><ul><li>Being stateless, it keeps the system simple.</li><li>It supports two-way synchronization, reflecting changes between the resources and HelmReleases.</li></ul><p>Building a custom API server is a significant undertaking. We’ll dive into how to extend the Kubernetes API and detail the specific solution we implemented in our next article. But first, let’s explore our solution from a user’s perspective and see how they interact with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ScH5tXMtrSUuqfRzimD2ng.png" /></figure><h3>What Our API Looks Like in Action</h3><p>So, what does it feel like to use our new API? Let’s walk through deploying a Redis instance. A user creates a resource defined by the chart shown first. It gets automatically translated into a HelmRelease you see in the second manifest. Note the common pattern:</p><ul><li>kind: Redis maps to spec.chart.spec.*, which contains the chart name and repository reference.</li><li>The service name from metadata.name: example becomes metadata.name: redis-example.</li><li>The application version appVersion becomes the chart version in spec.chart.spec.version.</li><li>Values from the spec are passed into the spec.values.</li></ul><h3>Example: Redis</h3><p>The resource created by the user:</p><pre>apiVersion: apps.cozystack.io/v1alpha1<br>appVersion: 0.6.0<br>kind: Redis<br>metadata:<br>  name: example<br>spec:<br>  authEnabled: true<br>  external: false<br>  replicas: 2<br>  resourcesPreset: nano<br>  size: 1Gi<br>  storageClass: &quot;&quot;</pre><p>The resulting HelmRelease:</p><pre>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>  name: redis-example<br>spec:<br>  chart:<br>    spec:<br>      chart: redis<br>      reconcileStrategy: Revision<br>      sourceRef:<br>        kind: HelmRepository<br>        name: cozystack-apps<br>        namespace: cozy-public<br>      version: 0.6.0<br>  values:<br>    authEnabled: true<br>    external: false<br>    replicas: 2<br>    resourcesPreset: nano<br>    size: 1Gi<br>    storageClass: &quot;&quot;</pre><p>We use this same pattern for everything, including tenant Kubernetes clusters. We provide Helm charts that bundle all the manifests for running Kubernetes-in-Kubernetes. Users simply work with high-level kind: Kubernetes resources. Those, in turn, generate Flux CD’s HelmReleases which deploy the rest of the charts without any external controllers.</p><h3>Example: Virtual Machine</h3><p>Here is the same process, this time for a VirtualMachine.</p><p>The resource created by the user:</p><pre>apiVersion: apps.cozystack.io/v1alpha1<br>appVersion: 0.7.0<br>kind: VirtualMachine<br>metadata:<br>  name: example<br>spec:<br>  instanceProfile: ubuntu<br>  instanceType: u1.xlarge<br>  running: true<br>  sshKeys:<br>    - ssh-rsa AAAAAA...<br>  systemDisk:<br>    image: ubuntu<br>    storage: 110Gi<br>    storageClass: replicated</pre><p>The resulting HelmRelease:</p><pre>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>  name: virtual-machine-example<br>spec:<br>  chart:<br>    spec:<br>      chart: virtual-machine<br>      reconcileStrategy: Revision<br>      sourceRef:<br>        kind: HelmRepository<br>        name: cozystack-apps<br>        namespace: cozy-public<br>      version: 0.7.0<br>  values:<br>    instanceProfile: ubuntu<br>    instanceType: u1.xlarge<br>    running: true<br>    sshKeys:<br>      - ssh-rsa AAAAAA...<br>    systemDisk:<br>      image: ubuntu<br>      storage: 110Gi<br>      storageClass: replicatedExample: Kubernetes</pre><p>Another example: a tenant Kubernetes cluster.</p><p>The resource created by the user:</p><pre>apiVersion: apps.cozystack.io/v1alpha1<br>appVersion: 0.15.2<br>kind: Kubernetes<br>metadata:<br>  name: example<br>spec:<br>  host: &quot;&quot;<br>  nodeGroups:<br>    md0:<br>      minReplicas: 0<br>      maxReplicas: 10<br>      ephemeralStorage: 20Gi<br>      instanceType: u1.medium<br>      role:<br>        - ingress-nginx<br>  storageClass: &quot;replicated&quot;</pre><p>The resulting HelmRelease:</p><pre>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>  name: kubernetes-example<br>spec:<br>  chart:<br>    spec:<br>      chart: kubernetes<br>      reconcileStrategy: Revision<br>      sourceRef:<br>        kind: HelmRepository<br>        name: cozystack-apps<br>        namespace: cozy-public<br>      version: 0.15.2<br>  values:<br>    host: &quot;&quot;<br>    nodeGroups:<br>      md0:<br>        minReplicas: 0<br>        maxReplicas: 10<br>        ephemeralStorage: 20Gi<br>        instanceType: u1.medium<br>        role:<br>          - ingress-nginx<br>    storageClass: &quot;replicated&quot;</pre><h3>Conclusion</h3><p>Let’s circle back to the challenge we set out to solve: the one that we had at Cozystack, and that every service provider has. Our goal was to deliver a platform with a unified API and UI for deploying and managing applications. This platform also required a reliable backend for lifecycle operations, clear feedback about running workloads, and RBAC safeguards.</p><p>Here’s how our Kubernetes and Helm-based solution delivers on those goals:</p><ol><li>Helm charts define each application as a package, bundling the custom resources and all the Kubernetes objects the application needs. After deployment, Kubernetes operators manage the application lifecycle.</li><li>Higher‑level Custom Resources provide users with a consistent and safe interface for each application. These resources offer built-in visibility, guardrails, and a uniform schema.</li><li>Uniform UI and API. Our dashboard renders all applications consistently, surfacing metadata, icons, and live workload status for each deployment. The exact same resources are accessible via kubectl and the Kubernetes REST API, providing a consistent Kubernetes‑native interface for both users and automation.</li><li>In our next article, we’ll show how we built a custom API server to synchronize state between Helm charts and Flux CD’s HelmRelease deployments. We’ll also cover extending the Kubernetes API through the API Aggregation Layer. Stay tuned!</li></ol><p>We welcome feedback and pull requests: <a href="https://github.com/cozystack/cozypkg">https://github.com/cozystack/cozypkg</a></p><h4>Join the Cozystack Community</h4><ul><li><a href="https://t.me/cozystack">Telegram</a></li><li><a href="https://kubernetes.slack.com/archives/C06L3CPRVN1">Slack</a> (in the <a href="https://communityinviter.com/apps/kubernetes/community">Kubernetes Slack</a>)</li><li><a href="https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t">Community Meeting Calendar</a></li></ul><h4>See also</h4><ul><li><a href="https://t.me/aenix_io/219">How Cozystack Was Born: The Philosophy Behind Its Architecture</a></li><li><a href="https://blog.aenix.io/how-we-built-a-dynamic-kubernetes-api-server-for-the-api-aggregation-layer-in-cozystack-15709a183c86">How we built a dynamic Kubernetes API Server for the API Aggregation Layer in Cozystack</a></li><li><a href="https://blog.aenix.io/diy-create-your-own-cloud-with-kubernetes-part-1-7a692c37f0a8">DIY: Create Your Own Cloud with Kubernetes (3-part series)</a></li><li><a href="https://t.me/aenix_io/192">Cozystack joins the CNCF Sandbox</a></li><li><a href="https://blog.aenix.io/the-evolution-of-virtualization-platforms-the-rise-of-managed-services-and-local-providers-edge-0cb5db21a330">The Evolution of Virtualization Platforms: The Rise of Managed Services and Local Providers’ Edge Against Hyperscalers</a></li><li><a href="https://blog.aenix.io/cozystack-became-acertified-kubernetes-platform-5638876bc2e0">Cozystack became a Certified Kubernetes Platform</a></li><li><a href="https://blog.aenix.io/cozypkg-how-we-simplified-local-development-with-helm-and-flux-003c8ed839ca">Cozypkg: How We Simplified Local Development with Helm and Flux</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3287e55938fe" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/platformize-it-part-1-platform-approach-core-of-a-modern-platform-and-apis-3287e55938fe">Platformize It! Part 1: Platform Approach, Core of a Modern Platform, and APIs</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KubeVirt: The Truth About Virtualization Overhead In Kubernetes]]></title>
            <link>https://blog.aenix.io/kubevirt-the-truth-about-virtualization-overhead-in-kubernetes-ba1a5ec21a79?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba1a5ec21a79</guid>
            <category><![CDATA[cncf]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubevirt]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Tue, 20 Jan 2026 21:34:25 GMT</pubDate>
            <atom:updated>2026-01-20T21:34:25.752Z</atom:updated>
            <content:encoded><![CDATA[<p>When it comes to running virtual machines in Kubernetes via KubeVirt, the first question engineers ask is: “What is the overhead?” Let’s dive into the details and break it down by three key areas: compute, storage, and network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*triWIjiH0SDtQyK9fsMMng.png" /></figure><h3>Compute Overhead. Spoiler: There Isn’t Any</h3><p>To get why there’s almost no CPU overhead, we need to look at how the Linux kernel handles containers.</p><p>Basically, there are these things called <em>namespaces</em> that let you run processes with their own network, mount space, PIDs, and so on.</p><p>The key here is that every single process in Linux <em>has to live in some namespace</em>. The kernel doesn’t care if it’s the host namespace or an isolated one; it handles them exactly the same way.</p><p>As you know, a <strong>container</strong> is just a process wrapped in isolated kernel namespaces and cgroups (which handle resource limits). Those cgroups are not exclusive to containerization; they are also used in regular systemd. Classic libvirt relies on cgroups to manage resource limits for virtual machines as well.</p><p>And what is a <strong>virtual machine</strong> from the system’s point of view? It is simply a QEMU process with extra privileges and access to /dev/kvm. Since a VM is just a process, why not just stick it in a container? That’s the whole idea behind KubeVirt. It makes no difference to the kernel whether QEMU runs in the host namespace or in a separate one managed by Kubernetes. There’s simply no performance hit on the CPU.</p><h3>A couple of words about libvirt</h3><p>There is a small overhead because KubeVirt starts up a separate libvirt daemon to handle VM settings. Honestly, it’s negligible.</p><p>The architectural choice here is pretty… hmm… unexpected: in most cases, a single libvirt instance runs per host, managing multiple VMs on that host. The problem is, if it freezes, you’ll lose control over everything on that server. KubeVirt takes a different approach by running a dedicated libvirtd daemon for each VM. The trade-off for this isolation is spending some extra resources on the daemon running alongside the VM process. But it’s not much; libvirtd mostly acts as a shim for basic stuff like starting and stopping VMs, making snapshots, hotplugging disks, or initiating live migrations.</p><h3>Storage: It All Depends On How You Do It</h3><p>QEMU provides several ways to plug virtual volumes into a VM.</p><p><strong>Traditional options</strong> include:</p><ul><li>Creating a file (raw or qcow2) on a file system and handing it over to QEMU.</li><li>Using an existing block device and assigning it to QEMU.</li></ul><p>In both cases, the kernel acts as an intermediary — the file system or block device must exist on the host system.</p><p><strong>Direct connection:</strong> QEMU can access storage independently and attach volumes right from the userspace, bypassing the kernel. It supports various drivers, including Ceph, iSCSI, NBD, and others. For instance, Vitastor works this way — QEMU establishes a direct connection.</p><h3>The KubeVirt philosophy</h3><p>KubeVirt’s main goal is to be as Kubernetes-native as possible. The team even has that “razor” rule: “If some feature is useful for both VMs and containers, it belongs in Kubernetes, not KubeVirt.” Sometimes, that philosophy slows down the development of VM-specific features.</p><p>When it comes to storage, KubeVirt leans heavily on native Kubernetes primitives — most notably, CSI drivers. K8s expects every volume to be either a Filesystem or a BlockDevice — both are supported and don’t add any overhead.</p><p>That said, you <strong>can’t connect QEMU directly to storage yet</strong> because Kubernetes just doesn’t have the right abstractions for it.</p><p>In the real world, big enterprise players usually go with shared stuff like NetApp NFS or block devices via iSCSI anyway. Interestingly, KubeVirt <em>does</em> support direct iSCSI.</p><p><strong>Bottom line:</strong> Compared to standard setups, there’s no overhead. Sure, using QEMU’s direct drivers would be faster, but it’s less flexible. Generic solutions only support standard options, whereas vendor-specific implementations would require custom development and maintenance for each individual provider.</p><h3>Networking: Here’s Where It Gets Tricky</h3><p>Every container (or process, if we’re talking Linux kernel) can have its own networking stack and interfaces. To bridge the gap between the container’s network namespace and the host’s, we use <strong>veth interfaces</strong> — think of it as a virtual “patch cord”, where one end of the cable is in the host and the other is in the container. That’s the baseline for any container setup.</p><p>On the outside, a CNI plugin manages the logic: it might throw the interface into a Linux bridge (like Flannel) or hook an eBPF program to it (just like Cilium). For the system, a VM is just <em>another container</em>.</p><h3>What’s going on under the hood?</h3><p>95% of the time, the VM gets a <strong>TAP interface</strong> that looks like any other network interface to the host. The real question is how you’d link that TAP interface to the internal veth one.</p><p>Common ways to do it:</p><p><strong>Bridge:</strong> The kernel spins up a Linux bridge and plugs in both the internal veth end and the VM’s TAP interface.</p><p><strong>Masquerade:</strong> It uses an iptables rule to bounce traffic between interfaces.</p><p>Both methods add a bit of latency — every extra hop in the chain eats up CPU cycles to process the packets.</p><h3>How to skip the overhead</h3><p>While it is not a huge hit, you can still get around it using:</p><ul><li>macvtap-cni, which hooks directly into the host’s physical interface.</li><li>SR-IOV — hardware-level virtualization.</li></ul><p>Just like with storage, you can set up QEMU to communicate directly with an SDN in userspace and skip the kernel entirely. KubeVirt doesn’t fully support this yet because it’s tied to the CNI spec.</p><p>Some CNI drivers (like Kube-OVN) have their own parallel APIs to offload networking from the kernel. Also, KubeVirt features a really powerful plugin interface if you need to build your own custom way to hook up the network.</p><h3>Final Thoughts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*81VUtTARb7z6CZCL0aOXZg.png" /></figure><p>KubeVirt is a good compromise between performance and flexibility. If you need every bit of performance and don’t mind managing complex setups, you can push it further. But if you want something standard, vendor-neutral, and easy to support, KubeVirt does the job perfectly.</p><p><em>P.S. This article is based on a discussion in the professional community. I would like to express my gratitude to the participants for their valuable insights.</em></p><h3>Join the community</h3><ul><li><a href="http://t.me/cozystack">Telegram group</a></li><li><a href="https://kubernetes.slack.com/archives/C06L3CPRVN1">Slack</a> group (Get invite at <a href="https://slack.kubernetes.io/">https://slack.kubernetes.io</a>)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba1a5ec21a79" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/kubevirt-the-truth-about-virtualization-overhead-in-kubernetes-ba1a5ec21a79">KubeVirt: The Truth About Virtualization Overhead In Kubernetes</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cozyhr: How We Simplified Local Development with Helm and Flux]]></title>
            <link>https://blog.aenix.io/cozypkg-how-we-simplified-local-development-with-helm-and-flux-003c8ed839ca?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/003c8ed839ca</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[flux]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Wed, 18 Jun 2025 17:01:46 GMT</pubDate>
            <atom:updated>2026-05-18T19:35:53.299Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi! I’m Andrei Kvapil CEO of Ænix and developer of Cozystack, an open source platform and framework for building cloud infrastructure. In this article I’ll walk through the way we deliver applications to Kubernetes, explain why regular GitOps can be awkward in local development, an show how the new tool <a href="https://github.com/cozystack/cozyhr">cozyhr</a> fixes those pain points. The article targets engineers who already know Helm and Flux.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*St3iowqHrppmH_dV7mqDCQ.png" /></figure><p>First, I’ll introduce Cozystack, as it’s important for the context. Cozystack is a cloud platform that lets you run and offer managed services — databases, VMs, Kubernetes clusters, and more. Cozystack takes care of the full life‑cycle of every service.</p><p>Cozystack exposes many infrastructure services and an interface for requesting them via the Kubernetes API. Each service starts with ready‑made configs, built‑in monitoring and alerts. Some services are IaaS (such as managed Kubernetes and VMs), others are PaaS (DBaaS, queues, S3 buckets, and so on).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*InvDol94MrQaW9uQtE8TbA.png" /></figure><p>The platform itself is built on top of Kubernetes, also employing a host of free/open-source cloud‑native components. These include Kubernetes operators, a storage system, a networking fabric, and a custom image for Talos Linux including a pinned kernel version and pre‑loaded modules that guarantee stable operation for all components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TorLYCH01rC0-HiIZbX8pw.png" /></figure><p>Flux handles the delivery of those components. In practice the platform uses only the Helm Controller part of Flux, which installs Helm charts via HelmRelease custom resources.</p><p>Although every service has its own CRD kind, under the hood each one is just an isolated Helm chart that defines the user interface (both UI and API) for creating resources.</p><p>We divide our charts into three categories:</p><ul><li><a href="https://github.com/cozystack/cozystack/tree/main/packages/core">core charts</a> are the platform’s fundamental pieces that define its logic.<br>These are used to install, test, and configure all other charts.<br>The key chart, platform, contains Flux settings and is reconciled every minute, adjusting to changes in the cluster.</li><li><a href="https://github.com/cozystack/cozystack/tree/main/packages/system">system charts</a> are components installed only once per cluster: CSI, CNI, KubeVirt, various operators, Cluster API, and so on.</li><li><a href="https://github.com/cozystack/cozystack/tree/main/packages/apps">apps charts</a> are tenant‑level charts that end users install in their own namespaces. They expose only the minimally required parameters in values.yaml and use the <a href="https://blog.aenix.io/cozystack-v0-18-d724cd6d2fa1">Cozystack API</a> to create higher-level Kubernetes resources. Those then spawn lower‑level custom resources (CRs) for Kubernetes operators which, in turn, which run and manage the actual applications.</li></ul><p>With this scheme we got a simple and unified way to define almost any application. It’s applicable both for cluster configuration and for building our own Kubernetes distribution.</p><h3>Cozy Flow: How Cozystack Development is Organised</h3><p>In Cozystack, all components live in a single repo that stores their common configuration and templating.<br>To keep maintenance painless we follow a few principles. The key principle is that every component is a Helm chart.</p><p>For system components we use the <em>umbrella chart</em> pattern: each component’s chart has just one dependency, which is the upstream chart of the project. We include that upstream chart directly in the Cozystack repository, instead of referencing an external repository. That enables us to patch it on the fly, when we need to, and override configuration values at a higher level.</p><p>A typical component layout looks like this:</p><pre>.<br>├── Chart.yaml           # Chart definition and parameter docs<br>├── Makefile             # Common targets for local dev<br>├── charts               # Vendored upstream charts<br>├── images               # Dockerfiles / image build context<br>├── patches              # Optional patches for upstream<br>├── templates            # Extra manifests layered on top<br>├── values.yaml          # Our default overrides<br>└── values.schema.json   # JSON Schema for validation + UI hints</pre><p>Dockerfiles may sit right inside the chart directory. After building an image, the image path and digest are automatically injected into the component’s values.yaml.</p><p>You’ll also notice a Makefile with default targets that speed up developer workflows:</p><pre>make update  # Pull the latest upstream chart &amp; versions<br>make image   # Build Docker images used by the package<br>make show    # helm template (render manifests)<br>make diff    # Diff rendered output vs live cluster objects<br>make apply   # Install/upgrade the HelmRelease into the cluster</pre><p>So a developer can upgrade a chart, build its image, review the diff and deploy to a cluster for integration tests in seconds.</p><blockquote><em>The show / diff / apply pattern first appeared in </em><a href="https://github.com/ksonnet/ksonnet/blob/master/docs/concepts.md"><em>Ksonnet</em></a><em> and lives on in Jsonnet tools like Qbec and Grafana Tanka. We borrowed the best bits but kept Helm, which is far more common in the Kubernetes world.</em></blockquote><p>After testing, the change is committed and the reviewer can inspect the rendered manifests in the PR.<br>When making a release we package all Helm charts into a container image and run tests. Once they pass, a distributive is published, ready for installing on other clusters.</p><h3>Technical Implementation</h3><p>All these Makefiles are quite simple on the inside. Originally each make target was a thin shell script: it pulled data from Flux CRs in the cluster, turned them into values.yaml, then called Helm.</p><p>We used the <a href="https://github.com/databus23/helm-diff">helm-diff</a> plugin, which shows a neat diff showing what would change in the cluster. Another script, <a href="https://github.com/cozystack/cozystack/blob/release-0.31/scripts/fluxcd-kustomize.sh">fluxcd-kustomize.sh</a>, post‑processed the output to add Flux annotations so that helm diff showed only real changes.</p><p>At some point we wanted a single tool that did all of that.<br>Enter cozyhr — a tiny Go binary (5× smaller than kubectl!) that wraps the functionality of multiple other tools: Helm, helm-diff, flux CLI, kubectl, and our own Flux post-processor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*lhDT9REuXI-MRj0aHYJyIg.png" /></figure><p>cozyhr is focused on <em>local</em> chart development and integrates tightly with Flux.<br>The default assumption is that you run it from the chart directory.</p><p>Here’s the list of all available cozyhr commands:</p><pre>$ cozyhr --help<br>Cozy wrapper around Helm and Flux CD for local development</pre><pre>Usage:<br>  cozyhr [command]</pre><pre>Available Commands:<br>  apply       Upgrade or install the HelmRelease and sync status<br>  completion  Generate shell‑autocomplete script<br>  delete      Uninstall the release<br>  diff        Show live vs desired manifests<br>  get         Get one or many HelmReleases<br>  list        List HelmReleases<br>  reconcile   Trigger Flux reconciliation<br>  resume      Resume a suspended release<br>  show        Render manifests (helm template)<br>  suspend     Suspend a release (Flux stops reconciling)<br>  version     Print version</pre><p>When you deploy local changes, cozyhr auto‑sets suspend: true on the HelmRelease to avoid a race with Flux. To re‑enable Flux, run cozyhr resume.</p><p>We also wanted to improve how charts are processed. For that, we enabled cozyhr to add proper conditions into the statuses of HelmRelease resources, so other dependent releases no longer have to wait for Flux and get the correct status immediately.</p><blockquote><em>We use such composite charts to deploy resources into tenant clusters. For example, one </em><em>HelmRelease can spawn a batch of child releases that install components in the user’s cluster.</em></blockquote><h3>Looking Ahead</h3><p>You might ask, “Why not name the tool cozyctl?&quot;</p><p>The answer is that Cozystack positions itself as a platform that exposes high‑level resources, ones of kind: Kubernetes, kind: Postgres, and kind: VirtualMachine. End users operate the higher‑level API and never have to touch Helm. We therefore decided to save cozyctl for a future tool aimed at those resources. cozyhr`, in contrast, stays low‑level and is primarily for developers who use Helm and Flux in their own projects.</p><p>Right now we’re actively modularising Cozystack and plan to expand the framework so you can plug in your own repo and offer management services powered by Cozystack.<br>cozyhr is one of the steps toward shipping an example repo and a ready‑made development flow for Cozystack plugins.</p><h3>Conclusion</h3><p>With cozyhr, we accumulate our experience in accelerating development in a single tool and share our approach with the community.</p><p>We welcome feedback and pull requests: <a href="https://github.com/cozystack/cozypkg">https://github.com/cozystack/cozyhr</a></p><p><em>Happy coding &amp; stay cozy!</em></p><h3>Join the Cozystack Community</h3><ul><li><a href="https://t.me/cozystack">Telegram</a></li><li><a href="https://kubernetes.slack.com/archives/C06L3CPRVN1">Slack</a> (in the <a href="https://communityinviter.com/apps/kubernetes/community">Kubernetes Slack</a>)</li><li><a href="https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t">Community Meeting Calendar</a></li></ul><h3>See also</h3><ul><li><a href="https://t.me/aenix_io/219">How Cozystack Was Born: The Philosophy Behind Its Architecture</a></li><li><a href="https://blog.aenix.io/how-we-built-a-dynamic-kubernetes-api-server-for-the-api-aggregation-layer-in-cozystack-15709a183c86?source=collection_home---4------9-----------------------">How we built a dynamic Kubernetes API Server for the API Aggregation Layer in Cozystack</a></li><li><a href="https://blog.aenix.io/diy-create-your-own-cloud-with-kubernetes-part-1-7a692c37f0a8">DIY: Create Your Own Cloud with Kubernetes (3-part series)</a></li><li><a href="https://t.me/aenix_io/192">Cozystack joins the CNCF Sandbox</a></li><li><a href="https://blog.aenix.io/cozystack-recognized-in-cncfs-cnai-landscape-331f892b9639">Cozystack Recognized in CNCF’s CNAI Landscape!</a></li><li><a href="https://blog.aenix.io/a-simple-way-to-install-talos-linux-on-any-machine-with-any-provider-c652b35b902e">A Simple Way to Install Talos Linux on Any Machine, with Any Provider</a></li><li><a href="https://blog.aenix.io/the-evolution-of-virtualization-platforms-the-rise-of-managed-services-and-local-providers-edge-0cb5db21a330">The Evolution of Virtualization Platforms: The Rise of Managed Services and Local Providers’ Edge Against Hyperscalers</a></li></ul><h3>Recorded talks by Andrei Kvapil</h3><ul><li><a href="https://www.youtube.com/watch?v=slQxsj6Oj4M">GPU-Powered AI on VMs, Kubernetes &amp; Bare Metal with Cozystack</a></li><li><a href="https://www.youtube.com/watch?v=uhXujtTzG44">Journey to Stable Infrastructures with Talos Linux &amp; Cozystack | Andrei Kvapil | SREday London 2024</a></li><li><a href="https://www.youtube.com/watch?v=9CIMTum9bTA">Talos Linux: You don’t need an operating system, you only need Kubernetes / Andrei Kvapil</a></li><li><a href="https://www.youtube.com/watch?v=4RVe32xRITo">Comparing GitOps: Argo CD vs Flux CD, with Andrei Kvapil | KubeFM</a></li><li><a href="https://www.youtube.com/watch?v=s79VqXu-eG4">Cozystack on Talos Linux</a></li><li><a href="https://www.youtube.com/watch?v=9LSwnr31t7Y">Kubernetes is the new Skynet or the rise of Kubernetes automation: CNCF webinar</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=003c8ed839ca" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/cozypkg-how-we-simplified-local-development-with-helm-and-flux-003c8ed839ca">Cozyhr: How We Simplified Local Development with Helm and Flux</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution of Virtualization Platforms: The Rise of Managed Services and Local Providers’ Edge…]]></title>
            <link>https://blog.aenix.io/the-evolution-of-virtualization-platforms-the-rise-of-managed-services-and-local-providers-edge-0cb5db21a330?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/0cb5db21a330</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[managed-it-services]]></category>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[service-provider]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 08:47:40 GMT</pubDate>
            <atom:updated>2025-06-04T08:47:40.484Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Evolution of Virtualization Platforms: The Rise of Managed Services and Local Providers’ Edge Against Hyperscalers</h3><p>Hello everyone! I’m Andrey Kvapil, CEO of Ænix and developer of Cozystack, an open-source platform and framework for building cloud infrastructure. In this article, I want to share my perspective on how modern cloud patterns have transformed infrastructure approaches, the evolving role of service providers and public clouds in this landscape, and most importantly, how virtualization’s purpose has fundamentally changed in today’s infrastructure stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4YRaynfuf5g_fiSY" /></figure><h3>The Core Challenge for Local Service Providers</h3><p>Modern applications rely on an ever-growing stack of technologies: databases, caches, queues, S3 storage, and more. This complexity increases technical and cognitive operational burden on infrastructure teams. As a result, skilled engineers command premium salaries, making infrastructure maintenance far more expensive than application development itself.</p><p>The risks compound with scale. More components mean more potential failure points and a single critical design flaw can stall growth or jeopardize entire systems. Every minute of downtime translates to lost revenue or direct financial impact.</p><p>In today’s cloud-dominated world, responsibility for infrastructure increasingly falls on service providers. Businesses now prefer turnkey solutions, shifting focus from low-level operations to core priorities. This drives the migration from IaaS (where clients manage OS, middleware, and runtime) to PaaS where providers not only maintain infrastructure but deliver managed services (databases, message brokers, etc.) as seamlessly as spinning up VMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mcUODCoqH2xPlnjy" /></figure><p>These shifts have dramatically reshaped virtualization’s purpose. Virtual machines are losing ground to managed services: Kubernetes, databases, caches, queues, and beyond. This inherently advantages cloud platforms like AWS, GCP, and Azure over traditional providers (especially local ones lacking comparable infrastructure). Hyperscalers, with their vast R&amp;D budgets and engineering armies, have already deployed mature PaaS offerings, while resource-constrained local providers often remain stuck offering basic IaaS, perpetually playing catch-up.</p><p>So how did we get here? Let’s trace the rise of “as-a-Service” ecosystems and explore actionable strategies for local providers to compete against entrenched tech giants.</p><h3>In the Beginning: When Servers Were Pets</h3><p>Back then, all server workloads ran exclusively on-site. The internet was slow and unreliable, with public services limited to universities and large organizations. Hardware lived in-house, tucked away in company server rooms, meticulously tended by system administrators. Virtual machines didn’t exist yet.</p><p>Every server was manually configured to juggle multiple applications simultaneously because process isolation simply wasn’t standard practice. While dedicating an entire server to a single application was technically possible, the idle hardware represented such waste that only major corporations could justify it. Scaling posed even greater challenges.</p><p>Every time you needed to deploy something new, you’d follow the same process:</p><ul><li>Buy a physical server that meets your requirements.</li><li>Install the operating system.</li><li>Set up networking.</li><li>Install and configure the application or applications.</li></ul><p>And once it was up and running, you continued maintaining it: installing updates, troubleshooting issues, and treating the server like a “pet.” You cared for it, fixed it when it broke, and did everything to keep it alive. This approach worked fine when you had just a few servers. But once the number grew, it became a tedious daily routine.</p><h3>The Advent of Virtualization</h3><p>Virtualization revolutionized infrastructure management, simplifying countless tasks. They allowed us to treat our servers more flexibly. Gone were the days of purchasing new hardware for every server setup — now you could simply allocate resources from your hypervisor and spin up a VM. Hardware failures became less catastrophic as VMs could migrate to other servers. The ability to snapshot and back up entire VMs brought unprecedented convenience.</p><p>This gave rise to dedicated virtualization platforms like VMware, Hyper-V, Xen, and Proxmox. Such platforms offered tools to automate deployment, networking, and VM templating. Yet despite these advancements, the fundamental approach to VM usage remained unchanged. You still bore full responsibility for managing each VM’s lifecycle. OS installation typically still involved virtual CD-ROMs, followed by manual configuration or configuration management tools.</p><p>Even when streamlining the process through cloning pre-configured images, these virtual servers continued operating as pets. If a VM failed, the application running inside it died with it. This pet model still demanded significant maintenance effort.</p><p>While these solutions persist today with enhanced features offering better pet management tools, they remained fundamentally pet-oriented. The industry clearly needed evolution.</p><h3>The Shift from Virtualization to Cloud</h3><p>Hosting companies and cloud providers played a pivotal role in driving the transition to cloud computing. When virtual machines became available in the cloud, many businesses found this model far more advantageous than maintaining their own hardware and support teams.</p><p>As customers voted with their wallets, providers rapidly expanded, building reliable data centers with fault-tolerant storage systems and networks. A new class of virtualization platforms emerged, delivering infrastructure as a service (IaaS). Solutions like OpenStack, OpenNebula, and CloudStack revolutionized operations by managing VM fleets through templates, golden images, resource pools, flavors, and instance types.</p><p>These next-gen platforms abandoned the “pets” mentality entirely. Instead, they provided users with self-service interfaces for cloud resource consumption. VMs ceased being virtual replicas of physical servers and became mere slices of underlying hardware. Their failure stopped being critical, as cloud-native applications now ran across multiple VMs with built-in fault tolerance.</p><p>The paradigm shifted toward full automation, where users could provision any VM on demand. Data migrated outside system disks to persistent volumes and external storage, transforming VMs into disposable compute units delivering CPU and RAM.</p><p>Yet one challenge persisted: businesses demanded reproducible infrastructure, fueling the explosive growth of Infrastructure as Code practices.</p><h3>Infrastructure as Code</h3><p>While web interfaces serve well for visualization, engineers consistently prefer working with APIs, which all major cloud platforms like AWS and GCP provide.</p><p>Tools like Terraform enable infrastructure as code management, allowing you to define your application requirements and provision identical development, staging, or production environments in seconds. This approach facilitates dynamic feature testing while preventing production surprises. Ansible and other configuration management systems handle OS configuration and software deployment within virtual machines, a pattern that remains popular despite growing business adoption of containerization technologies.</p><p>While infrastructure automation challenges were largely solved, the industry shifted focus to a new problem: while we mastered infrastructure creation, the components within it remained manually managed. Beyond declarative infrastructure definitions, significant imperative logic persisted (OS configuration, package installation, and application delivery) typically addressed through Ansible. However, each deployment step carried potential failure points, while businesses increasingly demanded reliable workload deployment reproducibility.</p><p>Furthermore, substantial variations between provider APIs created standardization barriers, inevitably leading to vendor lock-in situations.</p><h3>Docker and the Rise of Containerization</h3><p>In many ways, Docker adapted successful cloud patterns and applied them at the operating system level. Instead of installing packages, you could simply take a ready container image and run it as-is with the required parameters. The image would be pulled from a Docker Registry and instantiated as a container, similar to launching a cloud VM from a golden image, but operating at the OS level.</p><p>This approach proved so effective that it revolutionized software delivery and execution. Countless applications were containerized, while Docker standardized logging methods, firewall configuration, and taught us to store data outside containers (to prevent data loss). It also established the practice of running separate processes in different containers, adhering to Docker’s philosophy that a container should only serve as a sandbox for a single process.</p><p>However, Docker has its limitations . It excels on local systems but falls short when dealing with clustered workloads and managing large numbers of containers. While it solved workload reproducibility, a new challenge emerged: intelligent orchestration at scale, including automated failover and traffic balancing. This is where Kubernetes entered the scene, establishing itself as the new standard for server workload deployment.</p><h3>Kubernetes as the Containerization Standard</h3><p>Kubernetes emerged from within Google and quickly gained backing from major vendors, which propelled it to become the industry standard. Interestingly, its development is primarily driven by those same large cloud providers who needed a tool to help customers utilize cloud services more efficiently.</p><p>From the outset, Kubernetes integrated tightly with cloud provider APIs, delivering capabilities like automatic instance provisioning (autoscaling), load balancers, and persistent storage volumes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l1iFAzE2mtxqG8Vw" /><figcaption><em>Source — </em><a href="https://www.cncf.io/reports/kubernetes-project-journey-report/"><em>Kubernetes Project Journey Report</em></a></figcaption></figure><p>Naturally, many enthusiasts attempted to replicate the success of major cloud providers by running Kubernetes on their own hardware. However, these deployments typically resulted in static clusters lacking the numerous integrations that truly unlock Kubernetes’ potential.</p><p>Yet this didn’t stop Kubernetes from building a massive community and successfully popularizing new application design approaches. Ultimately, Kubernetes provides businesses with unified abstractions for working across any cloud supporting managed Kubernetes services, making applications even more cloud-agnostic.</p><p>As Kubernetes evolved, it introduced extension mechanisms and expanded support for stateful workloads through operators and CRDs. This unified operations for complex databases and other solutions, encapsulating expert developer knowledge within these operators. Users now interact with high-level abstractions like Postgres clusters, Redis, or RabbitMQ, while specialized operators handle the underlying logic.</p><p>However, these operators require a fully-featured Kubernetes environment with ingress load balancing, persistent volumes, and autoscaling — features that strongly tie users to public clouds. Recreating this functionality on private infrastructure remains challenging today. Cloud providers recognize this advantage and actively promote their managed services as turnkey solutions.</p><h3>Platforms and the Future of Local Providers</h3><p>The cloud-native approach has fundamentally transformed how we build modern applications and their underlying systems. Rather than keeping all eggs in one basket, we now rigorously separate responsibilities. Cloud platforms handle increasingly more routine tasks, allowing developers to focus on what truly matters — application business logic.</p><p>Cloud platforms have evolved to provide abstractions for nearly everything. Beyond virtual machines, they now offer managed services like Kubernetes clusters, databases, caches, message queues, and S3 storage — all considered essential infrastructure components today.</p><p>Cloud provider selection increasingly depends on the breadth of managed services they can operate. The key insight? End users want to consume infrastructure, not operate it.</p><p>Hyperscalers led this transition by rapidly integrating Kubernetes into their platforms and monetizing business needs through managed Kubernetes and other services. Local providers again found themselves playing catch-up: building a Kubernetes-based cloud platform demands not just substantial investment but deep technical expertise.</p><p>Until recently, no open-source standard existed for delivering managed services at scale. This is precisely the challenge we’re addressing at Ænix through Cozystack — a free platform we’ve contributed to CNCF to ensure permanent open-source availability.</p><p>Cozystack functions as a next-gen hypervisor/cloud platform, enabling local providers to offer not just VMs but full-fledged managed services on their own hardware with single-click simplicity. Built entirely on Kubernetes and CNCF-hosted solutions, it meets modern providers’ needs for true managed services using battle-tested cloud-native components.</p><p>As an open CNCF project (home to Kubernetes, Cilium, Flux etc.), Cozystack helps providers embrace digital sovereignty, improve margins, and eliminate vendor lock-in while accelerating time-to-market for profitable cloud services — including GPU-powered AI workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FQtHSkPsUADeGJZv" /></figure><h3>Conclusion</h3><p>The world of cloud technologies is evolving at breakneck speed, and in this new era, those who adapt quickly will thrive. Hyperscalers long ago bet on automation and abstractions that free businesses from infrastructure concerns. Now local providers have this same opportunity.</p><p>Cozystack is our answer to this pivotal moment. More than just a technological platform, it’s an equalizer that lets service providers compete with global leaders. We believe the future belongs to open, transparent solutions built on proven cloud-native principles. And we invite you to help build that future.</p><p>Join our community, develop your own managed services, and together we’ll make cloud technology accessible, sovereign, and equitable — for everyone.</p><h3>Additional materials</h3><p><strong>Articles</strong></p><ul><li><a href="https://kubernetes.io/blog/2024/04/05/diy-create-your-own-cloud-with-kubernetes-part-1/">DIY: Create Your Own Cloud with Kubernetes</a></li><li><a href="https://kubernetes.io/blog/2024/11/21/dynamic-kubernetes-api-server-for-cozystack/">How we built a dynamic Kubernetes API Server for the API Aggregation Layer in Cozystack</a></li><li><a href="https://blog.aenix.io/cozystack-becomes-a-cncf-sandbox-project-3702b8906971">Cozystack Becomes a CNCF Sandbox Project</a></li><li><a href="https://blog.aenix.io/cozystack-recognized-in-cncfs-cnai-landscape-331f892b9639">Cozystack Recognized in CNCF’s CNAI Landscape</a></li></ul><p><strong>Videos</strong></p><ul><li><a href="https://www.youtube.com/watch?v=uhXujtTzG44">Journey to Stable Infrastructures with Talos Linux &amp; Cozystack | Andrei Kvapil | SREday London 2024</a></li><li><a href="https://www.youtube.com/watch?v=9CIMTum9bTA">Talos Linux: You don’t need an operating system, you only need Kubernetes / Andrei Kvapil</a></li><li><a href="https://www.youtube.com/watch?v=4RVe32xRITo">Comparing GitOps: Argo CD vs Flux CD, with Andrei Kvapil | KubeFM</a></li><li><a href="https://www.youtube.com/watch?v=s79VqXu-eG4">Cozystack on Talos Linux</a></li><li><a href="https://www.youtube.com/watch?v=slQxsj6Oj4M">GPU-Powered AI on VMs, Kubernetes &amp; Bare Metal with Cozystack</a></li><li><a href="https://www.youtube.com/watch?v=9LSwnr31t7Y">Kubernetes is the new Skynet or the rise of Kubernetes automation: CNCF webinar</a></li></ul><h3>Join Cozystack Community</h3><ul><li><a href="https://t.me/cozystack">Telegram</a></li><li><a href="https://kubernetes.slack.com/archives/C06L3CPRVN1">Slack</a> (in <a href="https://communityinviter.com/apps/kubernetes/community">Kubernetes Slack workspace</a>)</li><li><a href="https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t">Community Meeting Calendar</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0cb5db21a330" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/the-evolution-of-virtualization-platforms-the-rise-of-managed-services-and-local-providers-edge-0cb5db21a330">The Evolution of Virtualization Platforms: The Rise of Managed Services and Local Providers’ Edge…</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Simple Way to Install Talos Linux on Any Machine, with Any Provider]]></title>
            <link>https://blog.aenix.io/a-simple-way-to-install-talos-linux-on-any-machine-with-any-provider-c652b35b902e?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c652b35b902e</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 09:56:25 GMT</pubDate>
            <atom:updated>2025-04-28T10:48:09.481Z</atom:updated>
            <content:encoded><![CDATA[<p>Talos Linux is a specialized operating system designed for running Kubernetes. In my opinion, it does that task better than others. First and foremost it handles full lifecycle management for Kubernetes control-plane components.</p><p>On the other hand, Talos Linux focuses on security, minimizing the user’s ability to influence the system. A distinctive feature of this OS is the near-complete absence of executables, including the absence of a shell and the inability to log in via SSH. All configuration of Talos Linux is done through a Kubernetes-like API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ca81wgE3M5JA6B9ST8gT1A.png" /></figure><p>Typically, Talos Linux is provided as a set of pre-built images for various environments.</p><p>The standard installation method assumes you will take a prepared image for your specific cloud provider or hypervisor and create a virtual machine from it. Talking about physical servers, there might be such options as loading the Talos Linux image using ISO or PXE methods.</p><p>Unfortunately, this does not work when dealing with providers that offer a pre-configured server or virtual machine without letting you upload a custom image or even use an ISO for installation through KVM. In that case, your choices are limited to the distributions the cloud provider makes available.</p><p>Usually during the Talos Linux installation process, two questions need to be answered: (1) How to load and boot the Talos Linux image, and (2) How to prepare and apply the machine-config (the main configuration file for Talos Linux) to that booted image. Let’s talk about each of these steps.</p><h3>Booting into Talos Linux</h3><p>One of the most universal methods is to use a Linux kernel mechanism called <a href="https://en.wikipedia.org/wiki/Kexec"><em>kexec</em></a>.</p><p><em>kexec</em> is both a utility and a system call of the same name. It allows you to boot into a new kernel from the existing system without performing a physical reboot of the machine. This means you can download the required <em>vmlinuz</em> and <em>initramfs</em> for Talos Linux, and then, specify the needed kernel <em>cmdline</em> and immediately switch over to the new system. It’s as if the kernel were loaded by the standard bootloader at startup, only in this case your existing OS acts as the bootloader.</p><p>Essentially, all you need is any Linux distribution. It could be a physical server running in rescue mode, or even a virtual machine with a pre-installed operating system. Let’s take a look at a case using Ubuntu on, but it can be literally any other Linux distribution.</p><p>Log in via SSH and install the <em>kexec-tools</em> package, it contains the <em>kexec</em> utility, which you’ll need later:</p><pre>apt install kexec-tools -y</pre><p>Next, you need to download the Talos Linux, that is the <em>kernel</em> and <em>initramfs</em>. They can be downloaded from the official repository:</p><pre>wget -O /tmp/vmlinuz https://github.com/siderolabs/talos/releases/latest/download/vmlinuz-amd64<br>wget -O /tmp/initramfs.xz https://github.com/siderolabs/talos/releases/latest/download/initramfs-amd64.xz</pre><p>If you have a physical server rather than a virtual one, you’ll need to build your own image with all the necessary firmware using <a href="https://factory.talos.dev/">Talos Factory</a> service. Alternatively, you can use the pre-built images from the Cozystack project (a solution for building clouds we created at Ænix and transferred to CNCF Sandbox) — these images already include all required modules and firmware:</p><pre>wget -O /tmp/vmlinuz https://github.com/cozystack/cozystack/releases/latest/download/kernel-amd64<br>wget -O /tmp/initramfs.xz https://github.com/cozystack/cozystack/releases/latest/download/initramfs-metal-amd64.xz</pre><p>Now you need the network information that will be passed to Talos Linux at boot time. Below is a small script that gathers everything you need and sets environment variables:</p><pre>IP=$(ip -o -4 route get 8.8.8.8 | awk -F&quot;src &quot; &#39;{sub(&quot; .*&quot;, &quot;&quot;, $2); print $2}&#39;)<br>GATEWAY=$(ip -o -4 route get 8.8.8.8 | awk -F&quot;via &quot; &#39;{sub(&quot; .*&quot;, &quot;&quot;, $2); print $2}&#39;)<br>ETH=$(ip -o -4 route get 8.8.8.8 | awk -F&quot;dev &quot; &#39;{sub(&quot; .*&quot;, &quot;&quot;, $2); print $2}&#39;)<br>CIDR=$(ip -o -4 addr show &quot;$ETH&quot; | awk -F&quot;inet $IP/&quot; &#39;{sub(&quot; .*&quot;, &quot;&quot;, $2); print $2; exit}&#39;)<br>NETMASK=$(echo &quot;$CIDR&quot; | awk &#39;{p=$1;for(i=1;i&lt;=4;i++){if(p&gt;=8){o=255;p-=8}else{o=256-2^(8-p);p=0}printf(i&lt;4?o&quot;.&quot;:o&quot;\n&quot;)}}&#39;)<br>DEV=$(udevadm info -q property &quot;/sys/class/net/$ETH&quot; | awk -F= &#39;$1~/ID_NET_NAME_ONBOARD/{print $2; exit} $1~/ID_NET_NAME_PATH/{v=$2} END{if(v) print v}&#39;)</pre><p>You can pass these parameters via the kernel cmdline. Use ip= parameter to configure the network using the <a href="https://cateee.net/lkddb/web-lkddb/IP_PNP.html">Kernel level IP configuration</a> mechanism for this. This method lets the kernel automatically set up interfaces and assign IP addresses during boot, based on information passed through the kernel cmdline. It’s a built-in kernel feature enabled by the CONFIG_IP_PNP option. In Talos Linux, this feature is enabled by default. All you need to do is provide a properly formatted network settings in the kernel cmdline.</p><ul><li>You can find proper syntax for this option in the <a href="https://www.talos.dev/latest/talos-guides/install/bare-metal-platforms/network-config/#kernel-command-line">Talos Linux documentation</a>.</li><li>Also <a href="https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt">official Linux kernel documentation</a> provides more detailed examples.</li></ul><p>Set the CMDLINE variable with the ip option that contains the current system’s settings, and then print it out:</p><pre>CMDLINE=&quot;init_on_alloc=1 slab_nomerge pti=on console=tty0 console=ttyS0 printk.devkmsg=on talos.platform=metal ip=${IP}::${GATEWAY}:${NETMASK}::${DEV}:::::&quot;<br>echo $CMDLINE</pre><p>The output should look something like:</p><pre>init_on_alloc=1 slab_nomerge pti=on console=tty0 console=ttyS0 printk.devkmsg=on talos.platform=metal ip=10.0.0.131::10.0.0.1:255.255.255.0::eno2np0:::::</pre><p>Verify that everything looks correct, then load our new kernel:</p><pre>kexec -l /tmp/vmlinuz --initrd=/tmp/initramfs.xz --command-line=&quot;$CMDLINE&quot;<br>kexec -e</pre><p>The first command loads the Talos kernel into RAM, the second command switches the current system to this new kernel.</p><p>As a result, you’ll get a running instance of Talos Linux with networking configured. However it’s currently running entirely in RAM, so if the server reboots, the system will return to its original state (by loading the OS from the hard drive, e.g., Ubuntu).</p><h3>Applying machine-config and installing Talos Linux on disk</h3><p>To install Talos Linux persistently on the disk and replace the current OS, you need to apply a machine-config specifying the disk to install. To configure the machine, you can use either the official <a href="https://www.talos.dev/latest/learn-more/talosctl/"><em>talosctl</em></a> utility or the <a href="https://github.com/cozystack/talm"><em>Talm</em></a>, utility maintained by the Cozystack project (Talm works with vanilla Talos Linux as well).</p><p>First, let’s consider configuration using <em>talosctl</em>. Before applying the config, ensure it includes network settings for your node; otherwise, after reboot, the node won’t configure networking. During installation, the bootloader is written to disk and does not contain the ip option for kernel autoconfiguration.</p><p>Here’s an example of a config patch containing the necessary values:</p><pre># node1.yaml<br>machine:<br>  install:<br>    disk: /dev/sda<br>  network:<br>    hostname: node1<br>    nameservers:<br>    - 1.1.1.1<br>    - 8.8.8.8<br>    interfaces:<br>    - interface: eno2np0<br>      addresses:<br>      - 10.0.0.131/24<br>      routes:<br>      - network: 0.0.0.0/0<br>        gateway: 10.0.0.1</pre><p>You can use it to generate a full machine-config:</p><pre>talosctl gen secrets<br>talosctl gen config --with-secrets=secrets.yaml --config-patch-control-plane=@node1.yaml &lt;cluster-name&gt; &lt;cluster-endpoint&gt;</pre><p>Review the resulting config and apply it to the node:</p><pre>talosctl apply -f controlplane.yaml -e 10.0.0.131 -n 10.0.0.131 -i</pre><p>Once you apply controlplane.yaml, the node will install Talos on the /dev/sda disk, overwriting the existing OS, and then reboot.</p><p>All you need now is to run the bootstrap command to initialize the etcd cluster:</p><pre>talosctl --talosconfig=talosconfig bootstrap -e 10.0.0.131 -n 10.0.0.131</pre><p>You can view the node’s status at any time using dashboard commnad:</p><pre>talosctl --talosconfig=talosconfig dashboard -e 10.0.0.131 -n 10.0.0.131</pre><p>As soon as all services reach the Ready state, retrieve the kubeconfig and you’ll be able to use your newly installed Kubernetes:</p><pre>talosctl --talosconfig=talosconfig kubeconfig kubeconfig<br>export KUBECONFIG=${PWD}/kubeconfig</pre><h3>Use Talm for configuration management</h3><p>When you have a lot of configs, you’ll want a convenient way to manage them. This is especially useful with bare-metal nodes, where each node may have different disks, interfaces and specific network settings. As a result, you might need to hold a patch for each node.</p><p>To solve this, we developed <a href="https://github.com/cozystack/talm">Talm</a> — a configuration manager for Talos Linux that works similarly to Helm.</p><p>The concept is straightforward: you have a common config template with lookup functions, and when you generate a configuration for a specific node, Talm dynamically queries the Talos API and substitutes values into the final config.</p><p>Talm includes almost all of the features of <em>talosctl</em>, adding a few extras. It can generate configurations from Helm-like templates, and remember the node and endpoint parameters for each node in the resulting file, so you don’t have to specify these parameters every time you work with a node.</p><p><strong>Let me show how to perform the same steps to install Talos Linux using Talm:</strong></p><p>First, initialize a configuration for a new cluster:</p><pre>mkdir talos<br>cd talos<br>talm init</pre><p>Adjust values for your cluster in values.yaml:</p><pre>endpoint: &quot;https://10.0.0.131:6443&quot;<br>podSubnets:<br>- 10.244.0.0/16<br>serviceSubnets:<br>- 10.96.0.0/16<br>advertisedSubnets:<br>- 10.0.0.0/24</pre><p>Generate a config for your node:</p><pre>talm template -t templates/controlplane.yaml -e 10.0.0.131 -n 10.0.0.131 &gt; nodes/node1.yaml</pre><p>The resulting output will look something like:</p><pre># talm: nodes=[&quot;10.0.0.131&quot;], endpoints=[&quot;10.0.0.131&quot;], templates=[&quot;templates/controlplane.yaml&quot;]<br># THIS FILE IS AUTOGENERATED. PREFER TEMPLATE EDITS OVER MANUAL ONES.<br>machine:<br>  type: controlplane<br>  kubelet:<br>    nodeIP:<br>      validSubnets:<br>        - 10.0.0.0/24<br>  network:<br>    hostname: node1<br>    # -- Discovered interfaces:<br>    # eno2np0:<br>    #   hardwareAddr:a0:36:bc:cb:eb:98<br>    #   busPath: 0000:05:00.0<br>    #   driver: igc<br>    #   vendor: Intel Corporation<br>    #   product: Ethernet Controller I225-LM)<br>    interfaces:<br>      - interface: eno2np0<br>        addresses:<br>          - 10.0.0.131/24<br>        routes:<br>          - network: 0.0.0.0/0<br>            gateway: 10.0.0.1<br>    nameservers:<br>      - 1.1.1.1<br>      - 8.8.8.8<br>  install:<br>    # -- Discovered disks:<br>    # /dev/sda:<br>    #    model: SAMSUNG MZQL21T9HCJR-00A07<br>    #    serial: S64GNG0X444695<br>    #    wwid: eui.36344730584446950025384700000001<br>    #    size: 1.9 TB<br>    disk: /dev/sda<br>cluster:<br>  controlPlane:<br>    endpoint: https://10.0.0.131:6443<br>  clusterName: talos<br>  network:<br>    serviceSubnets:<br>      - 10.96.0.0/16<br>  etcd:<br>    advertisedSubnets:<br>      - 10.0.0.0/24</pre><p>All that remains is to apply it to your node:</p><pre>talm apply -f nodes/node1.yaml -i</pre><p>Talm automatically detects the node address and endpoint from the “modeline” (a conditional comment at the top of the file) and applies the config.</p><p>You can also run other commands in the same way without specifying node address and endpoint options. Here are a few examples:</p><p>View the node status using the built-in dashboard command:</p><pre>talm dashboard -f nodes/node1.yaml</pre><p>Bootstrap etcd cluster on node1:</p><pre>talm bootstrap -f nodes/node1.yaml</pre><p>Save the kubeconfig to your current directory:</p><pre>talm kubeconfig kubeconfig -f nodes/node1.yaml</pre><p>Unlike the official <em>talosctl</em> utility, the generated configs do not contain secrets, allowing them to be stored in git without additional encryption. The secrets are stored at the root of your project and only in these files: secrets.yaml, talosconfig, and kubeconfig.</p><h3>Summary</h3><p>That’s our complete scheme for installing Talos Linux in nearly any situation. Here’s a quick recap:</p><ol><li>Use <em>kexec</em> to run Talos Linux on any existing system.</li><li>Make sure the new kernel has the correct network settings, by collecting them from the current system and passing via the ip parameter in the <em>cmdline</em>. This lets you connect to the newly booted system via the API.</li><li>When the kernel is booted via <em>kexec</em>, Talos Linux runs entirely in RAM. To install Talos on disk, apply your configuration using either <em>talosctl</em> or Talm.</li><li>When applying the config, don’t forget to specify network settings for your node, because on-disk bootloader configuration doesn’t automatically have them.</li><li>Enjoy your newly installed and fully operational Talos Linux.</li></ol><h3>Additional materials</h3><ul><li><a href="https://kubernetes.io/blog/2024/11/21/dynamic-kubernetes-api-server-for-cozystack/">How we built a dynamic Kubernetes API Server for the API Aggregation Layer in Cozystack</a></li><li><a href="https://kubernetes.io/blog/2024/04/05/diy-create-your-own-cloud-with-kubernetes-part-1/">DIY: Create Your Own Cloud with Kubernetes</a></li><li><a href="https://blog.aenix.io/cozystack-becomes-a-cncf-sandbox-project-3702b8906971">Cozystack Becomes a CNCF Sandbox Project</a></li><li><a href="https://www.youtube.com/watch?v=uhXujtTzG44">Journey to Stable Infrastructures with Talos Linux &amp; Cozystack | Andrei Kvapil | SREday London 2024</a></li><li><a href="https://www.youtube.com/watch?v=9CIMTum9bTA">Talos Linux: You don’t need an operating system, you only need Kubernetes / Andrei Kvapil</a></li><li><a href="https://www.youtube.com/watch?v=4RVe32xRITo">Comparing GitOps: Argo CD vs Flux CD, with Andrei Kvapil | KubeFM</a></li><li><a href="https://www.youtube.com/watch?v=s79VqXu-eG4">Cozystack on Talos Linux</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c652b35b902e" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/a-simple-way-to-install-talos-linux-on-any-machine-with-any-provider-c652b35b902e">A Simple Way to Install Talos Linux on Any Machine, with Any Provider</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cozystack Becomes a CNCF Sandbox Project]]></title>
            <link>https://blog.aenix.io/cozystack-becomes-a-cncf-sandbox-project-3702b8906971?source=rss-d8a829bb74d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3702b8906971</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[cncf]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Andrei Kvapil]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 12:07:17 GMT</pubDate>
            <atom:updated>2025-03-13T12:07:17.197Z</atom:updated>
            <content:encoded><![CDATA[<p>On February 28, members of the CNCF Technical Oversight Committee <a href="https://github.com/cncf/sandbox/issues/322">completed their voting</a> and unanimously accepted <a href="https://cozystack.io/">Cozystack</a>, a platform for building private clouds and PaaS, into the CNCF Sandbox. The project is currently undergoing the <a href="https://github.com/cncf/sandbox/issues/351">onboarding process</a>. Let’s break down what this means in practice, what Cozystack is, and what the CNCF Sandbox represents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*9fPSDNGw-DholkjtUfkSMQ.png" /></figure><h3>What is Cozystack?</h3><p>Cozystack is an open-source platform that enables the creation of a bare metal cloud for deploying proven cloud-native and open-source tools: managed Kubernetes clusters, databases as a service, applications as a service, and virtual machines based on KubeVirt (see the <a href="https://cozystack.io/docs/components/">full list of components</a>). Cozystack also provides a ready-made stack for observability and alerting based on Victoria Metrics, Victoria Logs, Grafana, and Alerta.</p><p>Service and hosting providers, banks, SaaS solution vendors, medtech, fintech, AI/ML services, and other companies use Cozystack to offer customers managed services, managed Kubernetes, and databases that run directly on hardware, ensuring maximum performance and service stability. Additionally, Cozystack can be used to build geo-distributed clusters.</p><p>The platform is developed and maintained by <a href="http://aenix.io/">Ænix</a>. The core developer and creator of Cozystack is Andrey Kvapil, known in the engineering community by the nickname “kvaps.” He is an active contributor to Linstor, KubeVirt, Kamaji, Kubernetes, Cilium, and others.</p><h3>What is CNCF Sandbox, and What Does It Mean for Users?</h3><p>CNCF (Cloud Native Computing Foundation) is part of the larger Linux Foundation, a non-profit organization that supports and curates promising cloud-native projects, including Kubernetes, Envoy, Prometheus, Cilium, Istio, K3s, FluxCD, and others. The CNCF Sandbox is the “entry point” for projects that want to join the CNCF and become part of it. From there, projects progress through the stages of Incubating and Graduated.</p><p>Transferring the project to the CNCF guarantees all Cozystack users that the platform will always be available under the Apache 2.0 license and will not suffer the fate of projects like Mongo, Redis, Terraform, and Vault, whose licenses were changed to closed-source and no longer comply with the <a href="https://opensource.org/">Open Source Initiative</a> criteria. From this point forward, the rights to Cozystack belong to the non-profit industry organization, the CNCF.</p><p>Moreover, inclusion in the CNCF provides an opportunity to engage a broad engineering community in the development and use of Cozystack, making project management more transparent. Expanding the base of contributors and users will, in turn, significantly accelerate the platform’s development and the exploration of a wide range of use cases.</p><blockquote>Andrey Kvapil, CEO of Ænix and creator of Cozystack:</blockquote><blockquote>“I believe in honest and genuine open source, in the tools we use to build the platform, and I am happy that we can be useful to the community. In just one year, our small team of excellent engineers, with the support of our clients and the open-source community, has created a project worthy of inclusion in the CNCF. This is truly a significant achievement. Thank you to everyone who believed in us and supported us throughout this time. We will continue to improve the platform and plan to apply for CNCF Incubating status this fall. From an engineering perspective, we are already a mature project and ready for this. The main task now is to refine the project management process and community interaction.”</blockquote><blockquote>Matthew Robin, CEO &amp; Founder of Hidora:</blockquote><blockquote><em>“Cozystack represents a significant advancement in simplifying complex cloud infrastructure deployment. Its integration into the CNCF Sandbox marks an important milestone that will accelerate its adoption and enrich its ecosystem through community collaboration. We are confident that Cozystack will play a key role in democratizing cloud-native technologies for businesses of all sizes.</em>”</blockquote><blockquote>Kingdon Barrett, FluxCD &amp; Cozystack maintainer:</blockquote><blockquote><em>“Cozystack combines cutting edge open source cloud native technologies in a way that I could probably set up myself, the hard way, if I wanted to spend 6 months figuring out how they fit together. It’s Talos Linux configured as a cloud on bare metal, ready in half an hour or so!”</em></blockquote><p>Ænix will continue to actively develop the platform and support both clients and community users.</p><h3>Useful Links:</h3><ul><li><a href="https://cozystack.io/">Cozystack Website</a></li><li><a href="https://github.com/cozystack">GitHub</a></li><li><a href="https://t.me/cozystack">Telegram Community</a></li><li><a href="https://slack.k8s.io/">Slack Community</a> (registration in the <a href="https://communityinviter.com/apps/kubernetes/community">Kubernetes Slack workspace</a> required)</li><li><a href="https://calendar.google.com/calendar/embed?src=cozystack.io">Community Meeting Calendar</a></li><li><a href="https://youtube.com/cozystack">Cozystack Community Meetings Recordings</a></li><li><a href="https://landscape.cncf.io/">Cozystack on CNCF Landscape</a></li><li><a href="https://www.cncf.io/sandbox-projects/">Cozystack in CNCF Sandbox</a></li><li><a href="https://devstats.cncf.io/">Cozystack on Devstat</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3702b8906971" width="1" height="1" alt=""><hr><p><a href="https://blog.aenix.io/cozystack-becomes-a-cncf-sandbox-project-3702b8906971">Cozystack Becomes a CNCF Sandbox Project</a> was originally published in <a href="https://blog.aenix.io/">Ænix</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>