<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nine Internet Solutions AG on Medium]]></title>
        <description><![CDATA[Stories by Nine Internet Solutions AG on Medium]]></description>
        <link>https://medium.com/@nine-internet-solutions?source=rss-4ce903f38443------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k_qVY8zJZpVEDN1t86MNqQ.png</url>
            <title>Stories by Nine Internet Solutions AG on Medium</title>
            <link>https://medium.com/@nine-internet-solutions?source=rss-4ce903f38443------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 18:39:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nine-internet-solutions/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Same Ingress, Different Defaults: What Changed When We Moved to HAProxy]]></title>
            <link>https://nine-internet-solutions.medium.com/same-ingress-different-defaults-what-changed-when-we-moved-to-haproxy-f142afb26c00?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/f142afb26c00</guid>
            <category><![CDATA[https]]></category>
            <category><![CDATA[nginx-ingress]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[paas]]></category>
            <category><![CDATA[ingress]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 08:31:01 GMT</pubDate>
            <atom:updated>2026-10-08T08:31:01.606Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gj68Jhld8DJXvMH2.jpg" /></figure><p>Over the last few weeks, we moved the customer-facing managed services on our <a href="https://nine.ch/en/products/kubernetes/">NKE</a> service clusters and <a href="https://nine.ch/en/products/deploio/">Deploio</a> from ingress-nginx to HAProxy: Grafana, Loki, ArgoCD, the container registry, Alertmanager, Tempo, OpenSearch, the Object Storage endpoints and all Deploio applications. Some clusters alone had more than 200 Ingresses.</p><p>We have written about the motivation twice before. <a href="https://nine.ch/en/blog/retirement-of-the-ingress-nginx-project/">In December</a>, we explained what the retirement of ingress-nginx meant and which migration paths were open. <a href="https://nine.ch/en/blog/techtalk-29-our-ingress-odyssey/">Our TechTalk #29 recap</a> covered our evaluation of Traefik, Gateway API and HAProxy, and why HAProxy became our default for classic Ingress on NKE.</p><p>This post is about what happened after that decision.</p><p>The <a href="https://docs.nine.ch/docs/managed-kubernetes/nke/ingress">annotation mapping table in our documentation</a> gets you a working Ingress on HAProxy in a few minutes. What it does not show is how the defaults shift underneath your workloads. Two ingress controllers can parse the same Ingress resource and still differ in how they talk to your backends, which status codes they send, how long they wait and how many connections they accept before refusing new ones.</p><p>We found most of these differences in production, after the switch. One of them broke a customer’s website asset delivery. We are sharing them so you can check your own setup before you migrate.</p><h3>HTTP to HTTPS Redirects</h3><p>While comparing responses from both controllers, we noticed that plain HTTP requests were redirected with 302 Found on HAProxy, whereas nginx returned 308 Permanent Redirect.</p><p>ingress-nginx sets http-redirect-code to 308 by default. haproxy-ingress defaults its corresponding ssl-redirect-code to a temporary 302. Browsers follow both, but API clients, search crawlers and intermediate caches treat them differently. A 302 allows clients to rewrite a POST into a GET on redirect, whereas a 308 preserves the request method and body.</p><p>A silent downgrade to temporary redirects can break API consumers, so we set ssl-redirect-code: &quot;308&quot; globally on our managed controllers to keep the previous behaviour.</p><h3>Connection Limits</h3><p>During peak traffic, requests to Deploio applications slowed down and some new connections were refused.</p><p>The cause was the connection limit. nginx and HAProxy calculate their ceilings differently:</p><ul><li>nginx multiplies worker_processes by worker_connections. With ingress-nginx setting worker_processes to the node&#39;s CPU count and worker_connections to 16384, even a modest node can handle tens of thousands of concurrent connections.</li><li>HAProxy enforces a single global maxconn, which haproxy-ingress exposes as max-connections and sets to 2,000 per pod by default.</li></ul><p>Once a pod reaches that ceiling, HAProxy stops accepting new connections. They then queue in the kernel’s accept backlog, which caused the slowdown. Once the backlog is full, they are dropped.</p><p>We raised max-connections to 20,000 across all managed controllers.</p><h3>Timeouts</h3><p>Shortly after we migrated Deploio, some customers saw requests fail with 504 Gateway Timeout.</p><p>haproxy-ingress defaults timeout-client and timeout-server to 50 seconds, while ingress-nginx waits 60 seconds. Ten seconds make no difference for typical web requests, but data exports or reports that reliably finished in 55 seconds suddenly failed with a 504.</p><p>To avoid cutting off requests that used to succeed, our managed controllers set both timeout-client and timeout-server to 60 seconds. You can still override the timeout per Ingress with haproxy-ingress.github.io/timeout-server in place of nginx.ingress.kubernetes.io/proxy-read-timeout.</p><h3>Upstream HTTP Version</h3><p>In July, we moved an S3-compatible <a href="https://nine.ch/en/products/objectstorage/">Object Storage</a> endpoint to HAProxy. A customer served website assets through their own nginx reverse proxy in front of that endpoint. Right after the cutover, every asset request failed with 400 Bad Request.</p><p>The cause: nginx’s proxy_pass talks to upstreams over <strong>HTTP/1.0</strong> by default. ingress-nginx always spoke <strong>HTTP/1.1</strong> to backends, regardless of the client&#39;s protocol, and so it silently upgraded the customer&#39;s HTTP/1.0 requests. HAProxy forwards the request with the protocol version the client sent. The Object Storage backend rejected HTTP/1.0 requests, which produced the 400 errors.</p><p>The fix was for the customer to add proxy_http_version 1.1 to their reverse proxy configuration:</p><pre>location / {<br>    proxy_pass http://storage-endpoint;<br>    proxy_http_version 1.1;<br>}</pre><p>If you run a reverse proxy in front of an NKE or Deploio Ingress, check which HTTP version it sends.</p><h3>Basic Auth</h3><p>After we had migrated most managed services, one of the controllers spiked to more than two full CPU cores while serving only 155 concurrent connections. Almost all traffic to that pod hit two endpoints protected by HTTP basic authentication, one of them our metrics ingestion pipeline.</p><p>HAProxy verifies basic authentication via crypt(3) synchronously on every request, without caching. A bcrypt hash with a cost factor of 10, the default in most standard libraries and in Helm&#39;s htpasswd template function, takes about 50 ms of CPU time to verify. At 50 ms per check, twenty requests per second are enough to saturate a whole CPU core.</p><p>This also applies to nginx, as its auth_basic module verifies every request synchronously and caches nothing either. Because each nginx worker is a single-threaded event loop, a 50 ms verification stalls every connection on that worker. But workers are separate processes, so a controller with eight workers can verify eight passwords in parallel.</p><p>HAProxy uses the thread-safe crypt_r(3) only on glibc and FreeBSD. On every other platform, including the Alpine-based image that haproxy-ingress ships, it wraps the plain crypt(3) call in a single process-wide spinlock. Only one thread can verify a password at a time, which caps the whole pod at roughly twenty verifications per second, no matter how many cores it has. And because it is a spinlock, waiting threads do not sleep. They keep their cores busy until it is their turn. That is how a pod serving 155 connections ended up using more than two cores.</p><p>The <a href="https://docs.haproxy.org/3.0/configuration.html#3.4-user">HAProxy manual</a> warns that hashed passwords “can quickly become a major factor in HAProxy’s overall CPU consumption” and notes that musl’s hash implementations are slower than glibc’s. An upstream request to cache successful basic authentication results has been open since 2019 in <a href="https://github.com/haproxy/haproxy/issues/370">haproxy/haproxy#370</a>.</p><p>For credentials we generate internally, we reduced the bcrypt cost to 4, where verification takes under one millisecond. This is safe because those passwords are long, machine-generated strings of 20 random characters, which makes brute-force attacks computationally infeasible regardless of the cost factor. Helm’s built-in htpasswd function always hashes with bcrypt&#39;s default cost of 10 and offers no way to change it ( <a href="https://github.com/Masterminds/sprig/blob/v3.3.0/crypto.go#L61">Sprig source</a>), so chart-generated credentials need to be hashed outside Helm.</p><p>The annotation syntax is simpler on HAProxy. Setting haproxy-ingress.github.io/auth-secret alone enables authentication, auth-type is deprecated. The secret format stays the same: an htpasswd file stored under the auth key.</p><h3>X-Forwarded-For</h3><p>On paper, this looks like a change in behaviour. In practice it is not one, but it does add a header.</p><p>ingress-nginx replaced any incoming X-Forwarded-For with the connecting IP unless the cluster-wide compute-full-forwarded-for option was enabled, and on our shared controllers it was off.</p><p>HAProxy’s option forwardfor on its own behaves differently: it appends the connecting IP, so if the client already sent an X-Forwarded-For, the backend receives both. Applications behind a CDN or reverse proxy would then see the whole proxy chain instead of the single value ingress-nginx gave them.</p><p>haproxy-ingress closes that gap. Its forwardfor key defaults to add, which deletes any incoming X-Forwarded-For before HAProxy inserts the connecting IP. Your application therefore sees the same as before: a single address, the one that connected directly to the ingress, for example your CDN.</p><p>What is new is one extra header. If the request already carried an X-Forwarded-For, haproxy-ingress copies its original value to X-Original-Forwarded-For before discarding it:</p><pre>X-Forwarded-For: &lt;cdn-or-proxy-ip&gt;<br>X-Original-Forwarded-For: &lt;whatever-the-client-sent&gt;</pre><p>On its own, X-Original-Forwarded-For is untrusted input. It is copied verbatim from whoever connected to the ingress, and anyone can put anything into it.</p><p>It becomes useful once your application checks who connected. X-Forwarded-For holds the address that opened the TCP connection to HAProxy, which the client cannot forge. If that address belongs to a CDN you trust, for example one of the published edge IPs of your provider, then X-Original-Forwarded-For is the header that CDN sent. Its rightmost entry is the address the CDN saw connecting to it.</p><h3>HSTS Headers</h3><p>haproxy-ingress defaults its Strict-Transport-Security header to max-age=15768000 (six months). ingress-nginx sent max-age=31536000 (one year).</p><p>RFC 6797 leaves the duration to the operator, and the two projects measure against different yardsticks. Six months is the minimum <a href="https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide">Qualys SSL Labs</a> has required for an A+ grade since January 2014. HAProxy’s own <a href="https://www.haproxy.com/blog/haproxy-and-http-strict-transport-security-hsts">documentation examples</a> use a value just above it, max-age=16000000, described as &quot;a bit more than 6 months&quot;. When HAProxy published those examples, the <a href="https://hstspreload.org/">HSTS preload list</a> accepted a max-age shorter than one year. It has since raised the minimum to one year.</p><p>For standard web traffic, this has no functional impact. If your domain is on the HSTS preload list, however, six months fails validation. We therefore override the haproxy-ingress default and send max-age=31536000 (one year), the same value ingress-nginx did.</p><p>If you need a different duration, set spec.forProvider.hsts.maxAge on the IngressHAProxy resource (a duration, for example 8760h). For a single Ingress, set haproxy-ingress.github.io/hsts-max-age (in seconds).</p><h3>Request Buffering</h3><p>Under ingress-nginx, high-volume write endpoints produced warnings like this in the controller logs:</p><pre>[warn] a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000123 request: &quot;POST /loki/api/v1/push HTTP/1.1&quot;</pre><p>A while back, one node in our service cluster ran at 98% CPU, with its nginx pod using 1.4 cores mainly to write and clean up temporary disk buffers. On nginx, the fix was to set nginx.ingress.kubernetes.io/proxy-request-buffering: &quot;off&quot; on write-heavy Ingresses such as Loki, Promscale and container registries.</p><p>HAProxy streams request bodies to backends by default. There is no equivalent buffering annotation, because HAProxy does not write incoming payloads to disk.</p><h3>Where This Leaves You</h3><p>If you still use our managed ingress-nginx on NKE: as announced, support ended on <strong>1 October 2026</strong>. HAProxy is available for every NKE cluster, and our <a href="https://docs.nine.ch/docs/managed-kubernetes/nke/ingress">migration documentation</a> is the place to start.</p><p>If you are unsure how one of these differences affects your workloads, write to us at <a href="/service/mailto:support@nine.ch">support@nine.ch</a> and we will look at it with you.</p><p><em>Originally published at </em><a href="https://nine.ch/en/blog/haproxy-defaults-after-ingress-nginx-migration/"><em>https://nine.ch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f142afb26c00" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ubuntu Resolute: Evolution Instead of Revolution]]></title>
            <link>https://nine-internet-solutions.medium.com/ubuntu-resolute-evolution-instead-of-revolution-47def2b42a43?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/47def2b42a43</guid>
            <category><![CDATA[resolute]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[implementation]]></category>
            <category><![CDATA[releases]]></category>
            <category><![CDATA[nine]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 11:54:49 GMT</pubDate>
            <atom:updated>2026-09-29T11:54:49.993Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ixXbxyZLsgx523EgUHQB7Q.png" /></figure><p>With the 2026 Ubuntu LTS release, we’re seeing evolution instead of revolution across the board. Mostly small version updates, a recent kernel in version 7, and many updates under the hood.</p><p>An overview of available software versions for the most commonly used services can be found in <a href="https://docs.nine.ch/docs/managed-server-services/operating-system-and-software-versions/which-software-versions-are-available-on-my-server">our documentation</a>.</p><h3>What Does It Take to Implement a New Ubuntu Version?</h3><p>Have you ever asked yourself what it takes us to implement a new Ubuntu LTS version in our ecosystem? It’s a little more complicated than imaging a USB stick. We’ll take you through our implementation journey.</p><h3>An Early Look</h3><p>On April 23, 2026, Ubuntu Resolute was publicly released, with a “Beta” preview a month prior.</p><p>Shortly after the Beta release, we had a first look at the new release. With clarity about the used Ruby version, we knew what it would take to update our self-service tools such as nine-manage-vhosts, nine-manage-databases, and FTPAdmin2. We also run some tools not directly used by customers but of importance to us and our automation landscape.</p><p>Following our first hands-on, we created the backlog items we considered mandatory for the implementation.</p><h3>Step by Step</h3><p>The first issue to solve was around the package manager and package sources. With some deprecations in apt 3.1, updates for our automation around our package mirrors (we mirror the Canonical repositories for all Ubuntu versions) as well as package sources became necessary.</p><p>After addressing these first issues, we needed to build a “packagebuilder” image for the new Ubuntu version. This container image is used in automated pipelines to build Debian packages like nine-manage-vhosts and nine-manage-databases, but also Prometheus exporters.</p><p>Prometheus is used for monitoring at Nine, and exporters are the metrics source our alerting is based on. We build more than 20 exporters that aren’t uniformly designed and need customisations in their build processes.</p><p>To test these packages in a “real” system environment (as opposed to a container, where things like systemd are missing), we built a VirtualBox VM image of Ubuntu Resolute, which we use to run tests of our self-built packages.</p><p>Once we had finished building our automation and monitoring tools, we could dive deeper into the automation of our Managed Services.</p><p>With Ubuntu Resolute, many services got updated systemd units and AppArmor profiles. This improves security but also required some changes and tests, for example to our Apache and MariaDB automation.</p><p>One of the changes to systemd resulted in /tmp partitions or folders that are no longer backed by system storage. Instead, /tmp uses a tmpfs partition type, which uses system memory to store data.</p><p>System memory is valuable, and many applications and services use /tmp for short-lived data that might require quite some storage. A tmpfs partition doesn&#39;t meet our requirements for a server environment, so we reverted to a storage-based solution.</p><p>Most of these tasks are independent of the type of system, virtual or dedicated. But of course, we also needed to adapt our automated dedicated server provisioning for Ubuntu Resolute.</p><h3>A Little Revolution</h3><p>In the past years, the companies behind Elasticsearch and Redis have tried to further commercialise their open-source services and changed the licensing model of their software.</p><p>Although both companies have since returned to open-source licenses, rebuilding lost trust is difficult. In the meantime, the open-source ecosystem has continued to evolve.</p><p>In our offering, Elasticsearch has since been superseded by OpenSearch.</p><p>After Redis announced a license change in 2024, we thought we were prepared, as we had already implemented KeyDB as an alternative a year before. What looked promising at first (e.g. Redis compatibility, the possibility to run multi-primary replication) has turned out to be a dead end: the KeyDB project has since been abandoned.</p><p>In 2024, following the license change, another Redis fork named Valkey was released. Broad support from industry leaders such as AWS, Google Cloud, Percona, Oracle, and more has helped Valkey gain momentum quickly and establish itself as the de facto alternative to Redis.</p><p>With Ubuntu Resolute, Valkey 9 has found its way into the Ubuntu “main” repository. The main repository contains software that’s supported by Canonical during the entire lifecycle of the Ubuntu release.</p><p>As Valkey in version 9 is fully compatible with Redis 7.2.4, and Redis remains in the “universe” repository (a “community maintained” repository), the decision to use Valkey from Ubuntu Resolute onward was an easy one for us: fully compatible with the latest Redis version we offered (7.0 in Ubuntu Noble), but actively supported throughout all lifecycle phases of Ubuntu.</p><h3>Engineering Summary</h3><p>This implementation journey doesn’t just provide a broad overview of some technical challenges, it also gives an idea of how Nine engineers work and which tools and processes we use.</p><p>Many of the backlog items iteratively built on each other. Besides the engineering and quality control effort, we also need to ensure that changes don’t cause side effects on older Ubuntu versions, since all versions are automated through the same codebase.</p><p>After 76 merge requests linked to 20 backlog items, the first Ubuntu Resolute systems were delivered to our customers on June 29, 2026.</p><p><em>Originally published at </em><a href="https://nine.ch/en/blog/ubuntu-resolute/"><em>https://nine.ch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47def2b42a43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three Days in August: What a DDoS Attack Exposed in Our Network]]></title>
            <link>https://nine-internet-solutions.medium.com/three-days-in-august-what-a-ddos-attack-exposed-in-our-network-d945d89db6a5?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/d945d89db6a5</guid>
            <category><![CDATA[ddos]]></category>
            <category><![CDATA[ddos-mitigation]]></category>
            <category><![CDATA[ddos-protection]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 11:48:54 GMT</pubDate>
            <atom:updated>2026-09-29T11:48:54.582Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T4JURqrOFqotwm78.jpg" /></figure><p>On the evening of 11 August 2026, one of our customers became the target of one of the largest attacks we have ever seen directed at our infrastructure. Because we provide that customer’s connection to the internet, the impact hit us directly as well, and a few hours later the attacker turned against our own services too. We already published a technical <a href="https://nine.ch/uploads/Postmortem_DDoS-attack_August-2026_EN.pdf">postmortem as a PDF</a> on this incident. In this post, we want to walk through what actually happened over those three days in more depth, why handling an attack of this size took as long as it did, and what we have changed in our infrastructure since.</p><h3>An attack that outsized our own capacity</h3><p>To put the scale into perspective: we estimate a peak volume of 500 to 600 Gbit/s hitting our network and the affected customer across all our links combined. Two of our upstream providers confirmed 260 Gbit/s of that directly. That is several times more than our own uplinks can carry in the first place, no matter how well the defences inside our own network are set up.</p><p>Over the course of the incident, different services were affected at different times: every application on Deploio, our own website, Cockpit and our ticketing system. The word “different times” matters here. This was not a single, continuous 42 hour outage, it was a series of waves, with normal operation in between. From Tuesday evening at around 19:15 to Thursday around 12:51, these waves kept recurring, each time with shifting targets and intensity.</p><p>One thing was never in question throughout: the safety of your data. There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.</p><h3>Confirmed by outside telemetry</h3><p>Independent telemetry from Nokia’s Deepfield threat research team backs up our own read of events. Their sensors are registered as bots on the same botnet command-and-control (C2) servers the attackers used, so they can log the actual attack commands as they go out, not just the traffic that later lands on a target. That data shows two botnet families involved, CECbot and Katana, and confirms the sequence we saw ourselves: the attack first targeted our customer directly, and only broadened to Nine’s own infrastructure roughly three hours later, exactly what you would expect if we were hit because we provide that customer’s internet connection, not because we were the actual target.</p><p>It also surfaced a detail worth calling out on its own: our customer announces its address block through two different providers, us and one other, so anyone trying to filter or attribute this attack by looking at our network alone would only ever see half the picture. Nokia’s data does not attribute the attack to a specific actor, and neither do we.</p><h3>How an attack like this actually works</h3><p>The attackers used a technique called UDP amplification. In simple terms: you send small requests to open services on the internet, but instead of using your own address as the sender, you spoof the victim’s address. The response, which is many times larger than the original request, then lands not with the attacker but directly with the target. A relatively small amount of the attacker’s own bandwidth turns into a massive attack volume, spread across tens of thousands of sources worldwide. If you want the mechanics in detail, CISA covers the method thoroughly under <a href="https://www.cisa.gov/news-events/alerts/2014/01/17/udp-based-amplification-attacks">“UDP-Based Amplification Attacks”</a>.</p><p>The point that matters for everything that follows: once your own internet uplink is full, filtering inside your own network no longer helps. The packets you would want to filter out have already clogged the line by the time they reach your filter, along with all the legitimate traffic trying to use the same line. From that point on, mitigation has to happen further out, at the providers that carry the traffic to you in the first place.</p><p>The tool for that is blackholing. We ask our upstream providers to stop delivering all traffic destined for a specific address to us entirely and drop it instead. That reliably protects the whole network and every customer who has nothing to do with the attack. The cost: the affected address becomes deliberately unreachable during that time, for everyone, including genuine visitors. That was, in many cases during this incident, the actual reason an application was down. Not because something was broken, but because we sacrificed that one address to keep everything else stable.</p><p>Making things harder, the attacker did not back off. When we moved our own website to a new IP address, that new address was under attack again within minutes. And it was not just about raw volume: the sheer number of requests was enough on its own to overload individual services, regardless of bandwidth. That called for a different tool than blackholing, one that can tell legitimate traffic apart from malicious traffic instead of blocking both indiscriminately.</p><h3>The way back to availability</h3><p>This is where the measure that ultimately ended the incident came in. Once it was clear that blackholing alone would not hold against an attacker also targeting us by sheer request volume, we started moving our exposed applications behind bunny.net, a European CDN with built in DDoS protection, one day into the incident, and had the migration complete by the next day. Our customer evaluated and rolled out additional protection measures on their end at the same time. Since then, attack traffic terminates at the protection provider, while legitimate requests still reach us.</p><p>One thing that concretely helped us during those three days: because we operate our own network, we could make routing changes directly while the attack was still ongoing, and coordinate the additional protection measures without waiting on changes to some underlying, third-party network infrastructure. That is part of why an attack of this magnitude did not turn into a multi-day total outage.</p><h3>Three gaps we found</h3><p>We found three concrete gaps during this incident, and we would rather be upfront about them than gloss over them.</p><p>First, our automated attack detection was scoped only to our own addresses. Networks that we announce on the internet on a customer’s behalf, but that belong to the customer, were not part of that automation. The very first wave hit exactly such a network, which is why it took around 90 minutes of manual mitigation to contain it. We closed that gap the same night.</p><p>Second, we had never systematically verified whether our blackhole signal actually takes effect on every single path our traffic travels. That signal only applies for peers with whom blackholing was already configured in advance, or for traffic reaching us through an internet exchange’s route servers. At the internet exchanges we connect to, we have no such agreement with any of our direct peers, so blackholing there only covers traffic sent through the route servers, not traffic exchanged directly. On top of that, we did not yet have a way to withhold a specific network from just one specific provider, so we had to build that capability while the attack was still ongoing, under full load. It worked, but building a tool in the middle of an emergency is always slower and riskier than already having it ready.</p><p>Third, and this was the most uncomfortable consequence for many customers: our own website runs on the same Deploio platform as your applications. When the attack targeted our website, unrelated applications on the same platform were automatically affected too. And because Cockpit and our ticketing system were later attacked directly as well, some customers temporarily lost their application, the ability to manage it, and part of their way of reaching us, all at the same time, exactly when they needed us most.</p><h3>What is different now</h3><p>All three gaps are either closed or actively being worked on. Every customer network we announce is now automatically part of our attack detection, that is a fixed part of our standard process. Our exposed applications remain permanently behind bunny.net’s CDN with DDoS protection, and migrating further services is now a prepared procedure rather than something we would have to improvise under pressure. We monitor the effectiveness of our mitigations across every link now, and we work more closely with our upstream providers to do so.</p><p>And because we deliberately keep running our own services on the Deploio platform, the same one we offer our customers, rather than moving them elsewhere, we are working instead on making it more resilient. We are evolving the architecture so individual applications depend less on shared addresses. Cockpit and the ticketing system are additionally hardened, so you can reach us even during an attack. And the scenario from this incident is now a fixed part of our regular exercises, not just theory on paper.</p><h3>What you can do yourself</h3><p>This incident leads to one concrete recommendation for anyone running their own domains on <a href="https://nine.ch/products/deploio">Deploio</a>.</p><p>Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained. On top of that, other applications are often reachable via the same shared address, which is exactly the risk described above. That is primarily an architectural issue on our end, and we are working on it. Until then, a CNAME or ALIAS is the single most effective step you can take yourself.</p><p>And wherever possible, put a <a href="https://nine.ch/products/security">CDN with its own DDoS protection</a> in front of your application. That is precisely what restored our availability. For most setups the effort is manageable and the running cost is low.</p><p>If you are not sure which option fits your application, reach out to <a href="/service/mailto:support@nine.ch">support@nine.ch</a>. We are happy to work through it with you.</p><p>For the full technical sequence of events, including the exact timeline, see the <a href="https://nine.ch/uploads/Postmortem_DDoS-attack_August-2026_EN.pdf">official postmortem as a PDF</a>.</p><h3>In closing</h3><p>We cannot fully prevent an attack of this magnitude, that is simply outside our control. What we can influence is how quickly we react and how few uninvolved customers get caught in the crossfire, and that is exactly what we have continued working on since this incident. We apologise for the inconvenience during those three days, and thank you for your patience.</p><p><em>Originally published at </em><a href="https://nine.ch/en/blog/ddos-attack-august-2026-postmortem/"><em>https://nine.ch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d945d89db6a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When the Red Light Goes On: How We Responded to the Linux Kernel 0-Day]]></title>
            <link>https://nine-internet-solutions.medium.com/when-the-red-light-goes-on-how-we-responded-to-the-linux-kernel-0-day-73b768693255?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/73b768693255</guid>
            <category><![CDATA[zero-day-vulnerability]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[nine]]></category>
            <category><![CDATA[kernel]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 11:57:21 GMT</pubDate>
            <atom:updated>2026-09-16T11:57:21.686Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*dA6yNmfzOOkcYWhEN__PzQ.png" /></figure><p>There are days when the red light goes on for system engineers. April 30, 2026 was one of them.</p><p>At 8:29, an article regarding a 0-day exploit in the Linux kernel, <a href="https://cybersecuritynews.com/linux-kernel-0-day-copy-fail/">publicly disclosed on cybersecuritynews.com</a>, was shared in our internal Slack. CVE-2026–31431, nicknamed “Copy Fail”. A local privilege escalation in the Linux kernel, introduced in kernel version 4.14 in 2017. And — the cherry on top — a working exploit is already available on <a href="https://github.com/theori-io/copy-fail-CVE-2026-31431">GitHub</a>. A handful of lines of Python, and any unprivileged user is able to become root:</p><pre>www-data@test-noble03:~ $ python3 copy_fail_exp.py<br># id<br>uid=0(root) gid=33(www-data) groups=33(www-data),4(adm),112(ssl-cert)</pre><p>That simple. That fast. On a recent Ubuntu Noble (24.04). And <a href="https://ubuntu.com/security/CVE-2026-31431">no updated kernel was available yet</a> for our Ubuntu fleet at that point.</p><h3>Why This Is a Problem</h3><p>On a typical web server, the application runs as an unprivileged user, usually www-data. That is defense in depth: even if an attacker manages to exploit a vulnerability in PHP, Node, or Ruby, they only get the rights of that one user. Unix file permissions then keep them out of other customers&#39; data and away from root-owned secrets - provided those permissions are set correctly. The blast radius stays bounded.</p><p>A local privilege escalation bridges that layer. “Compromised web app” turns into “compromised server”. While the kernel is the last line of defense, and application security is the first, we need to take precautions and secure that last line of defense with the tools available to us, especially when a working exploit is already public and trivial to run.</p><p>The team also quickly verified that the exploit works not only on virtual machines or dedicated servers, but also inside a container environment. Which means that our Kubernetes worker nodes might be exposed as well. A kernel bug that hands root to an unprivileged process undermines the isolation of a shared platform like Deploio: exactly the scenario you want to avoid at all costs.</p><h3>Mitigation Instead of Waiting</h3><p>Issues of that magnitude require instant attention and action. With no patch available yet, a mitigation needed to be implemented as soon as possible. The vulnerability is an out-of-bounds write in the kernel module algif_aead, only reachable through the AF_ALG socket interface of the Crypto User API. Unload the module and prevent it from being loaded again, and the attack vector is mitigated.</p><pre>echo &quot;install algif_aead /bin/false&quot; &gt; /etc/modprobe.d/disable-algif-aead.conf<br>rmmod algif_aead 2&gt;/dev/null</pre><p>The mitigation was shared in our internal communication, tested first, then integrated into our automation and rolled out to all our Managed Servers. After the rollout, we verified the mitigation again on multiple machines, ensuring it was working as intended. We were no longer able to reproduce the privilege escalation.</p><p>From the first Slack message to a confirmed-safe state across the fleet in just about an hour.</p><h3>And Kubernetes?</h3><p>Our Kubernetes platform <a href="https://nine.ch/en/products/kubernetes">NKE</a> and <a href="https://nine.ch/en/products/deploio">Deploio</a> do not run on Ubuntu, but on <a href="https://www.flatcar.org/">Flatcar Container Linux</a>. One engineer took a look at the kernel configuration of the versions we run, on staging and production. The result: CONFIG_CRYPTO_USER_API_AEAD is disabled, and algif_aead is not even shipped as a module. Only algif_hash and algif_skcipher are present.</p><p>In simple words: the attack vector via the kernel module algif_aead simply does not exist on our Kubernetes and Deploio environments. Flatcar&#39;s maintainers had disabled CONFIG_CRYPTO_USER_API_AEAD for their own reasons, and we happened to benefit. Flatcar themselves <a href="https://github.com/flatcar/Flatcar/issues/2119">confirmed</a> they are not affected by this CVE.</p><p>We will of course keep an eye on the Flatcar release notes in case anything changes. But this morning, we got to exhale on that front.</p><h3>What I’m Taking Away From Today</h3><p><strong>Mitigations buy time.</strong> When no patch is available yet, surgically removing the attack vector — here, unloading algif_aead and blocking it from being reloaded - can bridge the gap until the upstream patch lands. It&#39;s not a substitute for patching, but it&#39;s the difference between &quot;exposed for hours&quot; and &quot;exposed for days&quot;.</p><p><strong>A great team is everything.</strong> Around half a dozen engineers were active in the Slack thread and the discussion was a textbook example: someone finds the problem, someone verifies the exploit, someone researches the mitigation, someone rolls it out, someone audits the fleet, someone tests that it actually works. No escalation and no drama, only well orchestrated engineering. Hats off to everyone involved.</p><p>If you run a <a href="https://nine.ch/en/products/root-cloud-server">Root Server</a> with us, you’re at the wheel yourself: that’s the deal. If that’s you: the mitigation above ( install algif_aead /bin/false plus rmmod) takes about a minute to apply, no reboot required. As soon as your distribution ships the kernel patch, just update and reboot.</p><p>If you’ve ever wondered what the word “Managed” in <a href="https://nine.ch/en/products/managedserver">Managed Server</a> actually means in practice: this was a perfectly normal Thursday morning.</p><p><em>Originally published at </em><a href="https://nine.ch/en/blog/linux-kernel-0day-copy-fail/"><em>https://nine.ch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73b768693255" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Heroku to Deploio: How Renuo Mastered the Switch to Swiss Cloud Hosting]]></title>
            <link>https://nine-internet-solutions.medium.com/from-heroku-to-deploio-how-renuo-mastered-the-switch-to-swiss-cloud-hosting-41eee97e8bbe?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/41eee97e8bbe</guid>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[nine]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[deploio]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 11:54:24 GMT</pubDate>
            <atom:updated>2026-09-16T11:54:24.754Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="To guys using Deploio on their laptops" src="https://cdn-images-1.medium.com/max/1024/1*5K8Ruf0aXfBx7LY0qmBgdg.png" /></figure><p>One Slack message, a seamless migration — and finally cloud hosting from Switzerland.</p><h4>The Starting Point: Heroku Was No Longer Sustainable</h4><p>Renuo AG is a Swiss software engineering company with a clear mission: write great code — preferably in Ruby on Rails. Server administration? Not their thing. For years, Heroku fulfilled exactly this purpose: push code, deploy, done.</p><p>But as an owner-managed Swiss company, something fundamental bothered Renuo: Heroku is American and runs on AWS. That means the US CLOUD Act, data outside of Switzerland, and ultimately zero control.</p><p>“We often thought about how it should be possible to use a product as cool as Heroku in Switzerland — where we can kick down doors if we don’t like something,” explains Josua Schmid, CTO of Renuo. “Good luck doing that with Salesforce.”</p><p>On top of that: The enterprise contract with Heroku was, as Schmid puts it, “a construct from hell.” Rising prices, eliminated free tiers, and increasing outages didn’t help matters.</p><h4>The Solution: Deploio — The Swiss Heroku Alternative</h4><p>Deploio wasn’t born in isolation — it emerged from two independent ideas that converged at exactly the right time. Renuo had long wanted to move away from US hosting and run their applications in Switzerland. Meanwhile, we at Nine had already been considering building an application engine on top of our existing Kubernetes infrastructure in the summer of 2022 — an additional layer to make our services more accessible.</p><p>The trigger came in August 2022, when Heroku announced a pricing change that would have nearly quadrupled costs for Renuo. On September 23, 2022, Samuel Steiner, co-founder of Renuo, posted in an entrepreneur network asking whether others were experiencing similar pain with Heroku. Tom Hug, CEO of Nine, saw the post and replied: “We don’t have the same problem, but maybe we need to talk.”</p><p>What followed was a collaboration that benefited both sides. Nine brought the entire technical foundation — our own data centers, Nutanix infrastructure, and the Nine Kubernetes Engine (NKE). Renuo brought the customer perspective. “As engineers, we often built things we thought were cool,” Tom Hug shared at <a href="https://www.youtube.com/watch?v=0q_qaIdpsD0">TechTalkThursday #18</a>. “With Renuo on board, we had someone showing us what the market actually needs.”</p><p>The go decision for the MVP was made at the end of November 2022. The team additionally validated the idea with ten other agencies — the feedback was overwhelmingly positive. From July 2023, we invited 30 beta testers, in August 2023 we migrated docs.nine.ch as the first production application to Deploio, and by autumn 2023 the platform was gradually rolled out to all customers.</p><p>The result: a PaaS solution that takes a Git URL and returns a running application with SSL certificates — on Swiss infrastructure, with crucial differences to Heroku:</p><ul><li><strong>Swiss location</strong>: All data stays in Switzerland</li><li><strong>No CLOUD Act</strong>: No US government access</li><li><strong>Personal support</strong>: Real engineers instead of anonymous ticket systems</li><li><strong>Fair pricing structure</strong>: Transparent and without hidden costs</li></ul><p>For Renuo, the migration was seamless. The team could continue focusing on what they do best: writing code. The rest — Kubernetes, bare-metal servers, kernel updates, client separation — is in our hands.</p><h4>The Result: Focus on Code, Not Infrastructure</h4><p>Being in the software business doesn’t automatically mean knowing the infrastructure business inside and out. Renuo didn’t want to run Kubernetes themselves or have anything to do with bare-metal servers.</p><p>Today, Renuo’s workflow looks like this:</p><ol><li>Develop code</li><li>CI/CD runs automatically</li><li>Customers can test</li></ol><p>We — in this case with Deploio — take care of the entire Swiss infrastructure. Renuo can fully focus on writing excellent code for their customers.</p><h4>What Renuo Says About Deploio Today</h4><blockquote><em>“Cloud yes, but preferably in Switzerland from a Swiss company with capable engineers, not just Amazon resellers — that’s what we get with Deploio. Nine takes care of the infrastructure, is located in Switzerland, and the people there do their job very well. I haven’t had to kick down any doors on Stauffacher yet.”</em></blockquote><blockquote><strong><em>- Josua Schmid, CTO Renuo AG</em></strong></blockquote><p>What started with a single Slack message has become one of our most valuable partnerships. Beyond the migration, we work continuously with Renuo to further develop Deploio — based on regular feedback from real-world developer workflows.</p><p>This partnership is also visible publicly: At <a href="https://www.youtube.com/watch?v=Cm9gFmE6dKg">TechTalkThursday #26</a>, Josua Schmid spoke about “Mental Models vs. Deploio Reality” — an honest insight into the developer experience from a customer’s perspective. And at Swiss Python Summit 2025, we appeared together with Renuo as bronze sponsors.</p><h4>Why More and More Developers Are Leaving Heroku</h4><p>Renuo’s case is not an isolated incident. Since Heroku eliminated its free tier and prices continue to rise, more and more Swiss companies are looking for alternatives. The most common reasons:</p><ul><li><strong>Data protection</strong>: GDPR compliance and Swiss data sovereignty are becoming more important</li><li><strong>Costs</strong>: Heroku is increasingly expensive without offering additional value</li><li><strong>Stability</strong>: Outages are becoming more frequent</li><li><strong>Support</strong>: Problems lead to anonymous ticket systems</li></ul><p>Deploio addresses all these points — while offering the same developer experience that Heroku users are accustomed to.</p><h4>Update February 2026: Salesforce Freezes Heroku Development</h4><p>What had been looming for months is now official: in early February 2026, Salesforce announced that Heroku is transitioning to “sustaining engineering mode.” In practice, this means no new features. The platform remains online for now and existing customers can renew their contracts — but new enterprise contracts are no longer being offered.</p><p>Salesforce is instead investing in Agentforce, its in-house AI platform. Heroku, once the gold standard for developer-friendly cloud hosting, has effectively been put on ice. Industry analysts put it bluntly: exorbitant pricing weakened Heroku and the strategic pivot to AI drove the final nail in the coffin.</p><p>For teams still running on Heroku, the question is no longer whether to migrate, but when. Renuo’s story shows that switching to Deploio can be done quickly and without disruption — with the added benefit of landing on a platform that is actively being developed further.</p><h4>Conclusion: Cloud Hosting Doesn’t Have to Be Complicated</h4><p>Renuo’s story shows that switching from Heroku to a Swiss alternative doesn’t have to be a mammoth project. With the right platform and a partner who understands developers’ needs, the migration can be seamless.</p><p>For Renuo, Deploio was exactly that: a solution that allows them to focus on their core competency — writing great code — while we take care of the infrastructure.</p><p><strong>Thinking about switching from Heroku?</strong> Deploio offers you:</p><ul><li>Swiss data sovereignty</li><li>Personal support from real engineers</li><li>Fair, transparent pricing</li><li>A migration that’s doable in 30 minutes</li></ul><p>→ <a href="https://cockpit.nine.ch/en/signup?origin=deploio">Try Deploio for free</a></p><p>→ <a href="https://nine.ch/en/case-studies/renuo-ag/">Learn more about the Renuo case study</a></p><p><em>Originally published at </em><a href="https://nine.ch/en/blog/from-heroku-to-deploio-how-renuo-mastered-the-switch-to-swiss-cloud-hosting/"><em>https://nine.ch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41eee97e8bbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vibe Deploying: How Claude Code Migrated My 8-Year-Old Angular App to Deploio]]></title>
            <link>https://nine-internet-solutions.medium.com/vibe-deploying-how-claude-code-migrated-my-8-year-old-angular-app-to-deploio-62b7f0ab5afb?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/62b7f0ab5afb</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[angular-4]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[deploio]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Thu, 12 Mar 2026 07:42:15 GMT</pubDate>
            <atom:updated>2026-03-12T07:42:15.951Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PBXVirPBT5zaAxY8.png" /></figure><p>You know the drill: There’s this server you should have updated ages ago. But there’s still that one app running on it that probably won’t work on the new server without modifications. So you leave it. Again and again.</p><p>For me, it was an Angular 4 app — the frontend for my personal accounting application. Self-built in 2017. Yes, back then there was no AI to do it for you. Since then, it’s been running beautifully, doing exactly what it’s supposed to do.</p><h3>The Failed Attempts</h3><p>Since we released <a href="https://www.nine.ch/en/products/deploio">Deploio</a> as a product in 2023, I had wanted to migrate the app. I made a few attempts. I have to admit: I hadn’t invested much time. A bit of googling error messages, copy-pasting errors into ChatGPT, etc. The typical “let me just quickly try this”. It didn’t work. The app wouldn’t compile, some dependencies were outdated, TypeScript was complaining. From experience, I knew: This is going to be a can of worms.</p><h3>The Breakthrough with Claude Code</h3><p>After the great experiences with Claude Code — whether it was a connector for Odoo 19 (we’re migrating from Odoo14 to Odoo19 soon) or the <a href="https://nine.ch/en/blog/from-wordpress-to-hugo/">migration of our website from WordPress to Hugo</a> — I thought: Why not just let Claude have a look?</p><p>What’s special about Claude Code is that it doesn’t just generate code, but runs directly in the console and can operate tools. It sees the error messages when the app doesn’t compile. It understands the context. And it fixes the errors autonomously — iteratively, until it works.</p><h3>The Secret Ingredient: nctl</h3><p>It gets even more interesting with our CLI tool <a href="https://github.com/ninech/nctl">nctl</a>. With it, Claude Code can interact directly with Deploio: create apps, view logs, monitor deployments. That’s the real game-changer.</p><p>I allowed Claude to view logs:</p><pre>nctl logs app anbeda -l 500</pre><p>And I allowed it to commit and push. The code is on GitHub, and Deploio fetches the code directly from there — with every push, it automatically redeploys.</p><h3>4–5 Iterations — Then It Worked</h3><p>The migration then ran almost by itself:</p><ol><li>Claude analyses the old code and the outdated dependencies</li><li>Creates a Procfile for the buildpack</li><li>Pushes the changes</li><li>Deploio builds and deploys</li><li>Claude checks the logs, sees the error</li><li>Fixes the error, pushes again</li><li>Repeat</li></ol><p>After 4–5 iterations: The app runs (see screenshot). On Deploio. With current Node.js. Without me having to fight through hundreds of deprecation warnings!</p><h3>Bonus: Staging Environment in Seconds</h3><p>Since you can also create new apps with nctl, I had Claude set up a staging environment for me:</p><pre>nctl create app anbeda-dev \<br>  --git-url https://github.com/thomhug/anbeda-frontend \<br>  --git-revision dev</pre><p>The staging app simply fetches the dev branch instead of main. I could have done this quickly by hand, of course - but if Claude is already at it, why not?</p><h3>The Conclusion: Vibe Deploying</h3><p>What is “Vibe Deploying”? It’s like Vibe Coding, but for infrastructure. You give the AI access to the right tools, describe the goal, and let it iterate. Errors are detected and fixed without having to debug every line yourself.</p><p>Of course, you have to be careful what you allow the AI to do. But for migration projects like this — where an old app needs to move to a new platform — it’s an enormous productivity gain. Alternatively, you can also confirm each command individually and don’t have to worry about AI hallucinations suddenly causing chaos on your infrastructure!</p><p>My server can finally be updated. The app runs on Deploio. And I barely did anything except watch.</p><p>Do you have old apps that you’ve been meaning to migrate for ages? Give it a try — with <a href="https://www.nine.ch/en/products/deploio">Deploio</a>, <a href="https://github.com/ninech/nctl">nctl</a>, and an AI assistant of your choice.</p><p><em>Originally published at </em><a href="https://nine.ch/en/blog/vibe-deploying/"><em>https://nine.ch</em></a><em> by Thomas Hug, CEO &amp; Founder.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62b7f0ab5afb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Zero to Infra: Building a Production-Ready Setup Using Our API & Ansible]]></title>
            <link>https://nine-internet-solutions.medium.com/from-zero-to-infra-building-a-production-ready-setup-using-our-api-ansible-548caebe243c?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/548caebe243c</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-server]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Tue, 03 Jun 2025 09:40:55 GMT</pubDate>
            <atom:updated>2025-08-25T05:31:12.959Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XjTqChkaYAYivxrm.jpg" /></figure><h3>📖 The Story Begins</h3><p>One common pattern we’ve observed at Nine is that many of our customers run web applications — think content management systems, web shops, dashboards, or custom APIs. These apps often rely on a tried-and-true combination of infrastructure services: a database to persist data, a cache layer for performance, and a server to run the application logic.</p><p>With that in mind, the setup we chose here reflects that real-world use case. We’re not just launching a VM or showing off an API (ok, maybe a little 😉). But we’re also simulating the foundational components of what you’d find in production:</p><ul><li>A <strong>PostgreSQL database</strong> to store orders, users, and product data</li><li>A <strong>Redis key-value store</strong> to cache hot data and reduce load times</li><li>A <strong>CloudServer VM</strong> to run the actual application (e.g. web server, backend API, etc.)</li></ul><p>The goal is to show how these can be provisioned and managed easily, using our API and tools, in a way that mirrors your own workload and challenges.</p><p>As the Head of Engineering at <a href="https://nine.ch/de/">Nine</a>, I’ve witnessed our platform evolve, providing developers with powerful tools to automate infrastructure. One of the most potent tools is our <strong>public API</strong>, giving you the ability to provision and orchestrate your infrastructure programmatically.</p><p>This blog post is not just a tutorial; it’s a story about turning a blank slate into a working, production-like environment. We will build and validate the provisioning of the following infrastructure components:</p><ul><li>A <strong>PostgreSQL database</strong> for persistent storage</li><li>A <strong>Redis key-value store</strong> for high-speed caching</li><li>A <strong>CloudServer VM</strong> that could host any application you choose</li></ul><p>Each step is written not just to instruct but to explain, giving context to every decision and command. This post assumes you’re a developer or DevOps engineer who may not yet be familiar with Nine’s API, Ansible, or our CLI tool `nctl`. We’ll guide you from first principles.</p><h3>🤖 Why Ansible?</h3><p>Before we get into the technical weeds, it’s worth discussing the tool we’ll use to glue everything together: <strong>Ansible</strong>.</p><p>Ansible is an automation engine used for configuration management, application deployment, and general infrastructure orchestration. What makes it so popular?</p><ul><li><strong>Agentless:</strong> Unlike some other tools, you don’t need to install anything on your remote servers. Ansible connects over SSH.</li><li><strong>Human-readable YAML syntax:</strong> This makes it easy to learn and collaborate on with others.</li><li><strong>Dynamic control flow:</strong> You can incorporate conditions, variables, loops, and even execute raw shell commands or API calls.</li></ul><p>We chose Ansible because it’s great for scenarios like ours: where we want to orchestrate not just the creation of infrastructure, but also follow-up steps like configuration and validation.</p><p>You may have heard of OpenTofu or Terraform — these tools excel at declaratively managing infrastructure states. In fact, since we offer a public API, Nine is also fully compatible with Terraform. You can find example configurations in our <a href="https://github.com/ninech/terraform-examples">Terraform examples repository</a>. Ansible, in contrast, is more procedural and event-driven, which suits this use case well where we need more step-by-step control and conditional logic.</p><h3>🛠️ Setting Up Your Toolkit (macOS)</h3><p>Our journey begins on a MacBook. We’ll install all the tools we need using <a href="https://brew.sh/">Homebrew</a>, the de facto package manager for macOS.</p><p>If you’re on <strong>Linux</strong>, you can follow the <a href="https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html">Ansible installation guide for Linux</a> and install `nctl` manually from <a href="https://github.com/ninech/nctl/releases">our GitHub releases</a>.</p><p>For <strong>Windows</strong> users, we recommend using <a href="https://learn.microsoft.com/en-us/windows/wsl/">WSL2 (Windows Subsystem for Linux)</a> and then following the Linux instructions above. This allows you to use the same Unix-like tooling and commands without leaving Windows.</p><h3>Install Homebrew</h3><p>If you don’t have Homebrew yet, install it with:</p><pre>/bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)&quot;</pre><p>Homebrew lets us easily install other dependencies.</p><h3>Install Ansible</h3><pre>brew install ansible</pre><p>Ansible is our primary tool to automate the provisioning, configuration, and validation of infrastructure resources.</p><h3>Install Nine’s CLI tool: `nctl`</h3><pre>brew tap ninech/nine<br>brew install nctl</pre><p>`nctl` is Nine’s command-line client. It wraps around our public API and simplifies tasks like creating a VM or database, retrieving secrets, and managing access control. We’ll use it heavily.</p><h3>Authenticate with Nine’s API</h3><p>To interact with our API, you’ll need a Bearer token. This token grants authenticated access and is required for both Ansible and `nctl`.</p><p>You can create a service account and retrieve your token either through Cockpit:</p><p>👉 <a href="https://cockpit.nine.ch/en/customer/api_service_accounts/">https://cockpit.nine.ch/en/customer/api\_service\_accounts/</a></p><p>Or via the CLI:</p><p>Login with your account</p><pre>nctl auth login</pre><p>We create an apiserviceaccount named `cicd`:</p><pre>nctl create apiserviceaccount cicd</pre><p>And we retrieve the token for the apiserviceaccount named `cicd`:</p><pre>nctl get apiserviceaccounts cicd --print-token</pre><p>Once you have your token, export it as an environment variable:</p><pre>export NINE_API_TOKEN=&quot;&lt;your-token&gt;&quot;</pre><p>And don’t forget to export your SSH public key too. We’ll need it to access the VM later:</p><pre>export SSH_PUBLIC_KEY=&quot;$(cat ~/.ssh/id_ed25519.pub)&quot;</pre><p>&gt; 🛡️ You should always treat your API token like a password. Don’t hardcode it in files.</p><h3>📊 Project Structure</h3><p>To keep things tidy, let’s create a project directory and define our reusable configuration.</p><pre>mkdir nine-infra &amp;&amp; cd nine-infra</pre><h3>Group Variables</h3><p>Ansible allows us to keep variables in one place using group vars. Create the folder with:</p><pre>mkdir group_vars</pre><p>Here’s what `group_vars/all.yml` could look like:</p><pre># group_vars/all.yml<br>postgres_name: example-postgres<br>postgres_version: 15<br>redis_name: example-redis<br>vm_name: example-vm<br>location: nine-cz41<br>machine_type: nine-small-1<br>os: ubuntu24.04</pre><p>This makes our playbooks more readable and makes it easier to reuse or update settings. You can of course adjust the values here — such as the OS version, machine type, or service versions — depending on your needs. To discover which values are supported, you can use the documentation we provide for our API:</p><ul><li><a href="https://docs.nine.ch/api/#tag/CloudVirtualMachine">Cloud VM machine types and locations</a></li><li><a href="https://docs.nine.ch/api/#tag/Postgres">PostgreSQL supported versions and types</a></li><li><a href="https://docs.nine.ch/api/#tag/KeyValueStore">KeyValueStore specs</a></li></ul><h3>🔺 Step 1: Provision PostgreSQL</h3><p>To provision a managed PostgreSQL instance, we’ll use `nctl` within an Ansible playbook. This ensures the database is consistently created with our desired configuration.</p><p>Create the following file and save it as `provision_postgres.yml`:</p><pre># provision_postgres.yml<br>- name: Provision PostgreSQL<br>  hosts: localhost<br>  gather_facts: false<br><br>  vars_files:<br>    - group_vars/all.yml<br><br>  tasks:<br>    - name: Check if PostgreSQL already exists<br>      ansible.builtin.shell: &quot;nctl get postgres {{ postgres_name }}&quot;<br>      changed_when: false<br>      failed_when: false<br>      register: pg_exists<br><br>    - name: Create PostgreSQL<br>      ansible.builtin.command: &gt;<br>        nctl create postgres {{ postgres_name }}<br>        --postgres-version {{ postgres_version }}<br>        --machine-type nine-db-xs<br>        --location {{ location }}<br>        --allowed-cidrs 0.0.0.0/0<br>        --wait<br>      changed_when: &quot;pg_result.rc == 0&quot;<br>      ignore_errors: true when: &quot;pg_exists.rc != 0&quot;<br>      register: pg_result<br><br>    - name: Get PostgreSQL connection string<br>      ansible.builtin.command: &quot;nctl get postgres {{ postgres_name }} --print-connection-string&quot;<br>      register: pg_conn_str<br>      changed_when: &quot;pg_conn_str.rc != 0&quot;<br><br>    - name: Set Postgres connection string fact<br>      ansible.builtin.set_fact:<br>        postgres_conn_string: &quot;{{ pg_conn_str.stdout }}/postgres&quot;<br><br>    - name: Debug Postgres connection string<br>      ansible.builtin.debug:<br>        var: postgres_conn_string</pre><p>Run it using:</p><pre>ansible-playbook provision_postgres.yml</pre><p>Be aware: This step may take some time to complete.</p><p>First up is our database. A PostgreSQL instance gives us a durable, reliable data store. It’s a common backend choice for modern apps.</p><p>We use `nctl` to create the instance, and then immediately retrieve its connection string so we can use it later.</p><h3>✅ Validation:</h3><p>Use the connection string to confirm the instance is reachable:</p><pre>psql &lt;postgres_connection_string&gt;</pre><p>If the connection succeeds, you’re ready to move on. Keep in mind that the creation of a DB can take some time since we always provision a dedicated database. Soon we will offer Shared DBs as well which will speed up that scenario by a lot. Also please be aware that we — for simplicity reasons of this blog post — did allow 0.0.0.0/0 to access all our services. I do recommend you to not set up your infrastructure in this way.</p><h3>🔺 Step 2: Provision Redis</h3><p>Redis will act as our key-value store. Just like PostgreSQL, we’ll use an Ansible playbook to automate its provisioning.</p><p>Create the following file as `provision_redis.yml`:</p><pre># provision_redis.yml<br>- name: Provision Redis<br>  hosts: localhost<br>  gather_facts: false<br><br>  vars_files:<br>    - group_vars/all.yml<br><br>  tasks:<br>    - name: Check if Redis already exists<br>      ansible.builtin.shell: &quot;nctl get keyvaluestore {{ redis_name }}&quot;<br>      changed_when: false<br>      failed_when: false<br>      register: redis_exists<br><br>    - name: Create Redis<br>      ansible.builtin.command: &gt;<br>        nctl create keyvaluestore {{ redis_name }}<br>        --memory-size 1Gi<br>        --location {{ location }}<br>        --allowed-cidrs 0.0.0.0/0<br>        --wait<br>      changed_when: &quot;redis_result.rc == 0&quot;<br>      ignore_errors: true<br>      when: &quot;redis_exists.rc != 0&quot;<br>      register: redis_result<br><br>    - name: Get Redis connection info<br>      ansible.builtin.command: nctl get keyvaluestore {{ redis_name }} -o yaml<br>      register: redis_info<br>      changed_when: &quot;redis_info.rc != 0&quot;<br><br>    - name: Parse Redis connection info<br>      ansible.builtin.set_fact:<br>        redis_host: &quot;{{ redis_info.stdout | from_yaml | json_query(&#39;status.atProvider.connection.address&#39;) }}&quot;<br>        redis_password: &quot;{{ redis_info.stdout | from_yaml | json_query(&#39;status.atProvider.connection.password&#39;) }}&quot;<br><br>    - name: Debug Redis details<br>      ansible.builtin.debug:<br>        msg: &quot;Redis host: {{ redis_host }}, password: {{ redis_password }}&quot;</pre><p>Run it using:</p><pre>ansible-playbook provision_redis.yml</pre><p>Be aware: This step may take some time to complete.</p><p>Next, we spin up a Redis key-value store. Redis is often used for caching, session storage, rate limiting, or pub/sub. It’s fast and memory-based.</p><p>We again use `nctl` to create it, then extract the address and password from the YAML output.</p><h3>✅ Validation:</h3><p>Try connecting to Redis using the CLI:</p><pre>redis-cli -h &lt;redis_host&gt; -a &lt;redis_password&gt;</pre><p>If the response is `PONG`, you’re good to go.</p><h3>🔺 Step 3: Provision Cloud VM</h3><p>With the database services in place, let’s bring up the virtual machine that will act as our compute node. This VM can later be configured to host applications or scripts.</p><p>Save the following playbook as `provision_vm.yml`:</p><pre># provision_vm.yml<br>- name: Provision Cloud VM<br>  hosts: localhost<br>  gather_facts: false<br><br>  vars_files:<br>    - group_vars/all.yml<br><br>  tasks:<br>    - name: Check if VM already exists<br>      ansible.builtin.shell: &quot;nctl get cloudvirtualmachine {{ vm_name }}&quot;<br>      changed_when: false<br>      failed_when: false<br>      register: vm_exists<br><br>    - name: Create VM using nctl (wait until ready)<br>      ansible.builtin.command: &gt;<br>        nctl create cloudvirtualmachine {{ vm_name }}<br>        --hostname {{ vm_name }}<br>        --location {{ location }}<br>        --machine-type {{ machine_type }}<br>        --os {{ os }}<br>        --public-keys &quot;{{ lookup(&#39;env&#39;, &#39;SSH_PUBLIC_KEY&#39;) }}&quot;<br>        --wait<br>      changed_when: &quot;vm_create_output.rc == 0&quot;<br>      ignore_errors: true<br>      when: &quot;vm_exists.rc != 0&quot;<br>      register: vm_create_output<br><br>    - name: Show VM creation output<br>      ansible.builtin.debug:<br>        var: vm_create_output.stdout_lines<br>      when: &quot;vm_exists.rc != 0 and vm_create_output.rc == 0&quot;</pre><p>Then apply it with:</p><pre>ansible-playbook provision_vm.yml</pre><p>Be aware: This step may take some time to complete.</p><p>Now let’s bring up the virtual machine. This will be our compute layer — where your app (or anything else) can run.</p><p>We’ll use `nctl` again, this time to provision a VM with your preferred OS image and SSH key.</p><h3>✅ Validation:</h3><p>Log in via SSH:</p><pre>ssh root@&lt;vm_ip&gt;</pre><p>This confirms that the server is alive, reachable, and ready for configuration.</p><h3>🧩 Putting it all together</h3><h3>🔚 Wrapping Up</h3><p>We’ve now laid the groundwork for a complete cloud-native deployment:</p><ul><li>A secure, managed PostgreSQL instance</li><li>A fast Redis service ready for caching or queues</li><li>A fully provisioned and reachable virtual machine</li></ul><p>All provisioned step-by-step via Ansible and Nine’s CLI and API — no manual clicking, no guessing.</p><p>From here, you can begin configuring your app, deploying code, and scaling out your architecture — all documented in code and easily accessible.</p><p><em>Originally published at </em><a href="https://nine.ch/from-zero-to-infra-building-a-production-ready-setup-using-our-api-and-ansible/"><em>https://nine.ch</em></a><em> on June 3, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=548caebe243c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Git to Release and Run Your Web App on Deploio]]></title>
            <link>https://nine-internet-solutions.medium.com/using-git-to-release-and-run-your-web-app-on-deploio-135b66bf2b4e?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/135b66bf2b4e</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 07:30:13 GMT</pubDate>
            <atom:updated>2025-08-26T06:50:28.388Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*PoRcvbZIfmh2BPbZ.jpg" /></figure><p>Web development has evolved significantly, with numerous tools and frameworks streamlining the coding process. However, building and deploying web applications has become increasingly complex, often involving a variety of configurations. Seemingly gone are the days of quick deployments with scp or rsync pointing to dedicated or shared hosting servers.</p><p>At the centre of all this progress is Git, which has become the industry standard for version control, code management and collaboration. Git has also become the foundation of modern software release lifecycles. This is where <a href="https://deplo.io/en">Deploio</a> comes in, providing the missing piece to streamline the release process from development to deployment to operating in production.</p><p>Deploio adapts to your workflow based on Git revisions, whether you are rapidly iterating in the early stages, have already established a strict release schedule or are maintaining a legacy project with infrequent updates.</p><p>And if you are new to automated deployments, you don’t need to set up your own CI/CD pipeline — Deploio will handle building your project and can even run tests beforehand, ensuring an uninterrupted service, all running on Nine’s infrastructure.</p><h3>Understanding Git Revisions: Branches, Commits and Tags</h3><p>The following three Git objects can also be referred to as revisions.</p><p>Commits: The smallest unit which can be used to make a change in a repository. Unless they are part of a main branch, they might eventually be squashed or replaced with other commits.</p><p>Branches: Branches allow additional lines of development to be started, so that different sets of commits can be worked on at the same time. Branches are often merged or rebased with other branches, resulting in a new revision.</p><p>Tags: Tags point to specific commits, usually on main branches, regularly used to define a release. They should be considered immutable.</p><h3>Managing Releases on Deploio with Git</h3><p>There are numerous software release strategies, often linked to other development practices. To focus on deployment aspects, we have categorized them into three main types, based on their frequency.</p><p>Keep in mind that ephemeral revisions such as commits and branches can disappear from your Git repository, causing Deploio builds to fail. In general, running (available) Deploio releases will not be replaced by failed builds or releases that cannot be started.</p><h3>Continuous Deployments</h3><p>Also known as rolling releases, continuous deployments are usually tied to a specific branch. Because they allow a high deployment frequency, they are ideal for unstable integration and development branches where fast feedback cycles are desirable.</p><p>In the following scenario, a separate development instance is being created for the app based on the ‘dev’ branch. Options such as -port have been stored in the presets of the project <a href="https://docs.nine.ch/docs/deplo-io/configuration/deploio-configuration-layers/">configuration layer</a> in advance, so they do not need to be specified:</p><pre>nctl create application urlshortener-dev \<br>  --project=marco-deployment-demo \<br>  --git-url=https://github.com/ninech/deploio-examples \<br>  --git-sub-path=dockerfile/java-kvs \<br>  --dockerfile \<br>  --env=SPRING_PROFILE_ENV=dev \<br>  --size=micro \<br>  --git-revision=dev # Git branch</pre><p>Changes made to Commits on the dev branch will automatically trigger a build- and release process on Deploio.</p><p>Continuous deployments can also be applied to production environments of course, particularly if mature upstream CI/CD practices and tooling are in place, for example based on <a href="https://github.blog/enterprise-software/ci-cd/build-ci-cd-pipeline-github-actions-four-steps/">GitHub Actions</a>.</p><h3>Scheduled Deployments</h3><p>Also known as stable or fixed releases, scheduled deployments roll out changes based on a defined version, which usually has undergone extensive testing. The release version can be reflected by a Git tag and might follow a well-known scheme, such as <a href="https://semver.org/#summary">semantic versioning</a>.</p><p>In the following example, a specific version tag is being rolled out:</p><pre>nctl update application urlshortener-prod \<br>  --project=marco-deployment-demo \<br>  --git-revision=v1.1.0 # Git tag</pre><h3>Ad-Hoc Deployments</h3><p>Sometimes, deployments are needed outside of regular release cycles. This might involve rolling back to a previous version due to production issues, backporting a change from the main branch to an older release, or directly deploying a specific branch or commit.</p><p>Here is an example where the development instance is being used to make a specific feature branch available for testing:</p><pre>nctl update application urlshortener-dev \<br>  --project=marco-deployment-demo \<br>  --git-revision=feature/dark-mode # Git branch</pre><p>If required, a specific commit can also be used:</p><pre>nctl update application urlshortener-dev \<br>  --project=marco-deployment-demo \<br>  --git-revision=e8273c3f # Git commit</pre><h3>Progressive Delivery</h3><p>As discussed earlier, software can be released in various ways. An advanced approach, particularly suitable for large and potentially distributed applications, is progressive delivery. This involves gradually releasing new features or versions to a subset of users or environments before full-scale deployment.</p><p>Popular techniques within progressive delivery include <a href="https://en.wikipedia.org/wiki/Feature_toggle#Canary_release">Canary releases</a> and <a href="https://en.wikipedia.org/wiki/Blue%E2%80%93green_deployment">Blue-green deployments</a>. At the time of writing, such deployments to Nine managed infrastructure are only possible with the Application being deployed to Kubernetes managed by our Argo CD service, for example, on <a href="https://docs.nine.ch/docs/managed-kubernetes/gke/argo-cd">Managed GKE</a> or <a href="https://docs.nine.ch/docs/managed-kubernetes/nke/argocd">NKE</a>.</p><p><em>Originally published at </em><a href="https://nine.ch/using-git-to-release-and-run-your-web-app-on-deploio/"><em>https://nine.ch</em></a><em> on January 28, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=135b66bf2b4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No More IP Limitations With Static Egress]]></title>
            <link>https://nine-internet-solutions.medium.com/no-more-ip-limitations-with-static-egress-009ceef5c187?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/009ceef5c187</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[vcluster]]></category>
            <category><![CDATA[cilium]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Mon, 11 Nov 2024 05:29:51 GMT</pubDate>
            <atom:updated>2025-08-26T07:01:35.697Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aVTAimjstIwV9_Ay.jpg" /></figure><p>In today’s rapidly evolving tech landscape, modern systems like Kubernetes are often seen as the cutting edge of infrastructure and application management. However, even in this world of containerization and dynamic scaling, there are moments when these advanced technologies need to interface with more traditional systems and practices.</p><p>One such area is the need for predictable network connectivity, especially when dealing with legacy firewalls and external services that require traffic to originate from a specific IP address. As we face these requirements multiple times at Nine, we had to come up with a solution. Let’s have a closer look into our new static egress feature.</p><h3>Why do we need a solution for this?</h3><p>When we designed our NKE (Nine Kubernetes Engine) product, two of the main goals were automation and the ability to offer self-service. Kubernetes and the corresponding add-ons already provide the foundation for that. For example, the cluster autoscaler adds/removes nodes based on the resource requirements of the cluster running workloads. One thing though, which is not really taken care of by Kubernetes out of the box, is maintaining the underlying nodes.</p><p>Traditionally, we update the installed software and the Linux kernels of our VMs with the help of a configuration system and the package manager of the underlying Linux distribution. This also involves upgrades from one major distribution release to the next (e.g. Ubuntu 22.04 to 24.04), which needs a lot of preparation and work in advance. When we started with NKE, we did not want to go down that path and searched for different solutions.</p><p>Luckily, when designing NKE, we already had quite some experience with managing Google Kubernetes Engine (GKE) clusters, and have been generally very happy with how node upgrades are done there. In a GKE cluster, node software itself is not updated in an already running instance, but instead machines are replaced with newer, updated ones, one after another. Our implementation of this workflow uses <a href="https://www.flatcar.org/">Flatcar Linux</a>, which provides a rolling release Linux distribution with enhanced security. One of the added security mechanisms is an immutable file system that doesn’t allow changes to the already installed software. This prevents version and configuration drift.</p><p>But let’s get back to the upgrade mechanism itself. As the sequential upgrade workflow first adds a new machine before removing the old one, both machines are running in parallel for a short amount of time. One consequence of this is that both nodes need to use different IP addresses, which in turn influences the workloads running in the Kubernetes cluster, as the outgoing IP address for cluster external traffic depends on the IP of the node itself (workload initiated traffic to external targets makes use of a node source NAT).</p><p>After we implemented and introduced our node upgrade mechanism, some of our customers asked for the IP addresses of their NKE cluster nodes as they wanted to configure them on external firewalls. But the node IP addresses couldn’t really be predicted anymore, as they might change during the next maintenance window. Even if they hadn’t changed during maintenance, using cluster autoscaled node pools could lead to new nodes with new IP addresses as well. It was clear that the dynamic world of Kubernetes doesn’t really fit into the already existing, more static environments.</p><p>For quite some time, the only solution to this problem was to allow all NKE subnets on external firewalls. Although this lowers the attack risk for external services, I think one can agree that this was not really a satisfying solution. Moreover, we have to add new NKE subnets over time, which complicates things even more.</p><h3>How did we solve it?</h3><p>When we investigated how this problem can be solved, we noticed that our CNI provider Cilium already provided a feature called <a href="https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/">Egress Gateway</a>. The documentation describes the feature like this:</p><p>“The egress gateway feature routes all IPv4 connections originating from pods and destined to specific cluster-external CIDRs through particular nodes, from now on called ‘gateway nodes’. When the egress gateway feature is enabled and egress gateway policies are in place, matching packets that leave the cluster are masqueraded with selected, predictable IPs associated with the gateway nodes.”</p><p>This feature could pretty much solve our problem, but there were still some issues which we needed to tackle.</p><p>First, the “gateway node” might just disappear during a maintenance window (as it gets replaced) and so another node has to take over the “gateway node” role. This new “gateway node” might have an IP address from a totally different subnet than the previous “gateway node”. So it was clear that we cannot use IP addresses from the already existing NKE subnets as static egress IPs.</p><p>Second, the IP address to use as egress IP needs to be configured on the “gateway node” by some mechanism. Cilium doesn’t set it automatically, but instead requires it to be already set.</p><p>To solve the first problem, we are using BGP (Border Gateway Protocol) to announce routes for special, independent egress IPs to our routers. We are using the same mechanism already for Kubernetes services of the type loadbalancer (for ingressing traffic) in our NKE clusters. It was therefore logical to use the same system again. This allows the static egress IPs to be totally independent from the subnets which we use for our NKE nodes.</p><p>Once we finished the investigation and did some preliminary tests, we knew what we had to do. The task was to write an agent that runs on the control-plane nodes (we run 3 of them) in every NKE cluster and selects one of the nodes to be the “gateway node”. It then adds a specific Kubernetes label to that node, does all the BGP work to establish a proper routing for a selected static egress IP and configures that IP on the node itself. Once the selected egress node gets removed during maintenance (or just doesn’t work anymore because of other issues), the agent chooses a different existing node and does a reconfiguration. Luckily, there are existing solutions for leader election (the leader is the selected egress node) which we were able to use in Kubernetes already. After a month of work and tests, the agent was finished.</p><p>The rest of the static egress feature consisted of integrating Cilium’s “egress gateway” and our agent configuration into one setup workflow and exposing it via our API. Because <a href="https://deplo.io/en.html">Deploio</a> is also based on NKE, it was easy to make the static egress feature available to it. The same goes for <a href="https://docs.nine.ch/docs/nke/kubernetes-cluster-backed-by-vcluster">vClusters</a>. Once the <a href="https://docs.nine.ch/docs/nke/static-egress-nke">documentation</a> was ready, it was time to make the feature available to customers.</p><p>Running the feature in production for quite some time now, it has proven to be a reliable solution to offer predictable egress identities.</p><p><em>Originally published at </em><a href="https://nine.ch/no-more-ip-limitations-with-static-egress/"><em>https://nine.ch</em></a><em> on November 11, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=009ceef5c187" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Added Dockerfile Support to Deploio]]></title>
            <link>https://nine-internet-solutions.medium.com/why-we-added-dockerfile-support-to-deploio-42117212c9d3?source=rss-4ce903f38443------2</link>
            <guid isPermaLink="false">https://medium.com/p/42117212c9d3</guid>
            <category><![CDATA[git-repositories]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[buildpack]]></category>
            <category><![CDATA[vcluster]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <dc:creator><![CDATA[Nine Internet Solutions AG]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 08:46:39 GMT</pubDate>
            <atom:updated>2025-08-26T07:07:02.836Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0JjX1T_FAq9fWBil.jpg" /></figure><h4>Introduction</h4><p>At Nine, we’ve been hard at work refining <a href="https://deplo.io/en.html">Deploio</a>, our platform for deploying applications with ease. If you’ve been following our journey, you already know about the significant advancements we’ve made. If you’re new, I recommend reading our <a href="https://nine.ch/elevate-your-app-development-with-deploio/">previous blog post</a> or having a look at our <a href="https://docs.nine.ch/docs/category/deploio-paas">docs</a>, where we introduced Deploio and explained its core functionality.</p><p>Today, we’re excited to announce a new addition to the platform: <strong>Dockerfile Support</strong>. This enhancement is a big step forward, particularly for teams and developers who need more control over and further customisation of their deployment process. But to truly appreciate the significance of this update, it’s important to understand how Deploio has been handling application builds up until now.</p><h4>How We Built Applications Before: Buildpacks</h4><p>From the beginning, Deploio has relied on <a href="https://buildpacks.io/"><strong>Buildpacks</strong></a> to automate the process of turning your source code into runnable container images. The process is straightforward: once you link your Git repository to Deploio and select the branch or environment, Deploio automatically detects the type of application in your codebase. Based on what it finds, the appropriate Buildpack is selected to handle the build process. In theory, this sounds quite easy but brings the complexity of always making an assumption about which plugins, extensions or plain dependencies could be expected.</p><h3>But What Exactly Are Buildpacks? And Why Do They Matter?</h3><h4>What Is a Buildpack?</h4><p>A Buildpack is essentially an automated tool that converts source code into a runnable container image without the need for a Dockerfile, or any other complex configuration. Think of it as a way to abstract away the complexities of building containers, making it an appealing option for teams that might not be familiar with Docker or container technologies in general.</p><p>The beauty of Buildpacks lies in their simplicity. They detect your code’s language and dependencies, and build your application on a pre-configured, trusted base image. This means you can go from writing code to having a fully functional container without ever touching a Dockerfile. Once the building step is complete, <a href="https://docs.nine.ch/docs/deplo-io/deploio-troubleshooting-and-adavanced-topics#inspecting-and-running-the-build-output-locally">you can even download your docker image.</a></p><h3>Why Buildpacks Have Been Great (Until Now)</h3><p>The simplicity of Buildpacks is their biggest strength — by abstracting away the container-building process, they allow developers to focus solely on writing code, without worrying about configuring the underlying infrastructure. This is especially beneficial for teams that want to move fast, without having to dive into the nitty-gritty of containerization.</p><p>Moreover, the pre-built base images that Buildpacks rely on are maintained by trusted sources, meaning that security and performance optimizations are handled behind the scenes. This further reduces the burden on developers, allowing them to focus on what they do best: building applications.</p><p>However, as convenient as Buildpacks are, they come with their own set of limitations, especially when dealing with more complex or non-standard workloads.</p><h4>The Limitations of Buildpacks</h4><p>While Buildpacks provide a quick and secure way to get applications running, they aren’t always flexible enough for every use case. The very abstraction that makes Buildpacks so user-friendly can also make them limiting when you need more granular control over your container builds.</p><p>For example, Buildpacks rely on pre-configured base images, which may not always suit applications with specific requirements. If your app needs a custom dependency or a particular version of a system library, Buildpacks might not be able to accommodate that. Moreover, debugging issues with Buildpacks can sometimes feel like shooting in the dark, since much of the build process happens behind the scenes. This leads to a back-and-forth between our customers and us, and at times even results in us forking Buildpacks to allow further customization.</p><p>In other words, while Buildpacks simplify deployment for standard applications, they can become a roadblock for teams in need of more fine-tuned control. That’s why we’ve introduced <strong>Dockerfile Support</strong> on Deploio.</p><h4>What Is a Dockerfile?</h4><p>If you are reading this blog, chances are you are familiar with the concept of Dockerfiles. Nevertheless, I want to provide you with a “simple” explanation of what they are and how they work.</p><p>A <a href="https://docs.docker.com/reference/dockerfile/#:~:text=A%20Dockerfile%20is%20a%20text,line%20to%20assemble%20an%20image."><strong>Dockerfile</strong></a> is a script that contains a series of instructions on how to build a Docker image. Unlike Buildpacks, which rely on predefined rules and base images, Dockerfiles give developers full control over every step of the containerization process.</p><p>With a Dockerfile, you can specify everything from the base image you want to use to how dependencies are installed, which ports should be exposed, and how the application is started. This level of control is invaluable when you’re working on complex applications or have specific performance, security, or environmental requirements.</p><h4>Why Dockerfiles Are a Game-Changer for Deploio</h4><p>By introducing Dockerfile support on Deploio, we’re giving you, the developer, more flexibility and power over how your applications are built and run. Here are some key reasons why Dockerfiles matter:</p><ol><li><strong>More Complex Workloads</strong>: With Dockerfiles, you can define your environment with precision. This is particularly important for applications with complex requirements, such as custom dependencies, specific operating system configurations, or legacy components. Dockerfiles give you the freedom to tailor the container to your application’s exact needs.</li><li><strong>Control Over the Build Process</strong>: While Buildpacks handle everything for you, Dockerfiles allow you to take control of the build process. This can lead to more optimized builds, as you can control caching, layer management, and even the size of your final container image.</li><li><strong>Flexibility in Base Images</strong>: Buildpacks force you to use pre-built base images, which may or may not be ideal for your specific use case. With Dockerfiles, you can choose any base image you like. Whether you want a minimal Alpine image to keep your container lightweight, or a custom-built base image for enterprise applications, the choice is yours.</li><li><strong>Debugging and Troubleshooting</strong>: Since Dockerfiles give you complete insight into how your container is built, they make it easier to troubleshoot issues. If something goes wrong during the build, you can pinpoint exactly which step caused the issue and fix it, without having to dig through layers of abstraction.</li></ol><p><strong>Using Dockerfiles on Deploio</strong></p><p>With Dockerfile support now fully integrated into Deploio, you’ve got the freedom to choose the best tool for the job — whether that’s sticking with the streamlined simplicity of Buildpacks or diving into the granular control that Dockerfiles offer. Buildpacks are still an excellent choice when you want to focus purely on development without worrying about the underlying infrastructure. They abstract away the complexities of containerization, saving time and minimizing configuration — perfect for getting applications up and running quickly and securely.</p><p>But for those times when you need that extra level of control — when your app demands custom dependencies, specific system configurations, or optimizations — Dockerfiles open up a whole new world of possibilities. Want to fine-tune your image layers? Use a specific base image? Optimize for minimal footprint? Dockerfiles let you do all of that and more, offering flexibility that’s ideal for complex, non-standard workloads.</p><p>Deploying with Dockerfiles is as simple as adding your Dockerfile to your repository. Deploio will automatically detect it, allowing you to customize everything from the OS down to the tiniest dependencies. If you’re ready to get started, head over to our<a href="https://docs.nine.ch/docs/deplo-io/dockerfile-build"> Dockerfile Build documentation</a> for a deep dive into the process.</p><p>Whether you’re running with Buildpacks for speed or leveraging Dockerfiles for customization, Deploio has you covered. Ready to take it for a spin? Let’s see what new ideas you bring to life with this added flexibility.</p><p><em>Originally published at </em><a href="https://nine.ch/why-we-added-dockerfile-support-to-deploio/"><em>https://nine.ch</em></a><em> on October 8, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42117212c9d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>