<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Midhun Krishna on Medium]]></title>
        <description><![CDATA[Stories by Midhun Krishna on Medium]]></description>
        <link>https://medium.com/@midhunkrishna-ops?source=rss-3b237047fa8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PHp1TogCaacS7feU0p65Ug.png</url>
            <title>Stories by Midhun Krishna on Medium</title>
            <link>https://medium.com/@midhunkrishna-ops?source=rss-3b237047fa8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:22:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@midhunkrishna-ops/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Azure Front Door Went Down: How We Built an Emergency Path to Keep AKS Available]]></title>
            <link>https://midhunkrishna-ops.medium.com/when-azure-front-door-went-down-how-we-built-an-emergency-path-to-keep-aks-available-ac655d320036?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac655d320036</guid>
            <category><![CDATA[azure-frontdoor]]></category>
            <category><![CDATA[disaster-response]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 16:26:46 GMT</pubDate>
            <atom:updated>2026-10-06T16:26:46.976Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Architecture is straightforward when everything works. The real test begins when a critical dependency becomes unavailable.</strong></p><p>On <strong>October 29, 2025</strong>, Azure Front Door experienced a major service incident that affected customers using Azure Front Door and Azure CDN.</p><p>Microsoft’s incident report states that the disruption began at <strong>15:41 UTC on October 29</strong> and was fully mitigated at <strong>00:05 UTC on October 30</strong>. Customers experienced connection timeouts and DNS resolution issues, and the incident affected multiple Azure services that depended on Front Door.</p><p>Microsoft later explained that incompatible configuration metadata had propagated through the Front Door platform. Because the failure in the data plane was asynchronous, the configuration passed the existing protection and health-validation mechanisms before eventually triggering crashes across edge sites.</p><p>For many Azure customers, this was an edge-service outage.</p><p>For us, it became an architecture and incident-response problem.</p><p>Because our application was designed around Azure Front Door as the public entry point, when Front Door became unavailable, <strong>the application itself was healthy — but customers could no longer reach it.</strong></p><p>And that’s where things got interesting.</p><h3>The Architecture We Had Designed</h3><p>Our application was running on <strong>Azure Kubernetes Service (AKS)</strong>.</p><p>We were using <strong>Istio</strong> as the internal ingress layer, and the AKS workload was deliberately not exposed directly to the public internet.</p><p>The intended traffic flow was:</p><pre>Internet<br>   |<br>   v<br>Azure Front Door + WAF<br>   |<br>   v<br>Private Link<br>   |<br>   v<br>Private Link Service<br>   |<br>   v<br>Internal Load Balancer<br>   |<br>   v<br>Istio Ingress<br>   |<br>   v<br>AKS Workload</pre><p>This architecture was based on Microsoft’s recommended pattern for exposing and protecting AKS workloads using <strong>Azure Front Door, Azure Web Application Firewall, and Azure Private Link</strong>.</p><p>The design gave us several important security properties:</p><ul><li>Azure Front Door as the global public entry point</li><li>WAF protection at the edge</li><li>Private connectivity to the AKS origin</li><li>No public AKS ingress</li><li>Internal load balancing</li><li>Istio-based ingress</li><li>Hub-and-spoke network segmentation</li><li>Forced tunneling for outbound traffic</li><li>Azure Firewall as the controlled egress path</li></ul><p>It was a deliberate security architecture.</p><p>And normally, it worked exactly as intended.</p><h3>Our Network Design</h3><p>The environment followed a <strong>hub-and-spoke architecture</strong>.</p><p>The hub VNet contained shared networking components, including Azure Firewall.</p><p>The AKS environment lived inside a separate spoke VNet with its own address space and dedicated subnets.</p><p>Conceptually:</p><pre>Azure<br>                           |<br>                    +------+------+<br>                    |             |<br>                 HUB VNet      SPOKE VNet<br>                    |             |<br>             Azure Firewall       |<br>                                  |<br>                              AKS Cluster<br>                                  |<br>                           Istio Ingress</pre><p>For outbound traffic, we had <strong>forced tunneling</strong> configured through UDRs.</p><p>The relevant AKS subnet routes forced outbound traffic through Azure Firewall.</p><p>That meant our normal outbound path was:</p><pre>AKS<br> |<br> v<br>UDR<br> |<br> v<br>Azure Firewall<br> |<br> v<br>Internet</pre><p>This was an intentional security control.</p><p>We didn’t want workloads to have unrestricted direct internet access.</p><h3>Then Front Door Went Down</h3><p>When Azure Front Door became unavailable, our normal path effectively became:</p><pre>Customer<br>   |<br>   X<br>Azure Front Door<br>   |<br>   X<br>Private Link<br>   |<br>   X<br>AKS</pre><p>But internally, almost everything was still healthy.</p><p>The AKS cluster was healthy.</p><p>The workloads were healthy.</p><p>The internal load balancer was healthy.</p><p>The database was healthy.</p><p>The network inside the environment was healthy.</p><p>The problem was the <strong>entry point</strong>.</p><p>This distinction is important in distributed systems:</p><blockquote><strong><em>A healthy application is not necessarily an accessible application.</em></strong></blockquote><p>Our dependency chain had failed before traffic could reach the application.</p><h3>The Incident Question</h3><p>At that point, the question wasn’t:</p><blockquote><em>“How do we redesign our architecture?”</em></blockquote><p>We didn’t have the luxury of doing that during an active incident.</p><p>The question was:</p><blockquote><strong><em>“What is the fastest safe path we can create to restore customer access without destroying the architecture we already have?”</em></strong></blockquote><p>We considered several possibilities.</p><p>None of them were perfect.</p><h3>Option 1 — Make the AKS Ingress Public</h3><p>The most obvious solution was to expose the Istio ingress publicly.</p><p>Something like:</p><pre>Internet<br>   |<br>   v<br>Public IP<br>   |<br>   v<br>Istio Ingress<br>   |<br>   v<br>AKS</pre><p>But our network architecture introduced a complication.</p><p>We had <strong>forced tunneling</strong>.</p><p>Simply making the ingress public didn’t automatically mean the complete request/response path would behave the way we wanted.</p><h3>Option 2 — Remove Forced Tunneling</h3><p>Another possibility was to temporarily remove the UDR responsible for forced tunneling.</p><p>That could make the public ingress approach easier.</p><p>But there was a trade-off.</p><p>We would be deliberately weakening one of our existing security controls.</p><p>Our outbound traffic was supposed to pass through Azure Firewall.</p><p>Removing the forced tunnel meant that workloads could potentially bypass that centralized egress control.</p><p>Could we do it? Yes.</p><p>Was it the first thing I wanted to do? No.</p><p>We wanted to preserve as much of our existing security posture as possible.</p><h3>Option 3 — Use Azure Firewall DNAT</h3><p>Another option was to use <strong>Azure Firewall DNAT</strong>.</p><p>Conceptually:</p><pre>Internet<br>   |<br>   v<br>Azure Firewall Public IP<br>   |<br>  DNAT<br>   |<br>   v<br>Internal AKS Service</pre><p>This would give us an alternate inbound path.</p><p>But it came with another consideration.</p><p>Application traffic passing through Azure Firewall would introduce <strong>additional data-processing cost</strong>.</p><p>More importantly, we were using our centralized security boundary as an emergency public ingress mechanism.</p><p>Technically possible. Operationally understandable. But not necessarily the simplest emergency solution.</p><h3>Then I Looked at the Network Differently</h3><p>This was the turning point.</p><p>Instead of asking:</p><blockquote><strong><em>“How do we replace Azure Front Door?”</em></strong></blockquote><p>I asked:</p><blockquote><strong><em>“What is the simplest path that can reach the application without changing the architecture we already have?”</em></strong></blockquote><p>That led me back to the network.</p><p>Our hub-and-spoke design had something very useful:</p><p><strong>clear network boundaries.</strong></p><p>The spoke VNet had its own address space.</p><p>The AKS environment was deployed into dedicated subnets.</p><p>The internal ingress endpoint already existed.</p><p>We were using <strong>Azure CNI Overlay</strong> for AKS networking.</p><p>And because we had already designed the VNet and subnet structure properly, we had an internal network path that could reach the existing ingress layer.</p><p>So instead of modifying AKS…</p><p>Why not add a temporary entry point <strong>beside</strong> the existing architecture?</p><h3>The Emergency Architecture</h3><p>We created a <strong>temporary subnet inside the AKS spoke VNet</strong>.</p><p>Inside that subnet, we deployed a small VM with a public IP.</p><p>We installed <strong>NGINX</strong> on the VM and configured it as a reverse proxy.</p><p>NGINX forwarded requests to the existing internal Istio ingress endpoint.</p><p>The emergency traffic path became:</p><pre>Internet<br>                            |<br>                            v<br>                    Public IP on VM<br>                            |<br>                            v<br>                       NGINX Proxy<br>                            |<br>                            v<br>                    Internal Istio IP<br>                            |<br>                            v<br>                  Internal Load Balancer<br>                            |<br>                            v<br>                       AKS Service<br>                            |<br>                            v<br>                        Workload</pre><p>The key decision was:</p><blockquote><strong><em>We didn’t replace the existing architecture. We added a temporary path around the failed dependency.</em></strong></blockquote><p>That made the change significantly easier to reason about.</p><h3>Why This Worked</h3><p>The network design we already had made this possible.</p><p>The VM was deployed inside the same spoke VNet.</p><p>The existing internal ingress endpoint was reachable from within that network.</p><p>Therefore, we didn’t need to:</p><ul><li>expose the AKS service directly,</li><li>redesign the AKS networking,</li><li>remove Private Link,</li><li>modify application workloads,</li><li>or completely change the routing topology.</li></ul><p>We introduced a simple path:</p><p><strong>Internet → Temporary VM → NGINX → Existing Internal Ingress → AKS</strong></p><p>This was one of the most valuable lessons from the incident:</p><blockquote><strong><em>Good network segmentation isn’t only about security. It can also give you controlled recovery options.</em></strong></blockquote><h3>The DNS Change</h3><p>Our normal application path was:</p><pre>application.example.com<br>        |<br>        v<br>Azure Front Door<br>        |<br>        v<br>Private Link<br>        |<br>        v<br>AKS</pre><p>For the emergency path, we temporarily changed DNS so that the application hostname resolved to the public IP of the temporary VM.</p><p>The traffic path became:</p><pre>application.example.com<br>        |<br>        v<br>Temporary VM Public IP<br>        |<br>        v<br>NGINX<br>        |<br>        v<br>Internal Istio<br>        |<br>        v<br>AKS</pre><p>We intentionally <strong>left the existing Front Door configuration intact</strong>.</p><p>That was important.</p><p>We weren’t replacing our architecture.</p><p>We were creating a temporary bypass.</p><h3>The Security Trade-Off</h3><p>This solution came with a very obvious trade-off.</p><p><strong>It was not the architecture we wanted for normal production traffic.</strong></p><p>Our original design provided an important security boundary:</p><pre>Internet<br>   |<br>   v<br>Front Door<br>   |<br>   v<br>WAF<br>   |<br>   v<br>Private Origin</pre><p>During the outage, the temporary path became:</p><pre>Internet<br>   |<br>   v<br>Public VM<br>   |<br>   v<br>NGINX<br>   |<br>   v<br>Private Origin</pre><p>That meant we were temporarily bypassing the normal Front Door/WAF protection layer.</p><p>The VM itself became a public-facing endpoint.</p><p>That was a known security risk.</p><p>But this was an incident.</p><p>The decision was not:</p><blockquote><em>“T</em><strong><em>his is a better architecture.</em></strong><em>”</em></blockquote><p>The decision was:</p><blockquote><strong><em>“Can we accept a controlled and temporary reduction in our security posture to restore application availability?”</em></strong></blockquote><p>For us, the answer was yes.</p><p>But only because we could keep the change temporary, controlled, monitored, and easily reversible.</p><h3>Why I Chose This Approach</h3><p>There were four things I cared about.</p><h3>1. Minimal Change</h3><p>We didn’t redesign AKS.</p><p>We didn’t remove forced tunneling.</p><p>We didn’t dismantle Private Link.</p><p>We didn’t change the application.</p><p>We added a temporary component.</p><h3>2. Limited Blast Radius</h3><p>The temporary infrastructure was isolated inside the existing spoke VNet.</p><p>We weren’t changing routing for the entire environment.</p><p>We weren’t changing unrelated workloads.</p><p>That reduced the risk of turning one outage into a larger network incident.</p><h3>3. Fast Recovery</h3><p>The solution was built from components we already understood:</p><p><strong>VM + Public IP + NGINX + existing internal ingress + DNS</strong></p><p>There was no need to introduce another complex architecture during the incident.</p><h3>4. Easy Rollback</h3><p>This was probably the most important property.</p><p>Once Front Door recovered:</p><pre>DNS<br> |<br> v<br>Azure Front Door<br> |<br> v<br>Private Link<br> |<br> v<br>AKS</pre><p>We could restore the original DNS configuration, validate the normal path, and then remove the temporary VM and subnet.</p><p>The original architecture remained intact.</p><h3>It Wasn’t the Perfect Solution</h3><p>And I want to be very clear about that.</p><p>This wasn’t some hidden Azure best practice.</p><p>It wasn’t the architecture I would recommend for permanent production traffic.</p><p>It was an <strong>incident-response workaround</strong>.</p><p>It introduced a public endpoint.</p><p>It bypassed the normal WAF path.</p><p>It introduced additional operational responsibility.</p><p>It temporarily increased the application’s attack surface.</p><p>But incident response isn’t always about finding the perfect architecture.</p><p>Sometimes it is about finding:</p><blockquote><strong><em>the safest workable solution you can implement quickly with the information and constraints you have.</em></strong></blockquote><p>The important part is understanding the trade-off.</p><p>We knew what we were bypassing.</p><p>We knew why we were doing it.</p><p>We knew it was temporary.</p><p>And we knew how to roll it back.</p><h3>What Microsoft Recommended Later</h3><p>Interestingly, Microsoft itself highlighted the importance of having an independent failover strategy following the October Front Door incidents.</p><p>Microsoft discussed using <strong>Azure Traffic Manager</strong> as one possible approach to redirect traffic away from Azure Front Door toward origin servers during an outage.</p><p>That is a useful architectural lesson.</p><p>The primary question shouldn’t only be:</p><blockquote><strong><em>“How highly available is Azure Front Door?”</em></strong></blockquote><p>It should also be:</p><blockquote><strong><em>“What happens to my application if Azure Front Door is unavailable?”</em></strong></blockquote><p>Because your application’s availability is ultimately determined by the entire dependency chain.</p><h3>What I Would Design Differently Today</h3><p>This incident changed one question I ask when reviewing cloud architectures.</p><p>We normally ask:</p><ul><li>Is it secure?</li><li>Is it scalable?</li><li>Is it highly available?</li><li>Is it cost effective?</li></ul><p>I would now add:</p><blockquote><strong><em>What happens if the managed service at the edge fails?</em></strong></blockquote><p>For our architecture:</p><pre>User<br> |<br> v<br>Azure Front Door + WAF<br> |<br> v<br>Private Link<br> |<br> v<br>AKS</pre><p>The failure path should also be explicitly designed:</p><pre>Azure Front Door unavailable<br>          |<br>          v<br>     What happens?<br>          |<br>          v<br>   Alternate path?</pre><p>A predefined failover mechanism — such as Traffic Manager or another independently managed origin path — could potentially eliminate the need for an improvised recovery architecture.</p><p>The exact solution depends on the application’s requirements, security model, cost constraints, and acceptable recovery objectives.</p><p>But the important part is to <strong>design and test the failure path before the failure happens.</strong></p><h3>Lessons From the Incident</h3><h3>1. Design the failure path, not only the happy path</h3><p>Architecture diagrams usually show:</p><pre>User → Front Door → WAF → Private Link → AKS</pre><p>But production architecture should also answer:</p><pre>Front Door unavailable → ?</pre><p>If the answer is:</p><blockquote><em>“We’ll figure it out during the outage.”</em></blockquote><p>Then there is probably an opportunity to improve the architecture.</p><h3>2. Understand the network beneath the services</h3><p>Knowing how to configure Azure Front Door is useful.</p><p>Understanding how:</p><ul><li>VNets</li><li>subnets</li><li>UDRs</li><li>Azure Firewall</li><li>Private Link</li><li>internal load balancer</li><li>AKS networking</li><li>Istio ingress</li><li>service IPs</li></ul><p>fit together is even more valuable during an incident.</p><p>The emergency solution wasn’t particularly sophisticated.</p><p>It worked because we understood <strong>why the network would allow it to work</strong>.</p><h3>3. Don’t immediately dismantle security controls</h3><p>During an outage, the easiest reaction can be:</p><blockquote><em>“</em><strong><em>Remove the security control so traffic can flow</em></strong><em>.”</em></blockquote><p>Sometimes that is necessary.</p><p>But when possible:</p><blockquote><strong><em>Add a temporary recovery path instead of dismantling the existing architecture.</em></strong></blockquote><p>That keeps the blast radius smaller and makes rollback easier.</p><h3>4. Security and availability can temporarily conflict</h3><p>Normally:</p><p><strong>Security should be built into the architecture.</strong></p><p>During an incident:</p><p><strong>You may have to temporarily accept additional risk to restore availability.</strong></p><p>But that risk should be:</p><ul><li>understood,</li><li>explicitly accepted,</li><li>monitored,</li><li>time-bound,</li><li>and removed after recovery.</li></ul><h3>5. Simple and reversible beats clever</h3><p>Our emergency solution was not complicated.</p><p>It was:</p><p><strong>Public IP → VM → NGINX → Internal Istio → AKS</strong></p><p>But it worked.</p><p>More importantly, we understood it.</p><p>And we could remove it.</p><p>During an incident, that matters.</p><h3>The Bigger Lesson</h3><p>The October 29 Azure Front Door outage reminded me of something important about cloud architecture.</p><p>We spend a lot of time asking:</p><blockquote><strong><em>How do I make this architecture secure?</em></strong></blockquote><blockquote><strong><em>How do I make it scalable?</em></strong></blockquote><blockquote><strong><em>How do I make it highly available?</em></strong></blockquote><p>But there is another question that deserves equal attention:</p><blockquote><strong><em>How does this architecture fail?</em></strong></blockquote><p>Managed cloud services remove enormous amounts of operational complexity.</p><p>But when a critical managed service becomes a dependency for your application’s entry point, its failure can become your application’s failure — even when every component you own is completely healthy.</p><p>Our AKS cluster didn’t fail that day.</p><p>Our application didn’t fail.</p><p>Our database didn’t fail.</p><p><strong>Our dependency failed.</strong></p><p>And we had to make a decision quickly.</p><p>We didn’t build the perfect replacement for Azure Front Door.</p><p>We built a temporary path that:</p><ul><li>restored customer access,</li><li>preserved the existing architecture,</li><li>minimized the blast radius,</li><li>accepted a known security trade-off,</li><li>and provided a simple rollback.</li></ul><p>That experience changed how I think about production architecture.</p><blockquote><strong><em>Design the happy path carefully.</em></strong></blockquote><blockquote><strong><em>Understand the failure path deeply.</em></strong></blockquote><blockquote><strong><em>And when an incident happens, make the smallest controlled change that gets you safely back to service.</em></strong></blockquote><p>That is where architecture turns into engineering.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac655d320036" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When Rate Limiting Stops Working: How I Used JA4 to Detect a Distributed Bot Attack]]></title>
            <link>https://midhunkrishna-ops.medium.com/when-rate-limiting-stops-working-how-i-used-ja4-to-detect-a-distributed-bot-attack-90a5f1cfbb59?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/90a5f1cfbb59</guid>
            <category><![CDATA[azure-front-door]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[fingerprint]]></category>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 15:21:13 GMT</pubDate>
            <atom:updated>2026-10-02T15:46:48.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S3M_NLYy2flI6UG5-RGlqw.jpeg" /></figure><h3>There are some production incidents where the usual security playbook simply stops working.</h3><p>Block the IP.</p><p>Rate-limit the endpoint.</p><p>Block the country.</p><p>Increase the WAF rules.</p><p>And yet, the traffic keeps coming.</p><p>I faced one such situation where an endpoint was being targeted by a highly distributed automated attack.</p><p>At first glance, it looked almost impossible to mitigate.</p><p>The requests were coming from <strong>50+ countries</strong>, using <strong>thousands of different IP addresses</strong>, and the attacker was generating different payloads for almost every request.</p><p>There was no obvious IP to block.</p><p>There was no single country to block.</p><p>And traditional rate limiting wasn’t enough.</p><p>The interesting part was discovering that although the requests looked completely different at the application layer, they had something in common much deeper in the network stack.</p><p>That was where <strong>JA4 fingerprinting</strong> became useful.</p><h3>The Business Problem</h3><p>The endpoint involved in the attack was related to an <strong>OTP-based workflow</strong>.</p><p>This is important because some endpoints cannot simply be treated like normal public APIs.</p><p>You may have endpoints that:</p><ul><li>Need to remain publicly accessible</li><li>Cannot be aggressively rate-limited</li><li>Are required by legitimate customers</li><li>Have business workflows that generate a large number of requests</li><li>Cannot simply be protected by authentication without breaking the product</li></ul><p>So the usual advice of:</p><blockquote><em>“Just put a rate limit on the endpoint.”</em></blockquote><p>isn’t always practical.</p><p>The attacker understood this too.</p><p>Instead of sending a few requests from one IP address, they distributed the traffic across a large number of sources.</p><h3>The Attack</h3><p>The traffic pattern initially looked something like this:</p><pre>50+ countries<br>       |<br>       +---- thousands of IP addresses<br>       |<br>       +---- different request payloads<br>       |<br>       +---- different names / random data<br>       |<br>       +---- same target endpoint<br>       |<br>       +---- hundreds of requests per second</pre><p>The attacker was essentially trying to exhaust the OTP workflow.</p><p>The requests contained randomized information, including names generated from a dataset.</p><p>For example:</p><pre>Request 1<br>Name: Michael Anderson<br>IP: X.X.X.X</pre><pre>Request 2<br>Name: Daniel Wilson<br>IP: Y.Y.Y.Y</pre><pre>Request 3<br>Name: Robert Thompson<br>IP: Z.Z.Z.Z</pre><p>Then another IP.</p><p>Another country.</p><p>Another name.</p><p>Another request.</p><p>And another.</p><p>From the application’s perspective, many of these requests looked unique.</p><h3>Why Traditional Rate Limiting Wasn’t Enough</h3><p>A simple IP-based rule might look like:</p><pre>IP A → 10 requests/minute → BLOCK<br>IP B → 10 requests/minute → BLOCK<br>IP C → 10 requests/minute → BLOCK</pre><p>But the attacker was effectively doing:</p><pre>IP 1<br>IP 2<br>IP 3<br>IP 4<br>...<br>IP 10,000+</pre><p>Even if each individual IP generated only a small number of requests, the aggregate traffic against the endpoint was significant.</p><p>So:</p><pre>Per-IP rate limiting<br>        ↓<br>       ❌</pre><p>wasn’t sufficient.</p><p>We also considered geographic blocking.</p><p>But that had the same fundamental problem.</p><p>The traffic was distributed across many countries.</p><p>Blocking one region simply meant that traffic from other regions continued.</p><p>And more importantly, legitimate users could also exist in those regions.</p><p>So:</p><pre>IP blocking      → insufficient<br>Geo blocking     → insufficient<br>Traditional rate limiting → insufficient</pre><p>We needed another dimension.</p><h3>Looking Beyond the IP Address</h3><p>This was the point where the investigation became interesting.</p><p>Instead of asking:</p><blockquote><em>“Where is this request coming from?”</em></blockquote><p>we started asking:</p><blockquote><strong><em>“What is actually generating these requests?”</em></strong></blockquote><p>An IP address identifies a network source.</p><p>But an automated client leaves other characteristics behind.</p><p>The TCP connection.</p><p>The TLS handshake.</p><p>The way the client negotiates TLS.</p><p>The cipher suites.</p><p>The extensions.</p><p>The overall characteristics of the TLS client.</p><p>And this led us to <strong>JA4 fingerprinting</strong>.</p><h3>What Is a JA4 Fingerprint?</h3><p>JA4 is a technique for creating a fingerprint from characteristics of a client’s TLS connection.</p><p>The important idea is that two requests can have completely different:</p><pre>IP<br>Country<br>URL parameters<br>Request body<br>User data</pre><p>and still originate from clients that have the same underlying TLS characteristics.</p><p>Conceptually:</p><pre>Application Layer<br>IP 1  ── Name A ──────────────┐<br>IP 2  ── Name B ──────────────┤<br>IP 3  ── Name C ──────────────┤<br>IP 4  ── Name D ──────────────┤<br>...                            │<br>IP 10000 ── Name X ───────────┤<br>                               ↓<br>                         TLS characteristics<br>                               ↓<br>                         JA4 fingerprint</pre><p>That gave us something very interesting.</p><p>The attacker could change the information inside the request.</p><p>They could change the IP.</p><p>They could change the country.</p><p>They could change the name.</p><p>But the underlying automated client infrastructure was still leaving a recognizable fingerprint.</p><h3>Azure Front Door Gave Us Another Dimension</h3><p>Our application was behind <strong>Azure Front Door</strong>.</p><p>This became particularly useful during the investigation because we could use the JA4 fingerprint exposed in the request/security telemetry.</p><p>When we analyzed the traffic, we found that although the requests were coming from a very large and geographically distributed set of IP addresses, the requests were associated with a <strong>common JA4 fingerprint</strong>.</p><p>That changed the way we looked at the attack.</p><p>Instead of:</p><pre>10,000 IP addresses</pre><p>we could reason about:</p><pre>10,000 IP addresses<br>        ↓<br>distributed infrastructure<br>        ↓<br>common client characteristics<br>        ↓<br>JA4 fingerprint</pre><p>The attack was distributed at the network level.</p><p>But it wasn’t necessarily distributed at the <strong>client fingerprint level</strong>.</p><p>That distinction was extremely useful.</p><h3>The “Aha” Moment</h3><p>This was the most interesting part of the investigation.</p><p>Initially:</p><pre>IP-based analysis<br>IP 1      → Attack<br>IP 2      → Attack<br>IP 3      → Attack<br>IP 4      → Attack<br>...<br>IP 10,000 → Attack</pre><p>It looked like we had 10,000 separate sources.</p><p>But after looking at the JA4 fingerprint:</p><pre>IP 1      ─┐<br>IP 2      ─┤<br>IP 3      ─┤<br>IP 4      ─┤<br>...        ├──→ Same JA4 fingerprint<br>IP 10,000 ─┘</pre><p>That gave us a much stronger signal.</p><p>The attack wasn’t simply:</p><blockquote><em>“10,000 bad IP addresses.”</em></blockquote><p>It was:</p><blockquote><strong><em>“A large number of network sources exhibiting the same client fingerprint and targeting the same business endpoint.”</em></strong></blockquote><p>That was a much more useful security signal.</p><h3>Blocking the Fingerprint</h3><p>Once we identified the recurring JA4 fingerprint associated with the malicious traffic, we could use it as an additional detection/mitigation signal at the edge.</p><p>Instead of trying to maintain a constantly changing list of thousands of IP addresses, we could target the characteristic that remained consistent across the attack traffic.</p><p>Conceptually:</p><pre>Incoming request<br>       |<br>       ↓<br>Azure Front Door<br>       |<br>       ↓<br>Inspect request characteristics<br>       |<br>       ↓<br>JA4 fingerprint<br>       |<br>       ├── Known/legitimate pattern → Continue<br>       |<br>       └── Attack fingerprint → Block</pre><p>This moved the mitigation closer to the edge.</p><p>The application didn’t need to process every malicious request.</p><p>The OTP workflow didn’t need to deal with the traffic.</p><p>And we didn’t need to continuously chase thousands of changing IP addresses.</p><h3>The Bigger Lesson</h3><p>The biggest lesson I took from this incident wasn’t simply:</p><blockquote><em>“Use JA4.”</em></blockquote><p>It was:</p><blockquote><strong><em>Don’t build your entire defense around a single identifier.</em></strong></blockquote><p>An IP address is useful. But it isn’t an identity.</p><p>A country is useful. But it isn’t an identity.</p><p>A request body is useful. But it isn’t an identity.</p><p>Attackers can change all of these.</p><p>Modern bot attacks can rotate:</p><pre>IP addresses<br>↓<br>Countries<br>↓<br>Proxies<br>↓<br>Request parameters<br>↓<br>Payloads<br>↓<br>User-agent values</pre><p>So the more interesting question becomes:</p><blockquote><strong><em>What characteristics remain stable while everything else changes?</em></strong></blockquote><p>That’s where fingerprinting and behavioral analysis become powerful.</p><h3>From IP-Based Security to Behavioral Security</h3><p>The mental model changed for us.</p><h3>Traditional approach</h3><pre>Source IP<br>   ↓<br>Rate limit<br>   ↓<br>Block</pre><h3>More advanced approach</h3><pre>IP<br>+<br>TLS fingerprint<br>+<br>Request pattern<br>+<br>Endpoint<br>+<br>Request frequency<br>+<br>Behavior<br>+<br>Application context<br>        ↓<br>Risk signal<br>        ↓<br>Mitigation</pre><p>No single signal is necessarily perfect.</p><p>But combining signals makes the detection much stronger.</p><h3>There Was Another Interesting Signal</h3><p>The payloads themselves were also interesting.</p><p>The attacker wasn’t simply sending identical requests.</p><p>They were generating randomized data using a large dataset of names and other information.</p><p>That was clearly intended to make the requests look different from one another.</p><p>From an application perspective:</p><pre>Request A ≠ Request B<br>Request B ≠ Request C<br>Request C ≠ Request D</pre><p>But from an infrastructure perspective:</p><pre>Request A ─┐<br>Request B ─┤<br>Request C ─┤── Same underlying automation characteristics<br>Request D ─┤<br>Request E ─┘</pre><p>This is a good example of why <strong>application-level uniqueness doesn’t necessarily mean infrastructure-level uniqueness</strong>.</p><h3>What I Learned</h3><p>This incident changed how I think about distributed attacks.</p><p>Previously, it was easy to think of an attack as:</p><pre>Bad IP → Block IP</pre><p>But large-scale automation doesn’t always work that way.</p><p>A better mental model is:</p><pre>┌── IP 1<br>             ├── IP 2<br>             ├── IP 3<br>Bot network ─┼── IP 4<br>             ├── IP 5<br>             ├── ...<br>             └── IP 10,000+</pre><pre>                    ↓</pre><pre>             Same infrastructure<br>             / client behavior<br>             / TLS characteristics</pre><pre>                    ↓</pre><pre>              JA4 fingerprint</pre><p>The attacker can distribute the traffic.</p><p>But distribution doesn’t automatically mean invisibility.</p><p>Sometimes the infrastructure leaves a pattern.</p><p>You just need to look at the right layer.</p><h3>The Real Challenge: Finding the Signal</h3><p>The hardest part wasn’t writing a blocking rule.</p><p>The difficult part was finding the signal that actually separated the attack traffic from legitimate traffic.</p><p>We went from:</p><pre>&quot;Why can&#39;t we block these IPs?&quot;</pre><p>to:</p><pre>&quot;Why are thousands of IPs behaving similarly?&quot;</pre><p>and eventually:</p><pre>&quot;What characteristic is common across these requests?&quot;</pre><p>That shift in thinking made the difference.</p><h3>Final Takeaway</h3><p>Distributed attacks force you to stop thinking only in terms of IP addresses.</p><p>When an attacker has:</p><ul><li>Thousands of IP addresses</li><li>Multiple geographic locations</li><li>Rotating proxies</li><li>Randomized payloads</li><li>High request volume</li><li>A sensitive business endpoint</li></ul><p>traditional controls may not be enough on their own.</p><p>The answer isn’t necessarily to block everything.</p><p>Instead, look for <strong>stable signals hidden underneath the changing data</strong>.</p><p>In our case, JA4 fingerprinting provided one such signal.</p><p>The attacker changed almost everything we could see at the application layer.</p><p>But the underlying client characteristics still gave us a way to connect the dots.</p><p>And that was probably the most valuable lesson from the incident:</p><blockquote><strong><em>When the attacker keeps changing the identity of the request, look for the identity of the client generating it.</em></strong></blockquote><p>Sometimes the best security signal isn’t what the request says.</p><p>It’s <strong>how the request was made</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90a5f1cfbb59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Redis-to-ELK Log Processing: From Static Logstash Servers to On-Demand Nomad Jobs]]></title>
            <link>https://midhunkrishna-ops.medium.com/scaling-redis-to-elk-log-processing-from-static-logstash-servers-to-on-demand-nomad-jobs-5b5e94d1ae81?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b5e94d1ae81</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[logstash]]></category>
            <category><![CDATA[nomad]]></category>
            <category><![CDATA[elk-stack]]></category>
            <category><![CDATA[hashicorp]]></category>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 09:03:03 GMT</pubDate>
            <atom:updated>2026-10-02T09:03:03.914Z</atom:updated>
            <content:encoded><![CDATA[<h3>When I first started working on one of our large data-processing platforms, one of the things I encountered was a logging architecture that used <strong>Redis as a temporary log buffer</strong>, <strong>Logstash for processing</strong>, and the <strong>ELK stack for search and visualization</strong>.</h3><p>At a high level, the flow looked like this:</p><pre>Application / Data Processing<br>          │<br>          ▼<br>        Redis<br>   (temporary log data)<br>          │<br>          ▼<br>      Logstash<br>          │<br>          ▼<br>     Elasticsearch<br>          │<br>          ▼<br>        Kibana</pre><p>Redis was used for different purposes across the platform. In this particular system, processed-data logs were temporarily stored in Redis lists under different keys.</p><p>Logstash consumed those keys and forwarded the logs to Elasticsearch, where they were indexed and eventually visualized through Kibana.</p><p>I can’t describe the exact business workflow or project because of the confidentiality of the system, but the infrastructure problem itself is an interesting engineering problem.</p><p>And it taught me an important lesson:</p><blockquote><strong><em>An architecture that works perfectly at one scale can become the bottleneck when the workload changes.</em></strong></blockquote><h3>The Initial Architecture</h3><p>Initially, we had more than 20 Logstash servers.</p><p>Each server contained Logstash configurations for multiple Redis keys.</p><p>A simplified configuration looked something like this:</p><pre>input {<br>  redis {<br>    host =&gt; &quot;REDIS_HOST&quot;<br>    data_type =&gt; &quot;list&quot;<br>    key =&gt; &quot;project-a&quot;<br>    codec =&gt; json<br><br>add_field =&gt; {<br>      &quot;service_name&quot; =&gt; &quot;project-a&quot;<br>    }<br>add_field =&gt; {<br>      &quot;logstash_host&quot; =&gt; &quot;${HOSTIP}&quot;<br>    }<br>  }<br>}</pre><p>The logs could then be filtered or throttled before being sent to Elasticsearch:</p><pre>filter {<br>  if [service_name] == &quot;project-a&quot; {</pre><pre>    throttle {<br>      after_count =&gt; 4<br>      period =&gt; 1<br>      max_age =&gt; 5<br>      key =&gt; &quot;project-a&quot;<br>      add_tag =&gt; &quot;throttled&quot;<br>    }</pre><pre>    if &quot;throttled&quot; in [tags] {<br>      drop { }<br>    }<br>  }<br>}</pre><p>Finally, the logs were sent to an Elasticsearch index:</p><pre>output {<br>  if [service_name] == &quot;project-a&quot; {</pre><pre>    elasticsearch {<br>      index =&gt; &quot;project-a-%{+YYYY-MM}&quot;</pre><pre>      hosts =&gt; [<br>        &quot;ELASTICSEARCH_HOST_1&quot;,<br>        &quot;ELASTICSEARCH_HOST_2&quot;<br>      ]</pre><pre>      user =&gt; &quot;${ELASTICSEARCH_USERNAME}&quot;<br>      password =&gt; &quot;${ELASTICSEARCH_PASSWORD}&quot;<br>    }<br>  }<br>}</pre><p>A single Logstash server could have many such configurations.</p><p>The same Redis key could also be consumed by multiple Logstash instances when additional read capacity was required.</p><p>At smaller scale, this worked reasonably well. Then the workload grew.</p><h3>When Scaling Became the Problem</h3><p>As the data-processing workload increased, the amount of logging increased with it.</p><p>More processing meant:</p><pre>More data<br>   ↓<br>More logs<br>   ↓<br>More Redis writes<br>   ↓<br>More Redis data waiting to be consumed</pre><p>The critical part of the system was the relationship between <strong>Redis write speed</strong> and <strong>Logstash consumption speed</strong>.</p><p>If Logstash could consume the data fast enough, everything was fine.</p><p>But when the incoming log volume increased beyond the available consumption capacity, Redis started accumulating more data.</p><p>That created a chain reaction.</p><pre>Higher workload<br>      ↓<br>Higher log volume<br>      ↓<br>Redis backlog increases<br>      ↓<br>Logstash cannot consume fast enough<br>      ↓<br>Redis storage pressure<br>      ↓<br>Processing instability</pre><p>The solution was often straightforward in theory:</p><blockquote><em>Increase Logstash consumption capacity.</em></blockquote><p>But our deployment model made that much harder than it sounds.</p><h3>The Old Way of Scaling</h3><p>If a particular project suddenly generated significantly more logs, we might need additional Logstash capacity for that project’s Redis keys.</p><p>The traditional process looked something like:</p><pre>Provision server<br>      ↓<br>Install Logstash<br>      ↓<br>Configure Logstash<br>      ↓<br>Add Redis key configuration<br>      ↓<br>Configure Elasticsearch output<br>      ↓<br>Deploy<br>      ↓<br>Monitor<br>      ↓<br>Scale down later</pre><p>This was manageable when the requirement was relatively stable.</p><p>But our workloads were not always stable.</p><p>Sometimes we needed additional capacity for a relatively short period.</p><p>That meant manually provisioning infrastructure just to handle a temporary increase in log volume.</p><p>The bigger problem was that this was not always predictable.</p><p>A project could suddenly experience a large increase in data processing.</p><p>At that point, we needed to react quickly.</p><p>And when Redis started accumulating a large backlog, we didn’t necessarily have the luxury of slowly provisioning and configuring several new servers.</p><h3>The Real Problem</h3><p>The problem wasn’t simply:</p><blockquote><em>“We need more Logstash servers.”</em></blockquote><p>The deeper problem was:</p><blockquote><strong><em>Logstash capacity was tightly coupled to manually provisioned infrastructure.</em></strong></blockquote><p>We had many Logstash servers.</p><p>Each server contained multiple configurations.</p><p>The configurations were distributed across machines.</p><p>The same project could have configurations running on different servers.</p><p>Finding exactly where a particular key was being consumed wasn’t always straightforward.</p><p>Scaling therefore meant dealing with several dimensions at once:</p><ul><li>Which Redis keys need additional capacity?</li><li>Which Logstash server should handle them?</li><li>How many instances are required?</li><li>Where should the new instances run?</li><li>How do we configure them?</li><li>How do we remove them later?</li><li>How do we avoid configuration drift?</li></ul><p>The architecture needed a better abstraction.</p><h3>Moving From Servers to Workloads</h3><p>At the time, we were already using <strong>HashiCorp Nomad</strong> for workload scheduling.</p><p>That gave us an interesting possibility.</p><p>Instead of thinking:</p><blockquote><em>“I need another Logstash server.”</em></blockquote><p>we could think:</p><blockquote><em>“I need another Logstash workload consuming this project’s Redis keys.”</em></blockquote><p>That was the turning point.</p><p>We decided to containerize Logstash and make each project’s Logstash workload independently deployable through Nomad.</p><p>Conceptually, the architecture became:</p><pre>┌──────────────────┐<br>                    │      Redis       │<br>                    │ Temporary Logs   │<br>                    └────────┬─────────┘<br>                             │<br>             ┌───────────────┼───────────────┐<br>             │               │               │<br>             ▼               ▼               ▼<br>       ┌───────────┐   ┌───────────┐   ┌───────────┐<br>       │ Logstash  │   │ Logstash  │   │ Logstash  │<br>       │ Project A │   │ Project B │   │ Project C │<br>       └─────┬─────┘   └─────┬─────┘   └─────┬─────┘<br>             │               │               │<br>             └───────────────┼───────────────┘<br>                             ▼<br>                     Elasticsearch<br>                             │<br>                             ▼<br>                          Kibana</pre><p>Each project became an independently deployable workload.</p><h3>Organizing the Configurations</h3><p>The first challenge was configuration management.</p><p>We had a large number of existing Logstash configurations scattered across servers.</p><p>Before automating deployment, I needed to understand and reorganize them.</p><p>I grouped configurations based on projects.</p><p>For example:</p><pre>logstash/<br>│<br>├── project-a/<br>│   ├── config-01.conf<br>│   ├── config-02.conf<br>│   └── config-03.conf<br>│<br>├── project-b/<br>│   ├── config-01.conf<br>│   └── config-02.conf<br>│<br>├── project-c/<br>│   ├── config-01.conf<br>│   ├── config-02.conf<br>│   └── config-03.conf<br>│<br>├── config/<br>│   ├── logstash.yml<br>│   └── other-configs<br>│<br>└── Dockerfile</pre><p>This gave us a much cleaner model.</p><p>A project directory contained everything required for that project’s Logstash pipelines.</p><p>Instead of configurations being tied to individual servers, they became <strong>version-controlled artifacts</strong>.</p><p>That was a major improvement by itself.</p><h3>Building the Logstash Image</h3><p>The next step was creating a reusable Logstash container image.</p><p>The image was based on the official Logstash image, while our project-specific configurations were copied into the container.</p><p>A simplified version looked like:</p><pre>FROM docker.elastic.co/logstash/logstash:&lt;VERSION&gt;</pre><pre>RUN rm -f /usr/share/logstash/pipeline/logstash.conf</pre><pre>RUN mkdir /usr/share/logstash/project-a<br>COPY project-a/ /usr/share/logstash/project-a/</pre><pre>RUN mkdir /usr/share/logstash/project-b<br>COPY project-b/ /usr/share/logstash/project-b/</pre><pre>COPY config/ /usr/share/logstash/config/</pre><p>The important idea wasn’t the Dockerfile itself.</p><p>It was the separation between:</p><p><strong>Common Logstash runtime</strong></p><p>and</p><p><strong>Project-specific configuration.</strong></p><p>The resulting image contained the configurations, but the Nomad job determined which project configuration Logstash should actually execute.</p><p>For example:</p><pre>/usr/share/logstash/project-a/*</pre><p>could be used for Project A.</p><p>Project B could use:</p><pre>/usr/share/logstash/project-b/*</pre><p>The same base image could therefore support multiple independent workloads.</p><h3>Nomad Became the Deployment Layer</h3><p>The next part was defining a Nomad job for each project.</p><p>A simplified job looked like:</p><pre>job &quot;project-a-production&quot; {</pre><pre>  namespace   = &quot;InfraTeam&quot;<br>  datacenters = [&quot;production-DC&quot;]</pre><pre>  type = &quot;service&quot;</pre><pre>  group &quot;project-a-logstash&quot; {</pre><pre>    count = 1</pre><pre>    task &quot;project-a-logstash&quot; {</pre><pre>      driver = &quot;docker&quot;</pre><pre>      config {<br>        image   = &quot;registry/logstash:${var.image_tag}&quot;<br>        command = &quot;bash&quot;</pre><pre>        args = [<br>          &quot;/usr/share/logstash/bin/logstash&quot;,<br>          &quot;-f&quot;,<br>          &quot;/usr/share/logstash/project-a/*&quot;<br>        ]<br>      }</pre><pre>      resources {<br>        cpu    = 1000<br>        memory = 1500<br>      }<br>    }<br>  }<br>}</pre><p>The important part here was:</p><pre>-f /usr/share/logstash/project-a/*</pre><p>The container had all the configurations, but the Nomad job controlled which project’s pipelines were started.</p><p>This created a clean boundary:</p><pre>Container Image<br>      │<br>      ├── Project A configs<br>      ├── Project B configs<br>      ├── Project C configs<br>      └── Common configuration<br>             │<br>             ▼<br>       Nomad Job decides<br>       what actually runs</pre><h3>Independent Scaling</h3><p>This was where the new architecture became much more useful.</p><p>Previously, scaling could mean:</p><blockquote><em>“Give me another server.”</em></blockquote><p>Now it became:</p><blockquote><em>“Increase the number of Logstash instances for Project A.”</em></blockquote><p>For example:</p><pre>Project A</pre><pre>1 instance<br>   ↓<br>2 instances<br>   ↓<br>4 instances<br>   ↓<br>8 instances</pre><p>while Project B could continue running independently:</p><pre>Project B → 1 instance</pre><p>There was no need to scale every project’s Logstash capacity just because one project experienced a spike.</p><p>The unit of scaling had changed.</p><h3>Before</h3><pre>Server → many projects</pre><h3>After</h3><pre>Project → independent workload</pre><p>That was the architectural improvement I was looking for.</p><h3>GitLab CI/CD</h3><p>The next challenge was deployment.</p><p>Since the configurations were already maintained in GitLab, I wanted deployment to happen through the same workflow.</p><p>The goal was:</p><pre>Developer / Engineer<br>        │<br>        ▼<br>     GitLab<br>        │<br>        ▼<br>      Build<br>        │<br>        ▼<br>     Docker Image<br>        │<br>        ▼<br>      Registry<br>        │<br>        ▼<br>   Nomad Deployment<br>        │<br>        ▼<br> Project-specific<br> Logstash workload</pre><p>The pipeline was divided into stages:</p><pre>stages:<br>  - build<br>  - push<br>  - deploy<br>  - clean</pre><p>The image was built first:</p><pre>docker_build:<br>  stage: build</pre><pre>  script:<br>    - docker build -t $CI_PROJECT_PATH_SLUG:$DOCKER_IMAGE_TAG .</pre><p>Then pushed to the container registry.</p><p>Finally, the deployment stage executed the appropriate Nomad job.</p><p>Conceptually:</p><pre>nomad job run deployment/production/$PROJECT.nomad</pre><p>The project selection determined which workload would be deployed.</p><h3>Secrets and Runtime Configuration</h3><p>Another important part was separating application configuration from secrets.</p><p>Credentials were not hardcoded inside the Logstash configuration or Nomad job.</p><p>The deployment integrated with our secret-management system, and runtime environment variables were injected when the workload started.</p><p>That meant the repository could contain the deployment definitions without exposing sensitive credentials.</p><p>Conceptually:</p><pre>GitLab<br>  │<br>  ├── Logstash configurations<br>  ├── Dockerfile<br>  └── Nomad job definitions<br>          │<br>          ▼<br>       CI/CD<br>          │<br>          ▼<br>       Nomad<br>          │<br>          ├── Runtime secrets<br>          │<br>          ▼<br>      Logstash Pod/Task</pre><p>This separation also made configuration review much easier.</p><h3>The Final Model</h3><p>The resulting system looked approximately like this:</p><pre>GitLab<br>                           │<br>                           ▼<br>                     GitLab CI/CD<br>                           │<br>              ┌────────────┴────────────┐<br>              │                         │<br>              ▼                         ▼<br>        Docker Build              Nomad Jobs<br>              │                         │<br>              ▼                         ▼<br>       Container Registry          Project A<br>                                     Project B<br>                                     Project C<br>                                        │<br>                                        ▼<br>                                   Logstash<br>                                        │<br>                                        ▼<br>                                      Redis<br>                                        │<br>                                        ▼<br>                                Elasticsearch<br>                                        │<br>                                        ▼<br>                                     Kibana</pre><p>Each project had its own deployment definition.</p><p>Each deployment definition pointed Logstash to the appropriate project configuration.</p><p>And each project could be scaled independently.</p><h3>What Changed Operationally?</h3><p>The biggest improvement wasn’t simply “we started using containers.”</p><p>The bigger change was <strong>how we thought about the infrastructure</strong>.</p><h3>Before</h3><pre>Redis key<br>   ↓<br>Logstash configuration<br>   ↓<br>Specific server</pre><p>The configuration and infrastructure were tightly coupled.</p><h3>After</h3><pre>Redis key<br>   ↓<br>Project configuration<br>   ↓<br>Container<br>   ↓<br>Nomad workload</pre><p>The configuration became portable.</p><p>The workload became schedulable.</p><p>And scaling became much easier.</p><h3>From Static Infrastructure to Elastic Workloads</h3><p>The old model required us to think in terms of servers.</p><p>If we needed more capacity, we had to provision more infrastructure.</p><p>The new model allowed us to think in terms of workloads.</p><p>If Project A needed more capacity:</p><pre>Project A<br>   │<br>   ├── Logstash instance 1<br>   ├── Logstash instance 2<br>   ├── Logstash instance 3<br>   └── ...</pre><p>Project B didn’t need to be affected.</p><p>This was particularly useful for workloads with unpredictable traffic patterns.</p><p>Instead of permanently maintaining infrastructure for peak capacity, we could scale the workload according to demand.</p><h3>Lessons I Took From This</h3><p>This was one of the projects that helped me understand that <strong>scaling isn’t always about adding more resources</strong>.</p><p>Sometimes the real problem is the way the workload is packaged and deployed.</p><p>A few lessons stood out to me.</p><h3>1. Find the actual scaling unit</h3><p>Initially, our scaling unit was the server.</p><p>But the actual requirement was:</p><blockquote><em>Log consumption capacity per project.</em></blockquote><p>Once we aligned the deployment model with the actual requirement, scaling became much simpler.</p><h3>2. Separate configuration from infrastructure</h3><p>Having Logstash configurations tied directly to individual servers made the environment harder to operate.</p><p>Moving configurations into Git and packaging them into containers made them version-controlled, reproducible, and portable.</p><h3>3. Containers are useful beyond application workloads</h3><p>Logstash isn’t a typical web application.</p><p>But it still benefited from containerization because the container gave us a consistent runtime and allowed the workload to be scheduled independently.</p><h3>4. Automation becomes more valuable as the workload grows</h3><p>Manually configuring one server isn’t necessarily a problem.</p><p>Manually configuring 20+ servers during an unexpected traffic spike is a very different problem.</p><p>The larger the environment became, the more expensive manual operations became.</p><h3>5. Scaling should be independent</h3><p>If Project A needs more Logstash capacity, Project B shouldn’t have to scale with it.</p><p>Independent workloads gave us that flexibility.</p><h3>Looking Back</h3><p>This was one of my earlier experiences working on a large-scale logging pipeline, and it was also one of the projects that pushed me deeper into infrastructure automation.</p><p>The interesting part wasn’t just moving Logstash into containers.</p><p>The real engineering challenge was recognizing that our original deployment model had become the bottleneck.</p><p>We moved from:</p><pre>Static servers<br>      ↓<br>Manually managed configurations<br>      ↓<br>Difficult scaling</pre><p>to:</p><pre>Version-controlled configurations<br>      ↓<br>Containerized Logstash<br>      ↓<br>Project-specific Nomad workloads<br>      ↓<br>GitLab CI/CD<br>      ↓<br>Independent scaling</pre><p>The biggest takeaway for me was simple:</p><blockquote><strong><em>When a system starts struggling to scale, don’t immediately ask, “How many more servers do we need?”</em></strong></blockquote><blockquote><strong><em>First ask, “What is the actual unit that needs to scale, and is our architecture built around that unit?”</em></strong></blockquote><p>Sometimes the solution isn’t adding more machines.</p><p>Sometimes it’s changing the way the workload is packaged, deployed, and scaled.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b5e94d1ae81" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Still Remember the Day SSH Refused to Work on Jio]]></title>
            <link>https://midhunkrishna-ops.medium.com/i-still-remember-the-day-ssh-refused-to-work-on-jio-31eefc791cc0?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/31eefc791cc0</guid>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[network-packets]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[jio]]></category>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 19:39:18 GMT</pubDate>
            <atom:updated>2026-10-01T19:43:43.478Z</atom:updated>
            <content:encoded><![CDATA[<h3>Every once in a while, I remember an old incident from my early DevOps days and think:</h3><p><strong>“That was actually a really good troubleshooting exercise.”</strong></p><p>This is one of those incidents.</p><p>It happened around <strong>four years ago</strong>.</p><p>I don’t remember every technical detail anymore, but I remember the problem, the investigation, the packet captures, the discussions, and most importantly, the feeling of finally figuring out what was happening.</p><p>The problem sounded simple:</p><blockquote><strong><em>SSH was not working.</em></strong></blockquote><p>But there was a catch.</p><p>It was working everywhere else.</p><h3>Everything Worked. Except SSH.</h3><p>The issue was happening only under a very specific combination:</p><p><strong>Jio network + Pritunl VPN + SSH</strong></p><p>If the client was using a different network, SSH was fine.</p><p>If we accessed other services, they were fine.</p><p>HTTP worked.</p><p>HTTPS worked.</p><p>The VPN itself worked.</p><p>But SSH was behaving strangely.</p><p>Something like:</p><pre>HTTP       → ✅<br>HTTPS      → ✅<br>VPN        → ✅<br>SSH        → ❌</pre><p>And that immediately made this a much more interesting problem.</p><p>If the server was completely unreachable, troubleshooting would have been straightforward.</p><p>But it wasn’t.</p><p>If the VPN was broken, we would expect other traffic to fail too.</p><p>But it wasn’t.</p><p>So we had a very specific question:</p><blockquote><strong><em>Why does SSH have a problem when the rest of the connectivity appears to be working?</em></strong></blockquote><h3>This Was My Opportunity to Go Deeper Into Networking</h3><p>At that point, <strong>I was leading the troubleshooting</strong>.</p><p>I knew this wasn’t going to be solved by simply checking SSH configuration and changing a few settings. The behavior was unusual, and I wanted to understand what was actually happening underneath.</p><p>Since <a href="https://www.linkedin.com/in/athirams/"><strong>Athira</strong></a><strong> had strong networking knowledge</strong>, I brought her into the investigation. I knew her networking experience would be valuable, especially because we were starting to move beyond the application layer and into the network itself.</p><p>For me, this became a great opportunity to go much deeper into networking.</p><p>Instead of guessing, we decided to look at what was actually happening on the wire.</p><p>We weren’t going to start by randomly changing SSH settings.</p><p><strong>We were going to watch the traffic.</strong></p><h3>Wireshark on One Side. tshark on the Other.</h3><p>On my machine, we used <strong>Wireshark</strong>.</p><p>It gave us the nice graphical view of the connection.</p><p>We could follow the TCP stream, inspect individual packets, check sequence numbers, acknowledgements, timestamps, retransmissions, and much more.</p><p>On the server, we used <strong>tshark</strong>.</p><p>The server didn’t have a GUI environment where we could conveniently run Wireshark, so tshark gave us packet-level visibility directly from the command line.</p><p>The setup looked something like this:</p><pre>Network Path</pre><pre>┌──────────────┐                                  ┌──────────────┐<br>│    Client    │                                  │    Server    │<br>│              │                                  │              │<br>│  Wireshark   │                                  │    tshark    │<br>└──────┬───────┘                                  └──────┬───────┘<br>       │                                                  │<br>       │              Jio Network                        │<br>       │                    ↓                             │<br>       │                Internet                         │<br>       │                    ↓                             │<br>       │              Pritunl VPN                        │<br>       │                    ↓                             │<br>       └───────────────────┴──────────────────────────────┘<br>                         SSH Traffic</pre><p>The objective was simple:</p><blockquote><strong><em>See what is happening at both ends of the connection.</em></strong></blockquote><h3>We Stopped Looking at SSH and Started Looking at Packets</h3><p>This was probably the most interesting part of the whole incident.</p><p>Instead of:</p><blockquote><em>“SSH isn’t connecting.”</em></blockquote><p>we started asking:</p><blockquote><strong><em>“What is happening to the packets?”</em></strong></blockquote><p>We inspected the traffic in real time.</p><p>We looked at the packets leaving the server.</p><p>Then we looked at what was arriving at the client.</p><p>We compared timestamps.</p><p>We looked at TCP sequence numbers.</p><p>We checked acknowledgements.</p><p>We watched for retransmissions.</p><p>We followed the SSH connection as it progressed.</p><p>It felt less like debugging an application and more like watching two machines have a conversation.</p><pre>Client                                  Server<br>  |                                       |<br>  | -------- TCP / SSH packet ----------&gt; |<br>  |                                       |<br>  | &lt;------- TCP / SSH response ----------|<br>  |                                       |<br>  | -------- ACK ------------------------&gt;|<br>  |                                       |</pre><p>Except sometimes, what the server said it had sent and what the client appeared to receive were not lining up the way we expected.</p><p>That was the clue we needed.</p><h3>The Server Wasn’t Simply “Slow”</h3><p>One of the important observations we made was around <strong>timing</strong>.</p><p>We could see the server transmitting traffic.</p><p>But the corresponding traffic wasn’t reaching the client at exactly the expected time.</p><p>That changed our thinking.</p><p>The question was no longer:</p><blockquote><em>“Why is the SSH server taking so long to respond?”</em></blockquote><p>It became:</p><blockquote><strong><em>“If the server sent the packet, what happened to it on the way to the client?”</em></strong></blockquote><p>That is a completely different troubleshooting problem.</p><p>Now we had to investigate the path:</p><pre>Server<br>   ↓<br>VPN<br>   ↓<br>Internet<br>   ↓<br>Jio network<br>   ↓<br>Client</pre><p>Somewhere in that path, something was behaving differently.</p><h3>Why Did HTTP and HTTPS Still Work?</h3><p>This was one of the things that made the issue confusing.</p><p>If there was a networking problem, why weren’t HTTP and HTTPS affected?</p><p>The answer isn’t necessarily that a network problem must affect every protocol equally.</p><h3>TCP Traffic Can Behave Differently Depending On</h3><ul><li><strong>Packet size</strong> — how much data is carried in each TCP segment</li><li><strong>MSS (Maximum Segment Size)</strong> — the maximum TCP payload size negotiated between endpoints</li><li><strong>MTU (Maximum Transmission Unit)</strong> — the maximum packet size supported by a network interface/path</li><li><strong>TCP segmentation</strong> — how larger application data is split into multiple TCP segments</li><li><strong>Retransmissions</strong> — what happens when packets or acknowledgements are lost</li><li><strong>Path MTU Discovery (PMTUD)</strong> — how endpoints determine the largest packet size that can traverse the path without fragmentation</li><li><strong>VPN encapsulation</strong> — additional headers introduced by VPN protocols, reducing the effective payload size</li><li><strong>Network-path characteristics</strong> — routing, packet loss, latency, filtering, and behavior of intermediate networks</li></ul><p>So it is entirely possible to have a situation where:</p><pre>Small / different traffic pattern<br>        ↓<br>       Works</pre><pre>Different packet behavior<br>        ↓<br>       Problem</pre><p>That’s why “HTTPS works” isn’t enough to conclude that the network path is healthy for every kind of traffic.</p><h3>Then We Looked at the VPN</h3><p>The next important piece was <strong>Pritunl</strong>.</p><p>The client wasn’t communicating with the server over a simple direct connection.</p><p>There was a VPN tunnel in between.</p><p>Conceptually:</p><pre>Client<br>  |<br>  | Jio<br>  |<br>  v<br>Internet<br>  |<br>  v<br>Pritunl<br>  |<br>  | VPN tunnel<br>  |<br>  v<br>Server</pre><p>And VPN tunnels introduce additional packet overhead.</p><p>A normal packet might look conceptually like:</p><pre>+-------------------------------+<br>| IP | TCP | Application Data   |<br>+-------------------------------+</pre><p>After VPN encapsulation, it becomes something more like:</p><pre>+------------------------------------------------+<br>| Outer IP | VPN | Inner IP | TCP | Application |<br>+------------------------------------------------+</pre><p>That extra encapsulation matters.</p><p>Because now the packet has to fit through the MTU available on the underlying network path.</p><p>And that brought us to one of the oldest networking concepts:</p><p><strong>MTU.</strong></p><h3>The MTU Rabbit Hole</h3><p>MTU stands for <strong>Maximum Transmission Unit</strong>.</p><p>In simple terms, it represents the maximum size of an IP packet that can be transmitted over a particular link without fragmentation.</p><p>A common Ethernet MTU is:</p><pre>1500 bytes</pre><p>But when traffic passes through VPN tunnels, additional headers consume part of that space.</p><p>So the effective packet size available to the original traffic can be smaller.</p><p>Conceptually:</p><pre>Normal network</pre><pre>[------------- 1500 bytes -------------]<br></pre><pre>VPN</pre><pre>[Outer][VPN][----------- Inner packet -----------]<br>       ↑<br>       Additional overhead</pre><p>And if the network path has an even smaller effective MTU, things can become complicated.</p><p>Packets may need to be fragmented, dropped, retransmitted, or otherwise handled differently depending on how Path MTU Discovery and the intermediate network behave.</p><p>We weren’t initially looking at MTU.</p><p>The packet observations gradually pushed us in that direction.</p><h3>Then Came the Experiment</h3><p>Eventually, the observations from our packet investigation were passed to <a href="https://www.linkedin.com/in/vibin-vinoy-mathew-38393923a/"><strong>Vibin</strong></a>. He Took the Investigation Further</p><p>At this point, I had shared the packet-level observations and everything we had found so far with him.</p><p>He continued digging into the network behavior and suggested that we test different <strong>MTU values</strong>.</p><p>That turned out to be an important direction.</p><p>We started with the default:</p><pre>MTU 1500</pre><p>The problem was still there.</p><p>Then we reduced it:</p><pre>MTU 1250</pre><p>And suddenly, <strong>SSH worked</strong>.</p><p>We tested another value:</p><pre>MTU 1200</pre><p>SSH worked there as well.</p><p>So the results were pretty clear:</p><pre>1500  → ❌ SSH problem<br>1250  → ✅ SSH working<br>1200  → ✅ SSH working</pre><p>At that point, we had a very strong practical clue.</p><p><strong>Reducing the MTU resolved the issue.</strong></p><p>We didn’t have to guess anymore. We had changed one specific network parameter, reproduced the problem, changed it, and watched the behavior change.</p><p>For me, this was one of the most interesting parts of the whole investigation. What initially looked like an <strong>SSH problem</strong> had taken us all the way down to <strong>packet captures, VPN behavior, network paths, and MTU</strong>.</p><h3>I Wish I Had Saved the Packet Capture</h3><p>This is the part where I need to add a disclaimer to my own memory.</p><p>This happened roughly four years ago.</p><p>I don’t have the original Wireshark capture or the complete RCA sitting in front of me today.</p><p>So I don’t want to claim that I remember the exact packet, exact SSH message, or exact reason for every delay.</p><p>I remember the observations.</p><p>I remember the troubleshooting.</p><p>I remember comparing the client-side Wireshark capture with the server-side tshark output.</p><p>And I remember that reducing the MTU to <strong>1250 or 1200</strong> resolved the problem.</p><p>The exact packet-level explanation is something I would want to verify from the original capture before claiming it as fact.</p><p>And honestly, that’s an important lesson too:</p><p><strong>Don’t turn an old memory into a confident RCA.</strong></p><h3>What Made This Incident Special</h3><p>Looking back, the actual fix wasn’t the only interesting part.</p><p>The investigation itself was the valuable experience.</p><p>It gave me an opportunity to explore areas of networking that I didn’t get to work with every day.</p><p>I learned to look at:</p><pre>Application<br>    ↓<br>SSH<br>    ↓<br>TCP<br>    ↓<br>IP<br>    ↓<br>VPN<br>    ↓<br>Network Path</pre><p>Instead of stopping at the application layer.</p><p>And I got to work alongside people with different strengths.</p><p><strong>Athira</strong> helped throughout the networking investigation and brought strong networking knowledge.</p><p>I led the troubleshooting and drove the packet-level investigation.</p><p>Then <strong>Vibin</strong> took our observations further and connected them to the MTU change that ultimately resolved the problem.</p><p>That’s how real troubleshooting often works.</p><p>Not:</p><blockquote><em>One person knows everything.</em></blockquote><p>But:</p><blockquote><em>One person observes something.<br>Another person notices a pattern.<br>Someone else tests a hypothesis.<br>And eventually, the pieces fit together.</em></blockquote><h3>Four Years Later, This Is What I Remember</h3><p>I don’t remember every packet.</p><p>I don’t remember every timestamp.</p><p>I don’t remember exactly which Wireshark filter we used.</p><p>But I remember sitting there and watching packets move between two machines.</p><p>I remember switching between Wireshark on the client and tshark on the server.</p><p>I remember trying to understand why a packet sent from one side wasn’t showing up on the other side when we expected it.</p><p>And I remember the moment when changing the MTU made the problem disappear.</p><p>That experience gave me something much more valuable than just fixing one SSH issue.</p><p>It taught me to <strong>go lower in the stack when the behavior doesn’t make sense.</strong></p><h3>Sometimes, You Just Have to Follow the Packet</h3><p>When an application says:</p><blockquote><strong><em>“Connection failed.”</em></strong></blockquote><p>that’s only the beginning of the investigation.</p><p>There could be something wrong with:</p><pre>Application<br>      ↓<br>Transport<br>      ↓<br>Network<br>      ↓<br>VPN<br>      ↓<br>Routing<br>      ↓<br>MTU<br>      ↓<br>Physical network</pre><p>The application only knows that something went wrong.</p><p>The packets can tell you <strong>where to start looking</strong>.</p><p>And that’s probably why I still remember this incident after four years.</p><p>It wasn’t just an SSH problem.</p><p>It was one of those rare troubleshooting opportunities where you get to step back, open Wireshark, inspect the traffic, compare both ends, and slowly work your way toward the answer.</p><p>Sometimes the most interesting debugging sessions begin with a very simple question:</p><p><em>Why the hell does SSH work everywhere except on this one network?</em></p><p>And sometimes, the answer is hiding somewhere between two packets….</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31eefc791cc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Server Provisioning: From a 3-Hour Manual Process to a 10-Minute Workflow]]></title>
            <link>https://midhunkrishna-ops.medium.com/automating-server-provisioning-from-a-3-hour-manual-process-to-a-10-minute-workflow-62d87a6142d0?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/62d87a6142d0</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 18:36:56 GMT</pubDate>
            <atom:updated>2026-10-01T18:36:56.464Z</atom:updated>
            <content:encoded><![CDATA[<p>In infrastructure engineering, some of the most valuable automation projects start with a very simple question:</p><blockquote><em>Why are we doing the same three-hour task every time a server is needed?</em></blockquote><p>Back in <strong>2022</strong>, I was working with an environment built around <strong>Apache Mesos / DC/OS</strong> for orchestration. At that time, automatic scaling was not available for the way our environment needed to operate.</p><p>Business requirements, however, meant that servers still had to be added and removed frequently.</p><p>The problem was that provisioning one server was a long manual process.</p><p>A single server could take <strong>2–3 hours or more</strong> to provision, configure, harden, install monitoring, prepare for DC/OS and make ready for use.</p><p>With guidance and support from my senior team members, I played a key role in designing and implementing an automation workflow that reduced the overall process to approximately <strong>10 minutes</strong>.</p><p>This is the story behind that design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oFYnXsKeBUChZD3YQJ2kcg.jpeg" /></figure><h3>The Requirement</h3><p>The servers were hosted on <strong>OVH</strong>.</p><p>Whenever additional capacity was required, the process involved several different activities:</p><ol><li>Provision the server.</li><li>Select the required server size.</li><li>Select the operating system.</li><li>Configure SSH access and keys.</li><li>Configure the infrastructure definition.</li><li>Wait for the server to become available.</li><li>Update and harden the operating system.</li><li>Install required packages.</li><li>Install monitoring.</li><li>Configure the server for DC/OS.</li><li>Add it to the required cluster group.</li><li>Share the server details with the team.</li></ol><p>None of these steps were particularly difficult.</p><p>The problem was that they were manual, sequential and repetitive.</p><p>The objective was therefore to turn the whole workflow into a <strong>repeatable self-service process</strong>.</p><h3>The Design</h3><p>The desired experience was simple:</p><pre>User<br>  ↓<br>Choose server details<br>  ↓<br>Submit<br>  ↓<br>Wait<br>  ↓<br>Server is ready</pre><p>Behind that simple interface, however, there was an entire automation chain:</p><pre>Appsmith<br>   ↓<br>Flask API<br>   ↓<br>Terraform Template<br>   ↓<br>Git Repository<br>   ↓<br>GitLab CI/CD<br>   ↓<br>Terraform<br>   ↓<br>OVH<br>   ↓<br>Ansible AWX<br>   ↓<br>Configured Server<br>   ↓<br>Google Chat</pre><p>The key design principle was to give every component a specific responsibility.</p><h3>1. Appsmith — The Self-Service UI</h3><p>I used <strong>Appsmith</strong> for the user interface.</p><p>Appsmith was running as a Docker container.</p><p>The UI was intentionally simple. The person requesting a server did not need to understand Terraform or the underlying OVH infrastructure.</p><p>The form accepted values such as:</p><ul><li>Server name</li><li>Server size</li><li>Operating system</li><li>Cluster group</li></ul><p>The server size options were predefined.</p><p>The operating system could be selected as Linux or Windows.</p><p>The cluster group was also part of the request.</p><p>This meant users could request infrastructure without manually editing Terraform files.</p><p>The UI became the front door of the entire automation.</p><h3>2. Appsmith → Flask API</h3><p>After the form was submitted, Appsmith sent the information to a <strong>Flask API</strong>.</p><p>Conceptually:</p><pre>Appsmith<br>   │<br>   │ API request<br>   ▼<br>Flask API</pre><p>The Flask application was responsible for processing the request and preparing the Terraform configuration.</p><p>This was the bridge between the user-facing application and the infrastructure-as-code workflow.</p><h3>3. Generating Terraform Automatically</h3><p>The Flask application used a <strong>predefined Terraform template</strong>.</p><p>Instead of generating infrastructure code from scratch every time, the template already contained the structure required to create a server.</p><p>The values supplied by Appsmith were inserted into that template.</p><p>Conceptually:</p><pre>Predefined Terraform Template<br>             +<br>       User Parameters<br>             │<br>             ▼<br>      Server-specific .tf</pre><p>For example, the request could contain:</p><pre>Server Name  → mesos-worker-01<br>Size         → predefined-size<br>OS           → Linux<br>Cluster      → cluster-a</pre><p>Flask would replace the corresponding values in the Terraform template.</p><p>A new .tf file was then created using the server name.</p><h3>4. Git Became the Trigger</h3><p>After generating the Terraform configuration, the Flask application pushed the new file into the Git repository used for server creation.</p><p>The repository already had GitLab CI/CD configured.</p><p>Therefore, the Git push automatically started the next stage.</p><pre>Flask<br>  │<br>  │ Generate .tf<br>  │<br>  │ Git push<br>  ▼<br>Git Repository<br>  │<br>  │ Change detected<br>  ▼<br>GitLab CI/CD</pre><p>This was an important part of the design.</p><p>The API did not have to directly execute Terraform.</p><p>Instead, it generated the infrastructure definition and handed it to the existing Git-based automation workflow.</p><h3>5. GitLab CI/CD + Terraform</h3><p>Once the repository changed, GitLab CI/CD automatically ran the pipeline.</p><p>The pipeline executed Terraform.</p><p>The infrastructure was hosted on OVH, and Terraform used the <strong>OpenStack provider</strong> to provision the server.</p><p>The creation workflow also included the required SSH keys and other information needed by the server.</p><p>The flow became:</p><pre>GitLab CI/CD<br>      ↓<br>Terraform<br>      ↓<br>OpenStack Provider<br>      ↓<br>OVH<br>      ↓<br>New Server</pre><p>At this point, the machine existed.</p><p>But that was not the end of the process.</p><h3>6. A VM Is Not a Ready Server</h3><p>One of the important lessons from this project was that provisioning a VM is only one part of server provisioning.</p><p>The newly created machine still needed to be prepared for the environment.</p><p>It needed:</p><ul><li>Updates</li><li>Hardening</li><li>Required packages</li><li>Monitoring</li><li>DC/OS configuration</li><li>Cluster configuration</li></ul><p>That is where <strong>Ansible AWX</strong> came into the design.</p><h3>7. Terraform → Ansible AWX</h3><p>After Terraform successfully created the server, information such as the server’s <strong>IP address and username</strong> was passed to the Ansible AWX server.</p><p>Terraform’s local-exec functionality was used to pass the information to the next stage.</p><p>Conceptually:</p><pre>Terraform<br>    │<br>    │ IP<br>    │ Username<br>    │ Request parameters<br>    ▼<br>Ansible AWX</pre><p>The AWX server contained the organisation&#39;s <strong>server-setup Ansible automation</strong>.</p><p>The parameters collected during the earlier stages were used again during configuration.</p><p>This created a clean separation:</p><pre>Terraform<br>   │<br>   └── Create infrastructure</pre><pre>Ansible AWX<br>   │<br>   └── Configure infrastructure</pre><p>Terraform created the machine.</p><p>Ansible prepared the machine.</p><h3>8. Server Setup with Ansible AWX</h3><p>The server-setup Ansible workflow performed the configuration required to make the server usable.</p><p>The process included tasks such as:</p><pre>New Server<br>    │<br>    ├── Update<br>    ├── Harden<br>    ├── Install required packages<br>    ├── Install Zabbix monitoring<br>    ├── Configure DC/OS requirements<br>    ├── Configure cluster requirements<br>    └── Complete server setup</pre><p>The important part was that the automation continued beyond VM creation.</p><p>The goal was not:</p><blockquote><em>“The VM exists.”</em></blockquote><p>The goal was:</p><blockquote><em>“The server is ready to be used.”</em></blockquote><h3>9. Google Chat — Making the Change Visible</h3><p>There was another operational requirement.</p><p>The team needed visibility into infrastructure changes.</p><p>So the workflow also sent a webhook to a <strong>Google Chat channel</strong>.</p><p>The message included information such as:</p><ul><li>Who created the server</li><li>When it was created</li><li>Server purpose</li><li>Server cost</li><li>Server IP address</li><li>Relevant server information</li></ul><p>Conceptually:</p><pre>Server Provisioned</pre><pre>Server: mesos-worker-01<br>IP: x.x.x.x<br>Purpose: DC/OS worker<br>Created by: &lt;user&gt;<br>Created at: &lt;timestamp&gt;<br>Cost: &lt;cost&gt;</pre><p>This provided a simple operational record without requiring someone to manually announce the provisioning.</p><h3>The Complete Architecture</h3><p>Putting all the pieces together:</p><pre>┌───────────────┐<br>                    │     User      │<br>                    └───────┬───────┘<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │   Appsmith    │<br>                    │  Self-Service │<br>                    └───────┬───────┘<br>                            │<br>                       API Request<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │  Flask API    │<br>                    └───────┬───────┘<br>                            │<br>                    Terraform Template<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │ Generated .tf │<br>                    └───────┬───────┘<br>                            │<br>                         Git Push<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │ Git + GitLab  │<br>                    │     CI/CD     │<br>                    └───────┬───────┘<br>                            │<br>                         Pipeline<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │   Terraform   │<br>                    └───────┬───────┘<br>                            │<br>                   OpenStack Provider<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │      OVH      │<br>                    │     Server    │<br>                    └───────┬───────┘<br>                            │<br>                       IP + Username<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │  Ansible AWX  │<br>                    │ Server Setup  │<br>                    └───────┬───────┘<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │   Configured  │<br>                    │    Server     │<br>                    └───────┬───────┘<br>                            │<br>                            ▼<br>                    ┌───────────────┐<br>                    │  Google Chat  │<br>                    │   Webhook     │<br>                    └───────────────┘</pre><h3>Before vs After</h3><p>The difference was significant. Earlier, the entire server provisioning process involved several manual steps. Server requests were handled manually, followed by manual provisioning, server creation, configuration, monitoring, DC/OS setup, and notifications. This made the overall process time-consuming and dependent on multiple manual interventions, often taking <strong>2–3 hours or more</strong>.</p><p>With the new workflow, we automated most of these steps using <strong>Appsmith, Terraform, GitLab CI/CD, Ansible, Ansible AWX, and Google Chat</strong>. Server requests could be initiated through Appsmith, infrastructure was provisioned using Terraform, configuration was handled through Ansible, and the subsequent setup, monitoring, and notifications were automated through the pipeline.</p><p>With guidance from our senior engineers, I contributed to developing this automation workflow, which brought the overall provisioning time down to approximately <strong>10 minutes</strong>.</p><p>The most important change was not one particular tool.</p><p>It was connecting the tools into a single workflow.</p><h3>Why I Chose This Architecture</h3><p>It would have been possible to write one large script that created the VM and configured everything.</p><p>I preferred to separate the responsibilities.</p><pre>Appsmith<br>   ↓<br>User interaction</pre><pre>Flask<br>   ↓<br>Request processing + Terraform generation</pre><pre>Git<br>   ↓<br>Infrastructure source of truth</pre><pre>GitLab CI/CD<br>   ↓<br>Automation execution</pre><pre>Terraform<br>   ↓<br>Infrastructure provisioning</pre><pre>OpenStack Provider<br>   ↓<br>OVH integration</pre><pre>Ansible AWX<br>   ↓<br>Server configuration</pre><pre>Google Chat<br>   ↓<br>Operational visibility</pre><p>Each component had a clear job.</p><p>That made the overall system easier to reason about and allowed existing tools to be connected instead of replacing them with one custom application.</p><h3>The Real Value of the Automation</h3><p>The obvious result was speed.</p><p>We went from approximately:</p><p><strong>2–3+ hours → ~10 minutes</strong></p><p>But the larger improvement was consistency.</p><p>The process became:</p><pre>Request<br>   ↓<br>Generate<br>   ↓<br>Version<br>   ↓<br>Provision<br>   ↓<br>Configure<br>   ↓<br>Monitor<br>   ↓<br>Notify</pre><p>The engineer requesting a server no longer needed to manually coordinate all those stages.</p><p>The automation handled the repetitive work.</p><h3>What I Learned</h3><h3>Automate the Workflow, Not Just the Task</h3><p>Creating the VM was not the actual problem.</p><p>The actual problem was the entire chain between:</p><blockquote><em>“We need a server”</em></blockquote><p>and</p><blockquote><em>“The server is ready.”</em></blockquote><p>That meant provisioning, configuration, monitoring, cluster setup and notification all needed to be part of the design.</p><h3>Self-Service Does Not Mean Uncontrolled</h3><p>The UI exposed predefined server sizes and controlled parameters.</p><p>Users could request infrastructure without receiving unrestricted access to the underlying Terraform configuration.</p><h3>Git Can Be an Automation Boundary</h3><p>Instead of directly running Terraform from the API, the generated infrastructure configuration was pushed to Git.</p><p>GitLab CI/CD then handled execution.</p><p>This provided a clean boundary between request processing and infrastructure execution.</p><h3>Infrastructure and Configuration Are Different Problems</h3><p>Terraform and Ansible solved different parts of the lifecycle.</p><p>That separation was one of the important architectural decisions in the project.</p><h3>Final Thoughts</h3><p>Looking back, this was one of those projects where the technology was less important than the workflow design.</p><p>The requirement was simple:</p><blockquote><em>We need to add servers frequently, but manually provisioning each one takes several hours.</em></blockquote><p>The solution was to remove the repetitive manual steps and connect the tools already available in the environment.</p><p>The final workflow became:</p><pre>Appsmith<br>   ↓<br>Flask<br>   ↓<br>Terraform<br>   ↓<br>GitLab CI/CD<br>   ↓<br>OVH<br>   ↓<br>Ansible AWX<br>   ↓<br>Google Chat</pre><p>A process that previously required <strong>2–3 hours or more</strong> could be completed in approximately <strong>10 minutes</strong>.</p><p>That was the real outcome of the design: turning a repetitive infrastructure task into a repeatable, self-service engineering workflow.</p><p><strong><em>A big thanks to my teammates for their valuable mentoring, guidance, suggestions, ideas, and contributions throughout this automation journey.</em></strong></p><ul><li><a href="https://www.linkedin.com/in/sachinpk46/">Sachin PK - Turbolab Technologies | LinkedIn</a></li><li><a href="https://www.linkedin.com/in/imlabeeb/">Mohammed Labeebudeen - Vonage | LinkedIn</a></li><li><a href="https://www.linkedin.com/in/athirams/">Athira M S - Turbolab Technologies | LinkedIn</a></li><li><a href="https://www.linkedin.com/in/abinbennym/">Abin Benny - Accenture | LinkedIn</a></li><li><a href="https://www.linkedin.com/in/vibin-vinoy-mathew-38393923a/">Vibin Vinoy Mathew - Nuvento Inc | LinkedIn</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62d87a6142d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying and Configuring BunkerWeb WAF on AKS — Part 1]]></title>
            <link>https://midhunkrishna-ops.medium.com/deploying-and-configuring-bunkerweb-waf-on-aks-part-1-34e9cb17c328?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/34e9cb17c328</guid>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Sun, 09 Feb 2025 17:52:51 GMT</pubDate>
            <atom:updated>2025-02-09T17:53:33.986Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploying and Configuring BunkerWeb WAF on AKS — Part 1</h3><p>BunkerWeb is a robust security-focused reverse proxy designed to enhance web application security with minimal configuration. In this guide, we will deploy BunkerWeb along with a web UI on an Azure Kubernetes Service (AKS) cluster. Unlike the <a href="https://docs.bunkerweb.io/latest/web-ui/#advanced-installation">official documentation</a>, we have added configurations such as a LoadBalancer to optimize performance and manage traffic efficiently.</p><h3>Prerequisites</h3><ul><li>An AKS cluster up and running</li><li>kubectl installed on your local machine</li><li>Basic understanding of Kubernetes resources</li></ul><h3>Step 1: Deploy BunkerWeb DaemonSet</h3><p>We will deploy BunkerWeb as a DaemonSet to ensure it runs on all worker nodes in our AKS cluster.</p><pre>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: bunkerweb<br>spec:<br>  selector:<br>    matchLabels:<br>      app: bunkerweb<br>  template:<br>    metadata:<br>      labels:<br>        app: bunkerweb<br>      annotations:<br>        bunkerweb.io/INSTANCE: &quot;yes&quot;<br>    spec:<br>      serviceAccountName: sa-bunkerweb<br>      containers:<br>        - name: bunkerweb<br>          image: bunkerity/bunkerweb:1.5.12<br>          imagePullPolicy: Always<br>          securityContext:<br>            runAsUser: 101<br>            runAsGroup: 101<br>            allowPrivilegeEscalation: false<br>            capabilities:<br>              drop:<br>                - ALL<br>          ports:<br>            - containerPort: 8080<br>            - containerPort: 8443<br>          env:<br>            - name: KUBERNETES_MODE<br>              value: &quot;yes&quot;<br>            - name: DNS_RESOLVERS<br>              value: &quot;kube-dns.kube-system.svc.cluster.local&quot;<br>            - name: USE_API<br>              value: &quot;yes&quot;<br>            - name: API_WHITELIST_IP<br>              value: &quot;127.0.0.0/8 10.0.0.0/8&quot;<br>            - name: MULTISITE<br>              value: &quot;yes&quot;<br>            - name: USE_REDIS<br>              value: &quot;yes&quot;<br>            - name: REDIS_HOST<br>              value: &quot;svc-bunkerweb-redis.default.svc.cluster.local&quot;<br>            - name: UI_HOST<br>              value: &quot;http://svc-bunkerweb-ui:7000&quot;<br>            - name: USE_WHITELIST<br>              value: &quot;yes&quot;<br>            - name: WHITELIST_IP<br>              value: &quot;0.0.0.0/0&quot;</pre><h3>Step 2: Configure RBAC And Deployments</h3><p>RBAC permissions are necessary for BunkerWeb to manage Kubernetes resources.</p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: cr-bunkerweb<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;services&quot;, &quot;pods&quot;, &quot;configmaps&quot;, &quot;secrets&quot;]<br>    verbs: [&quot;get&quot;, &quot;watch&quot;, &quot;list&quot;]<br>  - apiGroups: [&quot;networking.k8s.io&quot;]<br>    resources: [&quot;ingresses&quot;]<br>    verbs: [&quot;get&quot;, &quot;watch&quot;, &quot;list&quot;]<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  namespace: default<br>  name: role-bunkerweb-logs<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods/log&quot;]<br>    verbs: [&quot;get&quot;]<br>---<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: sa-bunkerweb<br>  namespace: default<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: crb-bunkerweb<br>subjects:<br>  - kind: ServiceAccount<br>    name: sa-bunkerweb<br>    namespace: default<br>    apiGroup: &quot;&quot;<br>roleRef:<br>  kind: ClusterRole<br>  name: cr-bunkerweb<br>  apiGroup: rbac.authorization.k8s.io<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: rolebinding-bunkerweb-logs<br>  namespace: default<br>subjects:<br>  - kind: ServiceAccount<br>    name: sa-bunkerweb<br>    namespace: default<br>roleRef:<br>  kind: Role<br>  name: role-bunkerweb-logs<br>  apiGroup: rbac.authorization.k8s.io<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: bunkerweb-controller<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: bunkerweb-controller<br>  template:<br>    metadata:<br>      labels:<br>        app: bunkerweb-controller<br>    spec:<br>      serviceAccountName: sa-bunkerweb<br>      containers:<br>        - name: bunkerweb-controller<br>          image: bunkerity/bunkerweb-autoconf:1.5.12<br>          imagePullPolicy: Always<br>          env:<br>            - name: KUBERNETES_MODE<br>              value: &quot;yes&quot;<br>            - name: DATABASE_URI<br>              value: &quot;mariadb+pymysql://bunkerweb:changeme@svc-bunkerweb-db:3306/db&quot;<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: bunkerweb-scheduler<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: bunkerweb-scheduler<br>  template:<br>    metadata:<br>      labels:<br>        app: bunkerweb-scheduler<br>    spec:<br>      serviceAccountName: sa-bunkerweb<br>      containers:<br>        - name: bunkerweb-scheduler<br>          image: bunkerity/bunkerweb-scheduler:1.5.12<br>          imagePullPolicy: Always<br>          env:<br>            - name: KUBERNETES_MODE<br>              value: &quot;yes&quot;<br>            - name: DATABASE_URI<br>              value: &quot;mariadb+pymysql://bunkerweb:changeme@svc-bunkerweb-db:3306/db&quot;<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: bunkerweb-redis<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: bunkerweb-redis<br>  template:<br>    metadata:<br>      labels:<br>        app: bunkerweb-redis<br>    spec:<br>      containers:<br>        - name: bunkerweb-redis<br>          image: redis:7-alpine<br>          imagePullPolicy: Always<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: bunkerweb-db<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: bunkerweb-db<br>  template:<br>    metadata:<br>      labels:<br>        app: bunkerweb-db<br>    spec:<br>      containers:<br>        - name: bunkerweb-db<br>          image: mariadb:10.10<br>          imagePullPolicy: Always<br>          env:<br>            - name: MYSQL_RANDOM_ROOT_PASSWORD<br>              value: &quot;yes&quot;<br>            - name: MYSQL_DATABASE<br>              value: &quot;db&quot;<br>            - name: MYSQL_USER<br>              value: &quot;bunkerweb&quot;<br>            - name: MYSQL_PASSWORD<br>              value: &quot;changeme&quot;<br>          volumeMounts:<br>            - mountPath: /var/lib/mysql<br>              name: vol-db<br>      volumes:<br>        - name: vol-db<br>          persistentVolumeClaim:<br>            claimName: pvc-bunkerweb<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: bunkerweb-ui<br>spec:<br>  replicas: 1<br>  strategy:<br>    type: Recreate<br>  selector:<br>    matchLabels:<br>      app: bunkerweb-ui<br>  template:<br>    metadata:<br>      labels:<br>        app: bunkerweb-ui<br>    spec:<br>      serviceAccountName: sa-bunkerweb<br>      containers:<br>        - name: bunkerweb-ui<br>          image: bunkerity/bunkerweb-ui:1.5.12<br>          imagePullPolicy: Always<br>          env:<br>            - name: KUBERNETES_MODE<br>              value: &quot;YES&quot;<br>            - name: DATABASE_URI<br>              value: &quot;mariadb+pymysql://bunkerweb:changeme@svc-bunkerweb-db:3306/db&quot;<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: svc-bunkerweb<br>spec:<br>  clusterIP: None<br>  selector:<br>    app: bunkerweb<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: svc-bunkerweb-db<br>spec:<br>  type: ClusterIP<br>  selector:<br>    app: bunkerweb-db<br>  ports:<br>    - name: sql<br>      protocol: TCP<br>      port: 3306<br>      targetPort: 3306<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: svc-bunkerweb-redis<br>spec:<br>  type: ClusterIP<br>  selector:<br>    app: bunkerweb-redis<br>  ports:<br>    - name: redis<br>      protocol: TCP<br>      port: 6379<br>      targetPort: 6379<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: svc-bunkerweb-ui<br>spec:<br>  type: ClusterIP<br>  selector:<br>    app: bunkerweb-ui<br>  ports:<br>    - name: http<br>      protocol: TCP<br>      port: 7000<br>      targetPort: 7000<br>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: pvc-bunkerweb<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 5Gi</pre><h3>Step 3: Configure Ingress Resource</h3><p>We need to create an Ingress resource to expose BunkerWeb UI through a specific domain.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: bunkerweb-ingress<br>spec:<br>  rules:<br>    - host: &lt;hostname&gt;<br>      http:<br>        paths:<br>          - path: /changeme<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: svc-bunkerweb-ui<br>                port:<br>                  number: 7000</pre><p>UI will be available at “<strong>https://hostname/changeme</strong>” after successful deployment</p><h3>Step 4: Add LoadBalancer Configuration</h3><p>We introduce a LoadBalancer service to ensure external access to BunkerWeb.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: bunker-lb<br>spec:<br>  type: LoadBalancer<br>  loadBalancerIP: &lt;lb-ip&gt;<br>  externalTrafficPolicy: Local<br>  selector:<br>    app: bunkerweb<br>  ports:<br>    - port: 80<br>      targetPort: 8080<br>      protocol: TCP<br>      name: http<br>    - port: 443<br>      targetPort: 8443<br>      protocol: TCP<br>      name: https<br></pre><p>After a successful deployment, we can sign up or sign in using the UI.</p><p>In the next part, we will configure our existing applications to work with BunkerWeb, monitor the traffic, and utilize the features of the WAF.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34e9cb17c328" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Wazuh SIEM on Azure Kubernetes Service (AKS)]]></title>
            <link>https://midhunkrishna-ops.medium.com/deploying-wazuh-siem-on-azure-kubernetes-service-aks-d21e9581c8b0?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d21e9581c8b0</guid>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Sun, 09 Feb 2025 17:01:19 GMT</pubDate>
            <atom:updated>2025-02-09T17:01:19.937Z</atom:updated>
            <content:encoded><![CDATA[<p>Wazuh is an open-source Security Information and Event Management (SIEM) solution that provides real-time threat detection, integrity monitoring, and security analytics. This guide will walk you through deploying Wazuh on an Azure Kubernetes Service (AKS) cluster with a customized storage class configuration.</p><h3>Prerequisites</h3><ul><li>An <strong>Azure Kubernetes Service (AKS)</strong> cluster up and running.</li><li><strong>kubectl</strong> installed and configured for the AKS cluster.</li></ul><h3>Step 1: Clone the Wazuh Kubernetes Repository</h3><p>First, clone the official Wazuh Kubernetes repository:</p><pre>git clone https://github.com/wazuh/wazuh-kubernetes.git -b v4.10.1 --depth=1<br> cd wazuh-kubernetes</pre><h3>Step 2: Setup SSL Certificates</h3><p>You can generate self-signed SSL certificates for the Wazuh indexer cluster using the provided script:</p><pre>bash wazuh/certs/indexer_cluster/generate_certs.sh</pre><p>Alternatively, if you have your own SSL certificates, place them in the appropriate directory within wazuh/certs/indexer_cluster/.</p><h3>Step 3: Modify Storage Class for Azure</h3><p>Inside the wazuh-kubernetes/envs/eks directory, remove the default storage-class.yaml file and create a new one with the following configuration:</p><pre>apiVersion: storage.k8s.io/v1<br>kind: StorageClass<br>metadata:<br>  name: wazuh-storage<br>provisioner: disk.csi.azure.com<br>parameters:<br>  cachingmode: ReadOnly<br>  kind: Managed<br>  storageaccounttype: StandardSSD_LRS<br>volumeBindingMode: WaitForFirstConsumer<br>reclaimPolicy: Delete</pre><p>Save this file as storage-class.yaml inside wazuh-kubernetes/envs/eks/.</p><h3>Step 4: Deploy Wazuh on AKS</h3><p>Now, apply the deployment configurations using kubectl:</p><pre>kubectl apply -k wazuh-kubernetes/envs/eks/</pre><p>This will deploy the necessary Wazuh components, including Wazuh Manager, Wazuh Indexer, and the Wazuh Dashboard.</p><h3>Step 5: Get Wazuh Service Details</h3><p>After deployment, retrieve the services running in the wazuh namespace:</p><pre>kubectl get services -n wazuh</pre><p>This will output details including an assigned public IP for the Wazuh Dashboard.</p><h3>Step 6: Access the Wazuh Dashboard</h3><p>Once the deployment is complete, access the Wazuh Dashboard using the assigned public IP:</p><pre>https://&lt;assigned-public-ip&gt;</pre><p>Use the following credentials to log in:</p><ul><li><strong>Username:</strong> admin</li><li><strong>Password:</strong> SecretPassword</li></ul><h3>Additional Resources</h3><p>For more details, refer to the official Wazuh documentation:<br><a href="https://documentation.wazuh.com/current/deployment-options/deploying-with-kubernetes/kubernetes-deployment.html">Wazuh Kubernetes Deployment</a></p><p>This guide customizes the storage class configuration specifically for Azure Kubernetes Service (AKS) while following the standard Wazuh deployment process. 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d21e9581c8b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Mini Cloud Environment on a Single Machine: Exploring Kubernetes Made Simple -3]]></title>
            <link>https://midhunkrishna-ops.medium.com/exploring-kubernetes-the-easy-way-for-beginners-part-3-a5c862d8ab71?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5c862d8ab71</guid>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Sat, 11 Jan 2025 16:18:08 GMT</pubDate>
            <atom:updated>2025-01-11T16:31:44.091Z</atom:updated>
            <content:encoded><![CDATA[<h3>Configuring Gaia in K3s on Rancher and Accessing the UI</h3><p><strong>Gaia</strong> is a Terraform UI designed to simplify managing Terraform modules and enabling self-service infrastructure. This guide demonstrates how to set up Gaia in a K3s cluster managed by Rancher and access its user interface.</p><h3>Installation Steps</h3><h3>1. Access Kubernetes Shell</h3><ol><li>Login to the Rancher Dashboard.</li><li>Navigate to the <strong>kubectl shell</strong> to execute Kubernetes commands.</li></ol><h3>2. Create Namespace for Gaia</h3><p>Run the following command in the <strong>kubectl shell</strong> to create a namespace for Gaia:</p><pre>kubectl create namespace gaia</pre><h3>3. Install Gaia Using Helm</h3><p>Install Gaia using the Helm chart by running the following command:</p><pre>helm install gaia https://github.com/gaia-app/chart/archive/refs/tags/v0.1.2.tar.gz --namespace gaia</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hqX86OMPTHD9oFX6InfGEw.png" /></figure><h3>4. Check Pod Status</h3><p>Verify that the Gaia pods are running with the following command:</p><pre>kubectl get pods -n gaia</pre><p>Wait until the status of all pods is <strong>Running</strong>.</p><h3>5. Retrieve the Service Information</h3><p>Once the pods are running, get the service details to access Gaia:</p><pre>kubectl get svc --namespace gaia</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kTYJpYMUyJhDN1zAipFrZw.png" /></figure><h3>6. Access Gaia UI</h3><ol><li>Use the service’s IP address and port number to access the Gaia UI in a browser.</li><li>For Our Docker setup, replace the service IP with your Docker container IP and include the port number.</li></ol><p>Example:</p><pre>http://&lt;docker-container-ip&gt;:&lt;port&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6h3LIx_sz7OxFEjWIEqfiA.png" /></figure><ol><li>Login to the Gaia UI with the following credentials:</li></ol><ul><li><strong>Username:</strong> admin</li><li><strong>Password:</strong> admin123</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e9a-qVdlicTGRED5E4krwg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5c862d8ab71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Mini Cloud Environment on a Single Machine: Exploring Kubernetes Made Simple -2]]></title>
            <link>https://midhunkrishna-ops.medium.com/exploring-kubernetes-the-easy-way-for-beginners-2-d96b8d86541e?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d96b8d86541e</guid>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 15:31:56 GMT</pubDate>
            <atom:updated>2025-01-11T16:32:04.674Z</atom:updated>
            <content:encoded><![CDATA[<h3>Configuring ArgoCD in K3s on Rancher and Accessing the UI</h3><p>ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. This guide will walk you through the process of setting up ArgoCD inside a <strong>K3s cluster</strong> running in a <a href="https://midhunkr1shna.medium.com/exploring-kubernetes-the-easy-way-for-beginners-1a5d2e77db86"><strong>Rancher</strong></a><strong> container</strong>, and accessing the user interface.</p><p>Part 1: <a href="https://midhunkr1shna.medium.com/exploring-kubernetes-the-easy-way-for-beginners-1a5d2e77db86">Rancher-setup</a></p><p>Part 3: Configuring <a href="https://midhunkr1shna.medium.com/exploring-kubernetes-the-easy-way-for-beginners-part-3-a5c862d8ab71"><em>gaia</em></a></p><h3>Step 1: Create a Namespace for ArgoCD</h3><p>Before installing ArgoCD, you need to create a dedicated namespace in your Kubernetes cluster:</p><pre>kubectl create namespace argocd</pre><p>This will set up a clean namespace for all the ArgoCD resources.</p><p>You can use the kubectl shell of rancher ui for this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Afne1tTigQTOd6p2z4KnQ.png" /></figure><h3>Step 2: Install ArgoCD</h3><p>Now, apply the ArgoCD installation manifest to your Kubernetes cluster. This will deploy all the necessary resources for ArgoCD to run.</p><pre>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</pre><ul><li>This command will deploy ArgoCD components such as the server, API server, and controller in the argocd namespace.</li><li>After the command is successfully executed, Wait for few minutes, ArgoCD will be running in your cluster.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9uIrkYo6ib4rRZlnAEK3Bg.png" /></figure><p>Select the namespace argocd , You can see all the pods running</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3nZDeuH3GbO24_3GbJP4w.png" /></figure><h3>Step 3: Expose the ArgoCD UI</h3><p>To access the ArgoCD UI, you need to expose the ArgoCD server service. By. You can change the service type to <strong>LoadBalancer</strong> to access the UI.</p><pre><br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: argocd-server<br>  namespace: argocd<br>spec:<br>  selector:<br>    app.kubernetes.io/name: argocd-server<br>  ports:<br>    - name: http<br>      port: 80<br>      targetPort: 8080<br>      protocol: TCP<br>    - name: https<br>      port: 443<br>      targetPort: 8080<br>      protocol: TCP<br>  type: LoadBalancer</pre><p>You can import this yaml using the <strong>import yaml</strong> option in rancher ui</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JUCxrq_wk4yz05aLNWUSHA.png" /></figure><p>Note: It may show some warnings !!!</p><h3>Step 4: Accessing the ArgoCD UI</h3><p>Get the service details using the command</p><pre>kubectl get svc -n argocd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*91niqRrOWmM0YlxrUZVzVw.png" /></figure><p>The ArgoCD server will be mapped to a port, for example, 31594 as shown in the screenshot. You can access it by using the IP of your container, like so.</p><pre>https://your-container-ip:port-number-of-argocdserver/<br>Example : https://172.17.0.2:31594/</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tng0YTzNJvL9iR83dC2zEA.png" /></figure><h3>Step 5: ArgoCD Login</h3><p>Get the argocd admin password using the command</p><pre>kubectl get secret -n argocd argocd-initial-admin-secret -o=jsonpath=&#39;{.data.password}&#39; | base64 -d</pre><p>Username is : admin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*IrV7H3Hb5t1ZkhAOjjx8ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QTIHDywpxoh_3mkJbV1Mmg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d96b8d86541e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a Mini Cloud Environment on a Single Machine: Exploring Kubernetes Made Simple -1]]></title>
            <link>https://midhunkrishna-ops.medium.com/exploring-kubernetes-the-easy-way-for-beginners-1a5d2e77db86?source=rss-3b237047fa8------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a5d2e77db86</guid>
            <dc:creator><![CDATA[Midhun Krishna]]></dc:creator>
            <pubDate>Wed, 08 Jan 2025 14:48:56 GMT</pubDate>
            <atom:updated>2025-01-11T16:28:26.713Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Rancher Single-Node Docker Setup with User Interface for Learning and Testing DevOps Tools</strong></h3><p><strong>A Beginner-Friendly Guide to Setting Up Kubernetes with Rancher’s Single-Node K3s Docker Setup, and Exploring Essential DevOps Tools</strong></p><p><strong>Introduction:</strong><br>Kubernetes can be complex for beginners, but Rancher’s single-node Docker setup simplifies the process. In this guide, you’ll learn how to set up Kubernetes using Rancher’s K3s in a Docker container, deploy and explore essential DevOps tools in an easy-to-manage environment.</p><p><strong>Why Rancher?</strong><br>Rancher’s single-node Docker installation provides a simple way to run K3s locally, enabling you to configure and explore popular DevOps tools without the complexity of managing multi-node clusters.</p><p>Part 2: Configuring <a href="https://midhunkr1shna.medium.com/exploring-kubernetes-the-easy-way-for-beginners-2-d96b8d86541e">Argocd</a></p><p>Part 3: Configuring <a href="https://medium.com/@midhunkr1shna/exploring-kubernetes-the-easy-way-for-beginners-1a5d2e77db86">gaia</a></p><h3>Setting Up Rancher on Your Machine</h3><ol><li><strong>Install Docker:</strong><br>Make sure Docker is installed on your system. Or install it using the commands</li></ol><pre>curl -fsSL https://get.docker.com -o get-docker.sh<br>sudo sh get-docker.sh</pre><ol><li><strong>Pull and Run Rancher:</strong><br>Use the following command to pull and run the Rancher container:</li></ol><pre>docker run -d  --privileged   -p 8080:80 -p 8443:443   rancher/rancher:latest</pre><p>This will expose Rancher on <a href="http://localhost:8080.">http://localhost:8</a>443</p><p><strong>3. Access Rancher Dashboard:</strong><br>Open your browser and navigate to http://localhost:8443 to access the Rancher UI.</p><p>After deploying Rancher, you need to retrieve the <strong>Bootstrap Password</strong> to log in for the first time and create a new admin password.</p><p><strong>4. List the Running Docker Containers:</strong><br>Run the following command to identify the container ID for the Rancher container:</p><pre>docker ps --filter &quot;ancestor=rancher/rancher:latest&quot; --format &quot;{{.ID}}&quot;</pre><p><strong>5. Retrieve the Bootstrap Password:</strong><br>Use the container ID from the previous step and run the following command:</p><pre>sudo docker logs &lt;container_id&gt; 2&gt;&amp;1 | grep &quot;Bootstrap Password:&quot;</pre><ul><li>Bootstrap Password: abc123xyz</li></ul><p><strong>6 . Log in to Rancher:</strong></p><ul><li>Open http://localhost:8443 in your browser.</li><li>Use the retrieved <strong>Bootstrap Password</strong> to log in as the admin user.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LRTpIyAS0VXwXjq_Po7TvQ.png" /></figure><p><strong>7 .Set a New Admin Password:</strong></p><ul><li>Rancher will prompt you to set a new admin password after the first login.</li><li>Set a secure password that you can easily remember.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-pxl-wLRNyl0tVeKilbLTA.png" /></figure><p><strong>8. Accessing Your K3s Cluster Dashboard</strong></p><p>Once the setup is complete, your <strong>K3s cluster</strong> will be up and running on your machine. Rancher makes it incredibly easy to manage your cluster through a user-friendly interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hu4AEl2NPImAXmg76y5oPg.png" /></figure><p><strong>9. View Your Cluster Dashboard:</strong></p><ul><li>In the Rancher UI, click on the <strong>“Local”</strong> cluster.</li><li>This will open the cluster dashboard, where you can monitor and manage your Kubernetes resources.</li></ul><p><strong>10. Check Node Details:</strong></p><ul><li>Navigate to the <strong>“Nodes”</strong> section from the dashboard.</li><li>Here, you’ll see a list of nodes along with their <strong>External IP</strong> addresses.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YD97nUtoI5si2IGXUSEkgw.png" /></figure><p><strong>11. Access the Dashboard via External IP:</strong></p><ul><li>Use the <strong>External IP</strong> displayed in the node details to access the Rancher dashboard directly.</li><li>For example, open http://&lt;External-IP&gt; in your browser to connect to the dashboard.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a5d2e77db86" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>