<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Laxman Pajjuri on Medium]]></title>
        <description><![CDATA[Stories by Laxman Pajjuri on Medium]]></description>
        <link>https://medium.com/@laxmanpajjuri?source=rss-4add164debc7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Laxman Pajjuri on Medium</title>
            <link>https://medium.com/@laxmanpajjuri?source=rss-4add164debc7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:59:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@laxmanpajjuri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Math for Machine Learning: Understanding Statistics, Data, and the Story Behind Numbers]]></title>
            <link>https://medium.com/@laxmanpajjuri/math-for-machine-learning-understanding-statistics-data-and-the-story-behind-numbers-f056e9670eae?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f056e9670eae</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 15:20:40 GMT</pubDate>
            <atom:updated>2026-11-15T15:20:40.139Z</atom:updated>
            <content:encoded><![CDATA[<p><em>My journey to understanding the mathematics behind Machine Learning</em></p><p>When I started learning Machine Learning, I quickly realised something: before understanding algorithms, models, or neural networks, I needed to understand the language they use — <strong>Mathematics</strong>.</p><p>I decided to start with Statistics.</p><p>At first, terms like mean, median, variance, and standard deviation looked like formulas I had to memorise. But I wanted something different.</p><p>I wanted to understand:</p><ul><li>Why do these formulas exist?</li><li>What problem do they solve?</li><li>How are they useful in real-world applications?</li><li>How does a Machine Learning model use these concepts?</li></ul><p>So, I started asking my brain simple questions.</p><p>Let’s explore these concepts together.</p><h3>1. What is Statistics?</h3><p>Imagine I run a performance test on an application using JMeter.</p><p>I send 1,000 requests and collect information such as:</p><ul><li>Response time</li><li>CPU utilization</li><li>Memory consumption</li><li>Throughput</li><li>Error rate</li></ul><p>After the test, I have thousands of numbers.</p><p>But simply collecting numbers doesn’t tell me whether the application performed well.</p><p>I need to understand those numbers, identify patterns, compare results, and make decisions.</p><p>That’s where Statistics comes in.</p><p><strong>Statistics is the science of collecting, organising, analysing, interpreting, and presenting data.</strong></p><p>In simple words:</p><p>Statistics helps us turn raw data into meaningful information.</p><p>Now, let’s connect this to Machine Learning.</p><p>Imagine I want to build a Machine Learning model that predicts house prices.</p><p>I collect information about:</p><ul><li>House area</li><li>Number of bedrooms</li><li>Location</li><li>Property age</li><li>Previous selling prices</li></ul><p>I have a lot of data, but how do I understand it?</p><p>I need to know the average house price, the range of house areas, the most common number of bedrooms, and how much prices vary.</p><p>Statistics helps me answer these questions before I train my model.</p><p><strong>Machine Learning uses statistical concepts to understand data, identify patterns, and make predictions.</strong></p><h3>2. What is Data?</h3><p>Before understanding statistics, I need to understand data.</p><p>Data is simply a collection of facts, observations, or measurements.</p><p>For example, imagine I execute five API requests and record their response times:</p><p>100 ms, 120 ms, 110 ms, 130 ms, 140 ms.</p><p>These five values are data.</p><p>Now imagine a house-price dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*_Vo_mBdbh-bXCi4bKsDIVA.png" /></figure><p>This is also data.</p><p>Each row represents an observation, and each column represents a feature or attribute.</p><p>In Machine Learning:</p><ul><li><strong>Features (X):</strong> Input variables used by the model, such as area and bedrooms.</li><li><strong>Target (y):</strong> The value the model is trying to predict, such as house price.</li></ul><p>For example:</p><p>X = House area, number of bedrooms, property age.</p><p>y = House price.</p><p>The model learns relationships between X and y using historical examples.</p><h3>3. Types of Data</h3><p>Not all data is the same. Understanding its type is important because different types of data require different analysis techniques.</p><p>I started by dividing data into two major categories.</p><h3>A. Qualitative Data (Categorical Data)</h3><p>This type of data describes qualities, labels, or categories.</p><p>Examples:</p><ul><li>Browser: Chrome, Firefox, Edge</li><li>Environment: QA, UAT, Production</li><li>Payment method: UPI, Credit Card, Cash</li><li>Property location: Hyderabad, Bengaluru, Chennai</li></ul><p>Categorical data can be further divided into two types.</p><p><strong>Nominal Data</strong></p><p>Categories without any natural order.</p><p>Example:</p><p>Browser = Chrome, Firefox, Edge.</p><p>There is no meaningful ranking between these browsers.</p><p><strong>Ordinal Data</strong></p><p>Categories that have a meaningful order.</p><p>Example:</p><p>Performance rating:</p><ul><li>Poor</li><li>Average</li><li>Good</li><li>Excellent</li></ul><p>There is an order, but the difference between categories cannot necessarily be measured numerically.</p><p><strong>Machine Learning example:</strong></p><p>Imagine I am building a model to predict whether a customer will purchase a product.</p><p>My dataset contains a feature called Payment Method.</p><p>Possible values:</p><ul><li>UPI</li><li>Credit Card</li><li>Cash</li></ul><p>This is nominal categorical data.</p><p>Another feature is Customer Satisfaction:</p><ul><li>Low</li><li>Medium</li><li>High</li></ul><p>This is ordinal data.</p><p>Machine Learning models generally need numerical representations of categorical features. Techniques such as one-hot encoding and ordinal encoding help convert them into suitable representations.</p><h3>B. Quantitative Data (Numerical Data)</h3><p>This type of data represents numbers and measurable quantities.</p><p>Examples:</p><ul><li>API response time = 120 ms</li><li>CPU utilization = 65%</li><li>Age = 29 years</li><li>Number of users = 500</li><li>House price = ₹75 lakh</li></ul><p>Quantitative data can be divided into two types.</p><p><strong>Discrete Data</strong></p><p>Data that is countable, usually in whole numbers.</p><p>Examples:</p><ul><li>Number of users = 500</li><li>Number of failed requests = 12</li><li>Number of bedrooms = 3</li><li>Number of transactions = 1,000</li></ul><p><strong>Continuous Data</strong></p><p>Data that can take any value within a range, including decimal values.</p><p>Examples:</p><ul><li>Response time = 120.5 ms</li><li>Temperature = 36.7°C</li><li>Memory consumption = 4.25 GB</li><li>Weight = 81.5 kg</li></ul><p><strong>Machine Learning example:</strong></p><p>Imagine I am building a model to predict house prices.</p><p>House area, property age, and price are numerical features or targets.</p><p>The model can learn how these numerical values relate to one another.</p><p>Why does this classification matter?</p><p>Because the type of data determines which statistical operations and machine learning techniques make sense.</p><p>For example, calculating the average of house prices makes sense. Calculating the average of browser names does not.</p><h3>4. Population and Sample</h3><p>Now imagine I want to understand the performance of an application.</p><p>I have 100,000 API requests generated during a complete performance test.</p><p>Do I always need to analyse every request?</p><p>Not necessarily.</p><p>This brings me to two important concepts: Population and Sample.</p><h3>Population</h3><p>Population means the complete group of observations that I want to study.</p><p>Example:</p><p>Suppose my performance test generates 100,000 requests.</p><p>If I want to study all requests generated during that test, those 100,000 requests form my population.</p><h3>Sample</h3><p>A sample is a smaller subset selected from the population.</p><p>Instead of analysing all 100,000 requests, imagine I select 1,000 requests for an initial analysis.</p><p>Those 1,000 requests form my sample.</p><p>Think about it this way:</p><ul><li>Population = All 100,000 requests.</li><li>Sample = Selected 1,000 requests.</li></ul><h3>Machine Learning example</h3><p>Imagine I want to build a model that predicts house prices in Hyderabad.</p><p>I have historical information about 100,000 houses.</p><p>That complete dataset is my population for this study.</p><p>For an initial experiment, I might select 10,000 houses as a sample.</p><p>I can analyse this sample to understand price patterns and test my approach.</p><p>However, my sample should represent the population properly.</p><p>If I select houses only from expensive areas, my sample will be biased and my model may not learn patterns that apply to the entire city.</p><p>In Machine Learning, datasets are also divided into training, validation, and test sets. These are different subsets used for different purposes, not simply three names for population and sample.</p><h3>Population vs Sample</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*oF-bxyP6tHZIk9GIkjM56A.png" /></figure><p>This distinction becomes important when learning statistical estimation and evaluating machine learning models.</p><h3>5. Mean — The Equal Sharing Concept</h3><p>Now let’s start with our first important statistical measure: Mean.</p><p>Suppose I have four numbers:</p><p>2, 4, 6, 8.</p><p>What is their mean?</p><p>Mean = Sum of all values / Number of values</p><p>Mean = (2 + 4 + 6 + 8) / 4</p><p>Mean = 20 / 4 = 5.</p><p>But I didn’t want to just memorise this formula. I wanted to understand why we divide by the number of values.</p><p>So I asked myself:</p><p><strong>What does the number 5 actually represent?</strong></p><p>Imagine four people have 20 apples in total.</p><p>If I distribute those apples equally:</p><ul><li>Person 1 gets 5 apples.</li><li>Person 2 gets 5 apples.</li><li>Person 3 gets 5 apples.</li><li>Person 4 gets 5 apples.</li></ul><p>The total is still 20.</p><p>The original distribution was:</p><p>2, 4, 6, 8.</p><p>The equal distribution becomes:</p><p>5, 5, 5, 5.</p><p>Nothing happened to the total. I simply redistributed it equally.</p><p>That’s the intuition behind the mean.</p><p><strong>The mean represents the equal-share value of a group of numbers.</strong></p><h3>Performance testing example</h3><p>Suppose my API response times are:</p><p>100 ms, 200 ms, 300 ms, 400 ms.</p><p>Mean = (100 + 200 + 300 + 400) / 4</p><p>Mean = 250 ms.</p><p>This tells me that the average response time is 250 ms.</p><p>But does it mean every request took 250 ms?</p><p>No!</p><p>It is the equal-share value, not the actual response time of every request.</p><h3>Machine Learning example</h3><p>Imagine I have a house-price dataset with the following prices:</p><p>₹40 lakh, ₹50 lakh, ₹60 lakh, ₹70 lakh, ₹80 lakh.</p><p>Mean price = ₹60 lakh.</p><p>This tells me the average price in this small dataset.</p><p>I can use the mean to understand the general price level of the dataset.</p><p>The mean can also be used to fill missing numerical values during data preprocessing.</p><p>For example, if some house-area values are missing, I might consider replacing them with the mean area.</p><p>However, I should calculate that mean using the training data only, not the entire dataset, to avoid data leakage.</p><h3>6. Median — Finding the Middle Value</h3><p>Now imagine I have the following response times:</p><p>100, 110, 120, 130, 500 ms.</p><p>The median is the middle value after arranging the data in ascending order.</p><p>Here, the median is 120 ms.</p><p>Why is this useful?</p><p>Because the last request took 500 ms, which is much higher than the others.</p><p>This unusually high value is called an outlier.</p><p>The mean is:</p><p>(100 + 110 + 120 + 130 + 500) / 5 = 192 ms.</p><p>But the median is still 120 ms.</p><p>Notice something interesting:</p><p>The mean increased significantly because of one unusually high value, while the median remained at the centre of the sorted data.</p><p>This is why the median is useful when data contains outliers.</p><p>For an even number of observations, the median is the average of the two middle values.</p><p>Example:</p><p>10, 20, 30, 40.</p><p>Median = (20 + 30) / 2 = 25.</p><h3>Machine Learning example</h3><p>Imagine I have the following house prices:</p><p>₹40 lakh, ₹45 lakh, ₹50 lakh, ₹55 lakh, ₹5 crore.</p><p>The ₹5 crore property is much more expensive than the others.</p><p>The mean will be pulled upwards by this expensive property.</p><p>The median, however, remains ₹50 lakh.</p><p>If I want to understand the typical house price in this dataset, the median may give me a more representative value.</p><p>The median is also useful when filling missing numerical values in datasets that contain extreme values.</p><p><strong>The mean tells me the equal-share value. The median tells me the middle position.</strong></p><h3>7. Mode — The Most Frequently Occurring Value</h3><p>Now consider these response times:</p><p>100, 100, 100, 120, 150, 200.</p><p>Which value occurs most frequently?</p><p>100 occurs three times.</p><p>Therefore, Mode = 100.</p><p>Mode helps identify the most frequently occurring value.</p><p>It is especially useful for categorical data.</p><p>For example:</p><p>Browser usage:</p><p>Chrome, Chrome, Firefox, Chrome, Edge.</p><p>Mode = Chrome.</p><p>Why?</p><p>Because Chrome appears more frequently than the other categories.</p><h3>Machine Learning example</h3><p>Imagine I have a customer dataset with a feature called Preferred Payment Method.</p><p>UPI, Credit Card, UPI, Cash, UPI.</p><p>The mode is UPI.</p><p>If some payment-method values are missing, I might use the mode to fill them.</p><p>This is a common approach for categorical data, although I should consider whether it makes sense for the particular dataset.</p><p>So now I understand three important measures:</p><ul><li>Mean = Equal-share value.</li><li>Median = Middle value.</li><li>Mode = Most frequently occurring value.</li></ul><h3>8. Range — Understanding the Spread of Data</h3><p>Now let’s say two applications have the same average response time.</p><p>Application A:</p><p>48, 49, 50, 51, 52 ms.</p><p>Application B:</p><p>10, 30, 50, 70, 90 ms.</p><p>Both have a mean of 50 ms.</p><p>But are they behaving similarly?</p><p>Definitely not.</p><p>Application A has response times close to 50 ms.</p><p>Application B has response times spread across a much wider range.</p><p>How do I measure this difference?</p><p>I use Range.</p><p><strong>Range = Maximum value — Minimum value</strong></p><p>For Application A:</p><p>Range = 52–48 = 4 ms.</p><p>For Application B:</p><p>Range = 90–10 = 80 ms.</p><p>This tells me that Application B has a much wider spread.</p><h3>Machine Learning example</h3><p>Imagine two datasets containing house areas.</p><p>Dataset A:</p><p>1,000, 1,100, 1,200, 1,300, 1,400 sq. ft.</p><p>Dataset B:</p><p>500, 1,000, 1,500, 2,000, 2,500 sq. ft.</p><p>Both have a mean of 1,200 sq. ft.</p><p>But their ranges are different:</p><p>Dataset A = 1,400–1,000 = 400 sq. ft.</p><p>Dataset B = 2,500–500 = 2,000 sq. ft.</p><p>This helps me understand how widely the house areas are distributed.</p><p>Range is also connected to Min-Max Scaling, a data preprocessing technique.</p><p>The formula is:</p><p>X_scaled = (X — X_min) / (X_max — X_min)</p><p>Let’s say house areas range from 800 to 3,000 sq. ft.</p><p>I want to scale a house area of 1,500 sq. ft.</p><p>X_scaled = (1500–800) / (3000–800)</p><p>X_scaled = 700 / 2200</p><p>X_scaled ≈ 0.318</p><p>The original value of 1,500 becomes approximately 0.318 on a scale from 0 to 1.</p><p>Why is this useful?</p><p>Some machine learning algorithms are sensitive to differences in feature scales.</p><p>For example, K-Nearest Neighbours, K-Means, and Support Vector Machines can be affected when one feature has values in the thousands, and another has values between 0 and 1.</p><p>Scaling helps put features on comparable scales.</p><p>One limitation: Min-Max Scaling is sensitive to outliers because it uses the minimum and maximum values.</p><h3>9. Variance — How Much Do Values Differ From the Mean?</h3><p>This concept initially confused me.</p><p>I understood mean and range, but why did I need variance?</p><p>Let’s use the same example.</p><p>Application A:</p><p>48, 49, 50, 51, 52.</p><p>Mean = 50.</p><p>Now I calculate how far each value is from the mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*quaslXf0HhUCBOvsFfho1w.png" /></figure><p>These differences are called deviations.</p><p>If I simply add them:</p><p>-2 + -1 + 0 + 1 + 2 = 0.</p><p>That’s a problem!</p><p>The negative and positive deviations cancel each other out.</p><p>So how can I measure the spread without allowing this cancellation?</p><p>I square the deviations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*JYYlLdsxnD_rN2EjOBfvWw.png" /></figure><p>Now I add the squared deviations:</p><p>4 + 1 + 0 + 1 + 4 = 10.</p><p>Since I have five observations:</p><p>Population Variance = 10 / 5 = 2.</p><p>This is the variance.</p><h3>Why do we square the deviations?</h3><p>I asked myself this question.</p><p>There are two important reasons:</p><ol><li>Squaring prevents positive and negative deviations from cancelling each other.</li><li>Squaring gives greater weight to larger deviations.</li></ol><p>So variance measures how spread out the values are around their mean.</p><h3>Comparing two applications</h3><p>Application A:</p><p>48, 49, 50, 51, 52.</p><p>Mean = 50.</p><p>Population variance = 2.</p><p>Application B:</p><p>10, 30, 50, 70, 90.</p><p>Mean = 50.</p><p>Squared deviations:</p><p>1600, 400, 0, 400, 1600.</p><p>Sum = 4,000.</p><p>Population variance = 4,000 / 5 = 800.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*NUoUafqVVI94pq207QqMAg.png" /></figure><p>Both applications have the same mean, but their variance is very different.</p><p><strong>Higher variance means greater spread around the mean.</strong></p><h3>Machine Learning example</h3><p>Imagine I am building a model to predict house prices.</p><p>I have two features:</p><ul><li>Number of bedrooms</li><li>House area</li></ul><p>The number of bedrooms might range from 1 to 5.</p><p>House area might range from 500 to 5,000 sq. ft.</p><p>Variance helps me understand how much each feature varies across the dataset.</p><p>It is also important in several machine learning concepts:</p><ul><li>Feature selection and data exploration.</li><li>Understanding how much information a feature varies across observations.</li><li>Principal Component Analysis (PCA), which uses variance to identify directions with high variation.</li><li>Understanding noise and uncertainty in data.</li></ul><p>A feature with zero variance has the same value for every observation. It generally provides no information for distinguishing between those observations.</p><h3>Population Variance vs Sample Variance</h3><p>One more important detail.</p><p>If I have the entire population, I divide by N.</p><p>If I have a sample and want to estimate the population variance, I generally divide by n— 1.</p><p>Population variance:</p><p>σ² = Σ(x — μ)² / N</p><p>Sample variance:</p><p>s² = Σ(x — x̄)² / (n — 1)</p><p>I don’t need to memorise these immediately. The important thing is understanding why the denominator changes depending on whether I have the entire population or a sample.</p><h3>10. Standard Deviation — Bringing Variance Back to the Original Unit</h3><p>Now I understand variance.</p><p>But I noticed something strange.</p><p>If my original data is measured in milliseconds, variance is measured in milliseconds squared.</p><p>For example:</p><p>Response times = 100, 110, 120, 130, 140 ms.</p><p>Mean = 120 ms.</p><p>Deviations:</p><p>-20, -10, 0, +10, +20.</p><p>Squared deviations:</p><p>400, 100, 0, 100, 400.</p><p>Sum = 1,000.</p><p>Population variance = 1,000 / 5 = 200 ms².</p><p>But what does 200 milliseconds squared mean in practical performance testing?</p><p>It isn’t easy to interpret.</p><p>I wanted a measurement in milliseconds again.</p><p>That’s why we take the square root of variance.</p><p><strong>Standard Deviation = √Variance</strong></p><p>Standard Deviation = √200</p><p>Standard Deviation ≈ 14.14 ms.</p><p>Now the result is back in milliseconds.</p><p>This was an important moment in my learning.</p><p>I understood that:</p><ul><li>Variance measures spread using squared deviations.</li><li>Standard deviation is the square root of variance.</li><li>Standard deviation returns the measurement to the original unit.</li></ul><h3>Why is Standard Deviation useful?</h3><p>Imagine two servers have the same average CPU utilization of 60%.</p><p>Server A:</p><p>Standard deviation = 2 percentage points.</p><p>Server B:</p><p>Standard deviation = 15 percentage points.</p><p>Which server is more consistent?</p><p>Server A.</p><p>Why?</p><p>Because its CPU utilization values stay much closer to the average.</p><p>Server B has much greater variation.</p><p><strong>Lower standard deviation generally means more consistent values. Higher standard deviation means greater variation.</strong></p><p>Remember: standard deviation does not mean every observation is exactly that distance from the mean. It is a measure of overall spread.</p><h3>Machine Learning example</h3><p>Imagine I am building a model that predicts house prices.</p><p>My dataset contains house areas:</p><p>1,000, 1,100, 1,200, 1,300, 1,400 sq. ft.</p><p>The mean is 1,200 sq. ft.</p><p>The population standard deviation is approximately 141.42 sq. ft.</p><p>This tells me how spread out the house areas are around their mean.</p><p>Standard deviation is also used in a preprocessing technique called <strong>Standardization</strong>.</p><p>The formula is:</p><p>z = (x — μ) / σ</p><p>Where:</p><ul><li>x = Original value</li><li>μ = Mean</li><li>σ = Standard deviation</li><li>z = Standardized value</li></ul><p>This transforms values into a scale based on how far they are from the mean, measured in standard deviations.</p><p>For example, if a house area is 1,400 sq. ft., with a mean of 1,200 and a standard deviation of approximately 141.42:</p><p>z = (1400–1200) / 141.42</p><p>z ≈ 1.41</p><p>This means the house area is approximately 1.41 standard deviations above the mean.</p><p>Standardization is useful for algorithms such as Logistic Regression, Support Vector Machines, and K-Means, among others.</p><p>Importantly, the mean and standard deviation used for standardization should be calculated from the training data and then applied to validation and test data. This helps prevent data leakage.</p><h3>11. What Have I Learned So Far?</h3><p>I started with a simple question:</p><p>How can I understand a collection of numbers?</p><p>Now I have a set of statistical tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*9GT2pwfG_6hSnlfqpgnEkA.png" /></figure><h3>12. How Do These Concepts Fit Into a Machine Learning Workflow?</h3><p>Let’s imagine I am building a Machine Learning model to predict house prices.</p><p>My dataset contains:</p><ul><li>House area</li><li>Number of bedrooms</li><li>Property age</li><li>Location</li><li>House price</li></ul><p>Before training the model, I need to understand my data.</p><p>Here is how the concepts we learned fit into the process.</p><p><strong>Step 1: Understand the data</strong></p><p>Identify whether each feature is numerical or categorical.</p><p>For example:</p><ul><li>House area = Numerical</li><li>Number of bedrooms = Discrete numerical</li><li>Location = Categorical</li><li>Property age = Numerical</li><li>House price = Numerical target</li></ul><p><strong>Step 2: Explore the data using statistics</strong></p><p>Calculate:</p><ul><li>Mean house price.</li><li>Median house price.</li><li>Most common number of bedrooms.</li><li>Minimum and maximum house area.</li><li>Variance and standard deviation of house prices.</li></ul><p>This helps me understand the dataset.</p><p><strong>Step 3: Handle missing values</strong></p><p>If house-area values are missing, I might use the mean or median.</p><p>If location values are missing, I might consider the mode or another suitable strategy.</p><p><strong>Step 4: Scale numerical features</strong></p><p>Use Min-Max Scaling or Standardization when appropriate for the selected algorithm.</p><p><strong>Step 5: Train the model</strong></p><p>Now the cleaned and prepared data can be used to train a Machine Learning model.</p><p>This is where the model starts learning relationships between features and the target.</p><p>The important thing is that statistics is not separate from Machine Learning. It is part of understanding and preparing the data that Machine Learning depends on.</p><h3>13. Final Thoughts</h3><p>I initially thought learning mathematics meant memorising formulas.</p><p>But now I am trying a different approach.</p><p>Whenever I see a formula, I ask myself:</p><ol><li>What problem is this formula solving?</li><li>Why do we need it?</li><li>Can I understand it using a real-world example?</li><li>Where would I use it in Machine Learning?</li></ol><p>That approach is helping me build intuition rather than just remembering calculations.</p><p>I started with basic numbers, and now I understand why mean, median, mode, range, variance, and standard deviation exist.</p><p>I can also connect these concepts to practical problems in performance testing and Machine Learning.</p><p>This is only the beginning of my Math for Machine Learning journey.</p><p>In the next part, I will explore <strong>percentiles, quartiles, and probability</strong> — concepts that will help me understand API response-time distributions, performance metrics, and eventually how machine learning models interpret uncertainty.</p><p><strong>My goal is simple: I don’t just want to learn the mathematics. I want to understand why the mathematics works.</strong></p><p><em>Learning one concept at a time.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f056e9670eae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Math for Machine Learning: Starting from Zero]]></title>
            <link>https://medium.com/@laxmanpajjuri/math-for-machine-learning-starting-from-zero-dbff6b462730?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbff6b462730</guid>
            <category><![CDATA[basic-math]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[math-for-ml]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 09:09:01 GMT</pubDate>
            <atom:updated>2026-11-02T14:35:24.573Z</atom:updated>
            <content:encoded><![CDATA[<p><em>My journey to understanding the mathematics behind Machine Learning, one concept at a time — used AI in this blog for content generation by giving prompts through my experience and dificulty for understanding math behind ML</em></p><h3>Introduction</h3><p>Have you ever wondered how a machine learns to recognize a face, recommend a movie, predict tomorrow’s weather, or generate an answer using an AI chatbot?</p><p>Behind these capabilities lies mathematics.</p><p>When I started exploring Machine Learning and Large Language Models (LLMs), I realized something important: learning the algorithms alone is not enough. To truly understand how these systems work, I need to understand the mathematics behind them.</p><p>But there is one challenge.</p><p><strong>What if you want to learn Machine Learning but your mathematical foundations are weak? Where should you start?</strong></p><p>That is exactly the question I want to answer through this blog series.</p><p>Instead of jumping directly into complex equations, I am starting from the basics: numbers, fractions, percentages, algebra, functions, vectors, probability, and calculus.</p><p>The goal is simple: understand the concepts first, then connect them to Machine Learning.</p><p>Let’s begin!</p><h3>1. Why Does Mathematics Matter in Machine Learning?</h3><p>Imagine that you want to build a system that predicts house prices.</p><p>You give the system information such as:</p><ul><li>The size of the house</li><li>The number of bedrooms</li><li>The location</li><li>The age of the building</li></ul><p>Based on this information, the system predicts a price.</p><p>But how does it make that prediction?</p><p>It identifies patterns in historical data, represents relationships using mathematical expressions, and adjusts its internal parameters to improve its predictions.</p><p>For example, a simple prediction model might look like this:</p><p><strong>Predicted Price = (House Size × Weight) + Bias</strong></p><p>Don’t worry about the words <em>weight</em> and <em>bias</em> just yet. We will understand them in later blogs.</p><p>For now, notice that the model uses multiplication and addition to calculate an answer.</p><p>That is our first important observation:</p><p><strong>Machine Learning is not magic. Mathematics helps us understand how it works.</strong></p><h3>2. Start With Numbers</h3><p>Before learning advanced mathematics, we need to understand what numbers represent.</p><p>Numbers help us describe quantities.</p><p>For example:</p><ul><li>5 apples means we have five apples.</li><li>100 students means there are one hundred students.</li><li>0 means there are none of something.</li><li>−5°C means the temperature is five degrees below zero.</li></ul><p>Numbers are more than symbols. They allow us to represent and compare information.</p><p>This becomes important in Machine Learning because computers work with numerical representations of data.</p><p>For example, a model might represent a house using values such as:</p><ul><li>House size: 1,500 square feet</li><li>Bedrooms: 3</li><li>Age: 10 years</li></ul><p>These numbers become inputs that a model can process.</p><p>Of course, not all real-world information starts as a number. Text, images, and audio need to be converted into suitable numerical representations before many Machine Learning models can process them.</p><p>Understanding numbers is our first step toward understanding that process.</p><h3>3. Understanding Zero</h3><p>Zero is one of the most important ideas in mathematics.</p><p>It represents the absence of a quantity.</p><p>If you have five apples and give all five away, how many apples remain?</p><p>The answer is zero.</p><p>[<br>5–5=0<br>]</p><p>Simple, right?</p><p>But zero also has an important role in mathematical operations.</p><p>Consider these examples:</p><p>[<br>0+5=5<br>]</p><p>[<br>5times0=0<br>]</p><p>[<br>0div5=0<br>]</p><p>However, dividing by zero is different:</p><p>[<br>5div0<br>]</p><p>This operation is undefined in ordinary arithmetic.</p><p>Why?</p><p>Division asks us to find a number that, when multiplied by the divisor, gives us the original quantity.</p><p>For example:</p><p>[<br>10div2=5<br>]</p><p>Because:</p><p>[<br>2times5=10<br>]</p><p>But for (5div0), we would need a number that satisfies:</p><p>[<br>0times x=5<br>]</p><p>No such number exists because zero multiplied by any ordinary number is always zero.</p><p>That is why division by zero is undefined.</p><p><strong>Why does this matter for Machine Learning?</strong></p><p>Mathematical calculations appear throughout Machine Learning algorithms. Understanding basic operations helps us recognize why certain calculations work, why others fail, and how numerical errors can occur.</p><h3>4. Understanding Negative Numbers</h3><p>Positive numbers represent quantities above zero, while negative numbers can represent quantities below zero or values in the opposite direction.</p><p>Consider a bank account.</p><p>If you have ₹1,000, your balance is positive:</p><p>[<br>+1000<br>]</p><p>If your account is overdrawn by ₹500, we can represent that amount as:</p><p>[<br>-500<br>]</p><p>Negative numbers are also useful for representing temperature, changes in value, and direction.</p><p>Let’s look at a simple example:</p><p>[<br>5–8=-3<br>]</p><p>Starting at 5 and subtracting 8 takes us three units below zero.</p><p>Now consider a Machine Learning example.</p><p>Suppose a model predicts a house price of ₹50 lakh, but the actual price is ₹55 lakh.</p><p>If we define the prediction error as:</p><p>[<br>text{Error}=text{Predicted}-text{Actual}<br>]</p><p>Then:</p><p>[<br>text{Error}=50–55=-5<br>]</p><p>The negative sign tells us that the model predicted a value lower than the actual price by ₹5 lakh.</p><p>If the prediction were ₹60 lakh instead, the error would be:</p><p>[<br>60–55=+5<br>]</p><p>Now the model has overpredicted by ₹5 lakh.</p><p>The sign of a number can tell us not just the size of a difference, but also its direction.</p><h3>5. Understanding Fractions</h3><p>A fraction represents a part of a whole.</p><p>Imagine that you have a pizza divided into four equal slices.</p><p>If you eat one slice, you have eaten:</p><p>[<br>{1}\{4}<br>]</p><p>If you eat two slices, you have eaten:</p><p>[<br>{2}\{4}<br>]</p><p>And if you eat all four slices, you have eaten:</p><p>[<br>{4}\{4}=1<br>]</p><p>A fraction has two parts:</p><ul><li><strong>Numerator:</strong> The number on top, representing how many parts we have.</li><li><strong>Denominator:</strong> The number on the bottom, representing how many equal parts make up the whole.</li></ul><p>For example, in ({3}\{4}), the numerator is 3 and the denominator is 4.</p><p>This means we have three out of four equal parts.</p><h3>Equivalent fractions</h3><p>Here is something interesting.</p><p>[<br>{1}\{2}={2}\{4}={3}\{6}<br>]</p><p>How can different fractions represent the same quantity?</p><p>Imagine cutting a pizza into two equal pieces and taking one piece.</p><p>Now imagine cutting the same pizza into four equal pieces and taking two pieces.</p><p>You still have the same amount of pizza.</p><p>The number of pieces changes, but the quantity does not.</p><p>Mathematically, we can multiply the numerator and denominator by the same nonzero number without changing the value of the fraction.</p><p>For example:</p><p>[<br>{1}\{2}\*{2}\{2}={2}\{4}<br>]</p><p>Since ({2}\{2}=1), we are effectively multiplying the original fraction by 1. Its value remains unchanged.</p><h3>Why fractions matter in Machine Learning</h3><p>Suppose a model correctly classifies 80 out of 100 images.</p><p>Its accuracy is:</p><p>[<br>{Accuracy}={80}\{100}=0.8<br>]</p><p>This fraction can also be expressed as 80%.</p><p>We will explore decimals and percentages next.</p><h3>6. Fractions, Decimals, and Percentages</h3><p>These three forms may look different, but they can represent the same quantity.</p><p>Consider the following examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*_nZEEJUJJlRJFOpJN1mzIQ.png" /></figure><p>Let’s understand the relationship using one example.</p><p>Suppose 25 out of 100 students passed an examination.</p><p>The fraction is:</p><p>[<br>{25}\{100}<br>]</p><p>Dividing 25 by 100 gives us:</p><p>[<br>0.25<br>]</p><p>To convert a decimal into a percentage, multiply it by 100:</p><p>[<br>0.25*100=25%<br>]</p><p>Therefore:</p><p>[<br>{25}\{100}=0.25=25%<br>]</p><p>All three expressions describe the same quantity.</p><h3>A Machine Learning example</h3><p>Suppose a model makes 1,000 predictions and gets 850 correct.</p><p>Its accuracy is:</p><p>[<br>{850}\{1000}=0.85<br>]</p><p>As a percentage:</p><p>[<br>0.85*100=85%<br>]</p><p>We can say that the model has an accuracy of 85%.</p><p>This is a simple example of how fractions, decimals, and percentages help us interpret Machine Learning results.</p><p>However, accuracy alone does not tell us everything about a model’s performance. We will explore other evaluation metrics in a future blog.</p><h3>7. Connecting the Basics to Machine Learning</h3><p>Let’s bring these concepts together.</p><p>Imagine that you are building a simple model to predict house prices.</p><p>Your dataset contains information about several houses, including their sizes and actual prices.</p><p>You need to understand:</p><ul><li><strong>Numbers</strong> to represent house measurements and prices.</li><li><strong>Negative numbers</strong> to understand whether predictions are above or below actual values.</li><li><strong>Fractions</strong> to represent parts, proportions, and ratios.</li><li><strong>Decimals</strong> to represent numerical values precisely.</li><li><strong>Percentages</strong> to understand metrics such as accuracy and percentage error.</li></ul><p>These are not advanced Machine Learning concepts. They are the mathematical building blocks that make more advanced concepts easier to understand.</p><p>Once these foundations are clear, we can move on to algebra, variables, equations, and functions.</p><p>These concepts will help us understand how a Machine Learning model combines inputs to produce a prediction.</p><h3>8. What Comes Next?</h3><p>In the next blog, I will explore algebra and functions.</p><p>We will answer questions such as:</p><ul><li>What is a variable, and why do we use symbols such as (x) and (y)?</li><li>What does an equation represent?</li><li>How do we solve an equation?</li><li>What is a function?</li><li>What does (y=mx+b) actually mean?</li><li>How does this simple equation connect to Machine Learning?</li></ul><p>Instead of memorizing formulas, we will understand why they work using simple examples.</p><h3>Final Thoughts</h3><p>Learning mathematics for Machine Learning does not have to begin with complicated equations or advanced calculus.</p><p>It can begin with something as simple as understanding numbers, fractions, and percentages.</p><p>The key is to build one concept at a time and understand how each concept connects to the bigger picture.</p><p>I am documenting this journey to strengthen my own mathematical foundations and, hopefully, help other beginners who want to understand Machine Learning from the ground up.</p><p><strong>You do not need to know all the mathematics before you begin. You need to be willing to learn it, one concept at a time.</strong></p><p>This is just the beginning.</p><p><em>Next: Understanding Algebra and Functions — From Unknown Values to Machine Learning Predictions.</em></p><p><em>If you are also learning mathematics for Machine Learning from scratch, follow along with this series. We will start with the basics and gradually work our way toward the mathematics behind neural networks and Large Language Models (LLMs).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbff6b462730" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[StatefulSets vs Deployments in Kubernetes: A Clear, Practical Guide]]></title>
            <link>https://medium.com/@laxmanpajjuri/statefulsets-vs-deployments-in-kubernetes-a-clear-practical-guide-2462bad42ade?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2462bad42ade</guid>
            <category><![CDATA[satefulset]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 07:30:38 GMT</pubDate>
            <atom:updated>2026-06-23T07:30:38.228Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Why your database and your API server need completely different Kubernetes objects — explained with real examples</em></blockquote><p>If you’ve worked with Kubernetes for more than a week, you’ve used a Deployment. It’s the default, go-to object for running anything. But the moment you try to run a database on Kubernetes, Deployment starts breaking in subtle, confusing ways. That’s where StatefulSet comes in.</p><p>This article explains the difference in the simplest possible terms, then backs it up with real YAML you can use today.</p><h3>The one-sentence difference</h3><pre>Deployment   →  treats pods as CATTLE — identical, disposable, interchangeable<br>StatefulSet  →  treats pods as PETS — each one has a name, an identity, and its own data</pre><p>That’s it. Everything else in this article is just explaining the consequences of that one idea.</p><h3>Why “cattle vs pets” actually matters</h3><p>Imagine you have 3 pods running your Spring Boot API. If pod #2 crashes, Kubernetes deletes it and creates a brand new one. Does it matter which pod handled which request? No. They’re all identical copies running the same code with no memory of anything. You could shuffle them, rename them, replace all three at once — nothing breaks. That’s a <strong>Deployment</strong> use case.</p><p>Now imagine you have 3 MySQL replicas. Pod #2 isn’t just “a MySQL pod” — it might be the one holding a specific slice of data, or it might be the primary that the other two replicate from. If Kubernetes deleted it and created a “new” pod with a random name and no memory of which data directory belonged to it, your database would break. That’s a <strong>StatefulSet</strong> use case.</p><h3>Visual comparison</h3><pre>DEPLOYMENT (3 replicas)<br>┌─────────────┐  ┌─────────────┐  ┌─────────────┐<br>│ pod-7x9k2    │  │ pod-3m8p1    │  │ pod-9q2r5    │<br>│ random name  │  │ random name  │  │ random name  │<br>│ no own data  │  │ no own data  │  │ no own data  │<br>└─────────────┘  └─────────────┘  └─────────────┘<br>   any can die and be replaced — nobody notices<br></pre><pre>STATEFULSET (3 replicas)<br>┌─────────────┐  ┌─────────────┐  ┌─────────────┐<br>│  mysql-0     │  │  mysql-1     │  │  mysql-2     │<br>│ fixed name   │  │ fixed name   │  │ fixed name   │<br>│ owns disk-0  │  │ owns disk-1  │  │ owns disk-2  │<br>└─────────────┘  └─────────────┘  └─────────────┘<br>   if mysql-1 restarts, it comes BACK as mysql-1<br>   and reconnects to disk-1 — same identity, same data</pre><h3>Five concrete differences</h3><h3>1. Pod names</h3><pre># Deployment — random suffix, changes every time a pod restarts<br>kubectl get pods -l app=backend<br>NAME                       <br>backend-5d8f9c7b6d-7x9k2   <br>backend-5d8f9c7b6d-3m8p1   </pre><pre># StatefulSet — predictable, numbered, NEVER changes<br>kubectl get pods -l app=mysql<br>NAME      <br>mysql-0   <br>mysql-1   <br>mysql-2</pre><p>If you delete mysql-1, Kubernetes creates a replacement pod and names it mysql-1 again — not some random string. The name is part of its identity.</p><h3>2. Storage</h3><p>This is the big one. With a Deployment, if multiple pods need storage, they typically share <strong>one</strong> volume, or each gets ephemeral storage that disappears on restart. With a StatefulSet, <strong>each pod automatically gets its own separate, persistent volume</strong> — and that exact volume follows that exact pod forever.</p><pre># Deployment — if you add storage at all, it&#39;s usually shared<br>volumes:<br>  - name: shared-storage<br>    persistentVolumeClaim:<br>      claimName: one-pvc-for-everyone</pre><pre># StatefulSet — each pod gets its OWN automatically created volume<br>volumeClaimTemplates:<br>  - metadata:<br>      name: data<br>    spec:<br>      accessModes: [&quot;ReadWriteOnce&quot;]<br>      resources:<br>        requests:<br>          storage: 5Gi<br># this single block creates THREE separate PVCs behind the scenes:<br># data-mysql-0, data-mysql-1, data-mysql-2 — one per pod</pre><h3>3. Startup and shutdown order</h3><pre>Deployment scaling to 3:<br>  all 3 pods start AT THE SAME TIME, in any order</pre><pre>StatefulSet scaling to 3:<br>  mysql-0 must be Running and Ready<br>      ↓ only then<br>  mysql-1 starts<br>      ↓ only then<br>  mysql-2 starts</pre><pre>StatefulSet scaling DOWN from 3 to 1:<br>  mysql-2 is removed FIRST<br>      ↓ only then<br>  mysql-1 is removed<br>  (mysql-0 is always the last one standing)</pre><p>This ordering matters for real databases. A MySQL primary (mysql-0) needs to exist and be ready before replicas (mysql-1, mysql-2) try to connect and start replicating from it.</p><h3>4. Network identity</h3><pre>Deployment pods reach each other only through a Service —<br>there&#39;s no way to address &quot;that one specific pod&quot; directly.</pre><pre>StatefulSet pods get their OWN individual DNS name:<br>  mysql-0.mysql-service.taskmanager.svc.cluster.local<br>  mysql-1.mysql-service.taskmanager.svc.cluster.local<br>  mysql-2.mysql-service.taskmanager.svc.cluster.local</pre><pre>Now your application can say &quot;always talk to mysql-0 for writes&quot;<br>and that name will ALWAYS point to the same pod, even after restarts.</pre><h3>5. What happens on restart</h3><pre>Deployment:<br>  pod dies → new pod gets a NEW random name<br>           → mounts whatever generic/shared storage was configured<br>           → has no &quot;memory&quot; of being anything special</pre><pre>StatefulSet:<br>  pod dies → new pod gets the SAME name (e.g. mysql-1)<br>           → automatically reconnects to the SAME PersistentVolumeClaim<br>             (data-mysql-1)<br>           → picks up exactly where the old one left off, same data</pre><h3>Side-by-side YAML</h3><p>Here’s the same idea expressed as both objects, so you can see exactly where they diverge.</p><h3>Deployment — for your stateless backend</h3><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: backend<br>  namespace: taskmanager<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: backend<br>  template:<br>    metadata:<br>      labels:<br>        app: backend<br>    spec:<br>      containers:<br>        - name: backend<br>          image: task-app:latest<br>          ports:<br>            - containerPort: 8080<br># No volumeClaimTemplates section at all — these pods don&#39;t need<br># individual storage. If they store anything, it&#39;s in the shared MySQL.</pre><h3>StatefulSet — for your MySQL database</h3><pre>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: mysql<br>  namespace: taskmanager<br>spec:<br>  serviceName: &quot;mysql-service&quot;     # ← REQUIRED, Deployment has no equivalent<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: mysql<br>  template:<br>    metadata:<br>      labels:<br>        app: mysql<br>    spec:<br>      containers:<br>        - name: mysql<br>          image: mysql:8.0<br>          ports:<br>            - containerPort: 3306<br>          volumeMounts:<br>            - name: data<br>              mountPath: /var/lib/mysql<br>  volumeClaimTemplates:             # ← ONLY StatefulSet has this<br>    - metadata:<br>        name: data<br>      spec:<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        resources:<br>          requests:<br>            storage: 5Gi</pre><p>Two extra fields — serviceName and volumeClaimTemplates — are the entire structural difference. Everything else looks almost identical. But those two fields completely change how Kubernetes manages the pods underneath.</p><h3>The headless Service requirement</h3><p>StatefulSets require something called a <strong>headless Service</strong> — this is what gives each pod its individual DNS name.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mysql-service<br>  namespace: taskmanager<br>spec:<br>  clusterIP: None        # ← THIS is what makes it &quot;headless&quot;<br>  selector:<br>    app: mysql<br>  ports:<br>    - port: 3306</pre><p>A normal Service gets one virtual IP (a ClusterIP) that load-balances across all matching pods — you never know or care which specific pod actually answered. Setting clusterIP: None removes that virtual IP entirely. Instead, DNS lookups return the <strong>individual pod IPs directly</strong>, which is exactly what gives you mysql-0.mysql-service... as a usable, stable address.</p><h3>When to use which — the simple rule</h3><p>Ask yourself: <strong>“If I delete this pod and it comes back with no memory of what it was, does anything break?”</strong></p><pre>&quot;No, totally fine&quot; → Deployment<br>  Examples: API servers, frontends, stateless workers,<br>            anything that reads/writes its data somewhere else</pre><pre>&quot;Yes, that would be a disaster&quot; → StatefulSet<br>  Examples: MySQL, PostgreSQL, MongoDB, Kafka, Zookeeper,<br>            Elasticsearch, Redis (in cluster mode)</pre><h3>A common mistake to avoid</h3><p>A lot of people running a single database instance think “I only have 1 replica, so Deployment is fine.” It’s not, even with replicas: 1. The moment that one pod restarts — and pods restart constantly, for node maintenance, upgrades, or just because Kubernetes feels like it — a Deployment gives you no guarantee that the new pod reconnects to the same storage in the same way. StatefulSet&#39;s volumeClaimTemplates mechanism is what reliably ties one specific pod identity to one specific piece of storage, even at replicas: 1.</p><h3>Quick reference table</h3><p>Deployment StatefulSet Pod names Random (backend-7x9k2) Fixed (mysql-0, mysql-1) Storage Shared or none Individual PVC per pod, automatic Scaling order All at once, any order One at a time, in sequence Network identity Service only Service + per-pod DNS name Restart behavior New pod, fresh identity Same name, reconnects to same data Use for APIs, frontends, workers Databases, message queues, caches Your TaskManager app React frontend, Spring Boot backend MySQL</p><h3>Conclusion</h3><p>Deployment and StatefulSet aren’t “the simple one” and “the complicated one” — they’re built for two genuinely different problems. Deployment assumes every pod is interchangeable, which is true for the vast majority of application workloads. StatefulSet exists specifically for the cases where that assumption breaks: anything where a pod’s identity and its data need to survive restarts, scaling, and rescheduling together.</p><p>The good news is you rarely need to decide carefully — the answer is almost always obvious once you ask “would this pod losing its identity actually break something?” If you’re running a database, the answer is yes, and you reach for StatefulSet. For everything else, Deployment remains the right default.</p><p><em>If this helped clear things up, leave a clap — it helps more people find clear explanations like this one.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2462bad42ade" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying TaskManager on Kubernetes: A Complete Line-by-Line Production Walkthrough]]></title>
            <link>https://medium.com/@laxmanpajjuri/deploying-taskmanager-on-kubernetes-a-complete-line-by-line-production-walkthrough-96be1f4f1ce0?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/96be1f4f1ce0</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[full-stack]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 14:55:23 GMT</pubDate>
            <atom:updated>2026-06-24T16:20:46.061Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Every YAML file explained, every line commented, every concept from Namespace to Distributed Tracing — built on a real React + Spring Boot + MySQL app</em></blockquote><p>This is the complete production deployment of a three-tier application on Kubernetes, with every single line of YAML explained. By the end, you will understand not just what each object does, but why each field exists and what breaks if you remove it.</p><p>The stack: <strong>React (Nginx) → Spring Boot → MySQL</strong>, deployed with full observability, security, scaling, and networking — exactly how it would be deployed in a real company.</p><h3>Architecture overview</h3><pre>Internet<br>    ↓<br>Ingress Controller (Nginx)<br>    ↓  taskmanager.local<br>    ↓  /api  →  backend-service:8080<br>    ↓  /     →  frontend-service:80<br>    ↓<br>┌─────────────────────────────────────────────────────┐<br>│  Namespace: taskmanager                             │<br>│                                                     │<br>│  Frontend Pods (Nginx, replicas=2)                  │<br>│    ↓ proxy_pass /api/                               │<br>│  Backend Pods (Spring Boot, replicas=2, HPA 2-10)  │<br>│    ↓ jdbc:mysql://mysql-service:3306/taskdb         │<br>│  MySQL StatefulSet (replicas=1)                     │<br>│                                                     │<br>│  NetworkPolicy: deny-all + selective allow          │<br>│  RBAC: least-privilege ServiceAccounts              │<br>│  HPA + VPA: autoscaling                             │<br>│  OTel + Jaeger: tracing                             │<br>│  Fluentd + ELK: logging                             │<br>│  Prometheus + Grafana: metrics                      │<br>└─────────────────────────────────────────────────────┘</pre><h3>Part 1: Namespace — the foundation</h3><p>Every resource we create lives inside this namespace. It is the first thing we deploy.</p><pre>apiVersion: v1                    # core API group — Namespace has always been here<br>kind: Namespace                   # the object type we are creating<br>metadata:<br>  name: taskmanager                # this is the name we reference everywhere else<br>                                    # every other YAML in this project says<br>                                    # namespace: taskmanager<br>  labels:<br>    # ── Pod Security Standards ──────────────────────────<br>    # These labels make Kubernetes itself enforce security rules<br>    # on every pod created in this namespace - no extra controller needed<br>    pod-security.kubernetes.io/enforce: baseline<br>    # baseline = blocks privileged containers, hostNetwork, hostPath volumes<br>    # any pod violating this is REJECTED with a 403 error at creation time<br>    pod-security.kubernetes.io/enforce-version: latest<br>    # pins which K8s version&#39;s definition of &quot;baseline&quot; to use<br>    # &quot;latest&quot; tracks whatever the cluster is currently running<br>    pod-security.kubernetes.io/warn: restricted<br>    # restricted is the strictest level - requires non-root, read-only FS<br>    # &quot;warn&quot; means: don&#39;t block, but print a warning if a pod would fail this<br>    # this lets you see what you&#39;d need to fix before actually enforcing it<br>    pod-security.kubernetes.io/warn-version: latest<br>    pod-security.kubernetes.io/audit: restricted<br>    # same as warn, but the violation goes into the audit log instead of<br>    # your terminal - useful for compliance review without blocking anyone<br>    pod-security.kubernetes.io/audit-version: latest<br>    # ── Used by NetworkPolicy later ──────────────────────<br>    kubernetes.io/metadata.name: taskmanager<br>    # NetworkPolicy objects later in this project reference this exact label<br>    # to say &quot;allow traffic FROM the taskmanager namespace&quot; - this label<br>    # is actually auto-added by Kubernetes itself since 1.21, we just rely on it<br>    app.kubernetes.io/managed-by: kubectl<br>    # informational only - tells anyone reading `kubectl get ns -o yaml`<br>    # that this object is managed by plain kubectl apply, not Helm or ArgoCD<br>    environment: production<br>    # informational/organizational label - not enforced by Kubernetes,<br>    # but useful for filtering: kubectl get ns -l environment=production</pre><p><strong>Why this matters:</strong> Without the PSS labels, any pod — including a misconfigured one — could run as root or mount the host filesystem. With enforce: baseline, Kubernetes itself blocks that at admission time, before the pod is ever scheduled.</p><pre>kubectl apply -f 00-namespace/namespace.yaml<br># Verify the labels landed<br>kubectl get namespace taskmanager --show-labels</pre><h3>Part 2: ConfigMap and Secret — configuration and credentials</h3><pre>apiVersion: v1<br>kind: ConfigMap                   # holds plain-text, non-sensitive config<br>metadata:<br>  name: taskmanager-config        # other objects reference this exact name<br>                                    # via configMapKeyRef<br>  namespace: taskmanager           # must match - ConfigMaps are namespace-scoped<br>  labels:<br>    app: taskmanager<br>    component: config              # organizational - lets you query<br>                                    # kubectl get cm -l component=config<br>data:<br>  # ── Database connection info (safe to expose - not secrets) ──<br>  DB_HOST: &quot;mysql-service&quot;<br>  # this is NOT an IP address - it&#39;s a Kubernetes Service NAME<br>  # CoreDNS resolves &quot;mysql-service&quot; to the Service&#39;s ClusterIP automatically<br>  # this value never needs to change even if the MySQL pod restarts<br>  DB_PORT: &quot;3306&quot;<br>  # MySQL&#39;s default port - stored as a string because ConfigMap<br>  # values are ALWAYS strings, even when they represent numbers<br>  DB_NAME: &quot;taskdb&quot;<br>  # the actual database name Spring Boot will connect to inside MySQL<br>  # ── Application behavior config ──────────────────────────<br>  LOG_LEVEL: &quot;INFO&quot;<br>  # controls Spring Boot&#39;s logging verbosity - read by application.properties<br>  # via ${LOG_LEVEL} placeholder<br>  SERVER_PORT: &quot;8080&quot;<br>  # the port Spring Boot&#39;s embedded Tomcat listens on inside the container<br>  SPRING_PROFILES_ACTIVE: &quot;production&quot;<br>  # tells Spring Boot which application-production.properties to load<br>  # this is how the SAME Docker image behaves differently in prod vs dev<br>  # ── JVM tuning ────────────────────────────────────────────<br>  JVM_OPTS: &quot;-Xms256m -Xmx512m -XX:+UseG1GC&quot;<br>  # -Xms256m  = JVM starts with 256MB heap already allocated<br>  # -Xmx512m  = JVM will never use more than 512MB heap<br>  #             this MUST be less than the container&#39;s memory limit (1Gi)<br>  #             or the JVM gets OOMKilled by the kernel, not by its own GC<br>  # -XX:+UseG1GC = use the G1 garbage collector - good default for<br>  #                containerized apps with limited, fixed memory<br>  # ── OpenTelemetry tracing config ─────────────────────────<br>  OTEL_EXPORTER_OTLP_ENDPOINT: &quot;http://otel-collector:4317&quot;<br>  # where Spring Boot sends trace spans - another Service DNS name,<br>  # resolved by CoreDNS, port 4317 is OTLP&#39;s standard gRPC port<br>  OTEL_TRACES_SAMPLER: &quot;always_on&quot;<br>  # 100% of requests get traced - fine for learning/staging<br>  # in real production you&#39;d use something like &quot;traceidratio&quot; at 10%<br>  # to avoid overwhelming the tracing backend with volume<br>  OTEL_RESOURCE_ATTRIBUTES: &quot;service.name=taskmanager-backend,environment=production&quot;<br>  # metadata attached to EVERY span this service creates<br>  # this is what makes &quot;taskmanager-backend&quot; show up as a service name<br>  # in the Jaeger UI&#39;s service dropdown<br>---<br>apiVersion: v1<br>kind: Secret                      # same shape as ConfigMap, different intent<br>                                    # values must be base64 encoded by you<br>metadata:<br>  name: taskmanager-secret<br>  namespace: taskmanager<br>  labels:<br>    app: taskmanager<br>    component: secret<br>type: Opaque<br># Opaque = generic key-value secret, no special K8s behavior attached<br># (contrast with type: kubernetes.io/tls which K8s treats specially)<br>data:<br>  DB_USER: cm9vdA==<br>  # this is base64 of the string &quot;root&quot; - NOT encryption<br>  # decode it yourself: echo cm9vdA== | base64 -d  →  root<br>  # anyone with `kubectl get secret -o yaml` access reads this in 2 seconds<br>  DB_PASSWORD: cm9vdDEyMw==<br>  # base64 of &quot;root123&quot; - same caveat applies<br>  # in a REAL production cluster this entire Secret object would be<br>  # replaced by an ExternalSecret synced from AWS Secrets Manager or Vault</pre><p><strong>Why ConfigMap and Secret are separate objects:</strong> They are functionally almost identical — both inject key-value pairs into pods. The split exists purely for <em>intent</em> and <em>access control</em>. You can write RBAC rules that allow reading ConfigMaps but block reading Secrets, and tools like kubectl get all deliberately don&#39;t print Secret values by default.</p><pre>kubectl apply -f 01-config/configmap-secret.yaml<br># Verify - Secret values show as base64, not plain text<br>kubectl get secret taskmanager-secret -n taskmanager -o yaml<br># Decode to verify<br>kubectl get secret taskmanager-secret -n taskmanager \<br>  -o jsonpath=&#39;{.data.DB_PASSWORD}&#39; | base64 -d</pre><h3>Part 3: PersistentVolume and PersistentVolumeClaim — storage</h3><p>If you are using minikube for practie use the below command.</p><pre>minikube ssh -- sudo mkdir -p /data/mysql &amp;&amp; sudo chmod 777 /data/mysql</pre><pre>apiVersion: v1<br>kind: PersistentVolume            # PV is a CLUSTER-scoped object — no namespace<br>                                    # it represents an actual piece of storage<br>metadata:<br>  name: mysql-pv                  # cluster-wide unique name<br>  labels:<br>    app: mysql<br>    type: local                    # informational - &quot;local&quot; means hostPath,<br>                                    # not network storage like EBS or NFS<br>spec:<br>  capacity:<br>    storage: 5Gi<br>    # how much storage THIS PV offers - a PVC can request up to this amount<br>  accessModes:<br>    - ReadWriteOnce<br>    # ReadWriteOnce (RWO) = only ONE node can mount this read-write at a time<br>    # this is correct for MySQL - a database should never be written<br>    # to from two places simultaneously<br>    # (ReadWriteMany would be for shared file storage like NFS)<br>  persistentVolumeReclaimPolicy: Retain<br>  # Retain = when the PVC is deleted, KEEP the underlying data<br>  #          you&#39;d have to manually delete it - this is the safe choice<br>  # Delete = automatically wipe the data when PVC is deleted - risky for a DB<br>  storageClassName: manual<br>  # &quot;manual&quot; is not a real provisioner - it just means &quot;this PV was created<br>  # by a human, not auto-provisioned&quot; - matches PVC&#39;s storageClassName below<br>  hostPath:<br>    path: /data/mysql<br>    # THIS is the actual storage backend - a folder on the Minikube VM&#39;s disk<br>    # in production this entire `hostPath:` block is replaced by something like:<br>    #   csi:<br>    #     driver: ebs.csi.aws.com<br>    #     volumeHandle: vol-0123456789<br>---<br>apiVersion: v1<br>kind: PersistentVolumeClaim       # PVC IS namespace-scoped - this is the<br>                                    # developer-facing &quot;I need storage&quot; request<br>metadata:<br>  name: mysql-pvc                 # MySQL&#39;s pod spec references this exact name<br>  namespace: taskmanager<br>  labels:<br>    app: mysql<br>    component: storage<br>spec:<br>  storageClassName: manual<br>  # MUST match the PV&#39;s storageClassName - this is how Kubernetes decides<br>  # which PV is even eligible to bind to this PVC<br>  accessModes:<br>    - ReadWriteOnce<br>    # MUST be compatible with what the PV offers<br>  resources:<br>    requests:<br>      storage: 5Gi<br>      # requesting exactly 5Gi - must be &lt;= the PV&#39;s capacity<br>      # if you requested 10Gi here, this PVC would stay Pending forever<br>      # because no PV in the cluster could satisfy it</pre><p><strong>The binding process:</strong> When you kubectl apply The PVC, Kubernetes&#39;s PV controller, scans every existing PV looking for one that matches storageClassName, has enough capacity, and supports the requested accessModes. When it finds mysql-pv, it binds them together — this binding is recorded as Bound status on both objects.</p><pre>kubectl apply -f 02-storage/pv-pvc.yaml<br>kubectl get pv<br># STATUS should show: Bound<br>kubectl get pvc -n taskmanager<br># STATUS should show: Bound, VOLUME should show: mysql-pv</pre><h3>Part 4: MySQL StatefulSet — the database</h3><p>This is the longest single object so far because it combines storage, networking, security, and health checking all at once.</p><pre>apiVersion: apps/v1<br>kind: StatefulSet<br># StatefulSet, not Deployment — because each MySQL pod needs a STABLE<br># identity and ITS OWN storage. A Deployment&#39;s pods are interchangeable;<br># StatefulSet pods are named mysql-0, mysql-1... and keep that name forever<br>metadata:<br>  name: mysql<br>  namespace: taskmanager<br>  labels:<br>    app: mysql<br>    tier: database<br>    version: &quot;8.0&quot;<br>spec:<br>  serviceName: &quot;mysql-service&quot;<br>  # REQUIRED for StatefulSet - links to a headless Service (defined below)<br>  # this is what gives each pod its own DNS name like<br>  # mysql-0.mysql-service.taskmanager.svc.cluster.local<br>  replicas: 1<br>  # single MySQL instance for this learning setup<br>  # a real production MySQL would use a managed service (RDS) or<br>  # a proper operator (Percona XtraDB Cluster, Vitess) for true HA<br>  selector:<br>    matchLabels:<br>      app: mysql<br>    # this tells the StatefulSet &quot;the pods I own all have label app: mysql&quot;<br>    # MUST exactly match template.metadata.labels below<br>  template:<br>    metadata:<br>      labels:<br>        app: mysql<br>        tier: database<br>        # every pod created from this template gets these labels<br>        # the mysql-service Service below uses these same labels to find pods<br>      annotations:<br>        # ── annotations are metadata that don&#39;t affect K8s scheduling ──<br>        # they&#39;re read by OTHER tools watching the cluster<br>        prometheus.io/scrape: &quot;true&quot;<br>        # tells Prometheus (if configured for annotation-based discovery)<br>        # &quot;yes, scrape metrics from this pod&quot;<br>        prometheus.io/port: &quot;9104&quot;<br>        # the port to scrape - this is the mysqld-exporter sidecar&#39;s port,<br>        # not MySQL&#39;s own port 3306<br>        cluster-autoscaler.kubernetes.io/safe-to-evict: &quot;false&quot;<br>        # tells AWS/GCP Cluster Autoscaler: &quot;NEVER evict this pod during<br>        # scale-down, even if the node looks underutilized&quot; - protects<br>        # the database from being killed just to save a few cents<br>    spec:<br>      # ── Pod-level security context ──────────────────────────<br>      securityContext:<br>        runAsUser: 999<br>        # MySQL&#39;s official Docker image creates a &quot;mysql&quot; user with UID 999<br>        # we run as this UID explicitly instead of letting it default to root<br>        runAsGroup: 999<br>        fsGroup: 999<br>        # fsGroup affects volume permissions - when the PVC is mounted,<br>        # K8s chgrps the volume to GID 999 so the mysql user can write to it<br>      # ── Node affinity ────────────────────────────────────────<br>      affinity:<br>        nodeAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>          # &quot;preferred&quot; = soft rule. If no SSD node exists, schedule anyway.<br>          # &quot;IgnoredDuringExecution&quot; = if the node&#39;s label changes AFTER<br>          # the pod is already running, don&#39;t evict it because of that<br>            - weight: 80<br>              # weight 0-100, higher = scheduler tries harder to satisfy this<br>              preference:<br>                matchExpressions:<br>                  - key: disk<br>                    operator: In<br>                    values: [ssd]<br>                    # &quot;if a node has label disk=ssd, strongly prefer it&quot;<br>                    # databases benefit enormously from fast disk I/O<br>      containers:<br>        # ── Container 1: MySQL itself ──────────────────────────<br>        - name: mysql<br>          image: mysql:8.0<br>          # official MySQL image, version pinned to 8.0 (not &quot;latest&quot; -<br>          # you never want a database to silently upgrade major versions)<br>          imagePullPolicy: IfNotPresent<br>          # pull from registry only if not already cached locally -<br>          # standard for public images (contrast with imagePullPolicy: Never<br>          # which we use for our own custom-built backend/frontend images)<br>          ports:<br>            - name: mysql<br>              containerPort: 3306<br>              # documents that this container listens on 3306 - purely<br>              # informational, does NOT actually open the port (the<br>              # container does that itself by running mysqld)<br>          env:<br>            - name: MYSQL_ROOT_PASSWORD<br>              valueFrom:<br>                secretKeyRef:<br>                  name: taskmanager-secret<br>                  key: DB_PASSWORD<br>              # MySQL&#39;s official image reads this specific env var name<br>              # to set the root password on first boot - pulled from our Secret<br>            - name: MYSQL_DATABASE<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: DB_NAME<br>              # MySQL&#39;s official image auto-creates this database on first boot<br>          volumeMounts:<br>            - name: mysql-storage<br>              mountPath: /var/lib/mysql<br>              # /var/lib/mysql is MySQL&#39;s hardcoded data directory -<br>              # by mounting our PVC here, all of MySQL&#39;s actual table files<br>              # live on persistent storage, not the container&#39;s ephemeral layer<br>          resources:<br>            requests:<br>              cpu: &quot;100m&quot;<br>              memory: &quot;512Mi&quot;<br>            limits:<br>              cpu: &quot;500m&quot;<br>              memory: &quot;512Mi&quot;<br>              # NOTICE: memory request EQUALS memory limit (512Mi = 512Mi)<br>              # this makes MySQL&#39;s QoS class &quot;Guaranteed&quot; - the highest<br>              # priority class, meaning it is the LAST pod the kubelet<br>              # would ever kill if the node runs low on memory<br>          livenessProbe:<br>            exec:<br>              command:<br>                - sh<br>                - -c<br>                - &quot;mysqladmin ping -uroot -p${MYSQL_ROOT_PASSWORD} --silent&quot;<br>              # runs this exact shell command INSIDE the container<br>              # mysqladmin ping returns exit code 0 if MySQL responds,<br>              # non-zero otherwise - kubelet restarts the container on failure<br>            initialDelaySeconds: 40<br>            # wait 40s after container start before the FIRST check -<br>            # MySQL&#39;s first boot (creating system tables) takes time<br>            periodSeconds: 10<br>            # check every 10 seconds after that<br>            failureThreshold: 5<br>            # must fail 5 times in a row (50 seconds) before restarting -<br>            # generous threshold because MySQL under heavy load can be slow<br>            # to respond without actually being broken<br>          readinessProbe:<br>            exec:<br>              command:<br>                - sh<br>                - -c<br>                - &quot;mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e &#39;SELECT 1&#39;&quot;<br>              # stricter check than liveness - actually runs a query,<br>              # not just a ping, proving the SQL engine is fully operational<br>            initialDelaySeconds: 20<br>            periodSeconds: 10<br>            failureThreshold: 5<br>            # if this fails, the pod is marked NotReady - the Service stops<br>            # sending traffic to it, but the container is NOT restarted<br>        # ── Container 2: metrics sidecar ───────────────────────<br>        - name: mysqld-exporter<br>          image: prom/mysqld-exporter:v0.15.0<br>          # a separate, purpose-built tool that connects to MySQL and<br>          # translates its internal stats into Prometheus&#39;s metric format<br>          ports:<br>            - name: metrics<br>              containerPort: 9104<br>              # mysqld-exporter&#39;s default metrics port<br>          env:<br>            - name: DATA_SOURCE_NAME<br>              value: &quot;root:root123@(localhost:3306)/&quot;<br>              # NOTE: &quot;localhost&quot; works here because this sidecar runs<br>              # in the SAME POD as the mysql container - they share network<br>              # namespace, so localhost:3306 reaches the mysql container directly<br>              # (this is hardcoded for simplicity; production would pull<br>              # this from the same Secret as the main container)<br>          resources:<br>            requests:<br>              cpu: &quot;10m&quot;<br>              memory: &quot;32Mi&quot;<br>            limits:<br>              cpu: &quot;50m&quot;<br>              memory: &quot;64Mi&quot;<br>              # tiny footprint - this sidecar does almost nothing<br>              # except periodically query MySQL&#39;s SHOW STATUS<br>      volumes:<br>        - name: mysql-storage<br>          persistentVolumeClaim:<br>            claimName: mysql-pvc<br>            # connects the volume name used in volumeMounts above<br>            # to the actual PVC object we created in Part 3<br>---<br>apiVersion: v1<br>kind: Service<br># this Service is technically a &quot;headless&quot; service candidate, but since<br># we only have replicas: 1 and don&#39;t need per-pod DNS yet, we use a normal<br># ClusterIP here - StatefulSet still works fine, just without per-pod addressing<br>metadata:<br>  name: mysql-service<br>  # THIS EXACT NAME is what every other component uses as DB_HOST<br>  namespace: taskmanager<br>  labels:<br>    app: mysql<br>    tier: database<br>spec:<br>  clusterIP: None          # Headless Service<br>  selector:<br>    app: mysql<br>    # finds pods with label app=mysql - that&#39;s the StatefulSet&#39;s pods<br>  ports:<br>    - name: mysql<br>      port: 3306<br>      # the port other pods use: mysql-service:3306<br>      targetPort: 3306<br>      # the port forwarded to on the actual pod<br>    - name: metrics<br>      port: 9104<br>      targetPort: 9104<br>      # exposes the sidecar&#39;s metrics port too, so Prometheus can reach it<br>      # via the Service if needed<br>---<br>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br># PDB doesn&#39;t prevent eviction outright - it tells voluntary disruption<br># mechanisms (like Cluster Autoscaler or `kubectl drain`) the MINIMUM<br># number of pods that must stay running at all times<br>metadata:<br>  name: mysql-pdb<br>  namespace: taskmanager<br>spec:<br>  minAvailable: 1<br>  # with replicas: 1, this effectively means &quot;never evict this pod<br>  # voluntarily&quot; since evicting it would bring availability to 0,<br>  # which violates minAvailable: 1<br>  selector:<br>    matchLabels:<br>      app: mysql<br>      # applies to any pod matching this label - currently just our<br>      # single MySQL StatefulSet pod</pre><p><strong>Why a sidecar instead of a separate Deployment for the exporter:</strong> The exporter needs to talk to MySQL overlocalhost, which only works if both containers are in the <em>same pod</em> (sharing a network namespace). Running it as a separate Deployment would require it to reach MySQL over the Service, which works too. Still, the sidecar pattern keeps the metrics collector&#39;s lifecycle tightly coupled to the database it&#39;s monitoring — they start and stop together.</p><pre>kubectl apply -f 03-database/mysql-statefulset.yaml<br># Watch the StatefulSet create mysql-0<br>kubectl get pods -n taskmanager -w<br># Once Running, verify both containers are up<br>kubectl get pod mysql-0 -n taskmanager<br># READY should show 2/2 (mysql + mysqld-exporter)</pre><p><em>Continued in Part 2 — Backend Deployment, Frontend, Networking, and beyond.</em></p><h3>Part 5: Backend Deployment — Spring Boot with everything attached</h3><p>This is the most heavily annotated object in the entire project because it combines workload management, security, scaling, scheduling, and observability in a single file.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br># Deployment, not StatefulSet — backend pods are STATELESS.<br># Any replica can handle any request; none of them has unique identity<br># or its own storage. This is the defining difference from MySQL above.<br>metadata:<br>  name: backend<br>  namespace: taskmanager<br>  labels:<br>    app: backend<br>    tier: api<br>    version: &quot;1.0.0&quot;<br>    # version label is informational - useful for filtering during<br>    # canary releases: kubectl get pods -l version=1.0.0<br>  annotations:<br>    kubernetes.io/change-cause: &quot;Initial production deployment&quot;<br>    # shows up in `kubectl rollout history deployment/backend` -<br>    # a human-readable note attached to THIS specific revision<br>spec:<br>  replicas: 2<br>  # baseline pod count - HPA (defined later) can scale this between 2 and 10<br>  # starting at 2 ensures basic HA even before any traffic-based scaling kicks in<br>  selector:<br>    matchLabels:<br>      app: backend<br>    # MUST exactly match template.metadata.labels - this is how the<br>    # Deployment&#39;s ReplicaSet controller finds &quot;its&quot; pods<br>  strategy:<br>    type: RollingUpdate<br>    # the default and correct choice - replaces old pods with new ones<br>    # gradually, never going to zero capacity (contrast with type: Recreate<br>    # which kills ALL old pods first, causing downtime)<br>    rollingUpdate:<br>      maxSurge: 1<br>      # during an update, K8s can create up to 1 EXTRA pod above the<br>      # desired replica count - so with replicas: 2, you could briefly<br>      # have 3 pods running while the new version rolls out<br>      maxUnavailable: 0<br>      # K8s must NEVER have fewer than the desired replica count Ready<br>      # combined with maxSurge: 1, this means: spin up 1 new pod, wait<br>      # for it to pass readiness, THEN kill 1 old pod, repeat<br>  template:<br>    metadata:<br>      labels:<br>        app: backend<br>        tier: api<br>        version: &quot;1.0.0&quot;<br>      annotations:<br>        prometheus.io/scrape: &quot;true&quot;<br>        prometheus.io/path: &quot;/actuator/prometheus&quot;<br>        prometheus.io/port: &quot;8080&quot;<br>        # annotation-based service discovery hints - some Prometheus<br>        # setups read these directly instead of requiring a ServiceMonitor<br>        # (we use BOTH approaches in this project for completeness)<br>    spec:<br>      serviceAccountName: backend-sa<br>      # without this line, the pod would use the &quot;default&quot; ServiceAccount<br>      # which typically has minimal/no permissions - by specifying backend-sa<br>      # (created in the RBAC section), this pod gets EXACTLY the permissions<br>      # we explicitly granted it, nothing more<br>      securityContext:<br>        runAsNonRoot: true<br>        # this is a HARD ASSERTION to the kubelet: &quot;if this image tries<br>        # to run as root, REFUSE to start the container&quot; - fails fast<br>        # rather than silently running as root<br>        runAsUser: 1000<br>        runAsGroup: 3000<br>        # specific non-root UID/GID - must match what&#39;s set up in the<br>        # Dockerfile (RUN adduser -u 1000 appuser)<br>        fsGroup: 2000<br>        # any volumes mounted into this pod get chgrp&#39;d to GID 2000<br>        seccompProfile:<br>          type: RuntimeDefault<br>          # applies the container runtime&#39;s default seccomp filter,<br>          # blocking a long list of dangerous syscalls - this is one<br>          # of the REQUIRED fields for Pod Security Standard &quot;restricted&quot;<br>      initContainers:<br>        - name: wait-for-mysql<br>          image: busybox:1.36<br>          # busybox is ~1MB - perfect for a throwaway init container<br>          # that just needs basic shell utilities, nothing more<br>          command:<br>            - sh<br>            - -c<br>            - |<br>              until nc -z mysql-service 3306; do<br>                echo &quot;Waiting for MySQL...&quot;; sleep 3;<br>              done<br>              echo &quot;MySQL is ready!&quot;<br>          # nc -z = &quot;netcat, zero I/O mode&quot; - just tests if the port<br>          # is open and accepting connections, doesn&#39;t send any data<br>          # this loop blocks here until MySQL&#39;s port is reachable,<br>          # which only happens once MySQL has fully booted<br>      # ── this entire initContainers block is WHY we don&#39;t see<br>      # CrashLoopBackOff anymore - without it, Spring Boot would try<br>      # to connect to MySQL immediately on pod start, fail because<br>      # MySQL takes ~20-30s to initialize, crash, and K8s would restart<br>      # it in a loop until MySQL happened to be ready by chance<br>      containers:<br>        - name: backend<br>          image: task-app:latest<br>          imagePullPolicy: Never<br>          # CRITICAL for Minikube - tells kubelet &quot;do NOT attempt to<br>          # pull this image from any registry, it MUST already exist<br>          # locally&quot; - this is what makes `minikube docker-env` + local<br>          # `docker build` actually usable. In production this would be<br>          # imagePullPolicy: IfNotPresent pointing at ECR/GCR/Docker Hub<br>          ports:<br>            - name: http<br>              containerPort: 8080<br>          securityContext:<br>            allowPrivilegeEscalation: false<br>            # blocks setuid/setgid binaries from granting MORE privilege<br>            # than the process started with - required for &quot;restricted&quot; PSS<br>            readOnlyRootFilesystem: false<br>            # NOTE: this is false, not true, because Spring Boot writes<br>            # temp files to /tmp during normal operation (file uploads,<br>            # JIT compilation caches, etc) - setting this true would need<br>            # an explicit emptyDir mounted at /tmp first<br>            runAsNonRoot: true<br>            runAsUser: 1000<br>            capabilities:<br>              drop: [ALL]<br>              # Linux capabilities are fine-grained root permissions<br>              # (CAP_NET_ADMIN, CAP_SYS_ADMIN, etc) - dropping ALL of them<br>              # means even if something exploited this container, it<br>              # would have none of root&#39;s special powers available<br>          env:<br>            - name: DB_HOST<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: DB_HOST<br>            - name: DB_PORT<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: DB_PORT<br>            - name: DB_NAME<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: DB_NAME<br>            - name: DB_USER<br>              valueFrom:<br>                secretKeyRef:<br>                  name: taskmanager-secret<br>                  key: DB_USER<br>            - name: DB_PASSWORD<br>              valueFrom:<br>                secretKeyRef:<br>                  name: taskmanager-secret<br>                  key: DB_PASSWORD<br>            # each of these pulls one specific key from ConfigMap or Secret<br>            # and exposes it as a normal environment variable INSIDE the<br>            # container - Spring Boot&#39;s application.properties then reads<br>            # them via ${DB_HOST}, ${DB_PASSWORD} etc<br>            - name: OTEL_EXPORTER_OTLP_ENDPOINT<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: OTEL_EXPORTER_OTLP_ENDPOINT<br>            - name: OTEL_RESOURCE_ATTRIBUTES<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: OTEL_RESOURCE_ATTRIBUTES<br>            # these two configure the OpenTelemetry Java agent automatically<br>            # - once set, Spring Boot&#39;s HTTP and JDBC calls get traced<br>            # with ZERO code changes, the OTel starter dependency does it all<br>            - name: JAVA_OPTS<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: taskmanager-config<br>                  key: JVM_OPTS<br>              # picked up by the container&#39;s ENTRYPOINT script -<br>              # controls heap size, GC algorithm (defined back in the ConfigMap)<br>          resources:<br>            requests:<br>              cpu: &quot;250m&quot;<br>              memory: &quot;512Mi&quot;<br>            limits:<br>              cpu: &quot;1000m&quot;<br>              memory: &quot;1Gi&quot;<br>              # request != limit here, unlike MySQL - this makes backend&#39;s<br>              # QoS class &quot;Burstable&quot;: guaranteed 250m/512Mi minimum, but<br>              # can burst up to 1 full core / 1Gi when the node has headroom<br>              # this is the RIGHT choice for a web API that has spiky load<br>          startupProbe:<br>            httpGet:<br>              path: /actuator/health<br>              port: 8080<br>            failureThreshold: 30<br>            periodSeconds: 10<br>            # 30 × 10s = up to 5 MINUTES allowed for the JVM to fully boot<br>            # WHILE this probe is failing, the liveness probe below is<br>            # COMPLETELY DISABLED - this prevents K8s from killing a pod<br>            # that&#39;s simply taking its time to start, mistaking it for a hang<br>          livenessProbe:<br>            httpGet:<br>              path: /actuator/health/liveness<br>              port: 8080<br>            initialDelaySeconds: 60<br>            periodSeconds: 15<br>            failureThreshold: 3<br>            # only takes effect AFTER startupProbe succeeds once<br>            # checks Spring Boot Actuator&#39;s dedicated liveness endpoint -<br>            # this is deliberately a different, narrower check than readiness<br>            # (it should basically just confirm &quot;the JVM thread is not dead&quot;)<br>          readinessProbe:<br>            httpGet:<br>              path: /actuator/health/readiness<br>              port: 8080<br>            initialDelaySeconds: 30<br>            periodSeconds: 10<br>            failureThreshold: 3<br>            # checks readiness - which Spring Boot Actuator computes by<br>            # asking EVERY registered HealthIndicator, including the<br>            # database connection pool - if MySQL becomes unreachable<br>            # AFTER startup, readiness fails and the Service stops<br>            # routing traffic here, WITHOUT restarting the container<br>      affinity:<br>        nodeAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - weight: 60<br>              preference:<br>                matchExpressions:<br>                  - key: disk<br>                    operator: In<br>                    values: [ssd]<br>              # softer preference than MySQL&#39;s weight: 80 - backend<br>              # benefits from SSD but isn&#39;t as disk-bound as the database<br>        podAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - weight: 50<br>              podAffinityTerm:<br>                labelSelector:<br>                  matchLabels:<br>                    app: frontend<br>                topologyKey: &quot;kubernetes.io/hostname&quot;<br>              # &quot;try to schedule backend pods on the SAME node as frontend<br>              # pods&quot; - when this succeeds, frontend→backend traffic<br>              # travels over the node&#39;s internal bridge instead of crossing<br>              # to another node, shaving off network latency<br>        podAntiAffinity:<br>          requiredDuringSchedulingIgnoredDuringExecution:<br>            - labelSelector:<br>                matchLabels:<br>                  app: backend<br>              topologyKey: &quot;kubernetes.io/hostname&quot;<br>              # THIS IS A HARD RULE (required, not preferred) - the<br>              # scheduler will NEVER place two backend pods on the same<br>              # node. If one node dies, you still have at least one<br>              # backend pod alive on a different node - true HA<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: backend-service<br>  namespace: taskmanager<br>  labels:<br>    app: backend<br>    tier: api<br>spec:<br>  selector:<br>    app: backend<br>  ports:<br>    - name: http<br>      port: 8080<br>      targetPort: 8080<br>  type: ClusterIP<br>  # internal only - backend should NEVER be directly reachable from<br>  # outside the cluster; all external traffic comes through the Ingress<br>  # which then talks to this Service internally<br>---<br>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: backend-hpa<br>  namespace: taskmanager<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: backend<br>    # points at the EXACT Deployment object above by name<br>  minReplicas: 2<br>  maxReplicas: 10<br>  # the Deployment&#39;s own replicas: 2 is just the STARTING point -<br>  # once HPA is active, IT controls the actual replica count between<br>  # these bounds, overriding whatever the Deployment spec says<br>  metrics:<br>    - type: Resource<br>      resource:<br>        name: cpu<br>        target:<br>          type: Utilization<br>          averageUtilization: 70<br>          # HPA&#39;s formula: desiredReplicas = ceil(currentReplicas<br>          #                  × currentCPUUtilization / 70)<br>          # CPU, not memory - memory is a deliberately AVOIDED metric here<br>          # because it&#39;s an incompressible resource that can cause<br>          # runaway scale-up loops (old pods don&#39;t release memory just<br>          # because new pods were added)<br>  behavior:<br>    scaleUp:<br>      stabilizationWindowSeconds: 15<br>      policies:<br>        - type: Pods<br>          value: 2<br>          periodSeconds: 60<br>          # &quot;add at most 2 pods per 60-second window&quot; - fast reaction<br>          # to traffic spikes, but with a sane upper bound on how<br>          # aggressively it scales in one step<br>    scaleDown:<br>      stabilizationWindowSeconds: 300<br>      policies:<br>        - type: Pods<br>          value: 1<br>          periodSeconds: 60<br>          # scaling DOWN is deliberately much slower (5-minute window,<br>          # only 1 pod removed per minute) - this prevents &quot;flapping&quot;<br>          # where load oscillates and HPA thrashes pods up and down<br>---<br>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: backend-pdb<br>  namespace: taskmanager<br>spec:<br>  minAvailable: 1<br>  # with HPA potentially having anywhere from 2-10 backend pods,<br>  # this PDB guarantees that no VOLUNTARY disruption (Cluster Autoscaler<br>  # scale-down, node drain for maintenance) ever takes the backend<br>  # below 1 running pod - combined with podAntiAffinity above, this<br>  # means losing one node never fully takes down the API<br>  selector:<br>    matchLabels:<br>      app: backend<br>---<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br># this is a CUSTOM RESOURCE - it only exists because the Prometheus<br># Operator (installed via the kube-prometheus-stack Helm chart) defines<br># this CRD and watches for objects of this kind<br>metadata:<br>  name: backend-monitor<br>  namespace: taskmanager<br>  labels:<br>    release: prometheus-stack<br>    # THIS LABEL IS NOT DECORATIVE - Prometheus Operator is configured<br>    # (by the Helm chart) to only watch ServiceMonitors with this exact<br>    # label, matching the Helm release name &quot;prometheus-stack&quot; -<br>    # without this label, this ServiceMonitor would be silently ignored<br>spec:<br>  selector:<br>    matchLabels:<br>      app: backend<br>      # finds the backend-service Service object (NOT the pods directly -<br>      # ServiceMonitor watches Services, then discovers pods through<br>      # the Service&#39;s endpoints)<br>  endpoints:<br>    - port: http<br>      # must match the PORT NAME defined in backend-service above,<br>      # not the port number directly<br>      path: /actuator/prometheus<br>      # Spring Boot Actuator&#39;s Prometheus-format metrics endpoint<br>      interval: 15s<br>      # how often Prometheus scrapes this - matches Prometheus&#39;s own<br>      # default global scrape interval<br>  namespaceSelector:<br>    matchNames:<br>      - taskmanager<br>      # restricts this ServiceMonitor to only watch Services inside<br>      # the taskmanager namespace - without this, by default a<br>      # ServiceMonitor only looks in its OWN namespace anyway, but<br>      # being explicit here documents the intent clearly</pre><p><strong>The full request lifecycle through this Deployment, step by step:</strong></p><pre>1. kubectl apply -f backend-deployment.yaml<br>2. Deployment controller creates a ReplicaSet<br>3. ReplicaSet controller creates 2 Pod objects<br>4. Scheduler evaluates affinity/anti-affinity rules, picks nodes<br>5. kubelet on each node pulls the init container, runs wait-for-mysql<br>6. Init container loops until mysql-service:3306 is reachable<br>7. Init container exits 0 — main container starts<br>8. startupProbe begins checking /actuator/health every 10s<br>9. Once startupProbe passes, livenessProbe and readinessProbe activate<br>10. Once readinessProbe passes, the pod&#39;s IP is added to backend-service&#39;s endpoints<br>11. HPA continuously watches CPU usage via the metrics-server<br>12. ServiceMonitor tells Prometheus to scrape /actuator/prometheus every 15s<br>13. Spring Boot&#39;s OTel instrumentation sends trace spans to otel-collector:4317</pre><pre>kubectl apply -f 04-backend/backend-deployment.yaml</pre><pre># Watch the init container sequence happen live<br>kubectl get pods -n taskmanager -w</pre><pre># Confirm HPA is reading metrics correctly<br>kubectl get hpa -n taskmanager</pre><p><em>Continued in Part 3 — Frontend, Networking, and Ingress.</em></p><h3>Part 6: Frontend Deployment — Nginx with mounted config</h3><pre>apiVersion: v1<br>kind: ConfigMap<br># we put the ENTIRE nginx.conf file inside a ConfigMap instead of<br># baking it into the Docker image — this lets us change Nginx&#39;s<br># routing rules with `kubectl edit configmap` + a pod restart,<br># with NO image rebuild required</pre><pre>metadata:<br>  name: nginx-config<br>  namespace: taskmanager<br>  labels:<br>    app: frontend<br>    component: config</pre><pre>data:<br>  default.conf: |<br>    # everything indented under this key becomes the FILE CONTENT<br>    # when this ConfigMap is mounted as a volume — the key name<br>    # &quot;default.conf&quot; becomes the actual filename on disk</pre><pre>    server {<br>        listen 80;<br>        server_name _;<br>        # &quot;_&quot; is a catch-all server_name — matches any Host header,<br>        # since we don&#39;t need virtual host routing inside this container</pre><pre>        root /usr/share/nginx/html;<br>        index index.html;<br>        # standard Nginx static file serving setup — this is where<br>        # the React build output (from `npm run build`) lives inside<br>        # the Docker image</pre><pre>        location / {<br>            try_files $uri $uri/ /index.html;<br>            # THIS LINE IS WHAT MAKES REACT ROUTING WORK<br>            # without it, refreshing the browser on a route like<br>            # /tasks/5 would 404, because that path doesn&#39;t exist as<br>            # an actual file — this directive says &quot;if the exact file<br>            # isn&#39;t found, fall back to index.html and let React&#39;s<br>            # client-side router handle it&quot;<br>        }</pre><pre>        location /api/ {<br>            proxy_pass <a href="http://backend-service:8080;">http://backend-service:8080;</a><br>            # THIS is the key line for avoiding CORS entirely —<br>            # the React app&#39;s JavaScript calls /api/tasks on the SAME<br>            # origin (whatever domain served the page), and Nginx<br>            # transparently forwards that to backend-service internally<br>            # the browser never knows two different services were involved</pre><pre>            proxy_set_header Host $host;<br>            proxy_set_header X-Real-IP $remote_addr;<br>            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>            # these headers preserve the ORIGINAL client info as the<br>            # request passes through Nginx — without them, Spring Boot<br>            # would see every request as coming from the Nginx pod&#39;s<br>            # internal IP, losing the real client&#39;s address entirely</pre><pre>            proxy_connect_timeout 10s;<br>            proxy_read_timeout 60s;<br>            # if the backend doesn&#39;t even accept the connection within 10s,<br>            # or doesn&#39;t finish responding within 60s, Nginx gives up —<br>            # prevents one slow backend call from hanging a client forever<br>        }</pre><pre>        location /health {<br>            return 200 &#39;OK&#39;;<br>            add_header Content-Type text/plain;<br>            # a synthetic endpoint that doesn&#39;t touch any real backend —<br>            # used by THIS pod&#39;s own livenessProbe/readinessProbe below,<br>            # so K8s can check &quot;is Nginx itself alive&quot; without depending<br>            # on the backend being healthy too<br>        }</pre><pre>        add_header X-Frame-Options &quot;SAMEORIGIN&quot;;<br>        # blocks this page from being embedded in an &lt;iframe&gt; on another<br>        # domain — defends against clickjacking attacks</pre><pre>        add_header X-Content-Type-Options &quot;nosniff&quot;;<br>        # stops browsers from &quot;MIME sniffing&quot; a response away from the<br>        # declared Content-Type — defends against certain XSS vectors</pre><pre>        add_header X-XSS-Protection &quot;1; mode=block&quot;;<br>        # legacy header, mostly superseded by Content-Security-Policy,<br>        # but still harmless to include for older browser compatibility</pre><pre>        gzip on;<br>        gzip_types text/plain text/css application/json application/javascript;<br>        # compress these MIME types before sending — meaningfully reduces<br>        # bandwidth for the React JS bundle and CSS on first page load<br>    }</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: frontend<br>  namespace: taskmanager<br>  labels:<br>    app: frontend<br>    tier: web<br>    version: &quot;1.0.0&quot;</pre><pre>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: frontend</pre><pre>  strategy:<br>    type: RollingUpdate<br>    rollingUpdate:<br>      maxSurge: 1<br>      maxUnavailable: 0<br>      # identical reasoning to the backend Deployment — zero-downtime<br>      # deploys for the frontend too</pre><pre>  template:<br>    metadata:<br>      labels:<br>        app: frontend<br>        tier: web</pre><pre>    spec:<br>      securityContext:<br>        runAsNonRoot: true<br>        runAsUser: 101<br>        # UID 101 is the &quot;nginx&quot; user inside the official nginx:alpine image —<br>        # NOT an arbitrary number, it must match what the base image<br>        # actually creates, or the container fails to start</pre><pre>        runAsGroup: 101<br>        seccompProfile:<br>          type: RuntimeDefault</pre><pre>      containers:<br>        - name: frontend<br>          image: taskmanager-frontend:latest<br>          imagePullPolicy: Never<br>          # same Minikube-local-build reasoning as the backend</pre><pre>          ports:<br>            - name: http<br>              containerPort: 80</pre><pre>          securityContext:<br>            allowPrivilegeEscalation: false<br>            capabilities:<br>              drop: [ALL]<br>              add: [NET_BIND_SERVICE]<br>              # THIS IS THE INTERESTING ONE — normally, binding to a<br>              # port below 1024 (like port 80) REQUIRES root privileges<br>              # on Linux. By dropping ALL capabilities but explicitly<br>              # adding back JUST NET_BIND_SERVICE, Nginx can bind to<br>              # port 80 as a non-root user (UID 101) WITHOUT having<br>              # any other root-level power — this is the textbook<br>              # example of least-privilege capability management</pre><pre>          volumeMounts:<br>            - name: nginx-config-vol<br>              mountPath: /etc/nginx/conf.d<br>              readOnly: true<br>              # mounts the ConfigMap&#39;s default.conf file directly into<br>              # Nginx&#39;s standard config-include directory — Nginx reads<br>              # everything in conf.d/ automatically on startup</pre><pre>          resources:<br>            requests:<br>              cpu: &quot;50m&quot;<br>              memory: &quot;64Mi&quot;<br>            limits:<br>              cpu: &quot;200m&quot;<br>              memory: &quot;128Mi&quot;<br>              # Nginx is extremely lightweight compared to the JVM-based<br>              # backend — these limits reflect that reality</pre><pre>          livenessProbe:<br>            httpGet:<br>              path: /health<br>              port: 80<br>            initialDelaySeconds: 10<br>            periodSeconds: 15<br>            failureThreshold: 3</pre><pre>          readinessProbe:<br>            httpGet:<br>              path: /health<br>              port: 80<br>            initialDelaySeconds: 5<br>            periodSeconds: 10<br>            failureThreshold: 3<br>            # both probes hit the synthetic /health location defined<br>            # in the ConfigMap above — this never depends on the backend<br>            # being available, only on Nginx itself being responsive</pre><pre>      volumes:<br>        - name: nginx-config-vol<br>          configMap:<br>            name: nginx-config<br>            # connects the volume name used in volumeMounts above to<br>            # the actual ConfigMap object — each KEY in the ConfigMap&#39;s<br>            # data becomes a FILE inside the mounted directory</pre><pre>      affinity:<br>        podAntiAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - weight: 100<br>              podAffinityTerm:<br>                labelSelector:<br>                  matchLabels:<br>                    app: frontend<br>                topologyKey: &quot;kubernetes.io/hostname&quot;<br>              # NOTE: this is &quot;preferred&quot;, not &quot;required&quot; like the<br>              # backend&#39;s anti-affinity — frontend pods are cheap and<br>              # disposable, so a soft preference for spreading is<br>              # enough; we don&#39;t NEED the hard guarantee here</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: frontend-service<br>  namespace: taskmanager<br>  labels:<br>    app: frontend<br>    tier: web</pre><pre>spec:<br>  selector:<br>    app: frontend<br>  ports:<br>    - name: http<br>      port: 80<br>      targetPort: 80<br>      nodePort: 30080<br>      # nodePort is ONLY used when type: NodePort — this opens port<br>      # 30080 on EVERY node in the cluster</pre><pre>  type: NodePort<br>  # this is a LOCAL DEVELOPMENT choice — in production this would be<br>  # type: ClusterIP, with the Ingress being the only external entry point<br>  # NodePort here exists as a convenient fallback for direct access on<br>  # Minikube without needing the Ingress controller running</pre><p><strong>Why mount nginx.conf as a ConfigMap instead of baking it into the image:</strong> If you bake configuration into a Docker image, changing even one line means rebuilding the image, pushing it to a registry, and rolling out a new Deployment. Mounting it as a ConfigMap volume means kubectl edit configmap nginx-config following kubectl rollout restart deployment/frontend — No Docker involved at all. This is the same pattern used for things like application.properties, logging configs, and feature flags in real production clusters.</p><pre>kubectl apply -f 05-frontend/frontend-deployment.yaml</pre><pre># Verify the ConfigMap is actually mounted correctly<br>kubectl exec -it $(kubectl get pod -l app=frontend \<br>  -n taskmanager -o jsonpath=&#39;{.items[0].metadata.name}&#39;) \<br>  -n taskmanager -- cat /etc/nginx/conf.d/default.conf</pre><h3>Part 7: Ingress and NetworkPolicies — controlling all traffic</h3><p>This file has two very different jobs: routing traffic IN from outside, and locking down which pods can talk to which pods INSIDE.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: taskmanager-ingress<br>  namespace: taskmanager<br>  labels:<br>    app: taskmanager<br>annotations:<br>    nginx.ingress.kubernetes.io/rewrite-target: /<br>    # tells the Nginx Ingress Controller HOW to rewrite the path before<br>    # forwarding - here it rewrites everything to / (mostly a no-op<br>    # for our path-based setup, but a common pattern when path prefixes<br>    # need stripping before reaching the backend)<br>    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;<br>    # explicitly disables forcing HTTP → HTTPS redirect - we have no<br>    # TLS certificate configured yet; flip this to &quot;true&quot; once you<br>    # add cert-manager + a real certificate<br>    nginx.ingress.kubernetes.io/proxy-read-timeout: &quot;60&quot;<br>    nginx.ingress.kubernetes.io/proxy-connect-timeout: &quot;10&quot;<br>    # same timeout reasoning as Nginx&#39;s own config in the frontend -<br>    # but this time at the CLUSTER ENTRY POINT level, protecting against<br>    # slow upstream responses from EITHER service<br>spec:<br>  ingressClassName: nginx<br>  # tells Kubernetes WHICH Ingress Controller should handle this object -<br>  # there could be multiple controllers installed (AWS ALB, Nginx, Traefik)<br>  # this explicitly picks the Nginx one<br>  rules:<br>    - host: taskmanager.local<br>      # ONLY requests with this exact Host header get matched by this rule<br>      # this requires an /etc/hosts entry pointing taskmanager.local<br>      # at the Minikube IP - in production this would be a real domain<br>      http:<br>        paths:<br>          - path: /api<br>            pathType: Prefix<br>            # Prefix means /api ALSO matches /api/tasks, /api/tasks/5, etc<br>            # (contrast with pathType: Exact, which would ONLY match /api<br>            # literally and nothing else)<br>            backend:<br>              service:<br>                name: backend-service<br>                port:<br>                  number: 8080<br>          - path: /<br>            pathType: Prefix<br>            # this is the CATCH-ALL - matches everything not already<br>            # matched by the more specific /api rule above<br>            # Kubernetes Ingress evaluates rules by longest-prefix-match,<br>            # so /api correctly takes priority over this broader /<br>            backend:<br>              service:<br>                name: frontend-service<br>                port:<br>                  number: 80<br>---<br># ════════════════════════════════════════════════════════════<br># NETWORK POLICIES - read these in ORDER, they build on each other<br># ════════════════════════════════════════════════════════════<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: deny-all<br>  namespace: taskmanager<br>spec:<br>  podSelector: {}<br>  # an EMPTY selector matches EVERY pod in this namespace - there is<br>  # no pod this policy doesn&#39;t apply to<br>  policyTypes:<br>    - Ingress<br>    - Egress<br>    # by declaring these policyTypes with NO corresponding ingress: or<br>    # egress: rules below, the effective behavior is &quot;block ALL traffic<br>    # in both directions for every pod&quot; - this is the &quot;default deny&quot;<br>    # baseline that every subsequent policy then carves exceptions into<br>---<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-ingress-to-frontend<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: frontend<br>    # this policy&#39;s rules apply specifically TO pods labeled app=frontend<br>  policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        - namespaceSelector:<br>            matchLabels:<br>              kubernetes.io/metadata.name: ingress-nginx<br>              # only allow incoming traffic if it originates from a pod<br>              # in the ingress-nginx namespace - this is where the<br>              # Nginx Ingress Controller itself runs<br>      ports:<br>        - port: 80<br>          # and ONLY on port 80 - nothing else gets through to frontend pods<br>---<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-frontend-to-backend<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: backend<br>    # this policy protects backend pods<br>  policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        - podSelector:<br>            matchLabels:<br>              app: frontend<br>              # ONLY pods labeled app=frontend can send traffic to backend -<br>              # not the Ingress controller directly, not any random pod,<br>              # ONLY frontend<br>      ports:<br>        - port: 8080<br>---<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-backend-to-mysql<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: mysql<br>    # this policy protects the MySQL pod - the most sensitive part<br>    # of the whole system<br>  policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        - podSelector:<br>            matchLabels:<br>              app: backend<br>              # ONLY backend pods can reach MySQL - frontend CANNOT<br>              # talk to MySQL directly even if it tried (which it never<br>              # should, but defense in depth means not relying on<br>              # &quot;the app would never do that&quot;)<br>      ports:<br>        - port: 3306<br>---<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-backend-egress<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: backend<br>    # this is the FIRST egress (outgoing) policy in this file -<br>    # everything above controlled incoming traffic; this controls<br>    # what backend pods are allowed to INITIATE connections to<br>  policyTypes:<br>    - Egress<br>  egress:<br>    - to:<br>        - podSelector:<br>            matchLabels:<br>              app: mysql<br>      ports:<br>        - protocol: TCP<br>          port: 3306<br>          # backend can reach mysql on 3306 - the egress-side mirror<br>          # of the ingress rule defined two policies above<br>    - to:<br>        - namespaceSelector:<br>            matchLabels:<br>              kubernetes.io/metadata.name: kube-system<br>      ports:<br>        - protocol: UDP<br>          port: 53<br>          # ═══ THE MOST COMMONLY FORGOTTEN RULE IN ALL OF K8S ═══<br>          # CoreDNS lives in kube-system and listens on UDP port 53<br>          # WITHOUT this rule, backend pods could not resolve<br>          # &quot;mysql-service&quot; to an IP address AT ALL - every single<br>          # DNS lookup would simply time out, and you&#39;d see connection<br>          # failures that look like a networking problem but are<br>          # actually a missing DNS egress rule<br>    - to:<br>        - podSelector:<br>            matchLabels:<br>              app: otel-collector<br>      ports:<br>        - protocol: TCP<br>          port: 4317<br>          # allows backend to send trace spans to the OTel Collector -<br>          # without this rule, distributed tracing would silently stop<br>          # working even though the application code is completely fine<br>---<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-prometheus-scrape<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: backend<br>    # back to an INGRESS rule - this one allows something INTO backend<br>  policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        - namespaceSelector:<br>            matchLabels:<br>              kubernetes.io/metadata.name: monitoring<br>              # Prometheus runs in the &quot;monitoring&quot; namespace (created<br>              # by the kube-prometheus-stack Helm install) - without<br>              # this explicit allow, Prometheus&#39;s scrape requests to<br>              # /actuator/prometheus would be silently dropped by the<br>              # deny-all baseline, and Grafana dashboards would show<br>              # no data with no obvious error message anywhere<br>      ports:<br>        - port: 8080</pre><p><strong>The complete traffic map this set of policies creates:</strong></p><pre>ingress-nginx namespace → frontend (port 80) ✅<br>frontend → backend (port 8080) ✅<br>backend → mysql (port 3306) ✅<br>backend → kube-system (port 53 UDP, DNS) ✅<br>backend → otel-collector (port 4317) ✅<br>monitoring namespace → backend (port 8080) ✅<br>frontend → mysql ❌ BLOCKED<br>any random pod → backend ❌ BLOCKED<br>any random pod → mysql ❌ BLOCKED<br>backend → internet (anything not explicitly listed) ❌ BLOCKED</pre><pre>kubectl apply -f 06-networking/ingress-networkpolicy.yaml<br># Verify the policies are active<br>kubectl get networkpolicies -n taskmanager<br># Prove frontend CANNOT reach mysql directly (should time out)<br>kubectl exec -it &lt;frontend-pod&gt; -n taskmanager -- nc -zv mysql-service 3306<br># Prove backend CAN reach mysql (should succeed)<br>kubectl exec -it &lt;backend-pod&gt; -n taskmanager -- nc -zv mysql-service 3306</pre><p><em>Continued in Part 4 — Scaling, Security/RBAC, and the full Observability stack.</em></p><h3>Part 8: VPA, ResourceQuota, and LimitRange — namespace-level scaling controls</h3><pre>apiVersion: autoscaling.k8s.io/v1<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: backend-vpa<br>  namespace: taskmanager</pre><pre>spec:<br>  targetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: backend<br>    # points at the same backend Deployment that HPA also watches —<br>    # this is deliberately safe because of the controlledResources<br>    # setting further down</pre><pre>  updatePolicy:<br>    updateMode: &quot;Off&quot;<br>    # &quot;Off&quot; means VPA calculates and SHOWS recommendations via<br>    # `kubectl describe vpa`, but NEVER actually touches running pods<br>    # this is the safest starting mode — you observe for days/weeks,<br>    # then apply the suggested values yourself during a normal rollout</pre><pre>  resourcePolicy:<br>    containerPolicies:<br>      - containerName: backend<br>        # VPA can have different policies per container if a pod has<br>        # multiple containers — here we only care about &quot;backend&quot;</pre><pre>        minAllowed:<br>          cpu: &quot;100m&quot;<br>          memory: &quot;256Mi&quot;<br>          # even if VPA&#39;s math suggests going lower, never recommend<br>          # below these floors — protects against a recommendation<br>          # that&#39;s technically &quot;accurate&quot; for current load but too<br>          # risky for unexpected spikes</pre><pre>        maxAllowed:<br>          cpu: &quot;2000m&quot;<br>          memory: &quot;2Gi&quot;<br>          # and never recommend ABOVE these ceilings either — caps<br>          # how big a single pod could ever get recommended to be</pre><pre>        controlledResources:<br>          - memory<br>          # ═══ THIS IS THE CRITICAL LINE ═══<br>          # by listing ONLY &quot;memory&quot; here, VPA is explicitly forbidden<br>          # from ever touching CPU requests/limits — CPU remains<br>          # ENTIRELY the responsibility of HPA above<br>          # if both VPA and HPA tried to control CPU simultaneously,<br>          # they would fight each other: HPA scales pod COUNT based on<br>          # CPU%, while VPA would be changing the CPU REQUEST that<br>          # percentage is calculated against — a moving target that<br>          # never stabilizes</pre><pre>        controlledValues: RequestsOnly<br>        # VPA will only ever recommend changes to &quot;requests&quot;, never<br>        # to &quot;limits&quot; — keeps the limit/request ratio (Burstable QoS<br>        # headroom) intentionally set by humans, not auto-adjusted</pre><pre>---<br>apiVersion: v1<br>kind: ResourceQuota<br>metadata:<br>  name: taskmanager-quota<br>  namespace: taskmanager</pre><pre>spec:<br>  hard:<br>    requests.cpu: &quot;2&quot;<br>    requests.memory: &quot;4Gi&quot;<br>    limits.cpu: &quot;8&quot;<br>    limits.memory: &quot;8Gi&quot;<br>    # these four lines cap the SUM of all requests/limits across<br>    # EVERY pod in this namespace combined — if backend scales to<br>    # 10 replicas via HPA and that pushes total requested CPU above<br>    # &quot;2&quot; cores, new pods would fail to schedule with a quota<br>    # exceeded error, even if the cluster itself has free capacity<br>    # this is how platform teams prevent one team&#39;s namespace from<br>    # silently consuming an entire shared cluster</pre><pre>    pods: &quot;20&quot;<br>    services: &quot;10&quot;<br>    persistentvolumeclaims: &quot;5&quot;<br>    secrets: &quot;20&quot;<br>    configmaps: &quot;20&quot;<br>    # object COUNT limits, separate from compute limits — prevents,<br>    # for example, someone accidentally running a script in a loop<br>    # that creates hundreds of ConfigMaps</pre><pre>---<br>apiVersion: v1<br>kind: LimitRange<br>metadata:<br>  name: taskmanager-limits<br>  namespace: taskmanager</pre><pre>spec:<br>  limits:<br>    - type: Container<br>      default:<br>        cpu: &quot;500m&quot;<br>        memory: &quot;512Mi&quot;<br>        # if ANY pod is created in this namespace WITHOUT specifying<br>        # resources.limits, Kubernetes silently injects these values<br>        # for it — this is a safety net against the &quot;BestEffort QoS&quot;<br>        # mistake of forgetting limits entirely</pre><pre>      defaultRequest:<br>        cpu: &quot;100m&quot;<br>        memory: &quot;128Mi&quot;<br>        # same idea, but for requests.* when they&#39;re missing</pre><pre>      max:<br>        cpu: &quot;2000m&quot;<br>        memory: &quot;2Gi&quot;<br>        # no container in this namespace can EVER specify a limit<br>        # higher than this, even if someone explicitly tries to —<br>        # `kubectl apply` would be REJECTED at admission time</pre><pre>      min:<br>        cpu: &quot;50m&quot;<br>        memory: &quot;64Mi&quot;<br>        # and no container can specify a limit lower than this either —<br>        # prevents someone from setting cpu: &quot;1m&quot; by mistake and<br>        # creating a pod that&#39;s functionally unusable</pre><p><strong>The relationship between these three objects and HPA/PDB from earlier:</strong></p><pre>LimitRange    → sets DEFAULTS and BOUNDS for individual containers<br>ResourceQuota → sets a CEILING for the whole namespace combined<br>VPA           → RECOMMENDS the right values based on real usage (memory only)<br>HPA           → controls pod COUNT based on real usage (CPU only)<br>PDB           → ensures minimum pods survive voluntary disruption</pre><pre>kubectl apply -f 07-scaling/vpa-quota.yaml</pre><pre># After a few hours of traffic, check VPA&#39;s recommendation<br>kubectl describe vpa backend-vpa -n taskmanager</pre><pre># Check current quota usage vs limit<br>kubectl describe resourcequota taskmanager-quota -n taskmanager</pre><h3>Part 9: RBAC — ServiceAccounts, Roles, and RoleBindings</h3><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: backend-sa<br>  namespace: taskmanager<br>  labels:<br>    app: backend</pre><pre>automountServiceAccountToken: false<br># by default, Kubernetes automatically mounts a token for the<br># ServiceAccount into every pod, even if the pod never calls the<br># K8s API. Setting this to false means: don&#39;t bother, unless a<br># specific pod spec explicitly opts back in — reduces the attack<br># surface for pods that have no business calling the API at all</pre><pre>---<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: frontend-sa<br>  namespace: taskmanager<br>  labels:<br>    app: frontend</pre><pre>automountServiceAccountToken: false<br># frontend (Nginx) NEVER needs to call the Kubernetes API — this<br># ServiceAccount exists mainly so it has its OWN identity distinct<br># from &quot;default&quot;, for clean RBAC auditing, even though it has no<br># Role bound to it at all in this setup</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: backend-role<br>  namespace: taskmanager</pre><pre>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;]<br>    resourceNames: [&quot;taskmanager-config&quot;]<br>    # NOTICE: verbs only has &quot;get&quot;, not &quot;list&quot; — and resourceNames<br>    # restricts this to ONE SPECIFIC ConfigMap by name, not all<br>    # ConfigMaps in the namespace. Even if backend-sa&#39;s token were<br>    # somehow stolen, the attacker could only read this ONE ConfigMap,<br>    # and could not even discover what OTHER ConfigMaps exist (no &quot;list&quot;)</pre><pre>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;secrets&quot;]<br>    verbs: [&quot;get&quot;]<br>    resourceNames: [&quot;taskmanager-secret&quot;]<br>    # identical reasoning for the Secret — this is about as narrow<br>    # as RBAC permissions get while still being useful</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: developer-role<br>  namespace: taskmanager</pre><pre>rules:<br>  - apiGroups: [&quot;&quot;, &quot;apps&quot;]<br>    resources: [&quot;pods&quot;, &quot;deployments&quot;, &quot;services&quot;, &quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;]<br>    # NOTICE: no &quot;delete&quot; verb anywhere in this Role — developers<br>    # using this Role can deploy new versions, scale things up,<br>    # tweak configs, but CANNOT delete a Deployment or Service<br>    # outright — a guardrail against an accidental<br>    # `kubectl delete deployment backend` during a busy day</pre><pre>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods/log&quot;, &quot;pods/exec&quot;]<br>    verbs: [&quot;get&quot;, &quot;create&quot;]<br>    # pods/log and pods/exec are SUBRESOURCES, treated as separate<br>    # permissions from &quot;pods&quot; itself — this grants the ability to<br>    # run `kubectl logs` and `kubectl exec` for debugging</pre><pre>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;secrets&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;]<br>    # developers CAN read secrets (useful for debugging config<br>    # issues) but cannot create/modify/delete them</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: readonly-role<br>  namespace: taskmanager</pre><pre>rules:<br>  - apiGroups: [&quot;&quot;, &quot;apps&quot;, &quot;autoscaling&quot;]<br>    resources:<br>      - pods<br>      - deployments<br>      - services<br>      - configmaps<br>      - horizontalpodautoscalers<br>      - replicasets<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>    # ONLY read verbs across the board — no create, update, patch,<br>    # or delete anywhere. This Role is for people who need VISIBILITY<br>    # (monitoring teams, auditors, on-call engineers checking status)<br>    # but should never be able to change anything in this namespace</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: backend-binding<br>  namespace: taskmanager</pre><pre>subjects:<br>  - kind: ServiceAccount<br>    name: backend-sa<br>    namespace: taskmanager<br>    # the SUBJECT — &quot;who&quot; is getting permissions — here it&#39;s a pod<br>    # identity, not a human user</pre><pre>roleRef:<br>  kind: Role<br>  name: backend-role<br>  apiGroup: rbac.authorization.k8s.io<br>  # the ROLE being granted — connects backend-sa to backend-role&#39;s<br>  # exact two narrow permissions defined above</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: developer-binding<br>  namespace: taskmanager</pre><pre>subjects:<br>  - kind: Group<br>    name: developers<br>    apiGroup: rbac.authorization.k8s.io<br>    # &quot;Group&quot; subjects come from YOUR authentication system, not<br>    # Kubernetes itself — if your cluster uses OIDC (e.g. integrated<br>    # with Google Workspace or Okta), &quot;developers&quot; would be a group<br>    # claim embedded in the user&#39;s auth token</pre><pre>roleRef:<br>  kind: Role<br>  name: developer-role<br>  apiGroup: rbac.authorization.k8s.io</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: monitoring-binding<br>  namespace: taskmanager</pre><pre>subjects:<br>  - kind: Group<br>    name: monitoring-team<br>    apiGroup: rbac.authorization.k8s.io</pre><pre>roleRef:<br>  kind: Role<br>  name: readonly-role<br>  apiGroup: rbac.authorization.k8s.io</pre><p><strong>How to attach a ServiceAccount to a pod (already done in Part 5):</strong> Notice the backend Deployment’s pod spec includes serviceAccountName: backend-sa. That single line is what connects everything in this file to the actual running backend pods — without it, pods default to the namespace&#39;s &quot;default&quot; ServiceAccount, which has no Role bound to it here at all.</p><pre>kubectl apply -f 08-security/rbac.yaml</pre><pre># Verify backend-sa can read its specific ConfigMap<br>kubectl auth can-i get configmaps/taskmanager-config \<br>  --as=system:serviceaccount:taskmanager:backend-sa -n taskmanager<br># yes</pre><pre># Verify it CANNOT list all configmaps (only get this one)<br>kubectl auth can-i list configmaps \<br>  --as=system:serviceaccount:taskmanager:backend-sa -n taskmanager<br># no</pre><h3>Part 10: Fluentd DaemonSet — cluster-wide log collection</h3><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: fluentd-sa<br>  namespace: logging<br>  # Fluentd lives in its OWN &quot;logging&quot; namespace, separate from<br>  # &quot;taskmanager&quot; — this is a cluster-wide infrastructure service,<br>  # not specific to one application</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br># ClusterRole, not Role — Fluentd needs to read pod metadata from<br># EVERY namespace in the cluster, not just &quot;logging&quot; or &quot;taskmanager&quot;</pre><pre>metadata:<br>  name: fluentd-role<br>  # no &quot;namespace:&quot; field — ClusterRoles are cluster-scoped objects</pre><pre>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;namespaces&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>    # Fluentd uses this to ENRICH raw log lines with context — when it<br>    # tails a log file, the filename only contains a pod&#39;s UID, not<br>    # its human-readable name or labels. This RBAC permission lets the<br>    # kubernetes_metadata filter (configured below) call the K8s API<br>    # and attach pod_name, namespace_name, and labels to every log entry</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: fluentd-binding<br>  # also cluster-scoped, no namespace field</pre><pre>roleRef:<br>  kind: ClusterRole<br>  name: fluentd-role<br>  apiGroup: rbac.authorization.k8s.io</pre><pre>subjects:<br>  - kind: ServiceAccount<br>    name: fluentd-sa<br>    namespace: logging<br>    # the ServiceAccount lives in &quot;logging&quot; namespace, but the<br>    # ClusterRoleBinding grants it permissions across THE ENTIRE CLUSTER</pre><pre>---<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: fluentd-config<br>  namespace: logging</pre><pre>data:<br>  fluent.conf: |<br>    &lt;source&gt;<br>      @type tail<br>      path /var/log/containers/*.log<br>      # THIS PATH is where every container runtime on a K8s node<br>      # symlinks container stdout/stderr — Fluentd doesn&#39;t talk to<br>      # any container directly, it simply tails files on the NODE&#39;s<br>      # own filesystem (which is why this needs hostPath volumes below)</pre><pre>      pos_file /var/log/fluentd-containers.log.pos<br>      # a &quot;position file&quot; — tracks exactly how far into each log file<br>      # Fluentd has already read, so if the Fluentd pod restarts, it<br>      # resumes from where it left off instead of re-sending everything<br>      # or losing logs written during the restart</pre><pre>      tag kubernetes.*<br>      # every log line gets tagged with this prefix — used later by<br>      # the &lt;filter&gt; and &lt;match&gt; blocks to select which logs they apply to</pre><pre>      read_from_head true<br>      # on FIRST start, read the entire existing file from the beginning,<br>      # not just new lines appended after Fluentd starts watching</pre><pre>      &lt;parse&gt;<br>        @type json<br>        time_format %Y-%m-%dT%H:%M:%S.%NZ<br>        # container runtimes write logs in a JSON wrapper format —<br>        # this tells Fluentd how to parse that JSON envelope and<br>        # extract the actual timestamp format used inside it<br>      &lt;/parse&gt;<br>    &lt;/source&gt;</pre><pre>    &lt;filter kubernetes.**&gt;<br>      @type kubernetes_metadata<br>      @id filter_kube_metadata<br>      # this is the filter that USES the ClusterRole permissions defined<br>      # above — for each log line, it looks up the originating pod via<br>      # the K8s API and attaches namespace_name, pod_name, labels, etc<br>      # as structured fields on the log record<br>    &lt;/filter&gt;</pre><pre>    &lt;filter kubernetes.**&gt;<br>      @type record_transformer<br>      &lt;record&gt;<br>        cluster_name &quot;minikube&quot;<br>        log_source &quot;kubernetes&quot;<br>        # manually injects two static fields onto every single log<br>        # record passing through — useful when you have MULTIPLE<br>        # clusters all shipping to the same Elasticsearch, so you can<br>        # filter by cluster_name in Kibana later<br>      &lt;/record&gt;<br>    &lt;/filter&gt;</pre><pre>    &lt;match kubernetes.**&gt;<br>      @type elasticsearch<br>      host elasticsearch.logging.svc.cluster.local<br>      port 9200<br>      # Service DNS name again — Fluentd reaches Elasticsearch the<br>      # exact same way any other pod would, through CoreDNS resolution</pre><pre>      logstash_format true<br>      logstash_prefix logstash<br>      logstash_dateformat %Y.%m.%d<br>      # creates a NEW Elasticsearch index every day, named like<br>      # &quot;logstash-2026.05.10&quot; — this is the long-established<br>      # &quot;logstash format&quot; convention, making old indices easy to<br>      # delete on a retention schedule (delete indices older than 30 days)</pre><pre>      flush_interval 5s<br>      # batches log lines for 5 seconds before actually sending them<br>      # to Elasticsearch — trades a small amount of latency for much<br>      # better throughput than sending every single line individually<br>    &lt;/match&gt;</pre><pre>---<br>apiVersion: apps/v1<br>kind: DaemonSet<br># DaemonSet, not Deployment — we want EXACTLY ONE Fluentd pod running<br># on EVERY node, automatically, including nodes added later by Cluster<br># Autoscaler. A Deployment has no concept of &quot;one per node&quot;</pre><pre>metadata:<br>  name: fluentd<br>  namespace: logging<br>  labels:<br>    app: fluentd</pre><pre>spec:<br>  selector:<br>    matchLabels:<br>      app: fluentd</pre><pre>  template:<br>    metadata:<br>      labels:<br>        app: fluentd</pre><pre>    spec:<br>      serviceAccountName: fluentd-sa<br>      # connects this pod to the ClusterRole permissions defined above</pre><pre>      tolerations:<br>        - key: node-role.kubernetes.io/master<br>          effect: NoSchedule<br>          # by default, K8s &quot;taints&quot; control-plane/master nodes so<br>          # normal pods don&#39;t get scheduled there — this toleration<br>          # says &quot;Fluentd is okay running on the master node too&quot;,<br>          # which matters because we want logs from EVERY node,<br>          # including the control plane, not just worker nodes</pre><pre>      containers:<br>        - name: fluentd<br>          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch<br>          # this specific image tag already bundles the Elasticsearch<br>          # output plugin — using the generic fluentd image would<br>          # require manually installing fluent-plugin-elasticsearch</pre><pre>          env:<br>            - name: FLUENT_ELASTICSEARCH_HOST<br>              value: &quot;elasticsearch.logging.svc.cluster.local&quot;<br>            - name: FLUENT_ELASTICSEARCH_PORT<br>              value: &quot;9200&quot;<br>            - name: FLUENT_ELASTICSEARCH_SCHEME<br>              value: &quot;http&quot;<br>              # these three env vars are read by the base image&#39;s<br>              # built-in config template — technically redundant with<br>              # our custom fluent.conf ConfigMap, but kept for<br>              # compatibility if the entrypoint script references them</pre><pre>          resources:<br>            requests:<br>              cpu: &quot;100m&quot;<br>              memory: &quot;200Mi&quot;<br>            limits:<br>              cpu: &quot;200m&quot;<br>              memory: &quot;500Mi&quot;<br>              # deliberately small — Fluentd&#39;s job is to move bytes<br>              # around, not transform them heavily, so it shouldn&#39;t<br>              # need much CPU even under decent log volume</pre><pre>          volumeMounts:<br>            - name: varlog<br>              mountPath: /var/log<br>              # mounts the NODE&#39;s actual /var/log directory into this<br>              # container — this is THE source of all log data</pre><pre>            - name: varlibdockercontainers<br>              mountPath: /var/lib/docker/containers<br>              readOnly: true<br>              # some container runtimes store additional log metadata<br>              # here that the symlinks in /var/log/containers point to</pre><pre>            - name: fluentd-config<br>              mountPath: /fluentd/etc/<br>              # mounts our custom fluent.conf ConfigMap into the path<br>              # where this image&#39;s entrypoint script expects to find it</pre><pre>      volumes:<br>        - name: varlog<br>          hostPath:<br>            path: /var/log<br>            # hostPath = mount a path from the NODE&#39;s own filesystem,<br>            # not a Kubernetes-managed volume — this is REQUIRED for<br>            # a DaemonSet whose entire job is reading node-local files<br>            # (this is one of the few legitimate, necessary uses of<br>            # hostPath — contrast with PSS blocking it for app workloads)</pre><pre>        - name: varlibdockercontainers<br>          hostPath:<br>            path: /var/lib/docker/containers</pre><pre>        - name: fluentd-config<br>          configMap:<br>            name: fluentd-config</pre><pre>kubectl create namespace logging<br>kubectl apply -f 09-observability/fluentd-daemonset.yaml</pre><pre># Confirm exactly one Fluentd pod per node<br>kubectl get pods -n logging -o wide</pre><h3>Part 11: Prometheus alerting rules — turning metrics into pages</h3><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: taskmanager-monitor<br>  namespace: taskmanager<br>  labels:<br>    release: prometheus-stack</pre><pre>spec:<br>  selector:<br>    matchLabels:<br>      app: backend<br>  endpoints:<br>    - port: http<br>      path: /actuator/prometheus<br>      interval: 15s<br>  namespaceSelector:<br>    matchNames:<br>      - taskmanager<br>  # identical structure to the ServiceMonitor already shown in Part 5 —<br>  # included again here as a STANDALONE file for clarity in the<br>  # monitoring-specific folder, in case you want to manage observability<br>  # configs separately from the application&#39;s own deployment file</pre><pre>---<br>apiVersion: monitoring.coreos.com/v1<br>kind: PrometheusRule<br># another Custom Resource, defined by the Prometheus Operator —<br># this is how you define alerting logic as YAML instead of clicking<br># through a UI</pre><pre>metadata:<br>  name: taskmanager-alerts<br>  namespace: taskmanager<br>  labels:<br>    release: prometheus-stack<br>    # same label-matching requirement as ServiceMonitor — without it,<br>    # Prometheus Operator ignores this PrometheusRule entirely</pre><pre>spec:<br>  groups:<br>    - name: taskmanager.rules<br>      # a &quot;group&quot; is just an organizational container for related rules —<br>      # all rules in a group are evaluated together on the same interval</pre><pre>      rules:<br>        - alert: HighErrorRate<br>          expr: |<br>            rate(http_server_requests_seconds_count{<br>              application=&quot;taskmanager&quot;,status=~&quot;5..&quot;<br>            }[5m])<br>            /<br>            rate(http_server_requests_seconds_count{<br>              application=&quot;taskmanager&quot;<br>            }[5m])<br>            &gt; 0.05<br>          # this is raw PromQL — the numerator counts 5xx responses<br>          # per second (rate over a 5-minute window), the denominator<br>          # counts ALL responses per second, the division gives a<br>          # ratio, and &gt; 0.05 means &quot;fire if more than 5% of requests<br>          # are failing&quot;</pre><pre>          for: 2m<br>          # the expression must be continuously TRUE for 2 full minutes<br>          # before Prometheus actually fires the alert — this prevents<br>          # a single brief blip from paging someone at 3am</pre><pre>          labels:<br>            severity: warning<br>            team: backend<br>            # these labels are NOT decorative — Alertmanager&#39;s routing<br>            # rules (configured separately) use exactly these labels<br>            # to decide WHERE to send the notification (which Slack<br>            # channel, which team&#39;s on-call rotation, etc)</pre><pre>          annotations:<br>            summary: &quot;High error rate in TaskManager&quot;<br>            description: &quot;Error rate is {{ $value | humanizePercentage }}&quot;<br>            # annotations support Go templating — {{ $value }} is<br>            # automatically substituted with the actual computed value<br>            # of the expr at alert time, so the Slack message shows<br>            # something like &quot;Error rate is 12.4%&quot; instead of a raw number</pre><pre>        - alert: BackendPodDown<br>          expr: |<br>            kube_deployment_status_replicas_available{<br>              namespace=&quot;taskmanager&quot;,<br>              deployment=&quot;backend&quot;<br>            } &lt; 1<br>          # this metric comes from kube-state-metrics (installed as<br>          # part of the kube-prometheus-stack), NOT from our own app —<br>          # it directly reflects the Deployment object&#39;s own status field</pre><pre>          for: 1m<br>          labels:<br>            severity: critical<br>            team: backend<br>          annotations:<br>            summary: &quot;TaskManager backend has no pods running&quot;</pre><pre>        - alert: HighJVMHeap<br>          expr: |<br>            jvm_memory_used_bytes{<br>              application=&quot;taskmanager&quot;,area=&quot;heap&quot;<br>            }<br>            /<br>            jvm_memory_max_bytes{<br>              application=&quot;taskmanager&quot;,area=&quot;heap&quot;<br>            }<br>            &gt; 0.85<br>          # these jvm_memory_* metrics come from Micrometer, exposed<br>          # automatically by Spring Boot Actuator — no custom code needed</pre><pre>          for: 5m<br>          labels:<br>            severity: warning<br>            team: backend<br>          annotations:<br>            summary: &quot;JVM heap usage above 85%&quot;<br>            description: &quot;Heap at {{ $value | humanizePercentage }}&quot;</pre><pre>        - alert: SlowResponseTime<br>          expr: |<br>            histogram_quantile(0.95,<br>              rate(http_server_requests_seconds_bucket{<br>                application=&quot;taskmanager&quot;<br>              }[5m])<br>            ) &gt; 2<br>          # histogram_quantile calculates the 95th percentile from<br>          # Micrometer&#39;s histogram buckets — meaning &quot;95% of requests<br>          # are faster than this value&quot; — if even THAT number exceeds<br>          # 2 seconds, something is genuinely slow for almost everyone,<br>          # not just one outlier request</pre><pre>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;P95 response time above 2 seconds&quot;</pre><pre>        - alert: DBConnectionPoolSaturated<br>          expr: |<br>            hikaricp_connections_active{application=&quot;taskmanager&quot;}<br>            /<br>            hikaricp_connections_max{application=&quot;taskmanager&quot;}<br>            &gt; 0.8<br>          # HikariCP is Spring Boot&#39;s default JDBC connection pool —<br>          # these metrics are also auto-exposed via Micrometer<br>          # if active/max exceeds 80%, the app is close to running<br>          # out of database connections, which would start queueing<br>          # or rejecting requests entirely</pre><pre>          for: 2m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;DB connection pool 80% utilised&quot;</pre><pre>        - alert: MySQLDown<br>          expr: |<br>            kube_statefulset_status_replicas_ready{<br>              namespace=&quot;taskmanager&quot;,<br>              statefulset=&quot;mysql&quot;<br>            } &lt; 1<br>          # again from kube-state-metrics, this time reflecting the<br>          # StatefulSet&#39;s own readiness status rather than the<br>          # Deployment&#39;s</pre><pre>          for: 1m<br>          labels:<br>            severity: critical<br>          annotations:<br>            summary: &quot;MySQL database is down&quot;</pre><pre>kubectl apply -f 10-monitoring/prometheus-rules.yaml</pre><pre># Verify rules loaded<br># Open Prometheus UI → Alerts tab → should list taskmanager.rules group<br>kubectl port-forward svc/prometheus-stack-kube-prom-prometheus 9090:9090 -n monitoring</pre><h3>Part 12: OpenTelemetry Collector and Jaeger — distributed tracing</h3><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: otel-collector-config<br>  namespace: taskmanager</pre><pre>data:<br>  config.yaml: |<br>    receivers:<br>      otlp:<br>        protocols:<br>          grpc:<br>            endpoint: 0.0.0.0:4317<br>            # this is the exact endpoint Spring Boot&#39;s<br>            # OTEL_EXPORTER_OTLP_ENDPOINT config (set back in Part 2)<br>            # sends trace data TO — 0.0.0.0 means &quot;listen on all<br>            # network interfaces inside this container&quot;<br>          http:<br>            endpoint: 0.0.0.0:4318<br>            # an alternative HTTP-based protocol for clients that<br>            # can&#39;t use gRPC — most modern SDKs default to gRPC though</pre><pre>    processors:<br>      batch:<br>        timeout: 5s<br>        send_batch_size: 512<br>        # buffers up to 512 spans OR 5 seconds (whichever comes first)<br>        # before forwarding them onward — much more efficient than<br>        # sending every single span as a separate network call</pre><pre>      resource:<br>        attributes:<br>          - key: k8s.cluster.name<br>            value: &quot;minikube&quot;<br>            action: insert<br>            # &quot;insert&quot; means: add this attribute, but don&#39;t overwrite<br>            # it if a span already has a k8s.cluster.name value set —<br>            # this tags EVERY span passing through this Collector with<br>            # which cluster it came from, useful when multiple clusters<br>            # feed the same Jaeger backend</pre><pre>    exporters:<br>      jaeger:<br>        endpoint: jaeger-collector:14250<br>        tls:<br>          insecure: true<br>          # disables TLS verification for this internal, same-cluster<br>          # connection — fine for learning, but a real production setup<br>          # would use mTLS between the Collector and Jaeger</pre><pre>      logging:<br>        loglevel: warn<br>        # ALSO logs spans to the Collector&#39;s own stdout — useful for<br>        # debugging &quot;is anything even arriving here&quot; without needing<br>        # to open the Jaeger UI</pre><pre>    service:<br>      pipelines:<br>        traces:<br>          receivers: [otlp]<br>          processors: [batch, resource]<br>          exporters: [jaeger]<br>          # this is the actual DATA FLOW definition: spans come IN via<br>          # otlp, get processed by batch then resource (in that order),<br>          # then get sent OUT to jaeger — OTel Collector pipelines are<br>          # explicitly declared like this, nothing is implicit</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: otel-collector<br>  namespace: taskmanager<br>  labels:<br>    app: otel-collector</pre><pre>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: otel-collector</pre><pre>  template:<br>    metadata:<br>      labels:<br>        app: otel-collector</pre><pre>    spec:<br>      containers:<br>        - name: otel-collector<br>          image: otel/opentelemetry-collector-contrib:0.91.0<br>          # the &quot;contrib&quot; image includes many more receivers/exporters<br>          # than the minimal &quot;core&quot; image — including the jaeger exporter<br>          # we use here</pre><pre>          args: [&quot;--config=/etc/otel/config.yaml&quot;]<br>          # tells the binary exactly where to find its config file —<br>          # matches the mountPath below</pre><pre>          ports:<br>            - name: grpc<br>              containerPort: 4317<br>            - name: http<br>              containerPort: 4318</pre><pre>          volumeMounts:<br>            - name: config<br>              mountPath: /etc/otel<br>              # mounts the ConfigMap&#39;s config.yaml key as an actual<br>              # file at this exact path</pre><pre>          resources:<br>            requests:<br>              cpu: &quot;50m&quot;<br>              memory: &quot;128Mi&quot;<br>            limits:<br>              cpu: &quot;200m&quot;<br>              memory: &quot;256Mi&quot;</pre><pre>      volumes:<br>        - name: config<br>          configMap:<br>            name: otel-collector-config</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-collector<br>  namespace: taskmanager<br>  labels:<br>    app: otel-collector</pre><pre>spec:<br>  selector:<br>    app: otel-collector<br>  ports:<br>    - name: grpc<br>      port: 4317<br>      targetPort: 4317<br>    - name: http<br>      port: 4318<br>      targetPort: 4318<br>  # this Service is what backend pods actually connect to via the<br>  # otel-collector:4317 DNS name set in the ConfigMap back in Part 2</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: jaeger<br>  namespace: taskmanager<br>  labels:<br>    app: jaeger</pre><pre>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: jaeger</pre><pre>  template:<br>    metadata:<br>      labels:<br>        app: jaeger</pre><pre>    spec:<br>      containers:<br>        - name: jaeger<br>          image: jaegertracing/all-in-one:1.52<br>          # &quot;all-in-one&quot; bundles the collector, query service, and<br>          # storage (in-memory by default) into ONE container — perfect<br>          # for learning, but production Jaeger would split these into<br>          # separate components backed by real Elasticsearch/Cassandra<br>          # storage instead of in-memory (which loses all data on restart)</pre><pre>          env:<br>            - name: COLLECTOR_OTLP_ENABLED<br>              value: &quot;true&quot;<br>              # without this, the all-in-one image would only accept<br>              # Jaeger&#39;s own legacy wire format, not the OTLP format<br>              # our OTel Collector is sending</pre><pre>          ports:<br>            - name: ui<br>              containerPort: 16686<br>              # the web UI port — where you&#39;d browse traces<br>            - name: grpc<br>              containerPort: 14250<br>              # what the OTel Collector connects TO (matches the<br>              # endpoint: jaeger-collector:14250 set above)<br>            - name: http<br>              containerPort: 14268</pre><pre>          resources:<br>            requests:<br>              cpu: &quot;100m&quot;<br>              memory: &quot;256Mi&quot;<br>            limits:<br>              cpu: &quot;500m&quot;<br>              memory: &quot;512Mi&quot;</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: jaeger-collector<br>  namespace: taskmanager<br>  labels:<br>    app: jaeger</pre><pre>spec:<br>  selector:<br>    app: jaeger<br>  ports:<br>    - name: ui<br>      port: 16686<br>      targetPort: 16686<br>    - name: grpc<br>      port: 14250<br>      targetPort: 14250</pre><p><strong>The complete tracing data path:</strong></p><pre>Spring Boot (OTel Java agent auto-instruments HTTP + JDBC calls)<br>    ↓ gRPC, port 4317<br>otel-collector Service → otel-collector pod<br>    ↓ batches spans, adds cluster.name attribute<br>    ↓ gRPC, port 14250<br>jaeger-collector Service → jaeger pod<br>    ↓ stores in-memory, serves via UI<br>http://localhost:16686 (after port-forward)</pre><pre>kubectl apply -f 11-tracing/otel-jaeger.yaml</pre><pre>kubectl port-forward svc/jaeger-collector 16686:16686 -n taskmanager<br># Open <a href="http://localhost:16686,">http://localhost:16686,</a> select &quot;taskmanager-backend&quot; from<br># the service dropdown, click Find Traces</pre><h3>Putting it all together — the deploy script</h3><pre>#!/bin/bash<br>set -e<br># &quot;set -e&quot; means: if ANY command in this script fails (non-zero exit<br># code), stop immediately instead of plowing ahead with a broken state</pre><pre>minikube start --driver=docker --cpus=4 --memory=8192<br># allocates 4 CPU cores and 8GB RAM to the Minikube VM — enough<br># headroom to run MySQL, 2 backend replicas, 2 frontend replicas,<br># Prometheus, Grafana, Fluentd, Jaeger all simultaneously</pre><pre>minikube addons enable ingress<br>minikube addons enable metrics-server<br># metrics-server is REQUIRED for HPA to function — without it,<br># `kubectl top pods` and HPA&#39;s CPU percentage calculations have<br># no data source at all</pre><pre>eval $(minikube -p minikube docker-env)<br># this is THE line that makes local image builds visible to Minikube —<br># it re-points your shell&#39;s docker CLI at Minikube&#39;s INTERNAL Docker<br># daemon instead of your host machine&#39;s Docker Desktop</pre><pre>docker build -t task-app:latest ../backend/<br>docker build -t taskmanager-frontend:latest ../frontend/<br>docker build -t taskmanager-mysql:latest ../sql/<br># because of the eval above, these builds happen INSIDE Minikube&#39;s<br># Docker, making the images immediately available to<br># imagePullPolicy: Never in our Deployments</pre><pre>minikube ssh -- sudo mkdir -p /data/mysql &amp;&amp; sudo chmod 777 /data/mysql<br># creates the actual directory on the Minikube VM&#39;s disk that our<br># PersistentVolume&#39;s hostPath: /data/mysql points to — without this,<br># the PV would reference a path that doesn&#39;t exist yet</pre><pre>helm install prometheus-stack prometheus-community/kube-prometheus-stack \<br>  --namespace monitoring --create-namespace \<br>  --set grafana.adminPassword=admin123<br># one Helm command installs Prometheus Operator, Prometheus itself,<br># Grafana, Alertmanager, kube-state-metrics, AND node-exporter — this<br># is why we can simply write ServiceMonitor and PrometheusRule objects<br># afterward and have them &quot;just work&quot;</pre><pre>kubectl apply -f 00-namespace/<br>kubectl apply -f 01-config/<br>kubectl apply -f 02-storage/<br>kubectl apply -f 03-database/</pre><pre>kubectl wait --for=condition=ready pod -l app=mysql \<br>  -n taskmanager --timeout=120s<br># THIS LINE is critical ordering — it BLOCKS the script here until<br># MySQL is actually Ready, before moving on to deploy the backend<br># (which would otherwise race against MySQL&#39;s own init container<br># logic, though the backend&#39;s OWN init container provides a second<br># layer of protection too)</pre><pre>kubectl apply -f 08-security/<br>kubectl apply -f 04-backend/<br>kubectl apply -f 05-frontend/<br>kubectl apply -f 06-networking/<br>kubectl apply -f 07-scaling/<br>kubectl apply -f 09-observability/<br>kubectl apply -f 10-monitoring/<br>kubectl apply -f 11-tracing/<br># RBAC is applied BEFORE backend — the backend pod spec references<br># serviceAccountName: backend-sa, which must already exist or the<br># pod creation would fail</pre><h3>Conclusion</h3><p>Every object in this project exists to solve one specific, real problem:</p><ul><li><strong>Namespace + PSS labels</strong> stop misconfigured pods before they even start</li><li><strong>ConfigMap + Secret</strong> separate “safe to read” config from “must protect” credentials</li><li><strong>PV + PVC</strong> lets MySQL’s data survive pod restarts, node failures, and rescheduling</li><li><strong>StatefulSet</strong> gives MySQL the stable identity that a Deployment cannot provide</li><li><strong>Init containers</strong> eliminate the entire class of “started before its dependency” crashes</li><li><strong>Three health probes</strong> (startup/liveness/readiness) each answer a different question about pod health</li><li><strong>Resource requests/limits</strong> turn “guess and hope” into deliberate, Guaranteed vs Burstable QoS decisions</li><li><strong>HPA + VPA together</strong> scale pod count by CPU while right-sizing memory, without fighting each other</li><li><strong>Affinity/anti-affinity</strong> turn “the scheduler does whatever” into “I control exactly where this runs”</li><li><strong>NetworkPolicy’s deny-all baseline</strong> means every single allowed connection was a deliberate choice, not an oversight</li><li><strong>RBAC’s narrow Roles</strong> mean a compromised pod or leaked token has the smallest possible blast radius</li><li><strong>Fluentd + Prometheus + Jaeger together</strong> answer “what happened,” “how bad is it,” and “where exactly,” respectively</li></ul><p>None of these concepts exists in isolation in a real cluster — they compound. The init container exists because of the StatefulSet’s startup time. The NetworkPolicy’s DNS rule exists because of how Services resolve names. The PodDisruptionBudget exists because of how Cluster Autoscaler interacts with affinity rules. Once you see how they connect, a 1000-line Kubernetes manifest stops looking like noise and starts looking like a single, coherent design.</p><p><em>All YAML and explanations in this article come from a real, working TaskManager application (React + Spring Boot + MySQL) deployed and tested on Minikube. Every line comment reflects something that was actually verified running, not theoretical behavior.</em></p><p><em>If this helped you understand production Kubernetes deployments, leave a clap.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96be1f4f1ce0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding TLS to Kubernetes Ingress with cert-manager]]></title>
            <link>https://medium.com/@laxmanpajjuri/adding-tls-to-kubernetes-ingress-with-cert-manager-737364b3aaa8?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/737364b3aaa8</guid>
            <category><![CDATA[certificate]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[tls]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Fri, 05 Jun 2026 05:19:11 GMT</pubDate>
            <atom:updated>2026-06-05T05:19:11.474Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Automate HTTPS certificate provisioning and renewal — you never manually create or rotate TLS certificates again</em></blockquote><p>A plain HTTP Ingress works for development but production always needs HTTPS. cert-manager is the standard Kubernetes solution — it watches for Certificate objects, talks to a certificate authority (self-signed for local dev, Let’s Encrypt for production), provisions the TLS cert, stores it in a Secret, and renews it automatically before it expires.</p><h3>How it fits into your existing Ingress setup</h3><p>Without TLS your Ingress looks like this:</p><pre>Browser → HTTP → Ingress Controller → backend-service → Spring Boot pod</pre><p>With TLS cert-manager adds one layer:</p><pre>Browser → HTTPS → Ingress Controller (terminates TLS here)<br>                      ↓ plain HTTP internally<br>              backend-service:8080<br>                      ↓ plain HTTP<br>              Spring Boot pod</pre><p>TLS terminates at the Ingress. Your Spring Boot app and MySQL pod never see encrypted traffic — they keep talking plain HTTP inside the cluster. The certificate never touches your application code.</p><h3>Four objects cert-manager needs</h3><pre>ClusterIssuer  →  who signs certificates<br>                  self-signed for Minikube, Let&#39;s Encrypt for production</pre><pre>Certificate    →  which domain to issue a cert for<br>                  cert-manager reads this and does all the work</pre><pre>Issuer         →  namespace-scoped certificate authority<br>                  backed by the root CA from ClusterIssuer</pre><pre>Secret         →  where cert-manager stores the TLS key and cert<br>                  auto-created, Ingress references it by name</pre><h3>Step 1 — Install cert-manager</h3><pre>kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.0/cert-manager.yaml</pre><pre># Wait for all three cert-manager pods to be ready<br>kubectl wait --for=condition=ready pod \<br>  -l app=cert-manager \<br>  -n cert-manager --timeout=120s</pre><pre># Verify<br>kubectl get pods -n cert-manager<br># NAME                                      READY   STATUS<br># cert-manager-xxx                          1/1     Running<br># cert-manager-cainjector-xxx               1/1     Running<br># cert-manager-webhook-xxx                  1/1     Running</pre><h3>Step 2 — Self-signed setup for Minikube</h3><pre># tls-ingress.yaml</pre><pre># ── ClusterIssuer — self-signed for local dev ────────────────<br>apiVersion: cert-manager.io/v1<br>kind: ClusterIssuer<br>metadata:<br>  name: selfsigned-issuer<br>spec:<br>  selfSigned: {}    # signs its own certificates — no external CA needed</pre><pre>---<br># ── Root CA Certificate ──────────────────────────────────────<br># cert-manager creates a CA cert and stores it in taskmanager-ca-secret<br>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: taskmanager-ca<br>  namespace: taskmanager<br>spec:<br>  isCA: true<br>  commonName: taskmanager-ca<br>  secretName: taskmanager-ca-secret    # cert-manager creates this Secret<br>  privateKey:<br>    algorithm: ECDSA<br>    size: 256<br>  issuerRef:<br>    name: selfsigned-issuer<br>    kind: ClusterIssuer<br>    group: cert-manager.io</pre><pre>---<br># ── Namespace Issuer backed by our CA ───────────────────────<br># Issues certificates signed by our root CA above<br>apiVersion: cert-manager.io/v1<br>kind: Issuer<br>metadata:<br>  name: taskmanager-ca-issuer<br>  namespace: taskmanager<br>spec:<br>  ca:<br>    secretName: taskmanager-ca-secret  # uses CA cert from above</pre><pre>---<br># ── TLS Certificate for taskmanager.local ───────────────────<br># cert-manager issues the cert and stores it in taskmanager-tls Secret<br># Auto-renews 15 days before the 90-day expiry — fully automatic<br>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: taskmanager-tls-cert<br>  namespace: taskmanager<br>spec:<br>  secretName: taskmanager-tls          # Ingress references this Secret<br>  duration: 2160h                      # 90 days validity<br>  renewBefore: 360h                    # renew 15 days before expiry<br>  commonName: taskmanager.local<br>  dnsNames:<br>    - taskmanager.local<br>    - api.taskmanager.local<br>  issuerRef:<br>    name: taskmanager-ca-issuer<br>    kind: Issuer<br>    group: cert-manager.io</pre><pre>---<br># ── Ingress with TLS block ───────────────────────────────────<br>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: taskmanager-ingress<br>  namespace: taskmanager<br>  annotations:<br>    # Force all HTTP requests to HTTPS<br>    nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;<br>    nginx.ingress.kubernetes.io/force-ssl-redirect: &quot;true&quot;<br>    nginx.ingress.kubernetes.io/proxy-read-timeout: &quot;60&quot;<br>    # Rate limit — protect API from abuse<br>    nginx.ingress.kubernetes.io/limit-rps: &quot;20&quot;<br>    # Tell cert-manager which Issuer to use<br>    cert-manager.io/issuer: &quot;taskmanager-ca-issuer&quot;</pre><pre>spec:<br>  ingressClassName: nginx</pre><pre>  # TLS — cert-manager populates this Secret automatically<br>  tls:<br>    - hosts:<br>        - taskmanager.local<br>        - api.taskmanager.local<br>      secretName: taskmanager-tls      # must match Certificate.spec.secretName</pre><pre>  rules:<br>    - host: taskmanager.local<br>      http:<br>        paths:<br>          # /api → Spring Boot backend<br>          - path: /api<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: backend-service<br>                port:<br>                  number: 8080</pre><pre>          # / → React frontend<br>          - path: /<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: frontend-service<br>                port:<br>                  number: 80</pre><pre>    # API subdomain — direct backend access<br>    - host: api.taskmanager.local<br>      http:<br>        paths:<br>          - path: /<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: backend-service<br>                port:<br>                  number: 8080</pre><h3>Step 3 — Apply and verify</h3><pre># Remove old plain HTTP ingress first<br>kubectl delete ingress taskmanager-ingress -n taskmanager</pre><pre># Apply TLS setup<br>kubectl apply -f tls-ingress.yaml</pre><pre># Watch cert-manager issue the certificate — takes about 30 seconds<br>kubectl get certificate -n taskmanager -w<br># NAME                   READY   SECRET            AGE<br># taskmanager-tls-cert   False   taskmanager-tls   5s   ← issuing<br># taskmanager-tls-cert   True    taskmanager-tls   28s  ← ready!</pre><pre># Verify the TLS Secret was created<br>kubectl get secret taskmanager-tls -n taskmanager<br># NAME              TYPE                DATA   AGE<br># taskmanager-tls   kubernetes.io/tls   2      30s<br>#                   ↑ tls.crt and tls.key inside</pre><pre># See certificate details — expiry, issuer, renewal time<br>kubectl describe certificate taskmanager-tls-cert -n taskmanager</pre><pre># Add domains to hosts file<br>echo &quot;$(minikube ip)  taskmanager.local api.taskmanager.local&quot; | sudo tee -a /etc/hosts</pre><pre># Test HTTPS — -k skips validation for self-signed cert<br>curl -k <a href="https://taskmanager.local/api/tasks">https://taskmanager.local/api/tasks</a><br>curl -k <a href="https://api.taskmanager.local/api/tasks">https://api.taskmanager.local/api/tasks</a></pre><pre># Open in browser<br># <a href="https://taskmanager.local/">https://taskmanager.local</a><br># Browser shows &quot;Not secure&quot; warning — expected for self-signed</pre><h3>What cert-manager does automatically</h3><pre>You apply Certificate object<br>    ↓<br>cert-manager reads it<br>    ↓<br>Talks to the Issuer (CA)<br>    ↓<br>Generates TLS key and certificate<br>    ↓<br>Stores in Kubernetes Secret: taskmanager-tls<br>    ↓<br>Ingress reads the Secret — HTTPS starts working</pre><pre>75 days later (15 days before expiry):<br>    ↓<br>cert-manager automatically renews<br>    ↓<br>Updates the Secret with new cert<br>    ↓<br>Nginx hot-reloads — zero downtime</pre><h3>Step 4 — Production switch to Let’s Encrypt</h3><p>For production with a real domain this is a one-annotation change in the Ingress. First create the Let’s Encrypt ClusterIssuer:</p><pre># letsencrypt-issuer.yaml<br>apiVersion: cert-manager.io/v1<br>kind: ClusterIssuer<br>metadata:<br>  name: letsencrypt-prod<br>spec:<br>  acme:<br>    # Let&#39;s Encrypt production endpoint<br>    server: https://acme-v02.api.letsencrypt.org/directory<br>    email: your-email@company.com      # gets expiry warnings<br>    privateKeySecretRef:<br>      name: letsencrypt-prod-key<br>    solvers:<br>      # HTTP-01 challenge — cert-manager creates a temporary pod<br>      # Let&#39;s Encrypt calls http://&lt;your-domain&gt;/.well-known/acme-challenge/<br>      - http01:<br>          ingress:<br>            class: nginx</pre><p>Then update two lines in your Ingress:</p><pre>annotations:<br>  # Change from:<br>  cert-manager.io/issuer: &quot;taskmanager-ca-issuer&quot;<br>  # Change to:<br>  cert-manager.io/cluster-issuer: &quot;letsencrypt-prod&quot;</pre><pre># And update Certificate issuerRef:<br>issuerRef:<br>  name: letsencrypt-prod<br>  kind: ClusterIssuer   # ClusterIssuer not Issuer</pre><p>cert-manager handles the ACME HTTP-01 challenge automatically — it creates a temporary Ingress path, Let’s Encrypt validates it, cert-manager gets the cert, stores it in the Secret, and cleans up. You never touch certificates again.</p><h3>Minikube vs Production comparison</h3><p>Minikube Production Issuer selfSigned ClusterIssuer Let’s Encrypt ACME Certificate trust Browser shows warning Fully trusted by browsers Domain taskmanager.local (hosts file) yourdomain.com (real DNS) Renewal cert-manager auto-renews cert-manager auto-renews Cost Free Free (Let’s Encrypt) Annotation cert-manager.io/issuer cert-manager.io/cluster-issuer</p><h3>Useful commands</h3><pre># Check all certificates in namespace<br>kubectl get certificate -n taskmanager</pre><pre># See certificate details including expiry and renewal time<br>kubectl describe certificate taskmanager-tls-cert -n taskmanager</pre><pre># See the TLS secret cert-manager created<br>kubectl get secret taskmanager-tls -n taskmanager</pre><pre># Decode the certificate to see expiry date<br>kubectl get secret taskmanager-tls -n taskmanager \<br>  -o jsonpath=&#39;{.data.tls\.crt}&#39; | base64 -d | openssl x509 -noout -dates</pre><pre># See cert-manager logs if certificate is stuck<br>kubectl logs -n cert-manager \<br>  -l app=cert-manager --tail=50</pre><pre># Force cert renewal (before expiry)<br>kubectl delete certificate taskmanager-tls-cert -n taskmanager<br>kubectl apply -f tls-ingress.yaml</pre><pre># Check all ClusterIssuers<br>kubectl get clusterissuers</pre><h3>Common issues and fixes</h3><pre>Certificate stuck in Ready: False<br>  → check cert-manager logs<br>  → kubectl describe certificate taskmanager-tls-cert -n taskmanager<br>  → look at Events section for exact error</pre><pre>Browser shows &quot;Your connection is not private&quot;<br>  → expected for self-signed cert on Minikube<br>  → click Advanced → Proceed → works fine for dev<br>  → use Let&#39;s Encrypt for production</pre><pre>Let&#39;s Encrypt HTTP-01 challenge failing<br>  → your domain must be publicly accessible<br>  → port 80 must be open to the internet<br>  → check Ingress is correctly routing /.well-known/acme-challenge/</pre><pre>Secret not found error in Ingress<br>  → cert-manager may still be issuing the cert<br>  → kubectl get certificate -n taskmanager — wait for Ready: True</pre><pre>Rate limit error from Let&#39;s Encrypt<br>  → Let&#39;s Encrypt limits 5 failures per hour per domain<br>  → use letsencrypt-staging for testing first<br>  → server: <a href="https://acme-staging-v02.api.letsencrypt.org/directory">https://acme-staging-v02.api.letsencrypt.org/directory</a></pre><p><em>This TLS setup is part of the complete production TaskManager Kubernetes deployment covered in the full series. cert-manager is a CNCF project and the standard way to handle TLS in Kubernetes — used by teams at all major cloud providers.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=737364b3aaa8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Observability: Logging, Monitoring, and Distributed Tracing]]></title>
            <link>https://medium.com/@laxmanpajjuri/kubernetes-observability-logging-monitoring-and-distributed-tracing-8090c665ac32?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/8090c665ac32</guid>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Tue, 26 May 2026 07:37:37 GMT</pubDate>
            <atom:updated>2026-05-26T07:37:37.471Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>A complete practical guide to seeing inside your Kubernetes cluster — with Fluentd, EFK stack, Prometheus, Grafana, OpenTelemetry, and Jaeger on a real Spring Boot + MySQL + React app</em></blockquote><p>When something breaks in production at 2 am, you need three things immediately: what happened (logs), how bad it is (metrics), and where exactly the request failed (traces). Without all three, you are debugging blind.</p><p>This article covers the three pillars of observability in Kubernetes:</p><ul><li><strong>Logging</strong> — collecting and searching logs with Fluentd and the EFK stack</li><li><strong>Monitoring</strong> — metrics collection and dashboards with Prometheus and Grafana</li><li><strong>Distributed Tracing</strong> — request journey visualization with OpenTelemetry and Jaeger</li></ul><p>All examples use a real TaskManager app (React + Spring Boot + MySQL) running on Minikube.</p><h3>The three pillars of observability</h3><pre>Logs    →  what happened? &quot;ERROR: Connection refused to mysql-service:3306&quot;<br>Metrics →  how bad is it? &quot;Error rate 12%, response time P99 = 2.3s&quot;<br>Traces  →  where exactly? &quot;Backend spent 295ms on SELECT * FROM tasks — missing index&quot;</pre><p>Each pillar answers a different question. You need all three to debug production issues quickly.</p><h3>Part 1: Logging with Fluentd and EFK Stack</h3><h3>The golden rule — always log to stdout</h3><p>Before touching Fluentd or Elasticsearch, your application must write logs to stdout/stderr — not to files inside the container:</p><pre>// Spring Boot — SLF4J logs to stdout automatically<br>// application.properties<br>logging.level.root=INFO<br>logging.level.com.taskmanager=DEBUG</pre><pre># Output goes to stdout automatically:<br># 2026-05-10 10:30:00 INFO  TaskController - GET /api/tasks - 200 OK 45ms</pre><p>Kubernetes saves everything written to stdout/stderr to /var/log/containers/ on each node. Fluentd reads from there — you never need to configure file paths inside containers.</p><h3>Three log collection patterns</h3><pre>1. Node-level (DaemonSet)  →  one Fluentd pod per node reads all container logs<br>                               most common, least overhead, what we use here</pre><pre>2. Sidecar container       →  each pod has its own logging container<br>                               use when app writes to files, not stdout</pre><pre>3. App pushes directly     →  app sends logs straight to Elasticsearch<br>                               most control, adds coupling to your code</pre><h3>The EFK Stack</h3><pre>E — Elasticsearch  →  stores and indexes logs<br>F — Fluentd        →  collects, parses, and forwards logs<br>K — Kibana         →  search, filter, and visualize logs</pre><h3>Deploy Elasticsearch</h3><pre># elasticsearch.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: elasticsearch<br>  namespace: logging<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: elasticsearch<br>  template:<br>    metadata:<br>      labels:<br>        app: elasticsearch<br>    spec:<br>      containers:<br>        - name: elasticsearch<br>          image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0<br>          ports:<br>            - containerPort: 9200<br>          env:<br>            - name: discovery.type<br>              value: single-node<br>            - name: ES_JAVA_OPTS<br>              value: &quot;-Xms512m -Xmx512m&quot;<br>            - name: xpack.security.enabled<br>              value: &quot;false&quot;        # disable auth for learning<br>          resources:<br>            limits:<br>              memory: &quot;2Gi&quot;</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: elasticsearch<br>  namespace: logging<br>spec:<br>  selector:<br>    app: elasticsearch<br>  ports:<br>    - port: 9200<br>      targetPort: 9200</pre><pre>kubectl create namespace logging<br>kubectl apply -f elasticsearch.yaml</pre><pre># Test Elasticsearch is up<br>kubectl port-forward svc/elasticsearch 9200:9200 -n logging<br>curl <a href="http://localhost:9200/">http://localhost:9200</a><br># {&quot;name&quot;:&quot;elasticsearch&quot;,&quot;cluster_name&quot;:&quot;docker-cluster&quot;,...}</pre><h3>Deploy Fluentd as a DaemonSet</h3><p>Fluentd runs as a DaemonSet — one pod per node — so it can read log files from the node’s filesystem:</p><pre># fluentd-configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: fluentd-config<br>  namespace: logging<br>data:<br>  fluent.conf: |<br>    &lt;source&gt;<br>      @type tail<br>      path /var/log/containers/*.log     # all container logs on this node<br>      pos_file /var/log/fluentd-containers.log.pos<br>      tag kubernetes.*<br>      read_from_head true<br>      &lt;parse&gt;<br>        @type json<br>        time_format %Y-%m-%dT%H:%M:%S.%NZ<br>      &lt;/parse&gt;<br>    &lt;/source&gt;</pre><pre>    &lt;filter kubernetes.**&gt;<br>      @type kubernetes_metadata           # enriches with pod/namespace info<br>      @id filter_kube_metadata<br>    &lt;/filter&gt;</pre><pre>    &lt;filter kubernetes.**&gt;<br>      @type record_transformer<br>      &lt;record&gt;<br>        cluster_name &quot;minikube&quot;<br>      &lt;/record&gt;<br>    &lt;/filter&gt;</pre><pre>    &lt;match kubernetes.**&gt;<br>      @type elasticsearch<br>      host elasticsearch.logging.svc.cluster.local<br>      port 9200<br>      logstash_format true<br>      logstash_prefix logstash<br>      logstash_dateformat %Y.%m.%d       # index per day: logstash-2026.05.10<br>      flush_interval 5s<br>    &lt;/match&gt;</pre><pre># fluentd-daemonset.yaml<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: fluentd<br>  namespace: logging<br>spec:<br>  selector:<br>    matchLabels:<br>      app: fluentd<br>  template:<br>    metadata:<br>      labels:<br>        app: fluentd<br>    spec:<br>      serviceAccountName: fluentd-sa<br>      tolerations:<br>        - key: node-role.kubernetes.io/master<br>          effect: NoSchedule<br>      containers:<br>        - name: fluentd<br>          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch<br>          env:<br>            - name: FLUENT_ELASTICSEARCH_HOST<br>              value: &quot;elasticsearch.logging.svc.cluster.local&quot;<br>            - name: FLUENT_ELASTICSEARCH_PORT<br>              value: &quot;9200&quot;<br>            - name: FLUENT_ELASTICSEARCH_SCHEME<br>              value: &quot;http&quot;<br>          resources:<br>            requests:<br>              cpu: &quot;100m&quot;<br>              memory: &quot;200Mi&quot;<br>            limits:<br>              cpu: &quot;200m&quot;<br>              memory: &quot;500Mi&quot;<br>          volumeMounts:<br>            - name: varlog<br>              mountPath: /var/log<br>            - name: varlibdockercontainers<br>              mountPath: /var/lib/docker/containers<br>              readOnly: true<br>            - name: fluentd-config<br>              mountPath: /fluentd/etc/<br>      volumes:<br>        - name: varlog<br>          hostPath:<br>            path: /var/log              # mount node&#39;s log directory<br>        - name: varlibdockercontainers<br>          hostPath:<br>            path: /var/lib/docker/containers<br>        - name: fluentd-config<br>          configMap:<br>            name: fluentd-config</pre><pre># fluentd-rbac.yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: fluentd-sa<br>  namespace: logging</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: fluentd-role<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;namespaces&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><pre>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: fluentd-binding<br>roleRef:<br>  kind: ClusterRole<br>  name: fluentd-role<br>  apiGroup: rbac.authorization.k8s.io<br>subjects:<br>  - kind: ServiceAccount<br>    name: fluentd-sa<br>    namespace: logging</pre><pre>kubectl apply -f fluentd-rbac.yaml<br>kubectl apply -f fluentd-configmap.yaml<br>kubectl apply -f fluentd-daemonset.yaml</pre><pre># Verify Fluentd is running on each node<br>kubectl get pods -n logging -o wide<br># fluentd-xxxxx   1/1   Running   0   minikube</pre><h3>Deploy Kibana</h3><pre># kibana.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: kibana<br>  namespace: logging<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: kibana<br>  template:<br>    metadata:<br>      labels:<br>        app: kibana<br>    spec:<br>      containers:<br>        - name: kibana<br>          image: docker.elastic.co/kibana/kibana:8.11.0<br>          ports:<br>            - containerPort: 5601<br>          env:<br>            - name: ELASTICSEARCH_HOSTS<br>              value: &quot;http://elasticsearch:9200&quot;<br>          resources:<br>            limits:<br>              memory: &quot;1Gi&quot;</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: kibana<br>  namespace: logging<br>spec:<br>  selector:<br>    app: kibana<br>  ports:<br>    - port: 5601<br>      targetPort: 5601<br>  type: NodePort</pre><pre>kubectl apply -f kibana.yaml</pre><pre># Access Kibana<br>kubectl port-forward svc/kibana 5601:5601 -n logging<br># Open <a href="http://localhost:5601/">http://localhost:5601</a></pre><h3>Search logs in Kibana</h3><pre>1. Stack Management → Index Patterns → Create index pattern<br>   Pattern: logstash-*<br>   Time field: @timestamp</pre><pre>2. Discover → useful KQL queries:<br>   kubernetes.namespace_name: taskmanager<br>   kubernetes.pod_name: backend*<br>   log: &quot;ERROR&quot;<br>   log: &quot;Exception&quot; AND kubernetes.namespace_name: taskmanager<br>   log: &quot;GET /api/tasks&quot; AND log: &quot;200&quot;</pre><h3>Structured logging — make logs searchable</h3><p>Instead of plain text, emit JSON so each field is individually indexed:</p><pre>&lt;!-- pom.xml --&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;net.logstash.logback&lt;/groupId&gt;<br>    &lt;artifactId&gt;logstash-logback-encoder&lt;/artifactId&gt;<br>    &lt;version&gt;7.4&lt;/version&gt;<br>&lt;/dependency&gt;</pre><pre>&lt;!-- logback-spring.xml --&gt;<br>&lt;configuration&gt;<br>  &lt;appender name=&quot;JSON_STDOUT&quot; class=&quot;ch.qos.logback.core.ConsoleAppender&quot;&gt;<br>    &lt;encoder class=&quot;net.logstash.logback.encoder.LogstashEncoder&quot;&gt;<br>      &lt;customFields&gt;{&quot;app&quot;:&quot;taskmanager&quot;,&quot;env&quot;:&quot;production&quot;}&lt;/customFields&gt;<br>    &lt;/encoder&gt;<br>  &lt;/appender&gt;<br>  &lt;root level=&quot;INFO&quot;&gt;<br>    &lt;appender-ref ref=&quot;JSON_STDOUT&quot;/&gt;<br>  &lt;/root&gt;<br>&lt;/configuration&gt;</pre><p>Each log line is now JSON — Elasticsearch indexes every field and you can search bylevel:ERROR, filter by response time, and build dashboards on specific fields.</p><h3>Useful kubectl log commands</h3><pre># Stream live logs<br>kubectl logs -l app=backend -n taskmanager --follow</pre><pre># See previous crash logs<br>kubectl logs backend-xxx-yyy -n taskmanager --previous</pre><pre># Filter with grep<br>kubectl logs -l app=backend -n taskmanager | grep ERROR</pre><pre># Last 100 lines<br>kubectl logs backend-xxx-yyy -n taskmanager --tail=100</pre><pre># Logs from last hour<br>kubectl logs backend-xxx-yyy -n taskmanager --since=1h</pre><pre># Init container logs<br>kubectl logs backend-xxx-yyy -c wait-for-mysql -n taskmanager</pre><h3>Part 2: Monitoring with Prometheus and Grafana</h3><h3>How Prometheus works — pull, not push</h3><p>Prometheus uses a pull model. Every 15 seconds, it calls /metrics on your pods and stores the result. This is the opposite of most monitoring systems that push data to a central server.</p><pre>Every 15 seconds Prometheus:<br>  GET http://backend-pod:8080/actuator/prometheus<br>  GET http://frontend-pod:9113/metrics<br>  GET http://kube-state-metrics:8080/metrics<br>  GET http://node-exporter:9100/metrics</pre><pre>Stores all data in TSDB (time-series database)<br>Evaluates alerting rules<br>Sends firing alerts to Alertmanager</pre><h3>Install the full Prometheus stack</h3><p>The kube-prometheus-stack Helm chart installs Prometheus, Grafana, Alertmanager, and all exporters in one command:</p><pre>helm repo add prometheus-community \<br>  https://prometheus-community.github.io/helm-charts<br>helm repo update</pre><pre>helm install prometheus-stack \<br>  prometheus-community/kube-prometheus-stack \<br>  --namespace monitoring \<br>  --create-namespace \<br>  --set grafana.adminPassword=admin123 \<br>  --set prometheus.prometheusSpec.retention=7d</pre><pre># Watch everything start<br>kubectl get pods -n monitoring -w</pre><pre># Expected:<br># alertmanager-prometheus-stack-xxx   2/2   Running<br># grafana-xxx                         3/3   Running<br># prometheus-prometheus-stack-xxx     2/2   Running<br># kube-state-metrics-xxx              1/1   Running<br># node-exporter-xxx                   2/2   Running  (one per node)</pre><pre># Access Prometheus UI<br>kubectl port-forward svc/prometheus-stack-kube-prom-prometheus \<br>  9090:9090 -n monitoring<br># Open http://localhost:9090</pre><pre># Access Grafana<br>kubectl port-forward svc/prometheus-stack-grafana \<br>  3000:80 -n monitoring<br># Open <a href="http://localhost:3000/">http://localhost:3000</a> — admin / admin123</pre><h3>Expose Spring Boot metrics to Prometheus</h3><pre>&lt;!-- pom.xml --&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.micrometer&lt;/groupId&gt;<br>    &lt;artifactId&gt;micrometer-registry-prometheus&lt;/artifactId&gt;<br>&lt;/dependency&gt;</pre><pre># application.properties<br>management.endpoints.web.exposure.include=health,info,prometheus,metrics<br>management.endpoint.prometheus.enabled=true<br>management.metrics.export.prometheus.enabled=true</pre><pre># Tag all metrics with app name<br>management.metrics.tags.application=taskmanager<br>management.metrics.tags.environment=production</pre><p>Verify:</p><pre>kubectl port-forward svc/backend-service 8080:8080 -n taskmanager<br>curl http://localhost:8080/actuator/prometheus</pre><pre># You see hundreds of metrics:<br># jvm_memory_used_bytes{area=&quot;heap&quot;,...} 3.45e+08<br># http_server_requests_seconds_count{method=&quot;GET&quot;,uri=&quot;/api/tasks&quot;,...} 42<br># hikaricp_connections_active{pool=&quot;HikariPool-1&quot;,...} 2</pre><h3>Tell Prometheus to scrape your backend</h3><pre># servicemonitor-backend.yaml<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: backend-monitor<br>  namespace: taskmanager<br>  labels:<br>    release: prometheus-stack      # must match Prometheus selector<br>spec:<br>  selector:<br>    matchLabels:<br>      app: backend<br>  endpoints:<br>    - port: http<br>      path: /actuator/prometheus<br>      interval: 15s<br>  namespaceSelector:<br>    matchNames:<br>      - taskmanager</pre><pre>kubectl apply -f servicemonitor-backend.yaml</pre><pre># Verify in Prometheus UI → Status → Targets<br># backend-monitor should show as UP</pre><h3>Useful PromQL queries</h3><pre># HTTP request rate<br>rate(http_server_requests_seconds_count{<br>  application=&quot;taskmanager&quot;<br>}[5m])</pre><pre># Error rate (5xx only)<br>rate(http_server_requests_seconds_count{<br>  application=&quot;taskmanager&quot;,<br>  status=~&quot;5..&quot;<br>}[5m])</pre><pre># Average response time in milliseconds<br>rate(http_server_requests_seconds_sum{<br>  application=&quot;taskmanager&quot;,<br>  uri=&quot;/api/tasks&quot;<br>}[5m])<br>/<br>rate(http_server_requests_seconds_count{<br>  application=&quot;taskmanager&quot;,<br>  uri=&quot;/api/tasks&quot;<br>}[5m])<br>* 1000</pre><pre># JVM heap usage percentage<br>jvm_memory_used_bytes{application=&quot;taskmanager&quot;,area=&quot;heap&quot;}<br>/<br>jvm_memory_max_bytes{application=&quot;taskmanager&quot;,area=&quot;heap&quot;}<br>* 100</pre><pre># Active database connections<br>hikaricp_connections_active{application=&quot;taskmanager&quot;}</pre><pre># Pod CPU usage by pod<br>sum(rate(container_cpu_usage_seconds_total{<br>  namespace=&quot;taskmanager&quot;<br>}[5m])) by (pod)</pre><h3>Import Grafana dashboards</h3><pre>Grafana → + → Import → enter Dashboard ID → Load</pre><pre>Useful dashboard IDs:<br>  6417  — Kubernetes Cluster Overview<br>  315   — Kubernetes All Nodes<br>  10280 — Spring Boot 2.1 Statistics<br>  4701  — JVM Micrometer<br>  7249  — Nginx Ingress Controller</pre><h3>Set up alerting rules</h3><pre># alerting-rules.yaml<br>apiVersion: monitoring.coreos.com/v1<br>kind: PrometheusRule<br>metadata:<br>  name: taskmanager-alerts<br>  namespace: taskmanager<br>  labels:<br>    release: prometheus-stack<br>spec:<br>  groups:<br>    - name: taskmanager.rules<br>      rules:<br>        - alert: HighErrorRate<br>          expr: |<br>            rate(http_server_requests_seconds_count{<br>              application=&quot;taskmanager&quot;,status=~&quot;5..&quot;<br>            }[5m])<br>            /<br>            rate(http_server_requests_seconds_count{<br>              application=&quot;taskmanager&quot;<br>            }[5m])<br>            &gt; 0.05<br>          for: 2m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;High error rate in TaskManager&quot;<br>            description: &quot;Error rate is {{ $value | humanizePercentage }}&quot;</pre><pre>        - alert: BackendPodDown<br>          expr: |<br>            kube_deployment_status_replicas_available{<br>              namespace=&quot;taskmanager&quot;,<br>              deployment=&quot;backend&quot;<br>            } &lt; 1<br>          for: 1m<br>          labels:<br>            severity: critical<br>          annotations:<br>            summary: &quot;TaskManager backend has no pods running&quot;</pre><pre>        - alert: HighJVMHeap<br>          expr: |<br>            jvm_memory_used_bytes{application=&quot;taskmanager&quot;,area=&quot;heap&quot;}<br>            /<br>            jvm_memory_max_bytes{application=&quot;taskmanager&quot;,area=&quot;heap&quot;}<br>            &gt; 0.85<br>          for: 5m<br>          labels:<br>            severity: warning<br>          annotations:<br>            summary: &quot;JVM heap usage above 85%&quot;</pre><h3>Configure Alertmanager for Slack</h3><pre># alertmanager-config.yaml<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: alertmanager-prometheus-stack-kube-prom-alertmanager<br>  namespace: monitoring<br>stringData:<br>  alertmanager.yaml: |<br>    global:<br>      slack_api_url: &#39;https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK&#39;</pre><pre>    route:<br>      group_by: [&#39;alertname&#39;, &#39;namespace&#39;]<br>      group_wait: 30s<br>      repeat_interval: 4h<br>      receiver: &#39;slack-notifications&#39;</pre><pre>    receivers:<br>      - name: &#39;slack-notifications&#39;<br>        slack_configs:<br>          - channel: &#39;#k8s-alerts&#39;<br>            title: &#39;{{ .Status | toUpper }} - {{ .CommonAnnotations.summary }}&#39;<br>            text: &#39;{{ .CommonAnnotations.description }}&#39;<br>            send_resolved: true</pre><h3>Key metrics to monitor</h3><pre>Infrastructure (node-exporter + kube-state-metrics):<br>  CPU and memory usage per pod<br>  Pod restart count<br>  Deployment replicas vs desired count</pre><pre>Application (Spring Boot Actuator):<br>  HTTP request rate<br>  HTTP error rate (5xx)<br>  Response time P50, P95, P99<br>  Active DB connections<br>  DB connection pool saturation</pre><pre>JVM (Micrometer):<br>  Heap memory used %<br>  GC pause duration<br>  Thread count</pre><h3>Part 3: Distributed Tracing with OpenTelemetry and Jaeger</h3><h3>What distributed tracing shows is that logs and metrics cannot</h3><pre>User calls GET /api/tasks — response takes 342ms</pre><pre>Logs show:   &quot;DB query executed&quot; on backend pod<br>Metrics show: P99 response time = 340ms<br>              → something is slow but WHERE?</pre><pre>Trace shows:<br>  frontend  0ms ──────────────────────────────────── 342ms<br>  backend      8ms ─────────────────────────────────  8ms setup + 22ms overhead<br>  mysql          13ms ──────────────────────────────  295ms ← HERE<br>  serialize              308ms ────────────────────    12ms</pre><pre>→ instantly see: 86% of time is in the MySQL query<br>→ add an index → problem solved</pre><h3>Key concepts</h3><pre>Trace    →  complete journey of one request end to end<br>             unique traceId shared by all spans</pre><pre>Span     →  one operation within the trace<br>             has spanId, parent spanId, start time, duration, attributes</pre><pre>Context  →  traceId + spanId passed between services via HTTP header<br>             traceparent: 00-&lt;traceId&gt;-&lt;spanId&gt;-01</pre><pre>Sampler  →  controls what % of requests get traced<br>             always_on = 100% (dev), 10% typical (production)</pre><h3>Add OpenTelemetry to Spring Boot</h3><pre>&lt;!-- pom.xml --&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.opentelemetry.instrumentation&lt;/groupId&gt;<br>    &lt;artifactId&gt;opentelemetry-spring-boot-starter&lt;/artifactId&gt;<br>    &lt;version&gt;2.0.0&lt;/version&gt;<br>&lt;/dependency&gt;</pre><pre># application.properties<br>spring.application.name=taskmanager-backend</pre><pre># Send to OTel Collector<br>otel.exporter.otlp.endpoint=http://otel-collector:4317<br>otel.exporter.otlp.protocol=grpc</pre><pre># Sample 100% in development<br>otel.traces.sampler=always_on</pre><pre># Add service metadata to all spans<br>otel.resource.attributes=service.name=taskmanager-backend,service.version=1.0.0,deployment.environment=production</pre><pre># Automatic instrumentation<br>otel.instrumentation.spring-web.enabled=true<br>otel.instrumentation.jdbc.enabled=true              # traces SQL queries<br>otel.instrumentation.logback-appender.enabled=true  # adds traceId to logs</pre><h3>Add custom spans to business logic</h3><pre>// TaskService.java<br>import io.opentelemetry.api.trace.Span;<br>import io.opentelemetry.api.trace.Tracer;<br>import io.opentelemetry.api.GlobalOpenTelemetry;</pre><pre>@Service<br>public class TaskService {</pre><pre>    private final Tracer tracer = GlobalOpenTelemetry.getTracer(&quot;taskmanager&quot;);</pre><pre>    public Task createTask(Task task) {<br>        Span span = tracer.spanBuilder(&quot;validate-and-create-task&quot;)<br>            .startSpan();</pre><pre>        try (var scope = span.makeCurrent()) {<br>            // Attributes are searchable in Jaeger<br>            span.setAttribute(&quot;task.title&quot;, task.getTitle());<br>            span.setAttribute(&quot;task.status&quot;, task.getStatus());</pre><pre>            if (task.getStatus() == null) {<br>                task.setStatus(&quot;TODO&quot;);<br>            }</pre><pre>            Task saved = taskRepository.save(task);<br>            span.setAttribute(&quot;task.id&quot;, saved.getId());<br>            return saved;</pre><pre>        } catch (Exception e) {<br>            span.recordException(e);            // records full stack trace in span<br>            span.setStatus(StatusCode.ERROR, e.getMessage());<br>            throw e;<br>        } finally {<br>            span.end();<br>        }<br>    }<br>}</pre><h3>Deploy OpenTelemetry Collector</h3><pre># otel-collector.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: otel-collector-config<br>  namespace: taskmanager<br>data:<br>  config.yaml: |<br>    receivers:<br>      otlp:<br>        protocols:<br>          grpc:<br>            endpoint: 0.0.0.0:4317<br>          http:<br>            endpoint: 0.0.0.0:4318</pre><pre>    processors:<br>      batch:<br>        timeout: 5s<br>        send_batch_size: 512</pre><pre>      resource:<br>        attributes:<br>          - key: k8s.cluster.name<br>            value: &quot;minikube&quot;<br>            action: insert</pre><pre>    exporters:<br>      jaeger:<br>        endpoint: jaeger-collector:14250<br>        tls:<br>          insecure: true</pre><pre>    service:<br>      pipelines:<br>        traces:<br>          receivers: [otlp]<br>          processors: [batch, resource]<br>          exporters: [jaeger]</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: otel-collector<br>  namespace: taskmanager<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: otel-collector<br>  template:<br>    metadata:<br>      labels:<br>        app: otel-collector<br>    spec:<br>      containers:<br>        - name: otel-collector<br>          image: otel/opentelemetry-collector-contrib:0.91.0<br>          args: [&quot;--config=/etc/otel/config.yaml&quot;]<br>          ports:<br>            - containerPort: 4317<br>            - containerPort: 4318<br>          volumeMounts:<br>            - name: config<br>              mountPath: /etc/otel<br>      volumes:<br>        - name: config<br>          configMap:<br>            name: otel-collector-config</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: otel-collector<br>  namespace: taskmanager<br>spec:<br>  selector:<br>    app: otel-collector<br>  ports:<br>    - name: grpc<br>      port: 4317<br>      targetPort: 4317<br>    - name: http<br>      port: 4318<br>      targetPort: 4318</pre><h3>Deploy Jaeger</h3><pre># jaeger.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: jaeger<br>  namespace: taskmanager<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: jaeger<br>  template:<br>    metadata:<br>      labels:<br>        app: jaeger<br>    spec:<br>      containers:<br>        - name: jaeger<br>          image: jaegertracing/all-in-one:1.52<br>          env:<br>            - name: COLLECTOR_OTLP_ENABLED<br>              value: &quot;true&quot;<br>          ports:<br>            - containerPort: 16686   # UI<br>            - containerPort: 14250   # gRPC collector<br>          resources:<br>            limits:<br>              memory: &quot;512Mi&quot;</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: jaeger-collector<br>  namespace: taskmanager<br>spec:<br>  selector:<br>    app: jaeger<br>  ports:<br>    - name: ui<br>      port: 16686<br>      targetPort: 16686<br>    - name: grpc<br>      port: 14250<br>      targetPort: 14250</pre><pre>kubectl apply -f otel-collector.yaml<br>kubectl apply -f jaeger.yaml</pre><pre># Access Jaeger UI<br>kubectl port-forward svc/jaeger-collector 16686:16686 -n taskmanager<br># Open <a href="http://localhost:16686/">http://localhost:16686</a></pre><h3>Connect traceId to logs</h3><p>Add traceId to every log line so you can jump from a log to the trace in Jaeger:</p><pre># application.properties<br>logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss} [%X{traceId}/%X{spanId}] %-5level %logger - %msg%n</pre><p>Now Spring Boot logs look like:</p><pre>2026-05-10 10:30:00 [a1b2c3d4e5f6a1b2/b3c4d5e6] INFO  TaskController - GET /api/tasks - 200 OK</pre><p>Search in Kibana for traceId:a1b2c3d4e5f6a1b2 → see all logs for that request. Copy the traceId to Jaeger → see the complete waterfall.</p><h3>View traces in Jaeger</h3><pre>1. Open http://localhost:16686<br>2. Select service: taskmanager-backend<br>3. Click Find Traces<br>4. Click any trace to see the waterfall diagram</pre><pre>Useful search filters:<br>  Operation: GET /api/tasks<br>  Tags: error=true           ← find all failing requests<br>  Min Duration: 500ms        ← find slow requests only<br>  Time range: 10:29 - 10:31  ← correlate with Grafana alert</pre><h3>OpenTelemetry vs Jaeger vs Tempo</h3><pre>OpenTelemetry   →  the vendor-neutral standard for instrumentation<br>                   you instrument code with OTel, never directly with Jaeger<br>                   SDK + protocol + Collector</pre><pre>Jaeger          →  trace storage and UI backend<br>                   most popular for Kubernetes, CNCF project<br>                   built by Uber</pre><pre>Zipkin          →  older trace backend, simpler<br>                   less features than Jaeger</pre><pre>Tempo (Grafana) →  trace storage that integrates with Grafana<br>                   best when you are already using Grafana for metrics</pre><pre>The pattern (always the same):<br>  App → OTel SDK → OTel Collector → Jaeger/Tempo/Zipkin</pre><h3>The full observability loop</h3><p>This is how these three pillars work together when something breaks:</p><pre>Step 1: Grafana alert fires<br>  Alert: HighErrorRate — error rate 12% (threshold 5%)<br>  Time: 10:30am<br>  Severity: warning</pre><pre>Step 2: Jump to Kibana<br>  KQL: kubernetes.namespace_name: taskmanager AND log: &quot;ERROR&quot;<br>  Time range: 10:28 - 10:32<br>  Find error log: &quot;ERROR TaskController - DB connection failed&quot;<br>  Notice traceId: a1b2c3d4e5f6</pre><pre>Step 3: Jump to Jaeger<br>  Search traceId: a1b2c3d4e5f6<br>  Open trace — waterfall shows:<br>    backend  0ms ──────────────────────── 2340ms<br>    mysql      8ms ──────────────────────  2332ms  ← 99% of time<br>    └ SELECT * FROM tasks WHERE status=&#39;TODO&#39;</pre><pre>Step 4: Fix<br>  Slow MySQL query — missing index on status column<br>  kubectl exec -it mysql-pod -n taskmanager -- \<br>    mysql -uroot -proot123 taskdb -e \<br>    &quot;CREATE INDEX idx_status ON tasks(status);&quot;</pre><pre>Step 5: Verify fix in Grafana<br>  Response time P99 drops from 2340ms to 45ms<br>  Error rate drops from 12% to 0%<br>  Alert resolves automatically<br>  Slack notification: &quot;RESOLVED - HighErrorRate&quot;</pre><h3>Quick reference — what each tool is for</h3><p>Tool Question: it answers. When to use kubectl logs What happened in this pod right now? Quick debugging Fluentd + EFK: What happened across all pods historically? Incident investigation Prometheus: How bad is it numerically? Alerting, capacity planning, Grafana. How does it look over time? Dashboards, trend analysis, OpenTelemetry. How did this request flow? Performance debugging Jaeger: Where exactly did it fail? Finding the root cause</p><h3>Architecture summary</h3><pre>Application pods<br>  ↓ stdout/stderr<br>  ↓ /actuator/prometheus<br>  ↓ OTel SDK traces</pre><pre>┌─────────────────────────────────────────────────┐<br>│  Collection layer                               │<br>│  Fluentd DaemonSet  →  Elasticsearch  →  Kibana │<br>│  Prometheus scrape  →  TSDB  →  Grafana         │<br>│  OTel Collector     →  Jaeger                   │<br>└─────────────────────────────────────────────────┘</pre><pre>All three connected by traceId in logs</pre><h3>Conclusion</h3><p>The three pillars of observability are not optional for production Kubernetes clusters. They are the difference between debugging an outage in 5 minutes and spending 3 hours reading raw pod logs.</p><p><strong>Fluentd + EFK</strong> gives you a searchable history of everything that happened across your entire cluster. Structured JSON logging with logstash-logback-encoder makes every field searchable in Kibana.</p><p><strong>Prometheus + Grafana</strong> gives you the numbers — request rates, error rates, response times, JVM heap, and DB connections. The kube-prometheus-stack Helm chart sets up everything in one command and includes pre-built dashboards for Kubernetes itself.</p><p><strong>OpenTelemetry + Jaeger</strong> shows you the complete journey of every request. Automatic instrumentation means Spring Boot generates traces for HTTP calls and SQL queries with zero code changes. The traceId in your logs connects all three pillars — find the error in Kibana, jump to the trace in Jaeger, and verify the fix in Grafana.</p><p>All three of these run on Minikube on your laptop. There is no reason to wait for a production cluster to set up observability — every concept here translates directly to EKS, GKE, or AKS.</p><p><em>All YAML and commands in this article come from a real TaskManager app (React + Spring Boot + MySQL) running on Minikube on Windows. Every tool was tested end-to-end.</em></p><p><em>If this helped you instrument your cluster, leave a clap.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8090c665ac32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Security Deep Dive: RBAC, Pod Security, Secrets Management, and Admission Controllers]]></title>
            <link>https://medium.com/@laxmanpajjuri/kubernetes-security-deep-dive-rbac-pod-security-secrets-management-and-admission-controllers-596246165cea?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/596246165cea</guid>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Tue, 26 May 2026 07:36:14 GMT</pubDate>
            <atom:updated>2026-06-23T06:11:22.842Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>A complete practical guide to securing your Kubernetes cluster — with real YAML, real mistakes to avoid, and real fixes from a running TaskManager app</em></blockquote><p>Security in Kubernetes is not a single feature — it is a layered system where each layer protects against a different threat. Get all four layers right and you have a cluster that is genuinely hard to breach. Miss any one of them and you have a gap that attackers will find before you do.</p><p>This article covers the four core security layers:</p><ul><li><strong>RBAC</strong> — who can do what in the cluster</li><li><strong>Pod Security Standards</strong> — what containers are allowed to do</li><li><strong>Secrets Management</strong> — how sensitive data is stored and accessed</li><li><strong>Admission Controllers</strong> — intercepting and enforcing policy on every request</li></ul><h3>Part 1: RBAC — Role-Based Access Control</h3><h3>What RBAC protects against</h3><p>Without RBAC, every user who has kubectl Access can do everything — delete namespaces, read secrets, terminate production pods. RBAC defines exactly what each person and each pod is allowed to do.</p><pre>Without RBAC:<br>  junior developer deletes production namespace by accident<br>  CI pipeline can read all secrets in all namespaces<br>  compromised pod can list every secret in the cluster</pre><pre>With RBAC:<br>  junior developer can deploy but cannot delete namespaces<br>  CI pipeline can only update images in its own namespace<br>  compromised pod can only read its own ConfigMap</pre><h3>Four objects in RBAC</h3><pre>Role              →  what can be done (namespace scoped)<br>ClusterRole       →  what can be done (cluster wide)<br>RoleBinding       →  connects subject to Role (namespace scoped)<br>ClusterRoleBinding →  connects subject to ClusterRole (cluster wide)</pre><p>The relationship is:</p><pre>Subject (User / Group / ServiceAccount)<br>    ↓ connected by<br>RoleBinding or ClusterRoleBinding<br>    ↓ points to<br>Role or ClusterRole<br>    ↓ grants permissions on<br>Resources (pods, deployments, secrets)</pre><h3>Role — namespace scoped permissions</h3><pre># role-developer.yaml<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: developer-role<br>  namespace: taskmanager       # only works in THIS namespace<br>rules:<br>  # Can read pods and view logs<br>  - apiGroups: [&quot;&quot;]            # empty string = core API group<br>    resources: [&quot;pods&quot;, &quot;pods/log&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><pre>  # Can manage deployments<br>  - apiGroups: [&quot;apps&quot;]<br>    resources: [&quot;deployments&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;]</pre><pre>  # Can read services and configmaps<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;services&quot;, &quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;]</pre><h3>ClusterRole — cluster-wide permissions</h3><pre># clusterrole-readonly.yaml<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: cluster-readonly       # no namespace — applies everywhere<br>rules:<br>  # Read pods and services across ALL namespaces<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;services&quot;, &quot;endpoints&quot;, &quot;namespaces&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><pre>  # Read nodes — node-level resources can only be in ClusterRole<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;nodes&quot;, &quot;persistentvolumes&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><pre>  # Read deployments cluster-wide<br>  - apiGroups: [&quot;apps&quot;]<br>    resources: [&quot;deployments&quot;, &quot;replicasets&quot;, &quot;statefulsets&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><h3>RoleBinding — attach Role to Subject</h3><pre># rolebinding-developer.yaml<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: developer-binding<br>  namespace: taskmanager<br>subjects:<br>  - kind: User<br>    name: john<br>    apiGroup: rbac.authorization.k8s.io<br>  - kind: User<br>    name: sarah<br>    apiGroup: rbac.authorization.k8s.io<br>roleRef:<br>  kind: Role<br>  name: developer-role<br>  apiGroup: rbac.authorization.k8s.io</pre><h3>ClusterRoleBinding — cluster-wide binding</h3><pre># clusterrolebinding-monitoring.yaml<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: monitoring-readonly<br>subjects:<br>  - kind: Group<br>    name: monitoring-team<br>    apiGroup: rbac.authorization.k8s.io<br>roleRef:<br>  kind: ClusterRole<br>  name: cluster-readonly<br>  apiGroup: rbac.authorization.k8s.io</pre><h3>ServiceAccount — RBAC for pods</h3><p>A ServiceAccount is an identity for <strong>pods</strong>, not humans. When your backend pod needs to call the Kubernetes API — to read a ConfigMap, list pods, or watch for changes — it authenticates as a ServiceAccount, not as you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8A5U-pgeniTE68lEKTu8g.png" /></figure><pre>User identity        →  for humans running kubectl<br>                         certificate or OIDC token in ~/.kube/config<br><br>ServiceAccount        →  for pods calling the K8s API<br>                         JWT token automatically mounted into the pod</pre><p>Every pod has one — even if you didn’t ask for it</p><pre># If you don&#39;t specify serviceAccountName, K8s uses &quot;default&quot;<br>kubectl get serviceaccount default -n taskmanager<br><br># See what&#39;s mounted<br>kubectl exec -it &lt;pod-name&gt; -n taskmanager -- ls /var/run/secrets/kubernetes.io/serviceaccount/<br># ca.crt      ← cluster&#39;s CA certificate<br># namespace   ← which namespace this pod is in<br># token       ← the JWT used to authenticate</pre><p>The default ServiceAccount has <strong>zero permissions</strong> by default — it can authenticate, but RBAC blocks everything until you bind it to a Role.</p><pre># serviceaccount.yaml<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: backend-sa<br>  namespace: taskmanager<br>---<br># Give backend read access to its own ConfigMap only<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: configmap-reader<br>  namespace: taskmanager<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;]<br>    resourceNames:<br>      - &quot;taskmanager-config&quot;   # ONLY this specific configmap<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: backend-configmap-binding<br>  namespace: taskmanager<br>subjects:<br>  - kind: ServiceAccount<br>    name: backend-sa<br>    namespace: taskmanager<br>roleRef:<br>  kind: Role<br>  name: configmap-reader<br>  apiGroup: rbac.authorization.k8s.io</pre><p>Attach ServiceAccount to your Deployment:</p><pre>spec:<br>  template:<br>    spec:<br>      serviceAccountName: backend-sa   # pod uses this identity<br>      containers:<br>        - name: backend<br>          image: task-app:latest</pre><h3>All verbs explained</h3><pre>get        →  read a specific resource by name<br>list       →  list all resources of a type<br>watch      →  stream changes in real time<br>create     →  create a new resource<br>update     →  replace entire resource<br>patch      →  partially update a resource<br>delete     →  delete a resource<br>exec       →  kubectl exec into a pod<br>portforward → kubectl port-forward</pre><h3>Practical RBAC for your TaskManager</h3><pre># Three roles for a real team:</pre><pre># 1 — Developer: deploy and debug, cannot delete<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: developer<br>  namespace: taskmanager<br>rules:<br>  - apiGroups: [&quot;&quot;, &quot;apps&quot;]<br>    resources: [&quot;pods&quot;, &quot;deployments&quot;, &quot;services&quot;, &quot;configmaps&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;]<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods/log&quot;, &quot;pods/exec&quot;]<br>    verbs: [&quot;get&quot;, &quot;create&quot;]</pre><pre>---<br># 2 — Read-only: monitoring and auditing<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: readonly<br>  namespace: taskmanager<br>rules:<br>  - apiGroups: [&quot;&quot;, &quot;apps&quot;]<br>    resources: [&quot;pods&quot;, &quot;deployments&quot;, &quot;services&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]</pre><pre>---<br># 3 — Namespace admin: full access in namespace only<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: namespace-admin<br>  namespace: taskmanager<br>rules:<br>  - apiGroups: [&quot;*&quot;]<br>    resources: [&quot;*&quot;]<br>    verbs: [&quot;*&quot;]</pre><h3>Verify and debug RBAC</h3><pre># Apply RBAC objects<br>kubectl apply -f role-developer.yaml<br>kubectl apply -f rolebinding-developer.yaml</pre><pre># Check if a user can perform an action<br>kubectl auth can-i create deployments \<br>  --namespace taskmanager \<br>  --as john<br># yes</pre><pre># Check what a ServiceAccount can do<br>kubectl auth can-i list pods \<br>  --namespace taskmanager \<br>  --as system:serviceaccount:taskmanager:backend-sa</pre><pre># See all your own permissions<br>kubectl auth can-i --list -n taskmanager</pre><pre># Describe role to see all rules<br>kubectl describe role developer -n taskmanager</pre><h3>The golden rule — least privilege</h3><pre>Start with zero permissions<br>Add only what is actually needed<br>Use Role over ClusterRole whenever possible<br>Use RoleBinding over ClusterRoleBinding whenever possible<br>Never grant verbs: [&quot;*&quot;] in production<br>Never grant resources: [&quot;*&quot;] in production<br>Audit regularly: kubectl auth can-i --list</pre><h3>Part 2: Pod Security Standards</h3><p>Pod Security Standards define <strong>three levels of security restrictions</strong> for Pods. Basically, rules about what a Pod is and isn’t allowed to do. They replaced the older, more complex <strong>PodSecurityPolicy</strong> (deprecated, removed in v1.25), and are enforced today via the built-in <strong>Pod Security Admission</strong> controller.</p><pre>Without PSS:<br>  container runs as root → writes to /etc/passwd on host<br>  container mounts hostPath → reads /etc/kubernetes/admin.conf<br>  container uses host network → sniffs all cluster traffic<br>With PSS:<br>  privileged containers rejected at the API server<br>  hostPath volumes blocked<br>  root containers blocked</pre><h3>The Three Levels</h3><h4>1. Privileged — “Anything goes”</h4><p>No restrictions at all. A Pod can do whatever the underlying container runtime/kernel allows — including running as root, accessing the host’s network, mounting host directories, escalating privileges, etc.</p><p><strong>When to use:</strong> System-level workloads like CNI plugins, storage drivers, or monitoring agents that genuinely need deep host access.</p><p>yaml</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: privileged-pod<br>spec:<br>  containers:<br>  - name: app<br>    image: busybox<br>    securityContext:<br>      privileged: true   # allowed under &quot;privileged&quot; level</pre><h4>2. Baseline — “Minimally restrictive”</h4><p>Blocks the most well-known privilege escalations, but stays flexible enough for typical applications. It disallows things like:</p><ul><li>Running privileged containers</li><li>Using the host’s network, PID, or IPC namespace</li><li>Mounting host paths</li><li>Certain dangerous Linux capabilities (e.g. SYS_ADMIN)</li></ul><p>But it <strong>still allows running as root</strong> and doesn’t require a read-only filesystem.</p><p>yaml</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: baseline-pod<br>spec:<br>  containers:<br>  - name: app<br>    image: nginx<br>    securityContext:<br>      privileged: false      # required<br>      # hostNetwork, hostPID, hostIPC not set =&gt; fine</pre><p>If you tried hostNetwork: true here, it would be <strong>rejected</strong> under Baseline.</p><h4>3. Restricted — “Heavily locked down” (best practice for most apps)</h4><p>Builds on Baseline and adds strong hardening:</p><ul><li>Must run as <strong>non-root</strong></li><li>Must <strong>disallow privilege escalation</strong></li><li>Must drop <strong>all Linux capabilities</strong> (can add back only NET_BIND_SERVICE)</li><li>Must use the <strong>default or RuntimeDefault seccomp profile</strong></li><li>Volume types are restricted (no exotic host-related volumes)</li></ul><p>yaml</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: restricted-pod<br>spec:<br>  containers:<br>  - name: app<br>    image: nginx<br>    securityContext:<br>      runAsNonRoot: true<br>      allowPrivilegeEscalation: false<br>      seccompProfile:<br>        type: RuntimeDefault<br>      capabilities:<br>        drop: [&quot;ALL&quot;]</pre><p>This is the level you want for most everyday business applications.</p><h3>How You Apply These Levels</h3><p>PSS is enforced <strong>per-namespace</strong> using labels — no separate policy object needed. You label the namespace, and the built-in admission controller checks every Pod created in it.</p><p>bash</p><pre>kubectl label namespace my-app \<br>  pod-security.kubernetes.io/enforce=restricted</pre><p>There are also <strong>three “modes”</strong> you can combine on the same namespace:</p><h3>Three security levels</h3><p><strong>Privileged — no restrictions.</strong> Use only for system components like CNI plugins and monitoring agents in kube-system. Never for application workloads.</p><p><strong>Baseline — prevents known privilege escalations.</strong> What is blocked: privileged: true containers, hostNetwork: true, hostPID: true, hostPath volumes, dangerous capabilities (NET_ADMIN, SYS_ADMIN). What is allowed: running as root (not ideal), most volume types, and common capabilities. Use for most application workloads.</p><p><strong>Restricted — hardened best practice.</strong> What is required: runAsNonRoot: true, drop ALL capabilities, read-only root filesystem, seccompProfile set, no privilege escalation. Use for production internet-facing workloads.</p><pre>Mode              What it does<br><br>enforce  -        Blocks non-compliant Pods from being created<br>audit    -        Allows the Pod, but logs a warning to the audit log<br>warn     -        Allows the Pod, but shows a warning to the kubectl user</pre><p>You can even pin a specific version, useful while rolling out a new policy gradually:</p><p>yaml</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: my-app<br>  labels:<br>    pod-security.kubernetes.io/enforce: baseline<br>    pod-security.kubernetes.io/enforce-version: latest<br>    pod-security.kubernetes.io/warn: restricted     # show warnings for stricter level<br>    pod-security.kubernetes.io/audit: restricted</pre><p>Let’s break this down piece by piece — it’s simpler than it looks once you separate the two concepts involved: <strong>modes</strong> and <strong>levels</strong>.</p><h3>The Two Separate Concepts</h3><p><strong>1. Levels</strong> — <em>how strict</em> the rule is</p><ul><li>privileged → no rules</li><li>baseline → basic protections</li><li>restricted → strict, hardened</li></ul><p><strong>2. Modes</strong> — <em>what happens</em> when a Pod breaks the rule</p><ul><li>enforce → <strong>block</strong> the Pod from being created</li><li>warn → <strong>allow</strong> it, but show a warning message in kubectl</li><li>audit → <strong>allow</strong> it, but write a note in the audit log (for admins to review later, no message to the user)</li></ul><p>Each mode is <strong>independent</strong> and can be set to a <strong>different level</strong>. That’s the confusing part — people assume there’s one “policy,” but actually there are three knobs, each pointing at a possibly different level.</p><h3>Reading Your Exact Example, Line by Line</h3><p>YAML</p><pre>pod-security.kubernetes.io/enforce: baseline</pre><p>→ “<strong>Actually block</strong> any Pod that breaks Baseline rules.” Baseline is fairly relaxed, so most normal Pods will pass this and get created.</p><p>yaml</p><pre>pod-security.kubernetes.io/enforce-version: latest</pre><p>→ Just pins which Kubernetes version’s <em>definition</em> of “baseline” to use. latest means &quot;always use whatever rules are current in this cluster&#39;s k8s version.&quot; (Not really part of the confusion — just metadata.)</p><p>yaml</p><pre>pod-security.kubernetes.io/warn: restricted</pre><p>→ “<strong>Don’t block</strong>, but if a Pod <em>would have failed</em> the stricter Restricted level, show the person running kubectl apply a warning.&quot;</p><p>yaml</p><pre>pod-security.kubernetes.io/audit: restricted</pre><p>→ “<strong>Don’t block</strong>, but if a Pod <em>would have failed</em> Restricted, silently log it” — visible only to whoever checks the cluster’s audit logs, not to the person deploying.</p><h3>Putting It All Together — What Actually Happens</h3><p>When someone creates a Pod in this namespace, <strong>three checks run at the same time</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*wYkblJbKOyh5z_beOU0M1w.png" /></figure><h3>A Concrete Walkthrough</h3><p>Say someone runs:</p><p>yaml</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: demo<br>spec:<br>  containers:<br>  - name: app<br>    image: nginx<br>    securityContext:<br>      privileged: false<br>      # runs as root, no seccomp profile set, capabilities not dropped</pre><p>Here’s what happens:</p><ol><li><strong>Enforce (baseline) check:</strong> This Pod doesn’t do anything baseline forbids (no privileged mode, no host networking, etc.) → ✅ <strong>passes</strong>, Pod gets created.</li><li><strong>Warn (restricted) check:</strong> Restricted requires non-root, dropped capabilities, and a seccomp profile — this Pod has none of that → ❌ fails Restricted, so kubectl prints something like:</li></ol><pre>Warning: would violate PodSecurity &quot;restricted:latest&quot;: allowPrivilegeEscalation != false, <br>   unrestricted capabilities, runAsNonRoot != true, seccompProfile</pre><p>But the Pod <strong>still gets created</strong> — it’s just a heads-up.</p><ol><li><strong>Audit (restricted) check:</strong> Same failure as above, but instead of printing to the terminal, it’s recorded in the cluster’s audit log for later review.</li></ol><h3>Why Would Anyone Set It Up This Way?</h3><p>This is a common <strong>migration pattern</strong>. The idea:</p><ul><li>You’re not ready to fully enforce restricted yet (it might break existing workloads).</li><li>But you <em>want visibility</em> into which Pods would fail if you did.</li><li>So you enforce the lenient level (baseline) to keep things running, while using warn/audit at the strict level (restricted) to surface problems without breaking anything.</li></ul><p>Over time, your team fixes the Pods that show warnings. Once no more warnings appear, you flip enforce to restricted too — now fully locked down with confidence, nothing will break.</p><h3>Simple Analogy</h3><p>Think of it like a spell-checker in a document:</p><ul><li><strong>enforce: baseline</strong> = a strict grammar rule that <strong>blocks you from saving</strong> the file if you violate it (rare violations).</li><li><strong>warn: restricted</strong> = a wavy red underline that says “this <em>could</em> be improved,” but doesn’t stop you from saving.</li><li><strong>audit: restricted</strong> = a hidden log file that records every wavy-underline issue, for someone else to review later — you don’t even see it.</li></ul><h3>Apply PSS to your TaskManager namespace</h3><pre># namespace-taskmanager.yaml<br>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: taskmanager<br>  labels:<br>    # Enforce baseline — pods violating this are REJECTED<br>    pod-security.kubernetes.io/enforce: baseline<br>    pod-security.kubernetes.io/enforce-version: latest<br># Warn if pod would fail restricted - shown in kubectl output<br>    pod-security.kubernetes.io/warn: restricted<br>    pod-security.kubernetes.io/warn-version: latest<br>    # Audit restricted - logged for review without blocking<br>    pod-security.kubernetes.io/audit: restricted<br>    pod-security.kubernetes.io/audit-version: latest</pre><pre>kubectl apply -f namespace-taskmanager.yaml<br># Or label an existing namespace<br>kubectl label namespace taskmanager \<br>  pod-security.kubernetes.io/enforce=baseline \<br>  pod-security.kubernetes.io/warn=restricted</pre><h3>SecurityContext — comply with Restricted PSS</h3><pre># Fully compliant pod for Restricted level<br>spec:<br>  securityContext:<br>    runAsNonRoot: true<br>    runAsUser: 1000<br>    runAsGroup: 3000<br>    fsGroup: 2000<br>    seccompProfile:<br>      type: RuntimeDefault      # required for Restricted<br>containers:<br>    - name: backend<br>      image: task-app:latest<br>      securityContext:<br>        allowPrivilegeEscalation: false   # required for Restricted<br>        readOnlyRootFilesystem: true      # required for Restricted<br>        runAsNonRoot: true<br>        runAsUser: 1000<br>        capabilities:<br>          drop:<br>            - ALL                         # required for Restricted<br>          add:<br>            - NET_BIND_SERVICE            # add back only what you need</pre><h3>Update your Dockerfile for non-root</h3><pre>FROM eclipse-temurin:21-jre-alpine<br>WORKDIR /app<br># Create non-root user - required for Restricted PSS<br>RUN addgroup -S appgroup &amp;&amp; adduser -S appuser -G appgroup<br># Switch to non-root before running<br>USER appuser<br>COPY --from=builder /app/target/*.jar app.jar<br>EXPOSE 8080<br>ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;app.jar&quot;]</pre><h3>Test PSS is working</h3><pre># Try a privileged pod — should be blocked<br>cat &lt;&lt; &#39;EOF&#39; | kubectl apply -f -<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: bad-pod<br>  namespace: taskmanager<br>spec:<br>  containers:<br>    - name: bad<br>      image: nginx<br>      securityContext:<br>        privileged: true<br>EOF<br># Error: violates PodSecurity &quot;baseline:latest&quot;:<br># privileged containers not allowed<br># Apply a compliant pod - should succeed<br>cat &lt;&lt; &#39;EOF&#39; | kubectl apply -f -<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: good-pod<br>  namespace: taskmanager<br>spec:<br>  securityContext:<br>    runAsNonRoot: true<br>    runAsUser: 1000<br>    seccompProfile:<br>      type: RuntimeDefault<br>  containers:<br>    - name: good<br>      image: nginx<br>      securityContext:<br>        allowPrivilegeEscalation: false<br>        readOnlyRootFilesystem: true<br>        capabilities:<br>          drop: [&quot;ALL&quot;]<br>EOF</pre><h3>Common violations and fixes</h3><pre>Violation: privileged container<br>Fix: remove securityContext.privileged: true</pre><pre>Violation: running as root<br>Fix: add runAsNonRoot: true, set runAsUser: 1000 in Dockerfile</pre><pre>Violation: hostPath volume<br>Fix: use emptyDir, configMap, or PVC instead</pre><pre>Violation: missing seccompProfile<br>Fix: add seccompProfile.type: RuntimeDefault</pre><pre>Violation: allowPrivilegeEscalation not false<br>Fix: add allowPrivilegeEscalation: false</pre><pre>Violation: capabilities not dropped<br>Fix: add capabilities.drop: [&quot;ALL&quot;]</pre><h3>Recommended PSS per namespace</h3><pre>kube-system     →  privileged  (system components need full access)<br>monitoring      →  baseline    (prometheus needs some host access)<br>taskmanager     →  baseline enforce + restricted warn<br>production      →  restricted  (maximum security)<br>development     →  baseline    (developers need flexibility)</pre><p>By simply adding the Pod Security Standard labels to a namespace, Kubernetes can enforce the corresponding security policies. The <strong>Pod Security Admission Controller</strong> (enabled by default in Kubernetes 1.25 and later) monitors these namespace labels. Whenever a pod is created, the admission controller validates it against the configured <strong>Baseline</strong> or <strong>Restricted</strong> policies and determines whether the pod should be <strong>allowed, warned, audited, or rejected</strong>. No additional Kubernetes resources or special objects are required. However, it is important to ensure that the Pod Security Admission Controller is enabled. While it is enabled by default in Kubernetes 1.25+, clusters running older versions or custom configurations may ignore these labels.</p><h3>Part 3: Secrets Management</h3><h3>The core problem</h3><p>K8s Secrets are not truly secret out of the box:</p><pre># Anyone with kubectl access can decode a secret instantly<br>kubectl get secret taskmanager-secret -n taskmanager \<br>  -o jsonpath=&#39;{.data.DB_PASSWORD}&#39; | base64 -d<br># root123  ← plain text, no encryption required</pre><p>Base64 is encoding, not encryption. A K8s Secret stored in etcd without encryption at rest is essentially plain text that anyone with etcd access can read.</p><h3>Five levels of secrets management</h3><p><strong>Level 1 — Hardcoded in YAML (never do this)</strong></p><pre># WRONG — visible in Git, kubectl get pod -o yaml, and logs<br>containers:<br>  - name: backend<br>    env:<br>      - name: DB_PASSWORD<br>        value: &quot;root123&quot;</pre><p><strong>Level 2 — K8s native Secret (what most teams start with)</strong></p><pre>kubectl create secret generic taskmanager-secret \<br>  --from-literal=DB_PASSWORD=root123 \<br>  -n taskmanager</pre><pre># Reference in pod — value never appears in YAML<br>env:<br>  - name: DB_PASSWORD<br>    valueFrom:<br>      secretKeyRef:<br>        name: taskmanager-secret<br>        key: DB_PASSWORD</pre><p>Problems with native Secrets:</p><pre>etcd stores base64 — anyone with etcd access reads everything<br>kubectl get secret — anyone with RBAC list/get reads everything<br>no automatic rotation<br>no audit trail of who read the secret</pre><p><strong>Level 3 — etcd encryption at rest</strong></p><p>Encrypts secrets in etcd storage. On EKS:</p><pre>eksctl create cluster \<br>  --name demo-eks \<br>  --region us-east-1 \<br>  --encrypt-secrets \<br>  --encryption-key-arn arn:aws:kms:us-east-1:ACCOUNT:key/KEY-ID</pre><p>For self-managed clusters:</p><pre># /etc/kubernetes/encryption-config.yaml<br>apiVersion: apiserver.config.k8s.io/v1<br>kind: EncryptionConfiguration<br>resources:<br>  - resources:<br>      - secrets<br>    providers:<br>      - aescbc:<br>          keys:<br>            - name: key1<br>              secret: &lt;base64-encoded-32-byte-key&gt;<br>      - identity: {}</pre><p><strong>Level 4 — Sealed Secrets (safe in Git)</strong></p><pre># Install controller and CLI<br>helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system<br>brew install kubeseal</pre><pre># Encrypt a secret — safe to commit to Git<br>kubectl create secret generic db-secret \<br>  --from-literal=DB_PASSWORD=root123 \<br>  --dry-run=client -o yaml | \<br>  kubeseal --format yaml &gt; sealed-db-secret.yaml</pre><pre># Commit the sealed secret<br>git add sealed-db-secret.yaml<br>git commit -m &quot;add database credentials&quot;</pre><pre># Apply — controller decrypts and creates real Secret<br>kubectl apply -f sealed-db-secret.yaml</pre><p><strong>Level 5 — External Secrets Operator (production best practice)</strong></p><pre># Install External Secrets Operator<br>helm install external-secrets \<br>  external-secrets/external-secrets \<br>  -n external-secrets --create-namespace</pre><pre># Connect to AWS Secrets Manager<br>apiVersion: external-secrets.io/v1beta1<br>kind: SecretStore<br>metadata:<br>  name: aws-secret-store<br>  namespace: taskmanager<br>spec:<br>  provider:<br>    aws:<br>      service: SecretsManager<br>      region: us-east-1<br>      auth:<br>        secretRef:<br>          accessKeyIDSecretRef:<br>            name: aws-credentials<br>            key: access-key-id<br>          secretAccessKeySecretRef:<br>            name: aws-credentials<br>            key: secret-access-key</pre><pre>---<br># Sync from AWS into K8s<br>apiVersion: external-secrets.io/v1beta1<br>kind: ExternalSecret<br>metadata:<br>  name: taskmanager-db-secret<br>  namespace: taskmanager<br>spec:<br>  refreshInterval: 1h              # sync every hour — picks up rotations<br>  secretStoreRef:<br>    name: aws-secret-store<br>    kind: SecretStore<br>  target:<br>    name: taskmanager-secret       # K8s Secret created automatically<br>    creationPolicy: Owner<br>  data:<br>    - secretKey: DB_PASSWORD<br>      remoteRef:<br>        key: taskmanager/production<br>        property: DB_PASSWORD<br>    - secretKey: DB_USER<br>      remoteRef:<br>        key: taskmanager/production<br>        property: DB_USER</pre><h3>HashiCorp Vault — dynamic credentials</h3><p>Vault goes beyond storage. It generates short-lived credentials that expire automatically:</p><pre># Enable database secrets engine<br>vault secrets enable database</pre><pre># Configure MySQL connection<br>vault write database/config/mysql \<br>  plugin_name=mysql-database-plugin \<br>  connection_url=&quot;{{username}}:{{password}}@tcp(mysql-service:3306)/&quot; \<br>  allowed_roles=&quot;taskmanager-role&quot; \<br>  username=&quot;vault-admin&quot; \<br>  password=&quot;vault-admin-password&quot;</pre><pre># Create role — generates credentials valid for 1 hour only<br>vault write database/roles/taskmanager-role \<br>  db_name=mysql \<br>  creation_statements=&quot;CREATE USER &#39;{{name}}&#39;@&#39;%&#39; IDENTIFIED BY &#39;{{password}}&#39;; GRANT SELECT, INSERT, UPDATE ON taskdb.* TO &#39;{{name}}&#39;@&#39;%&#39;;&quot; \<br>  default_ttl=&quot;1h&quot; \<br>  max_ttl=&quot;24h&quot;</pre><pre># Vault agent sidecar injects fresh credentials before each pod starts<br>spec:<br>  annotations:<br>    vault.hashicorp.com/agent-inject: &quot;true&quot;<br>    vault.hashicorp.com/role: &quot;taskmanager-role&quot;<br>    vault.hashicorp.com/agent-inject-secret-db: &quot;database/creds/taskmanager-role&quot;</pre><h3>RBAC for Secrets — restrict access</h3><pre># Only this ServiceAccount can read only this specific secret<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: Role<br>metadata:<br>  name: secret-reader<br>  namespace: taskmanager<br>rules:<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;secrets&quot;]<br>    verbs: [&quot;get&quot;]                 # get only — not list<br>    resourceNames:<br>      - &quot;taskmanager-secret&quot;       # ONLY this specific secret</pre><p>Never grant these in production:</p><pre>verbs: [&quot;list&quot;]              # exposes all secret names<br>verbs: [&quot;watch&quot;]             # streams all secret changes<br>resources: [&quot;secrets&quot;]       # without resourceNames = all secrets</pre><h3>Mount as a file, not an env var</h3><pre># Environment variables appear in:<br># - /proc/&lt;pid&gt;/environ inside the container<br># - crash dumps and core files<br># - some logging frameworks accidentally log all env vars</pre><pre># More secure — mount as a file<br>volumeMounts:<br>  - name: db-secret-vol<br>    mountPath: /secrets<br>    readOnly: true</pre><pre>volumes:<br>  - name: db-secret-vol<br>    secret:<br>      secretName: taskmanager-secret<br>      defaultMode: 0400   # owner read-only — nobody else can read</pre><pre># Spring Boot reads from file:<br># spring.datasource.password=${file:/secrets/DB_PASSWORD}</pre><h3>Best practices summary</h3><pre>1. Never hardcode secrets in YAML or Dockerfiles<br>2. Never commit plain K8s Secret YAML to Git<br>   → use Sealed Secrets or External Secrets instead<br>3. Enable etcd encryption at rest<br>4. Use RBAC resourceNames to restrict secret access<br>5. Use External Secrets with cloud managers for cloud clusters<br>6. Use Vault for dynamic short-lived credentials in production<br>7. Rotate secrets regularly — Vault automates this<br>8. Audit who accessed secrets — Vault provides full audit log<br>9. Never log secrets — set Spring Boot log level carefully<br>10. Mount as files not env vars wherever possible</pre><h3>Part 4: Admission Controllers and Webhooks</h3><h3>What admission controllers do</h3><p>Every kubectl apply passes through three gates before being saved to etcd:</p><pre>kubectl apply<br>    ↓<br>1. Authentication  — who are you? (certificate, token, OIDC)<br>    ↓<br>2. Authorization   — are you allowed? (RBAC)<br>    ↓<br>3. Admission Control — should this be accepted? ← this section<br>    ↓<br>etcd — saved to cluster state</pre><p>Admission controllers intercept requests after AuthN/AuthZ. They can modify the object (mutating) or reject it (validating) before it is saved.</p><h3>Mutating vs validating — order matters</h3><pre>Mutating webhooks run FIRST<br>  → can modify the request (add labels, inject containers, set defaults)<br>  → Istio uses this to inject Envoy sidecar into every pod<br>  → resource-defaults webhook uses this to add CPU/memory limits</pre><pre>Validating webhooks run SECOND<br>  → can only approve or reject — cannot modify<br>  → OPA Gatekeeper uses this to enforce policies<br>  → image validator uses this to block unsigned images</pre><p>This order is critical. Mutating runs first so that validating webhooks see the final modified object — not the raw user input.</p><h3>Built-in admission controllers</h3><p>K8s ships with built-in controllers enabled by default:</p><pre>LimitRanger         →  enforces resource limits and defaults<br>ResourceQuota       →  enforces namespace resource quotas<br>NamespaceLifecycle  →  prevents objects in terminating namespaces<br>ServiceAccount      →  auto-mounts service account tokens<br>NodeRestriction     →  limits what kubelets can modify<br>PodSecurity         →  enforces Pod Security Standards<br>DefaultStorageClass →  adds default storage class to PVCs<br>MutatingAdmission   →  runs your custom mutating webhooks<br>ValidatingAdmission →  runs your custom validating webhooks</pre><h3>Custom validating webhook — require resource limits</h3><pre>// webhook-server.js<br>const express = require(&#39;express&#39;);<br>const app = express();<br>app.use(express.json());</pre><pre>app.post(&#39;/validate-resources&#39;, (req, res) =&gt; {<br>  const pod = req.body.request.object;<br>  const containers = pod.spec?.containers || [];</pre><pre>  for (const container of containers) {<br>    if (!container.resources?.limits?.memory) {<br>      return res.json({<br>        apiVersion: &#39;admission.k8s.io/v1&#39;,<br>        kind: &#39;AdmissionReview&#39;,<br>        response: {<br>          uid: req.body.request.uid,<br>          allowed: false,           // REJECT the pod<br>          status: {<br>            message: `Container ${container.name} missing memory limit`<br>          }<br>        }<br>      });<br>    }<br>  }</pre><pre>  res.json({<br>    apiVersion: &#39;admission.k8s.io/v1&#39;,<br>    kind: &#39;AdmissionReview&#39;,<br>    response: {<br>      uid: req.body.request.uid,<br>      allowed: true               // ALLOW the pod<br>    }<br>  });<br>});</pre><pre>app.listen(8443);</pre><p>Register it with K8s:</p><pre># validating-webhook.yaml<br>apiVersion: admissionregistration.k8s.io/v1<br>kind: ValidatingWebhookConfiguration<br>metadata:<br>  name: resource-limits-validator<br>webhooks:<br>  - name: validate-resources.taskmanager.io<br>    rules:<br>      - operations: [&quot;CREATE&quot;, &quot;UPDATE&quot;]<br>        apiGroups: [&quot;&quot;]<br>        apiVersions: [&quot;v1&quot;]<br>        resources: [&quot;pods&quot;]<br>    clientConfig:<br>      service:<br>        name: webhook-service<br>        namespace: taskmanager<br>        path: /validate-resources<br>      caBundle: &lt;base64-encoded-CA-cert&gt;<br>    failurePolicy: Fail          # webhook down = reject all requests<br>    namespaceSelector:<br>      matchLabels:<br>        kubernetes.io/metadata.name: taskmanager<br>    admissionReviewVersions: [&quot;v1&quot;]<br>    sideEffects: None</pre><h3>Custom mutating webhook — inject default labels</h3><pre>app.post(&#39;/inject-labels&#39;, (req, res) =&gt; {<br>  const patch = [<br>    { op: &#39;add&#39;, path: &#39;/metadata/labels/team&#39;, value: &#39;backend-team&#39; },<br>    { op: &#39;add&#39;, path: &#39;/metadata/labels/environment&#39;, value: &#39;production&#39; }<br>  ];</pre><pre>  res.json({<br>    apiVersion: &#39;admission.k8s.io/v1&#39;,<br>    kind: &#39;AdmissionReview&#39;,<br>    response: {<br>      uid: req.body.request.uid,<br>      allowed: true,<br>      patchType: &#39;JSONPatch&#39;,<br>      patch: Buffer.from(JSON.stringify(patch)).toString(&#39;base64&#39;)<br>    }<br>  });<br>});</pre><h3>OPA Gatekeeper — policy as code</h3><pre># Install Gatekeeper<br>kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.14/deploy/gatekeeper.yaml</pre><pre># Step 1 — define the policy template<br>apiVersion: templates.gatekeeper.sh/v1<br>kind: ConstraintTemplate<br>metadata:<br>  name: requireresourcelimits<br>spec:<br>  crd:<br>    spec:<br>      names:<br>        kind: RequireResourceLimits<br>  targets:<br>    - target: admission.k8s.gatekeeper.sh<br>      rego: |<br>        package requireresourcelimits</pre><pre>        violation[{&quot;msg&quot;: msg}] {<br>          container := input.review.object.spec.containers[_]<br>          not container.resources.limits.memory<br>          msg := sprintf(&quot;Container %v is missing memory limit&quot;, [container.name])<br>        }</pre><pre>---<br># Step 2 — enforce the policy<br>apiVersion: constraints.gatekeeper.sh/v1beta1<br>kind: RequireResourceLimits<br>metadata:<br>  name: require-resource-limits<br>spec:<br>  match:<br>    kinds:<br>      - apiGroups: [&quot;&quot;]<br>        kinds: [&quot;Pod&quot;]<br>    namespaces:<br>      - taskmanager</pre><h3>Kyverno — policy as YAML (simpler than OPA)</h3><pre># kyverno-policy.yaml<br>apiVersion: kyverno.io/v1<br>kind: ClusterPolicy<br>metadata:<br>  name: taskmanager-policy<br>spec:<br>  validationFailureAction: Enforce<br>  rules:<br>    # Require team label on every pod<br>    - name: require-team-label<br>      match:<br>        resources:<br>          kinds: [&quot;Pod&quot;]<br>      validate:<br>        message: &quot;Pod must have a team label&quot;<br>        pattern:<br>          metadata:<br>            labels:<br>              team: &quot;?*&quot;</pre><pre>    # Auto-add environment label<br>    - name: add-environment-label<br>      match:<br>        resources:<br>          kinds: [&quot;Pod&quot;]<br>          namespaces: [&quot;taskmanager&quot;]<br>      mutate:<br>        patchStrategicMerge:<br>          metadata:<br>            labels:<br>              environment: production</pre><pre>    # Require non-root containers<br>    - name: require-non-root<br>      match:<br>        resources:<br>          kinds: [&quot;Pod&quot;]<br>      validate:<br>        message: &quot;Containers must not run as root&quot;<br>        pattern:<br>          spec:<br>            containers:<br>              - securityContext:<br>                  runAsNonRoot: true</pre><h3>failurePolicy — critical decision</h3><pre>failurePolicy: Fail    # webhook down → reject ALL requests<br>                       # safest for security — nothing sneaks through<br>                       # risk: webhook outage blocks all deployments</pre><pre>failurePolicy: Ignore  # webhook down → allow ALL requests<br>                       # deployments never blocked by outage<br>                       # risk: security bypassed during outage</pre><p>Production recommendation:</p><pre>Security-critical validating webhooks  →  failurePolicy: Fail<br>Non-critical mutating webhooks         →  failurePolicy: Ignore</pre><h3>Debug webhooks</h3><pre># See all webhook configurations<br>kubectl get mutatingwebhookconfigurations<br>kubectl get validatingwebhookconfigurations</pre><pre># Test policy before enforcing<br>kubectl apply -f pod.yaml --dry-run=server</pre><pre># See Gatekeeper violations<br>kubectl get constraints<br>kubectl describe requireresourcelimits require-resource-limits</pre><pre># See Kyverno policy reports<br>kubectl get policyreport -A</pre><pre># Check if webhook is blocking something<br>kubectl describe pod bad-pod -n taskmanager<br># Events section shows the exact rejection message</pre><h3>The four layers work together</h3><pre>Developer runs:  kubectl apply -f deployment.yaml<br>                              ↓<br>RBAC check:      is this user allowed to create Deployments?<br>                 NO → 403 Forbidden<br>                 YES → continue<br>                              ↓<br>Mutating webhooks run:<br>  Istio injects Envoy sidecar<br>  Resource-defaults adds cpu/memory limits<br>  Label-injector adds team and environment labels<br>                              ↓<br>Validating webhooks run:<br>  OPA checks: does pod have resource limits? (after mutation added them ✅)<br>  PSS checks: does pod run as non-root? (checked against namespace label)<br>  Image validator: is this image signed?<br>  NO to any → 403 rejected<br>  YES to all → continue<br>                              ↓<br>Object saved to etcd — pod scheduled and running<br>                              ↓<br>Pod runtime:<br>  Container runs as non-root (PSS + SecurityContext)<br>  Secrets pulled from Vault (External Secrets Operator)<br>  ServiceAccount has only the permissions it needs (RBAC)</pre><h3>Quick reference</h3><p>Layer Protects against Key objects RBAC Unauthorised API access Role, ClusterRole, RoleBinding, ServiceAccount Pod Security Standards Container privilege escalation Namespace labels, SecurityContext Secrets Management Credential exposure ExternalSecret, SealedSecret, Vault Admission Controllers Policy violations at deploy time ValidatingWebhookConfiguration, OPA, Kyverno</p><h3>Conclusion</h3><p>Security in Kubernetes is not a single switch — it is four overlapping layers that each catch a different class of threat.</p><p><strong>RBAC</strong> stops the wrong person from doing the wrong thing. Start with zero permissions and add only what is needed. Use kubectl auth can-i to audit regularly.</p><p><strong>Pod Security Standards</strong> stop containers from escaping their sandbox. Start with baseline enforcement and warn on restricted areas to find violations before you enforce them.</p><p><strong>Secrets Management</strong> stops credentials from being read by the wrong person. Never commit plain secrets to Git. Use External Secrets with your cloud provider or Sealed Secrets for GitOps. Use Vault for dynamic short-lived credentials in high-security environments.</p><p><strong>Admission Controllers</strong> enforce policy at the moment of deployment — before anything reaches production. OPA Gatekeeper and Kyverno are the most popular options. Mutating webhooks inject sane defaults automatically, so developers do not need to remember every security setting.</p><p>None of this is optional in a production cluster. The good news is that each layer can be added incrementally — start with RBAC and PSS, add Sealed Secrets when you set up GitOps, add Gatekeeper or Kyverno when you need organisation-wide policy enforcement.</p><p><em>All YAML in this article comes from a real TaskManager app (React + Spring Boot + MySQL) running on Minikube and EKS. Every security control was tested against a live cluster.</em></p><p><em>If this helped you secure your cluster, leave a clap.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=596246165cea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Autoscaling and Scheduling Deep Dive: HPA, VPA, Cluster Autoscaler, and Affinity]]></title>
            <link>https://medium.com/@laxmanpajjuri/kubernetes-autoscaling-and-scheduling-deep-dive-hpa-vpa-cluster-autoscaler-and-affinity-2feac95d5127?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2feac95d5127</guid>
            <category><![CDATA[hpa]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[vpa]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Sat, 16 May 2026 12:09:58 GMT</pubDate>
            <atom:updated>2026-05-16T12:09:58.986Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>A complete practical guide to making your Kubernetes cluster self-managing — with real YAML, real errors, and real fixes from a running TaskManager app</em></blockquote><p>One of the most powerful things about Kubernetes is that it can manage itself. Traffic spikes at 9 am — pods scale up automatically. Traffic drops at midnight — pods and nodes scale back down to save cost. Your app always gets the right resources — not your initial guess, but actual measured values.</p><p>This article covers the four features that make this possible:</p><ul><li><strong>HPA</strong> — automatically scales pod count based on CPU and memory</li><li><strong>Cluster Autoscaler</strong> — automatically scales node count</li><li><strong>VPA</strong> — automatically right-sizes pod resource requests</li><li><strong>Node and Pod Affinity</strong> — controls where pods are placed</li></ul><h3>Part 1: HorizontalPodAutoscaler (HPA)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*_Demrz1ixDTPW4l4v-lw1Q.png" /></figure><h3>What problem HPA solves</h3><pre>Without HPA:<br>  9am  — traffic spikes — 2 pods overwhelmed — users see 500 errors<br>  You notice alert at 9:15am — run kubectl scale — too late<br>With HPA:<br>  9am  - traffic spikes - CPU hits 70%<br>  9:00:15am - HPA already adding pod-3<br>  9:01am  - pod-3 Running - traffic distributed - users unaffected</pre><h3>The scaling formula</h3><p>HPA uses this formula every 15 seconds:</p><pre>desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric)<br>Scale up example:<br>  current replicas: 2<br>  current CPU:      90%<br>  target CPU:       70%<br>  desired = ceil(2 × 90/70) = ceil(2.57) = 3  →  scale to 3 pods<br>Scale down example:<br>  current replicas: 5<br>  current CPU:      20%<br>  target CPU:       70%<br>  desired = ceil(5 × 20/70) = ceil(1.43) = 2  →  scale to 2 pods</pre><h3>Full HPA YAML for your TaskManager backend</h3><pre># hpa-backend.yaml<br>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: backend-hpa<br>  namespace: taskmanager<br>spec:<br>  # Which Deployment to scale<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: backend<br># Never go below 2 or above 10 pods<br>  minReplicas: 2<br>  maxReplicas: 10<br>  metrics:<br>    # Scale on CPU - safe and predictable<br>    - type: Resource<br>      resource:<br>        name: cpu<br>        target:<br>          type: Utilization<br>          averageUtilization: 70    # keep average CPU at 70%<br>  # Prevent flapping - scale fast up, slow down<br>  behavior:<br>    scaleUp:<br>      stabilizationWindowSeconds: 15     # scale up fast<br>      policies:<br>        - type: Pods<br>          value: 2                       # add max 2 pods at a time<br>          periodSeconds: 60<br>    scaleDown:<br>      stabilizationWindowSeconds: 300    # scale down slow - 5 min window<br>      policies:<br>        - type: Pods<br>          value: 1                       # remove max 1 pod at a time<br>          periodSeconds: 60</pre><h3>Why HPA on memory is risky</h3><p>A common question is whether HPA can scale on memory. Technically, yes — but it has a critical problem:</p><pre>CPU is compressible:<br>  High CPU → add pod → CPU spreads across pods → usage drops → HPA scales down ✅<br>Memory is incompressible:<br>  High memory → add pod → OLD pod STILL holds its memory<br>  Memory doesn&#39;t drop → HPA sees still high → adds MORE pods<br>  Infinite scale up loop until maxReplicas! ❌</pre><p>The rule:</p><pre>HPA  →  CPU utilisation   (safe — compressible resource)<br>VPA  →  memory right-size (safe — handles incompressible resource)</pre><h3>Install Metrics Server (required for HPA)</h3><pre># On Minikube<br>minikube addons enable metrics-server</pre><pre># Verify metrics are flowing<br>kubectl top pods -n taskmanager<br># NAME               CPU(cores)   MEMORY<br># backend-xxx        45m          380Mi</pre><h3>Apply and watch HPA work</h3><pre># Apply HPA<br>kubectl apply -f hpa-backend.yaml</pre><pre># Check HPA status<br>kubectl get hpa -n taskmanager<br># NAME          REFERENCE             TARGETS    MINPODS   MAXPODS   REPLICAS<br># backend-hpa   Deployment/backend    45%/70%    2         10        2</pre><pre># Simulate traffic spike — watch HPA respond<br>kubectl run load-test \<br>  --image=busybox:1.36 \<br>  --restart=Never \<br>  -n taskmanager \<br>  -- sh -c &quot;while true; do wget -q -O- <a href="http://backend-service:8080/api/tasks;">http://backend-service:8080/api/tasks;</a> done&quot;</pre><pre># Watch in real time<br>kubectl get hpa -n taskmanager -w<br># TARGETS goes 45%/70% → 90%/70% → scaling!<br># REPLICAS goes 2 → 3 → 4</pre><pre># Stop load test<br>kubectl delete pod load-test -n taskmanager</pre><pre># Watch scale down (takes 5 minutes — stabilization window)<br>kubectl get hpa -n taskmanager -w</pre><h3>Common HPA issues</h3><pre>TARGETS shows unknown/70%<br>  → Metrics Server not installed<br>  → Resource requests not set on Deployment (required for % calculation)</pre><pre>HPA never scales down<br>  → stabilizationWindowSeconds too high (default 300s)<br>  → CPU never drops below target</pre><pre>Pods crash after scale up<br>  → Memory limit too low → OOMKilled under load<br>  → Fix: increase memory limit or tune JVM heap</pre><h3>Part 2: Cluster Autoscaler</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*uSE6yH9ip2ElHx198yZynw.png" /></figure><h3>What problem Cluster Autoscaler solves</h3><p>HPA adds pods. But what if the cluster runs out of nodes to place them on?</p><pre>HPA creates 5 new backend pods<br>    ↓<br>Scheduler: no node has enough CPU for these pods<br>    ↓<br>Pods stuck in Pending state — not serving traffic<br>    ↓<br>Cluster Autoscaler sees Pending pods<br>    ↓<br>Calls cloud API: &quot;add 2 more EC2 nodes&quot;<br>    ↓<br>New nodes ready in ~2 minutes<br>    ↓<br>Scheduler places pending pods — all Running ✅</pre><p>And in reverse:</p><pre>Traffic drops → HPA removes pods<br>    ↓<br>Node-3 is now 80% empty — wasting $0.04/hr<br>    ↓<br>Cluster Autoscaler: can those remaining pods fit elsewhere?<br>    YES → evict pods from Node-3<br>    ↓<br>Pods rescheduled on Node-1 and Node-2<br>    ↓<br>Node-3 terminated — cost saved ✅</pre><h3>HPA + Cluster Autoscaler working together</h3><pre>9am — traffic spike<br>  HPA:  2 pods → 8 pods (CPU too high)<br>  CA:   2 nodes → 4 nodes (pods can&#39;t schedule)</pre><pre>12am — traffic drops<br>  HPA:  8 pods → 2 pods (CPU too low)<br>  CA:   4 nodes → 2 nodes (nodes now idle)</pre><p>This is the complete self-managing cluster — no human intervention needed for scaling.</p><h3>Scale up flow — step by step</h3><pre>1. HPA creates new pods<br>2. Scheduler tries to place them — no capacity on existing nodes<br>3. Pods go to Pending state<br>4. CA checks every 10 seconds for Pending pods<br>5. CA simulates: &quot;which node group could fit these pods?&quot;<br>6. CA calls cloud API to add nodes<br>7. New EC2 instances boot (~2 minutes)<br>8. Scheduler places pending pods on new nodes<br>9. Pods: Pending → Running</pre><h3>Scale down flow — step by step</h3><pre>1. CA checks nodes every 10 seconds<br>2. Node utilisation below 50% for 10 continuous minutes<br>3. CA checks: can all pods on this node fit elsewhere?<br>   YES → safe to remove<br>   NO  → keep the node (protects against downtime)<br>4. CA evicts pods gracefully<br>5. Pods rescheduled on remaining nodes<br>6. CA terminates the empty node</pre><h3>Pods CA will NOT evict</h3><pre>1. Pods with annotation safe-to-evict: false<br>2. Pods with local storage (emptyDir, hostPath)<br>3. Pods protected by PodDisruptionBudget (minAvailable)<br>4. Pods in kube-system namespace (if configured)<br>5. Standalone pods not managed by a controller</pre><h3>Install CA on EKS</h3><pre># Tag your Auto Scaling Group (CA uses these tags to find node groups)<br>aws autoscaling create-or-update-tags \<br>  --tags \<br>    ResourceId=&lt;your-asg-name&gt; \<br>    ResourceType=auto-scaling-group \<br>    Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=true \<br>    ResourceId=&lt;your-asg-name&gt; \<br>    ResourceType=auto-scaling-group \<br>    Key=k8s.io/cluster-autoscaler/demo-eks,Value=owned,PropagateAtLaunch=true</pre><pre># Deploy Cluster Autoscaler<br>kubectl apply -f <a href="https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml">https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml</a></pre><pre># Update cluster name<br>kubectl set env deployment cluster-autoscaler \<br>  -n kube-system \<br>  CLUSTER_NAME=demo-eks</pre><h3>Full CA configuration</h3><pre>containers:<br>  - name: cluster-autoscaler<br>    image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.31.0<br>    command:<br>      - ./cluster-autoscaler<br>      - --cloud-provider=aws<br>      - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/demo-eks<br>      # Scale down config<br>      - --scale-down-utilization-threshold=0.5   # remove if below 50%<br>      - --scale-down-unneeded-time=10m           # idle for 10 minutes<br>      - --scale-down-delay-after-add=10m         # wait after scaling up<br>      # Safety<br>      - --skip-nodes-with-local-storage=true<br>      - --skip-nodes-with-system-pods=true</pre><h3>Protect critical pods from eviction</h3><pre># MySQL pod — never evict the database!<br>metadata:<br>  annotations:<br>    cluster-autoscaler.kubernetes.io/safe-to-evict: &quot;false&quot;<br># Backend pods - CA can evict freely<br>metadata:<br>  annotations:<br>    cluster-autoscaler.kubernetes.io/safe-to-evict: &quot;true&quot;</pre><h3>PodDisruptionBudget — always keep a minimum number of pods running</h3><pre># pdb-backend.yaml<br>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: backend-pdb<br>  namespace: taskmanager<br>spec:<br>  minAvailable: 1       # always keep at least 1 pod running<br>  selector:<br>    matchLabels:<br>      app: backend</pre><h3>Watch CA in action</h3><pre># See current nodes<br>kubectl get nodes</pre><pre># Exhaust capacity — force CA to add nodes<br>kubectl scale deployment backend --replicas=20 -n taskmanager</pre><pre># Watch pods go Pending<br>kubectl get pods -n taskmanager -w<br># backend-xxx   0/1   Pending   0   ← no room!</pre><pre># Watch CA add nodes<br>kubectl get nodes -w<br># ip-10-0-1-102   NotReady   &lt;none&gt;   10s  ← CA just added this<br># ip-10-0-1-102   Ready      &lt;none&gt;   90s  ← now ready</pre><pre># Scale back down<br>kubectl scale deployment backend --replicas=2 -n taskmanager</pre><pre># Wait 10 min — watch CA remove the empty node<br>kubectl get nodes -w</pre><pre># See CA decisions in logs<br>kubectl logs -n kube-system \<br>  -l app=cluster-autoscaler \<br>  --follow<br># &quot;Scale up triggered&quot;<br># &quot;Removing node ip-xxx&quot;</pre><h3>Part 3: VerticalPodAutoscaler (VPA)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*WCGe5WY6FJ8iNTYxh3E0mg.png" /></figure><h3>What problem VPA solves</h3><p>When you write Kubernetes YAML, you guess resource values:</p><pre>resources:<br>  requests:<br>    cpu: &quot;250m&quot;      # ← your guess<br>    memory: &quot;512Mi&quot;  # ← your guess<br>  limits:<br>    cpu: &quot;1000m&quot;<br>    memory: &quot;1Gi&quot;</pre><p>Your guess is almost always wrong. Over-provisioned pods waste money. Under-provisioned pods get OOMKilled. VPA watches your actual usage and tells you — or automatically sets — the correct values.</p><h3>VPA vs HPA — completely different</h3><pre>HPA:  traffic spikes  → add MORE pods (same size)<br>VPA:  pod uses 120m   → reduce request FROM 250m TO 120m (right size)<br>      pod uses 380Mi  → reduce request FROM 512Mi TO 380Mi (right size)</pre><h3>Three VPA components</h3><p><strong>Recommender</strong> — watches metrics server, analyzes historical usage, calculates optimal resource values. Updates status.recommendation in the VPA object.</p><p><strong>Updater</strong> — checks if running pods match the recommendation. If too far off, it evicts the pod so it restarts with correct resources.</p><p><strong>Admission Controller</strong> — intercepts pod creation and injects recommended values before the pod starts.</p><h3>VPA YAML</h3><pre># vpa-backend.yaml<br>apiVersion: autoscaling.k8s.io/v1<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: backend-vpa<br>  namespace: taskmanager<br>spec:<br>  targetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: backend<br>updatePolicy:<br>    updateMode: &quot;Off&quot;     # start here - recommend only, no changes<br>  resourcePolicy:<br>    containerPolicies:<br>      - containerName: backend<br>        minAllowed:<br>          cpu: &quot;100m&quot;<br>          memory: &quot;256Mi&quot;<br>        maxAllowed:<br>          cpu: &quot;2000m&quot;<br>          memory: &quot;2Gi&quot;<br>        controlledResources:<br>          - cpu<br>          - memory</pre><h3>Four update modes</h3><pre>Off       →  show recommendations only<br>             YOU apply the values manually — zero downtime<br>             best for: learning, production first step</pre><pre>Initial   →  set resources at pod creation only<br>             no eviction of running pods<br>             best for: stateful apps that can&#39;t be restarted</pre><pre>Recreate  →  evict pods when recommendation differs significantly<br>             at least one pod stays running (with enough replicas)<br>             best for: Deployments with 2+ replicas</pre><pre>Auto      →  same as Recreate today<br>             future: in-place resize without restart<br>             best for: stateless apps comfortable with rolling restarts</pre><h3>Does VPA cause downtime?</h3><pre>Mode Off      →  zero downtime (no pod changes)<br>Mode Initial  →  zero downtime (only new pods)<br>Mode Recreate →  brief gap per pod (30-60 seconds)<br>               replicas &gt; 1 + PDB = minimal impact</pre><pre>Future K8s 1.27+ in-place resize:<br>  VPA can update resources WITHOUT pod restart<br>  zero downtime — coming to Auto mode when stable</pre><h3>VPA in practice — step by step</h3><pre># Step 1 — install VPA<br>git clone https://github.com/kubernetes/autoscaler.git<br>cd autoscaler/vertical-pod-autoscaler<br>./hack/vpa-up.sh</pre><pre># Verify<br>kubectl get pods -n kube-system | grep vpa<br># vpa-admission-controller   1/1   Running<br># vpa-recommender            1/1   Running<br># vpa-updater                1/1   Running</pre><pre># Step 2 — apply VPA in Off mode<br>kubectl apply -f vpa-backend.yaml</pre><pre># Step 3 — generate traffic for history<br>kubectl run load-test \<br>  --image=busybox:1.36 \<br>  --restart=Never \<br>  -n taskmanager \<br>  -- sh -c &quot;for i in $(seq 1 200); do wget -q -O- <a href="http://backend-service:8080/api/tasks;">http://backend-service:8080/api/tasks;</a> done&quot;</pre><pre># Step 4 — check recommendation after 10 minutes<br>kubectl describe vpa backend-vpa -n taskmanager<br># Status:<br>#   Recommendation:<br>#     Container Recommendations:<br>#       Container Name: backend<br>#       Lower Bound:<br>#         Cpu:     80m<br>#         Memory:  300Mi<br>#       Target:            ← apply these values<br>#         Cpu:     120m<br>#         Memory:  380Mi<br>#       Upper Bound:<br>#         Cpu:     600m<br>#         Memory:  800Mi</pre><pre># Step 5 — apply recommendation via rolling update (zero downtime)<br>kubectl set resources deployment backend \<br>  -n taskmanager \<br>  --requests=cpu=120m,memory=380Mi \<br>  --limits=cpu=600m,memory=700Mi</pre><pre>kubectl rollout status deployment/backend -n taskmanager</pre><h3>Combining HPA and VPA safely</h3><pre>WRONG — conflict:<br>  HPA scaling on CPU + VPA adjusting CPU requests<br>  VPA changes the baseline HPA uses → unstable scaling</pre><pre>RIGHT — no conflict:<br>  HPA on CPU          → controls pod count<br>  VPA on memory only  → right-sizes memory</pre><pre>RIGHT — no conflict:<br>  HPA on requests/sec (custom metric) → controls pod count<br>  VPA on CPU + memory                 → right-sizes everything</pre><pre># Safe combination in VPA<br>resourcePolicy:<br>  containerPolicies:<br>    - containerName: backend<br>      controlledResources:<br>        - memory          # VPA only manages memory<br>      controlledValues: RequestsOnly<br>      # CPU left entirely to HPA</pre><h3>Part 4: Node Affinity and Pod Affinity</h3><h3>Why do you need affinity rules</h3><p>Without affinity, all backend replicas might land on the same node. If that node dies, all replicas die together. Zero high availability.</p><p>Without affinity, your database might land on a slow HDD node instead of an SSD. Performance suffers.</p><p>Affinity lets you control where pods run.</p><h3>Three types</h3><pre>Node affinity      →  pod chooses WHICH NODE to run on<br>                      &quot;schedule me only on disk=ssd nodes&quot;</pre><pre>Pod affinity       →  pod runs NEAR other pods<br>                      &quot;schedule me on same node as frontend pod&quot;</pre><pre>Pod anti-affinity  →  pod runs FAR from other pods<br>                      &quot;never put two backend replicas on same node&quot;</pre><h3>Two strictness levels</h3><pre>required   →  hard rule — pod stays Pending if no match found<br>              use when: hardware requirement is non-negotiable (MySQL needs SSD)</pre><pre>preferred  →  soft rule — try to match, schedule anywhere if no match<br>              use when: preference, not requirement (prefer same zone)</pre><h3>Label your nodes first</h3><pre># Label nodes with their hardware and zone properties<br>kubectl label node node-1 disk=ssd zone=us-east-1a<br>kubectl label node node-2 disk=hdd zone=us-east-1b<br>kubectl label node node-3 disk=ssd gpu=true</pre><pre># Verify<br>kubectl get nodes --show-labels</pre><h3>Node affinity YAML</h3><pre># MySQL must run on SSD nodes<br>spec:<br>  affinity:<br>    nodeAffinity:<br># HARD RULE - Pending if no SSD node available<br>      requiredDuringSchedulingIgnoredDuringExecution:<br>        nodeSelectorTerms:<br>          - matchExpressions:<br>              - key: disk<br>                operator: In<br>                values:<br>                  - ssd<br>      # SOFT RULE - prefer us-east-1a but schedule anywhere<br>      preferredDuringSchedulingIgnoredDuringExecution:<br>        - weight: 80              # 0-100, higher = stronger preference<br>          preference:<br>            matchExpressions:<br>              - key: zone<br>                operator: In<br>                values:<br>                  - us-east-1a</pre><h3>Node affinity operators</h3><pre>operator: In          # label value is in this list<br>operator: NotIn       # label value is NOT in this list<br>operator: Exists      # label exists (any value)<br>operator: DoesNotExist # label does not exist<br>operator: Gt          # numeric greater than<br>operator: Lt          # numeric less than</pre><h3>Pod affinity — co-locate for low latency</h3><p>Frontend and backend on the same node = no network hop between them = lower latency:</p><pre># backend — prefer same node as frontend<br>spec:<br>  affinity:<br>    podAffinity:<br>      preferredDuringSchedulingIgnoredDuringExecution:<br>        - weight: 100<br>          podAffinityTerm:<br>            labelSelector:<br>              matchExpressions:<br>                - key: app<br>                  operator: In<br>                  values:<br>                    - frontend<br>            topologyKey: &quot;kubernetes.io/hostname&quot;  # same node</pre><h3>Pod anti-affinity — spread for high availability</h3><p>Never put two backend replicas on the same node — if a node dies you still have pods running:</p><pre># backend — spread replicas across nodes (HA)<br>spec:<br>  affinity:<br>    podAntiAffinity:<br>      # HARD RULE — one backend pod per node maximum<br>      requiredDuringSchedulingIgnoredDuringExecution:<br>        - labelSelector:<br>            matchExpressions:<br>              - key: app<br>                operator: In<br>                values:<br>                  - backend<br>          topologyKey: &quot;kubernetes.io/hostname&quot;</pre><p>Result:</p><pre>3 nodes:<br>  Node 1 → backend-pod-1   ✅<br>  Node 2 → backend-pod-2   ✅<br>  Node 3 → backend-pod-3   ✅</pre><pre>Try to add 4th backend pod with only 3 nodes:<br>  All nodes blocked → pod stays Pending<br>  Add a 4th node → Cluster Autoscaler handles this!</pre><h3>topologyKey — what near and spread mean</h3><pre>topologyKey: &quot;kubernetes.io/hostname&quot;         # same physical node<br>topologyKey: &quot;topology.kubernetes.io/zone&quot;    # same availability zone<br>topologyKey: &quot;topology.kubernetes.io/region&quot;  # same region</pre><h3>Full production YAML — all three affinities together</h3><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: backend<br>  namespace: taskmanager<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: backend<br>  template:<br>    metadata:<br>      labels:<br>        app: backend<br>    spec:<br>      affinity:<br># 1 - prefer SSD nodes for better IO<br>        nodeAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - weight: 80<br>              preference:<br>                matchExpressions:<br>                  - key: disk<br>                    operator: In<br>                    values:<br>                      - ssd<br>        # 2 - prefer same node as frontend (low latency)<br>        podAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - weight: 50<br>              podAffinityTerm:<br>                labelSelector:<br>                  matchLabels:<br>                    app: frontend<br>                topologyKey: &quot;kubernetes.io/hostname&quot;<br>        # 3 - never two backends on same node (HA)<br>        podAntiAffinity:<br>          requiredDuringSchedulingIgnoredDuringExecution:<br>            - labelSelector:<br>                matchLabels:<br>                  app: backend<br>              topologyKey: &quot;kubernetes.io/hostname&quot;<br>      containers:<br>        - name: backend<br>          image: task-app:latest<br>          imagePullPolicy: Never<br>          resources:<br>            requests:<br>              cpu: &quot;250m&quot;<br>              memory: &quot;512Mi&quot;<br>            limits:<br>              cpu: &quot;1000m&quot;<br>              memory: &quot;1Gi&quot;</pre><h3>Taints and Tolerations — the opposite of affinity</h3><p>Affinity is the pod saying “I want to go there.” Taints are the node saying “you can’t come here unless you tolerate me.”</p><pre># Reserve node-3 for GPU workloads only<br>kubectl taint node node-3 gpu=true:NoSchedule</pre><pre># Normal pods → blocked from node-3<br># GPU pods → allowed if they have this toleration:</pre><pre>spec:<br>  tolerations:<br>    - key: &quot;gpu&quot;<br>      operator: &quot;Equal&quot;<br>      value: &quot;true&quot;<br>      effect: &quot;NoSchedule&quot;</pre><h3>Debug scheduling failures</h3><pre># Pod stuck in Pending? Find out why<br>kubectl describe pod &lt;pod-name&gt; -n taskmanager<br># Events section shows:<br># Warning  FailedScheduling  0/3 nodes available:<br>#   1 node(s) didn&#39;t match pod affinity rules<br>#   2 node(s) had untolerated taint</pre><pre># Add missing label to fix<br>kubectl label node minikube disk=ssd</pre><h3>Quick decision guide</h3><pre>Need pod on specific hardware (SSD, GPU)?<br>  → nodeAffinity required</pre><pre>Prefer a zone but OK anywhere?<br>  → nodeAffinity preferred</pre><pre>Frontend and backend close for speed?<br>  → podAffinity preferred, topologyKey: hostname</pre><pre>Backend replicas never share a node?<br>  → podAntiAffinity required, topologyKey: hostname</pre><pre>Reserve nodes for specific workloads?<br>  → Taints + Tolerations</pre><h3>Everything working together — your TaskManager</h3><pre>9am — traffic spike hits</pre><pre>HPA:  backend 2 pods → 6 pods (CPU hit 85%)<br>CA:   2 nodes → 3 nodes (new pods can&#39;t schedule)<br>Affinity: each backend pod lands on different node (anti-affinity)<br>          pods prefer SSD nodes (node affinity)</pre><pre>12pm — traffic normalizes</pre><pre>HPA:  backend 6 pods → 2 pods (CPU back to 30%)<br>CA:   3 nodes → 2 nodes (node-3 idle for 10 min)</pre><pre>Nightly — VPA checks recommendations<br>VPA:  sees backend only used 120m CPU not 250m<br>      recommends: reduce cpu request from 250m to 120m<br>      you apply during next maintenance window<br>      each node now fits MORE pods → CA needs fewer nodes → cost savings</pre><h3>Full comparison table</h3><p>Feature HPA VPA Cluster Autoscaler Affinity What it changes Pod count Pod size Node count Pod placement Trigger CPU/memory/custom Historical usage Pending/idle Your rules Speed Seconds Minutes 2–5 minutes At scheduling Downtime None Brief (if Auto mode) None None Cost impact Medium High savings High savings None Use for Traffic spikes Right-sizing Node capacity HA/hardware</p><h3>Conclusion</h3><p>These four features together make Kubernetes truly self-managing:</p><p><strong>HPA</strong> responds to demand in seconds — adding pods when traffic spikes and removing them when it drops. Always use CPU as the primary metric, not memory.</p><p><strong>Cluster Autoscaler</strong> responds to capacity in minutes — adding nodes when pods can’t schedule and removing them when they sit idle. Pair it with PodDisruptionBudgets to protect against data loss during scale down.</p><p><strong>VPA</strong> responds to data over time — replacing your guessed resource values with measured ones. Use it in Off mode first to learn, then apply recommendations manually until you’re comfortable with Auto mode.</p><p><strong>Affinity rules</strong> respond to your architecture requirements — spreading replicas for high availability, co-locating services for low latency, and reserving specific hardware for specific workloads.</p><p>None of this requires changing your application code. Spring Boot keeps calling mysql-service:3306. React keeps calling /api/tasks. Kubernetes handles the scaling, placement, and right-sizing automatically underneath.</p><h3>Diagrams to add in Medium</h3><p>Add screenshots of these diagrams from the conversation at these points:</p><ol><li><strong>After HPA introduction</strong>, the HPA scaling flow diagram showing metrics server, HPA controller, and Deployment</li><li><strong>After the HPA formula</strong>, the scale-up and scale-down scenario boxes</li><li><strong>After Cluster Autoscaler introduction</strong> — the CA scale-up and scale-down flow diagram</li><li><strong>After VPA introduction</strong> — the VPA three components diagram (Recommender, Updater, Admission Controller)</li><li><strong>After VPA modes</strong> — the before/after resource comparison</li><li><strong>After Affinity introduction</strong> — the node affinity, pod affinity, and anti-affinity diagram</li><li><strong>After full production YAML</strong> — the required vs preferred diagram</li></ol><p><em>All YAML and commands in this article come from a real TaskManager app (React + Spring Boot + MySQL) deployed on Minikube on Windows. Every scaling scenario was tested against a live cluster.</em></p><p><em>If this helped you understand K8s autoscaling, leave a clap.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2feac95d5127" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Networking Deep Dive: Pod Communication, DNS, Ingress, and NetworkPolicy]]></title>
            <link>https://medium.com/@laxmanpajjuri/kubernetes-networking-deep-dive-pod-communication-dns-ingress-and-networkpolicy-e28ec36f08a5?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e28ec36f08a5</guid>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[fqdn]]></category>
            <category><![CDATA[network-policies]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Thu, 14 May 2026 16:58:57 GMT</pubDate>
            <atom:updated>2026-05-16T12:10:52.407Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Everything you need to know about how traffic flows inside and outside a Kubernetes cluster — with real YAML from a running TaskManager app</em></blockquote><p>Networking is the part of Kubernetes that confuses most people. Pods talking to pods, services resolving by name, traffic entering from the internet, and security policies blocking unwanted connections — it all feels like magic until you understand the model underneath.</p><p>This article breaks down four core networking concepts with real examples from a Spring Boot + MySQL + React app running on Minikube:</p><ul><li>Pod-to-pod communication — how pods talk to each other</li><li>DNS and service discovery — how pods find each other by name</li><li>Ingress and Ingress controllers — how external traffic enters the cluster</li><li>NetworkPolicy — how to control who can talk to whom</li></ul><h3>Part 1: Pod-to-Pod Communication</h3><h3>The flat network model</h3><p>Kubernetes enforces three golden rules for networking, regardless of which underlying network plugin you use:</p><pre>Rule 1 — Every pod gets its own unique IP address<br>Rule 2 — Pods communicate directly — no NAT (Network Address Translation)<br>Rule 3 — Any pod can reach any other pod anywhere in the cluster</pre><p>This is called the <strong>flat network model</strong>. From any pod’s perspective, every other pod is directly reachable by IP — just like machines on the same LAN, even if they are on completely different physical nodes.</p><h3>How a pod gets its IP</h3><p>When Kubernetes schedules a pod onto a node, this happens automatically:</p><pre>Scheduler assigns pod to Node 1<br>    ↓<br>kubelet calls the CNI plugin (Calico / Flannel / Cilium)<br>    ↓<br>CNI creates a veth pair — a virtual ethernet cable<br>    one end lives inside the pod&#39;s network namespace<br>    one end connects to the node&#39;s bridge (cbr0)<br>    ↓<br>CNI assigns an IP from that node&#39;s subnet<br>    Node 1 pods → 192.168.1.0/24<br>    Node 2 pods → 192.168.2.0/24<br>    ↓<br>Pod is now reachable at that IP from anywhere in the cluster</pre><h3>Same-node communication — direct via bridge</h3><p>When your frontend pod (192.168.1.10) calls your backend pod (192.168.1.11) and both are on the same node:</p><pre>Frontend pod<br>    ↓  packet to 192.168.1.11<br>veth pair out of pod namespace<br>    ↓<br>cbr0 bridge (virtual switch on the node)<br>    ↓  destination is on same bridge — direct delivery<br>veth pair into backend pod namespace<br>    ↓<br>Backend pod receives the packet</pre><pre>No network hops. Latency: microseconds.</pre><h3>Cross-node communication — via cluster network</h3><p>When your backend pod (192.168.1.11) on Node 1 calls your MySQL pod (192.168.2.10) on Node 2:</p><pre>Backend pod (Node 1)<br>    ↓  packet to 192.168.2.10<br>veth pair → cbr0 bridge on Node 1<br>    ↓  destination not on this node&#39;s bridge<br>Node 1 routing table → 192.168.2.0/24 is on Node 2<br>    ↓<br>Physical / virtual cluster network between nodes<br>    ↓<br>Node 2 routing table → 192.168.2.10 is here<br>    ↓<br>cbr0 bridge on Node 2 → veth pair<br>    ↓<br>MySQL pod receives the packet</pre><pre>One network hop. Latency: sub-millisecond on same AZ.</pre><h3>CNI plugins — who does the actual work</h3><p>The CNI (Container Network Interface) plugin implements the three golden rules. Kubernetes defines the contract, CNI delivers it:</p><p>CNI Plugin Mechanism Best for Calico BGP routing Production, NetworkPolicy Flannel VXLAN overlay Simple clusters, learning Cilium eBPF High performance, advanced policy WeaveNet Gossip protocol Encrypted by default</p><p>Your Minikube cluster uses Calico. Your pod YAML never changes regardless of which CNI you use — the networking is completely transparent to your application.</p><h3>See it in action</h3><pre># See actual pod IPs assigned by CNI<br>kubectl get pods -n taskmanager -o wide<br># NAME           IP              NODE<br># backend-xxx    192.168.1.11    minikube<br># frontend-xxx   192.168.1.10    minikube<br># mysql-pod      192.168.1.12    minikube</pre><pre># Ping between pods directly by IP<br>kubectl exec -it &lt;frontend-pod&gt; -n taskmanager -- \<br>  ping 192.168.1.11</pre><pre># See the node&#39;s network bridges and veth pairs<br>minikube ssh -- bridge link show<br>minikube ssh -- ip route show</pre><h3>Part 2: DNS and Service Discovery</h3><h3>The problem DNS solves</h3><p>Pod IPs change every time a pod restarts, scales, or gets rescheduled. You can never hardcode pod IPs in your application. This is why Services exist — and DNS is how you find them.</p><pre>Without DNS:  backend connects to 192.168.1.12:3306 ← changes on restart<br>With DNS:     backend connects to mysql-service:3306 ← never changes</pre><h3>CoreDNS — the DNS server inside K8s</h3><p>CoreDNS runs as a Deployment inside the kube-system namespace. Its ClusterIP (usually 10.96.0.10) is automatically injected into every pod&#39;s /etc/resolv.conf as the nameserver.</p><pre># See CoreDNS running<br>kubectl get pods -n kube-system | grep coredns</pre><pre># See CoreDNS Service — this is the DNS server IP<br>kubectl get svc -n kube-system | grep kube-dns<br># kube-dns   ClusterIP   10.96.0.10   53/UDP,53/TCP</pre><h3>What is inside every pod’s DNS config</h3><pre># Shell into your backend pod<br>kubectl exec -it &lt;backend-pod&gt; -n taskmanager -- cat /etc/resolv.conf</pre><pre># Output:<br>nameserver 10.96.0.10                           ← CoreDNS<br>search taskmanager.svc.cluster.local \          ← search these domains<br>       svc.cluster.local \                         in order<br>       cluster.local<br>options ndots:5</pre><p>The search line is why mysql-service works as a short name — K8s automatically appends each search domain and tries to resolve until it succeeds.</p><h3>Full DNS resolution flow</h3><p>When your Spring Boot backend calls jdbc:mysql://mysql-service:3306/taskdb:</p><pre>1. Pod calls: mysql-service<br>              ↓<br>2. Pod asks CoreDNS at 10.96.0.10:<br>   &quot;What is the IP for mysql-service?&quot;<br>              ↓<br>3. CoreDNS searches:<br>   mysql-service.taskmanager.svc.cluster.local → found! 10.96.1.25<br>              ↓<br>4. CoreDNS returns 10.96.1.25 to the pod<br>              ↓<br>5. Pod connects to 10.96.1.25:3306<br>   MySQL Service receives the connection and routes to the MySQL pod</pre><h3>FQDN structure — every service gets one</h3><p>Every Kubernetes Service gets a DNS entry following this exact pattern:</p><pre>&lt;service-name&gt;.&lt;namespace&gt;.svc.cluster.local</pre><pre>Your TaskManager services:<br>mysql-service.taskmanager.svc.cluster.local   → 10.96.1.25<br>backend-service.taskmanager.svc.cluster.local → 10.96.2.10<br>frontend-service.taskmanager.svc.cluster.local → 10.96.3.15</pre><p>You can use short forms within the same namespace:</p><pre># All these work from inside the taskmanager namespace:<br>mysql-service                                    ← shortest (same ns)<br>mysql-service.taskmanager                        ← with namespace<br>mysql-service.taskmanager.svc                    ← with type<br>mysql-service.taskmanager.svc.cluster.local      ← full FQDN</pre><pre># From a DIFFERENT namespace — must include namespace at minimum:<br>mysql-service.taskmanager</pre><h3>application.properties uses short DNS name</h3><pre># Spring Boot just uses the short service name<br># CoreDNS resolves it to the actual ClusterIP<br>spring.datasource.url=jdbc:mysql://mysql-service:3306/taskdb</pre><h3>Headless Service — DNS for StatefulSets</h3><p>A regular Service DNS returns the ClusterIP (virtual IP). A headless Service returns actual pod IPs directly. StatefulSets use this so each pod gets its own stable DNS name:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mysql-headless<br>  namespace: taskmanager<br>spec:<br>  clusterIP: None    # ← headless — no virtual IP<br>  selector:<br>    app: mysql<br>  ports:<br>    - port: 3306</pre><p>With headless service each StatefulSet pod gets its own DNS entry:</p><pre>mysql-0.mysql-headless.taskmanager.svc.cluster.local → pod-IP-1<br>mysql-1.mysql-headless.taskmanager.svc.cluster.local → pod-IP-2<br>mysql-2.mysql-headless.taskmanager.svc.cluster.local → pod-IP-3</pre><p>This is how MySQL replicas find each other. mysql-0 is always the primary — stable name, stable identity, regardless of what IP the pod gets.</p><h3>Verify DNS in your cluster</h3><pre># Run a temporary DNS test pod<br>kubectl run dns-test --image=busybox:1.36 --rm -it \<br>  --restart=Never -n taskmanager \<br>  -- sh</pre><pre># Inside the pod<br>nslookup mysql-service<br># Server:  10.96.0.10<br># Name:    mysql-service.taskmanager.svc.cluster.local<br># Address: 10.96.1.25  ← the ClusterIP</pre><pre>nslookup backend-service.taskmanager<br>nslookup google.com    # external DNS also works via CoreDNS forward</pre><pre>exit</pre><h3>Part 3: Ingress and Ingress Controllers</h3><h3>Why Ingress exists — the cost problem</h3><p>Without Ingress every service that needs external access requires its own cloud load balancer:</p><pre>Without Ingress:<br>  frontend  → LoadBalancer → $0.02/hr<br>  backend   → LoadBalancer → $0.02/hr<br>  monitoring → LoadBalancer → $0.02/hr<br>  Total: 3 load balancers ≈ $43/month</pre><pre>With Ingress:<br>  ONE LoadBalancer → Ingress Controller → routes to all services<br>  Total: 1 load balancer ≈ $14/month</pre><p>Ingress also gives you host-based routing, path-based routing, TLS termination, and redirects — none of which a basic Service can provide.</p><h3>Ingress vs Ingress Controller — they are different things</h3><pre>Ingress resource    =  the RULES you write in YAML<br>Ingress Controller  =  the thing that READS and ENFORCES those rules</pre><pre>Without an Ingress Controller your Ingress YAML does nothing.<br>The rules exist but nobody reads them.</pre><h3>How traffic flows with Ingress</h3><pre>Internet request: GET http://taskmanager.local/api/tasks<br>    ↓<br>Cloud Load Balancer (or NodePort on Minikube)<br>    ↓<br>Ingress Controller pod (Nginx running in cluster)<br>    ↓  reads Ingress rules<br>    path /api → backend-service:8080<br>    path /    → frontend-service:80<br>    ↓<br>Routes to correct Service<br>    ↓<br>Service routes to healthy Pod<br>    ↓<br>Response returns same path</pre><h3>Full Ingress YAML for TaskManager</h3><pre># ingress-taskmanager.yaml<br>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: taskmanager-ingress<br>  namespace: taskmanager<br>  annotations:<br>    # Which Ingress Controller handles this<br>    nginx.ingress.kubernetes.io/rewrite-target: /<br>    # Redirect HTTP to HTTPS automatically<br>    nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;</pre><pre>spec:<br>  ingressClassName: nginx    # use Nginx Ingress Controller</pre><pre>  # TLS — HTTPS termination at the Ingress<br>  tls:<br>    - hosts:<br>        - taskmanager.local<br>      secretName: taskmanager-tls   # Secret with cert and key</pre><pre>  rules:<br>    - host: taskmanager.local</pre><pre>      http:<br>        paths:<br>          # /api → Spring Boot backend<br>          - path: /api<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: backend-service<br>                port:<br>                  number: 8080</pre><pre>          # / → React frontend (catches everything else)<br>          - path: /<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: frontend-service<br>                port:<br>                  number: 80</pre><h3>Install Nginx Ingress on Minikube</h3><pre># Minikube has it built in<br>minikube addons enable ingress</pre><pre># Wait for controller to be Running (~2 minutes)<br>kubectl get pods -n ingress-nginx -w</pre><pre># Verify ingress class<br>kubectl get ingressclass<br># NAME    CONTROLLER<br># nginx   k8s.io/ingress-nginx</pre><h3>Apply and test on Minikube</h3><pre>kubectl apply -f ingress-taskmanager.yaml</pre><pre># Check ingress created<br>kubectl get ingress -n taskmanager<br># NAME                  CLASS   HOSTS               ADDRESS        PORTS<br># taskmanager-ingress   nginx   taskmanager.local   192.168.49.2   80</pre><pre># Add to hosts file (Linux/Mac)<br>echo &quot;$(minikube ip)  taskmanager.local&quot; | sudo tee -a /etc/hosts</pre><pre># Test routing<br>curl <a href="http://taskmanager.local/">http://taskmanager.local/</a>           # → frontend HTML<br>curl <a href="http://taskmanager.local/api/tasks">http://taskmanager.local/api/tasks</a>  # → Spring Boot JSON</pre><h3>Path types — important difference</h3><pre># Exact — only this exact path<br>pathType: Exact<br>path: /api/tasks   # matches ONLY /api/tasks<br>                   # does NOT match /api/tasks/1</pre><pre># Prefix — path and everything under it<br>pathType: Prefix<br>path: /api         # matches /api<br>                   # matches /api/tasks<br>                   # matches /api/tasks/1/comments</pre><h3>Host-based routing — multiple apps on one cluster</h3><pre>rules:<br>  # App 1 — TaskManager<br>  - host: taskmanager.local<br>    http:<br>      paths:<br>        - path: /<br>          pathType: Prefix<br>          backend:<br>            service:<br>              name: frontend-service<br>              port:<br>                number: 80</pre><pre>  # App 2 — Grafana on the same cluster<br>  - host: grafana.local<br>    http:<br>      paths:<br>        - path: /<br>          pathType: Prefix<br>          backend:<br>            service:<br>              name: grafana-service<br>              port:<br>                number: 3000</pre><p>Two completely different apps, one Ingress Controller, one load balancer. Traffic is routed by the Host HTTP header.</p><h3>Popular Ingress Controllers</h3><p>Controller Best for Nginx Ingress General purpose, Minikube, most teams Traefik Auto-discovery, microservices AWS ALB EKS on AWS — native cloud integration Istio Gateway Service mesh, advanced traffic management HAProxy Very high performance, financial services</p><h3>Part 4: NetworkPolicy — Kubernetes Firewall</h3><h3>The default is wide open</h3><p>By default every pod in Kubernetes can talk to every other pod with no restrictions. Your frontend can directly query MySQL. A compromised pod can reach every service in the cluster.</p><pre>No NetworkPolicy:<br>frontend pod → mysql pod  ← allowed (risky!)<br>any pod      → any pod    ← allowed (very risky!)</pre><p>NetworkPolicy is a firewall rule for pods. It says “this pod can only receive traffic from X and can only send traffic to Y.”</p><h3>Start with deny-all</h3><p>Best practice is to lock everything down first then explicitly allow only what is needed:</p><pre># 01-deny-all.yaml<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: deny-all<br>  namespace: taskmanager<br>spec:<br>  podSelector: {}      # empty = selects ALL pods in namespace</pre><pre>  policyTypes:<br>    - Ingress          # block all incoming<br>    - Egress           # block all outgoing</pre><pre>  # No rules = deny everything<br>  # This is intentional</pre><pre>kubectl apply -f 01-deny-all.yaml<br># Every pod is now completely isolated<br># Nothing can reach anything<br># Now selectively open what is needed</pre><h3>Allow frontend → backend only</h3><pre># 02-allow-frontend-to-backend.yaml<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-frontend-to-backend<br>  namespace: taskmanager<br>spec:<br>  # This policy protects the backend pod<br>  podSelector:<br>    matchLabels:<br>      app: backend<br>policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        # Only pods with this label can call backend<br>        - podSelector:<br>            matchLabels:<br>              app: frontend<br>      ports:<br>        - protocol: TCP<br>          port: 8080</pre><h3>Allow backend → MySQL only</h3><pre># 03-allow-backend-to-mysql.yaml<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-backend-to-mysql<br>  namespace: taskmanager<br>spec:<br>  # This policy protects the MySQL pod<br>  podSelector:<br>    matchLabels:<br>      app: mysql<br>policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        # Only backend pods can reach MySQL<br>        - podSelector:<br>            matchLabels:<br>              app: backend<br>      ports:<br>        - protocol: TCP<br>          port: 3306</pre><h3>Allow backend egress — including DNS</h3><p>Backend needs to make outgoing calls. Always include a DNS rule or nothing resolves:</p><pre># 04-allow-backend-egress.yaml<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-backend-egress<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: backend<br>policyTypes:<br>    - Egress<br>  egress:<br>    # Allow calling MySQL<br>    - to:<br>        - podSelector:<br>            matchLabels:<br>              app: mysql<br>      ports:<br>        - protocol: TCP<br>          port: 3306<br>    # CRITICAL - allow DNS resolution<br>    # Without this nothing resolves by name<br>    - to:<br>        - namespaceSelector:<br>            matchLabels:<br>              kubernetes.io/metadata.name: kube-system<br>      ports:<br>        - protocol: UDP<br>          port: 53<br>    # Allow external HTTPS calls (payment APIs etc)<br>    - to:<br>        - ipBlock:<br>            cidr: 0.0.0.0/0<br>            except:<br>              - 10.0.0.0/8     # except internal range<br>      ports:<br>        - protocol: TCP<br>          port: 443</pre><h3>Apply all policies in order</h3><pre>kubectl apply -f 01-deny-all.yaml<br>kubectl apply -f 02-allow-frontend-to-backend.yaml<br>kubectl apply -f 03-allow-backend-to-mysql.yaml<br>kubectl apply -f 04-allow-backend-egress.yaml</pre><pre># Allow Ingress controller to reach frontend<br>cat &lt;&lt; &#39;EOF&#39; | kubectl apply -f -<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-ingress-to-frontend<br>  namespace: taskmanager<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: frontend<br>  policyTypes:<br>    - Ingress<br>  ingress:<br>    - from:<br>        - namespaceSelector:<br>            matchLabels:<br>              kubernetes.io/metadata.name: ingress-nginx<br>      ports:<br>        - port: 80<br>EOF</pre><h3>Verify NetworkPolicy is enforced</h3><pre># See all policies<br>kubectl get networkpolicy -n taskmanager<br># Test frontend CAN reach backend (should succeed)<br>kubectl exec -it &lt;frontend-pod&gt; -n taskmanager -- \<br>  wget -qO- http://backend-service:8080/api/tasks<br># Test frontend CANNOT reach MySQL (should fail/timeout)<br>kubectl exec -it &lt;frontend-pod&gt; -n taskmanager -- \<br>  nc -zv mysql-service 3306<br># Expected: connection timed out or refused<br># Test backend CAN reach MySQL (should succeed)<br>kubectl exec -it &lt;backend-pod&gt; -n taskmanager -- \<br>  nc -zv mysql-service 3306<br># Expected: Connection to mysql-service 3306 port [tcp] succeeded</pre><h3>Three selector types in NetworkPolicy</h3><pre>ingress:<br>  - from:<br>      # 1 — podSelector: allow specific pods by label<br>      - podSelector:<br>          matchLabels:<br>            app: frontend<br># 2 - namespaceSelector: allow entire namespace<br>      - namespaceSelector:<br>          matchLabels:<br>            env: production<br>      # 3 - ipBlock: allow external IP range<br>      - ipBlock:<br>          cidr: 10.0.0.0/8<br>          except:<br>            - 10.1.0.0/16    # except this subnet</pre><h3>Most common mistake — forgetting DNS egress</h3><pre># WRONG — pod cannot resolve hostnames!<br>egress:<br>  - to:<br>      - podSelector:<br>          matchLabels:<br>            app: mysql<br>    ports:<br>      - port: 3306<br># RIGHT - always add DNS alongside other egress rules<br>egress:<br>  - to:<br>      - podSelector:<br>          matchLabels:<br>            app: mysql<br>    ports:<br>      - port: 3306<br>  - to:                          # DNS must be a separate rule<br>      - namespaceSelector:<br>          matchLabels:<br>            kubernetes.io/metadata.name: kube-system<br>    ports:<br>      - protocol: UDP<br>        port: 53</pre><p>Without the DNS rule mysql-service cannot be resolved to an IP and every connection fails — even though the MySQL policy looks correct.</p><h3>NetworkPolicy requires a supporting CNI</h3><p>NetworkPolicy only works if the CNI plugin enforces it:</p><p>CNI NetworkPolicy Calico ✅ Full support Cilium ✅ Full support + extras WeaveNet ✅ Full support Flannel ❌ No support — policies apply but do nothing</p><h3>Putting it all together — your TaskManager traffic flow</h3><pre>User opens http://taskmanager.local<br>    ↓<br>DNS resolves taskmanager.local → Minikube IP (via /etc/hosts)<br>    ↓<br>Nginx Ingress Controller receives request<br>    ↓  Ingress rule: / → frontend-service:80<br>NetworkPolicy: ingress-nginx namespace allowed → frontend pod<br>    ↓<br>Frontend pod serves React HTML/JS<br>    ↓<br>Browser runs React, calls /api/tasks<br>    ↓<br>Nginx Ingress: /api → backend-service:8080<br>NetworkPolicy: frontend pod allowed → backend pod<br>    ↓<br>Spring Boot backend receives request<br>    ↓<br>Backend calls jdbc:mysql://mysql-service:3306/taskdb<br>CoreDNS resolves mysql-service → 10.96.1.25 (ClusterIP)<br>NetworkPolicy: backend pod allowed → mysql pod<br>    ↓<br>MySQL pod processes query, returns data<br>    ↓<br>Response travels back the same path<br>    ↓<br>User sees their tasks</pre><h3>Quick reference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*gbf55amlSDHDGaTwRTFYIQ.png" /></figure><p>Concept: What it does Your TaskManager example Pod network Flat L3 network, every pod has unique IP All pods reachable cluster-wide CNI plugin Implements pod networking Calico on Minikube CoreDNS Resolves service names to ClusterIPs mysql-service → 10.96.1.25 Short DNS name Works within same namespace mysql-service Full FQDN Works cross-namespace mysql-service.taskmanager.svc.cluster.local Headless service DNS returns pod IPs directly StatefulSet MySQL replicas Ingress resource HTTP routing rules /api → backend, / → frontend Ingress Controller Enforces Ingress rules Nginx in ingress-nginx namespace NetworkPolicy Pod firewall frontend → backend → mysql only deny-all Block everything, allow selectively First policy applied</p><h3>Conclusion</h3><p>Kubernetes networking feels complex but follows a clean mental model once you see it clearly:</p><ul><li><strong>Flat pod network</strong> — any pod talks to any pod by IP, CNI makes it work</li><li><strong>DNS</strong> — CoreDNS resolves service names so you never hardcode IPs</li><li><strong>Ingress</strong> — one entry point routes all external HTTP traffic by host and path</li><li><strong>NetworkPolicy</strong> — firewall rules that lock down which pods can talk to which</li></ul><p>The most important takeaway: your application code never deals with any of this. Spring Boot calls mysql-service:3306. React calls /api/tasks. Kubernetes handles everything in between — name resolution, routing, load balancing, and security.</p><p><em>All examples in this article come from a real TaskManager app (React + Spring Boot + MySQL) running on Minikube on Windows. Every command was tested against a live cluster.</em></p><p><em>If this helped you understand K8s networking, leave a clap — it helps more people find this.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e28ec36f08a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CPU Throttling, Server Capping, and Server Choking — explained simply with real examples]]></title>
            <link>https://medium.com/@laxmanpajjuri/when-your-server-hits-the-wall-9725f0050985?source=rss-4add164debc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9725f0050985</guid>
            <category><![CDATA[performance-engineering]]></category>
            <category><![CDATA[servers]]></category>
            <category><![CDATA[performance-testing]]></category>
            <dc:creator><![CDATA[Laxman Pajjuri]]></dc:creator>
            <pubDate>Thu, 14 May 2026 16:30:07 GMT</pubDate>
            <atom:updated>2026-06-05T05:20:41.601Z</atom:updated>
            <content:encoded><![CDATA[<p><em>CPU Throttling, Server Capping, and Server Choking — explained simply with real examples</em></p><p>You’re running a load test. Response times look fine for the first 60 seconds — then suddenly everything slows down. Is it your code? Your database? Your network? Maybe. But there’s a quieter, sneakier culprit that kills performance tests silently: the CPU never got a fair chance to work.</p><p>In this post, we’ll walk through three concepts that every performance engineer must understand — <strong>CPU Throttling</strong>, <strong>Server Capping</strong>, and <strong>Server Choking</strong>. They sound similar but are fundamentally different problems with different root causes and different fixes.</p><h3>CPU Throttling</h3><p>CPU throttling is when the system intentionally limits how much CPU time a process can use — even when more CPU power is physically available on the machine.</p><blockquote><em>The CPU has capacity. But your process has been given a speed limiter — like a car that can do 120 mph but is governed to 40.</em></blockquote><h4>How it actually works — the time window model</h4><p>Linux uses a scheduler called <strong>CFS (Completely Fair Scheduler)</strong>. It gives every process a time window and a quota of CPU time inside that window.</p><pre>cpu.cfs_period_us = 100000   # Window size = 100ms<br>cpu.cfs_quota_us  = 50000    # Your allowed CPU time = 50ms<br><br># Translation: you get 0.5 CPU cores effectively<br># Use your 50ms early? You are FROZEN for the rest.</pre><p>This freeze is the throttle. It happens silently — no error, no log, no warning. Your process just stops getting scheduled until the next window opens.</p><pre>Time ──────────────────────────────────────────▶<br><br>Window 1 (100ms)       Window 2 (100ms)<br>|───────────────────────|───────────────────────|<br>[ RUNNING  ][ FROZEN   ][ RUNNING  ][ FROZEN   ]<br><br>RUNNING = your process is executing<br>FROZEN  = quota exhausted, forcibly paused</pre><p>From the outside, this looks like random latency spikes. Requests take longer. GC pauses stretch. But your CPU monitor shows “normal” — because the process is using its full quota, just not continuously.</p><h4>Real-world example: JVM in Kubernetes</h4><pre>resources:<br>  requests:<br>    cpu: &quot;500m&quot;<br>  limits:<br>    cpu: &quot;1&quot;     # 1 core limit</pre><p>The JVM doesn’t know about this limit. It looks at the node and sees 32 cores. It spins up 32 GC threads. But when GC kicks in, all those threads hit the quota wall — the GC gets throttled mid-collection, causing GC pauses to stretch from 50ms to 500ms+.</p><blockquote><em>💡 Analogy: You hired 32 cleaners to tidy your office. But they’re only allowed to work 30 minutes per hour total — so they spend half the time waiting outside. The office takes forever to clean, not because they’re slow, but because they’re throttled.</em></blockquote><p><strong>The fix:</strong> Tell the JVM the truth about available CPUs.</p><pre>-XX:ActiveProcessorCount=1</pre><p>How to detect throttling</p><pre># Check cgroup throttle stats inside your container<br>cat /sys/fs/cgroup/cpu/cpu.stat<br><br># Output to look for:<br>nr_periods     1000<br>nr_throttled    420    # 42% of periods were throttled!<br>throttled_time  8400000000    # nanoseconds frozen</pre><pre># Prometheus query for Kubernetes monitoring<br>rate(container_cpu_cfs_throttled_periods_total[5m])<br>  /<br>rate(container_cpu_cfs_periods_total[5m])<br><br># If this is &gt; 0.25 (25%), you have a throttling problem</pre><h3>Server Capping</h3><p>Server capping is a deliberate ceiling placed on a server’s total resources — CPU, memory, network, or disk — ensuring the workload never exceeds its allocation regardless of what’s available on the machine.</p><p>Where throttling is about time slices, capping is about total budget. It’s a policy decision, usually made at the infrastructure level.</p><blockquote><em>💡 Analogy: Imagine a highway with 8 lanes, but you’re only allowed to drive in 2. The other 6 are empty, right there — but they’re off-limits to you by rule. That’s capping.</em></blockquote><h4>Where you see capping in practice</h4><p><strong>Kubernetes</strong> — resources.limits in pod spec → Pod capped at 2 CPU cores on a 64-core node</p><p><strong>Docker</strong> — --cpus and --memory flags → docker run --cpus=0.5 myapp</p><p><strong>AWS EC2</strong> — Instance type limits → t3.medium = 2 vCPUs, even on a 96-core host</p><p><strong>Shared hosting</strong> — Tenant isolation → Each tenant gets 10% of server resources</p><p><strong>JMeter agents</strong> — VM resource limits → Load generator capped, can’t hit target TPS</p><h4>The sneaky AWS burstable instance trap</h4><p>AWS T-series instances (t3, t2) are burstable. They run at full speed when you have CPU credits. Exhaust the credits and they’re hard-capped at a baseline — sometimes as low as 10–20% of a core.</p><pre># t3.micro baseline = only 10% of 1 vCPU when credits run out<br>Before credits exhausted:  TPS = 450, p99 = 120ms<br>After credits exhausted:   TPS = 87,  p99 = 980ms  ← same code!</pre><p>⚠️ Always check whether your SUT or load generator is on a burstable instance before running a sustained load test. T-series instances will give you misleading results. Use compute-optimized instances (C5, C6) for performance benchmarking.</p><h4>Capping vs Throttling</h4><p><strong>CPU Throttling</strong> — CPU time per window — enforced by CFS quota — like a speed camera that stops you mid-journey</p><p><strong>Server Capping</strong> — Total resource budget — enforced by policy or instance type — like a speed limit sign before you start</p><h3>Server Choking</h3><p>Server choking is when a server is genuinely overwhelmed — it has no artificial limits, it simply cannot handle the load. Unlike capping and throttling, choking is not a policy. It’s the server hitting its real physical ceiling.</p><blockquote><em>Capping is a rule. Throttling enforces a quota. Choking is the server screaming for help.</em></blockquote><blockquote><em>💡 Analogy: A waiter handles 5 tables comfortably. You add 50 tables at once. He doesn’t stop — but orders get delayed, some get wrong, some customers walk out. He’s not limited by a rule. He’s genuinely maxed out. That’s choking.</em></blockquote><h4>What happens when a server chokes</h4><p>→ <strong>CPU sustains 95–100%</strong> — no headroom, every cycle is used</p><p>→ <strong>Thread pool exhausted</strong> — all workers busy, new requests queue up</p><p>→ <strong>Memory climbs continuously</strong> — heap fills up, GC runs constantly but can’t keep up</p><p>→ <strong>Response times spike sharply</strong> — requests wait in queue before even being processed</p><p>→ <strong>Connection refused / 503 errors</strong> — server stops accepting new connections</p><p>→ <strong>Cascading failures</strong> — downstream services time out, triggering a chain reaction</p><h4>A real choking scenario in a Spring Boot app</h4><pre># Normal state — healthy<br>Thread pool active:  12 / 200<br>Heap used:          1.2 GB / 4 GB<br>GC time:            0.3%<br>p99 latency:        95ms<br># 10x load spike - choking begins<br>Thread pool active:  200 / 200  ← pool exhausted<br>Queue size:          4800        ← requests piling up<br>Heap used:          3.9 GB / 4 GB  ← near OOM<br>GC time:            68%         ← GC thrashing<br>p99 latency:        12,400ms    ← server is choking</pre><h4>Choking is what you WANT to find in load testing</h4><p>Unlike capping and throttling — which are problems to fix — choking is the goal of certain test types:</p><p><strong>Stress Test</strong> — Push until the server chokes — find the breaking point</p><p><strong>Capacity Test</strong> — Find the safe load level just before choking begins</p><p><strong>Soak Test</strong> — Check if server chokes slowly over hours (memory leaks, thread leaks)</p><p><strong>Spike Test</strong> — Does a sudden 10x load choke the server instantly?</p><h3>The Full Picture — Side by Side</h3><p><strong>Throttling</strong> — Process frozen periodically because it hit its CPU time quota within a scheduling window. Resources available, but access is time-sliced. Fix: raise quota or reduce thread count.</p><p><strong>Capping</strong> — Hard ceiling placed on total resources by policy — cloud instance type, K8s limits, Docker flags. Resources may exist on the host but are off-limits. Fix: resize instance or update limits.</p><p><strong>Choking</strong> — Server genuinely overwhelmed — no artificial limits, just real resource exhaustion. CPU maxed, threads exhausted, memory full. Fix: scale out, optimize code, or reduce load.</p><h4>The golden rule for performance engineers</h4><blockquote><em>Before blaming your code, always verify: is the server throttled? Is it capped below its true capacity? Or is it genuinely at its limit?</em></blockquote><blockquote><em>These three questions will save you hours of debugging the wrong thing.</em></blockquote><p>Always monitor CPU frequency, cgroup throttle stats, and resource limits before starting a load test. A test run under artificial constraints gives you artificial results — and false confidence in production.</p><p><em>Thanks for reading. If you found this useful, share it with a fellow engineer who’s ever blamed their app for a problem that was really the infrastructure’s fault. It happens more than we’d like to admit.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9725f0050985" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>