<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by RAHUL GUPTA on Medium]]></title>
        <description><![CDATA[Stories by RAHUL GUPTA on Medium]]></description>
        <link>https://medium.com/@rahul.gupta395?source=rss-2ddc680db047------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bcelM6Q1lVEQa50c.</url>
            <title>Stories by RAHUL GUPTA on Medium</title>
            <link>https://medium.com/@rahul.gupta395?source=rss-2ddc680db047------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:22:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rahul.gupta395/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Keep Your Kubernetes Deployments Balanced Across Multiple zones]]></title>
            <link>https://medium.com/expedia-group-tech/how-to-keep-your-kubernetes-deployments-balanced-across-multiple-zones-dfe719847b41?source=rss-2ddc680db047------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfe719847b41</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[RAHUL GUPTA]]></dc:creator>
            <pubDate>Tue, 27 Aug 2019 13:01:03 GMT</pubDate>
            <atom:updated>2019-08-27T13:01:03.406Z</atom:updated>
            <content:encoded><![CDATA[<h4>Customise the Kubernetes Scheduler policy to meet your desired SLOs</h4><p>Kubernetes is a container orchestration engine which schedules and manages containerized applications on a set of worker nodes. The Kubernetes API is used to specify a Kubernetes container pod and its scheduling requirements. The Kubernetes Scheduler is responsible for making pod scheduling decisions based on the pods that have been specified to run on the cluster. The Scheduler takes into account the pod’s resource requirements, QoS requirements, affinity, anti-affinity specifications and so on to make effective scheduling decisions<em>.</em></p><h4><strong>Imbalanced cluster</strong></h4><p>In a multi-zone Kubernetes cluster, Scheduler tries its best to balance the pods in a replication controller or service across zones when the zone information is included in the node’s label, to reduce the impact of zone failures.</p><p>The following is quoted from the Kubernetes documentation.</p><blockquote><em>Scheduler will automatically spread the pods in a replication controller or service across nodes in a single-zone cluster (to reduce the impact of failures). With multiple-zone clusters, this spreading behaviour is extended across zones (to reduce the impact of zone failures).</em></blockquote><p>However in my experience even though Kubernetes tries its best to balance the pods across zones, it does not try hard enough. Apart from looking at the zone information attached to a node’s label, the default Scheduler also considers the nodes with the least requested resources, affinity / anti-affinity in pod specs and so on. So you could end up in a situation where pods placement are uneven across zones putting your service level objectives at risk during zone failures.</p><p>Let&#39;s consider a very simple tested example of multi-zone Kubernetes cluster with 3 worker nodes, one in each zone and hosting 3 services A, Band C. Let&#39;s consider service Band Cis running with 2 pods each and service Cpods needs to be running along with service Bpods using the pod affinity. The following diagram depicts the pods placement of all three services. Let&#39;s assume service Apods have requested 10% of the node’s resources while the Service Band Chave requested 30% of the resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*OxYWVv-EUQk7rmAm68pqmQ.png" /><figcaption>Pods placement with requested resources details</figcaption></figure><p>Now if you deploy a service D with 6 pods with each pod having resource requirement of 10% of the node’s resources. As the resources on node in Zone3 are least requested (10%) whereas its 70% in Zone2 &amp; Zone3, it places 3 pods of service D in Zone3, 2 pods in Zone2and 1 pod in Zone1. The reason behind this is while Scheduler tries to balance the pods across zones it also tries to balance the requested resources (CPU, memory) across nodes making the service Dpods distribution un-even across zones as shown in below image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*5qECKr__LbReMmQ_Auvt6A.png" /><figcaption>Pod placement after service D is deployed</figcaption></figure><p>Now if you were to initiate a rolling update on your service D, as the rolling update tries to bring up new pods before taking out the older pods, you would end up with the majority of service pods 4podsin zone3and 1podeach in remaining zones. Based on my experience there are plenty of other scenarios where you may end up having no service pods in a particular zone or having all the pods from a service running in a single zone.</p><h4><strong>So what if my service replicas are not evenly distributed?</strong></h4><p>Kubernetes is smart enough to spin up new pods should the individual node or a zone goes down. Well that is true, but Kubernetes may take some time to recover from such <a href="https://medium.com/expedia-group-tech/improving-kubernetes-responsiveness-under-network-partitioning-df00403a6d97">failure</a> and to provision extra capacity in available zones, if they do not already have enough capacity to shift your entire workload from the failed zone to the available zones. If the pods in your service are not evenly balanced, zone failure can bring down majority or all of your service pods causing breaches in the latency and availability SLOs your service may have to offer. The level of disruption could fall in the following categories:</p><ol><li>Complete loss of service if all the service replicas are in one zone.</li><li>Major service degradation if the majority of your pods are impacted, but could lead to complete service failure if the remaining pods are unable to handle the load.</li><li>Major service degradation impacting customers for not meeting service latency and throughput objectives</li><li>Minor service degradation with little or no impact on latency and throughput.</li></ol><p>Generally you should decide on the availability level your component should adhere to based on service level commitments to your customers and configure your component accordingly i.e replica count, data placement and level of over-provisioning (replica count) for your service. If the requirement of your service is to be in 4th category above, configuring your service for high availability becomes easier if your service pods are evenly distributed.</p><h4><strong>How can I make sure my pods are evenly distributed?</strong></h4><p>Let’s first have a look at the steps Scheduler takes to make the scheduling decision.</p><p><strong>Node Filtering:</strong> For each pod in pending state, Scheduler applies a set of <em>predicate</em> functions to filter out the nodes on which the pod can not be scheduled. <em>Predicate</em> functions filter the nodes based on the constraints or requirements specified on pod’s spec like taints, affinity, anti-affinity and nodes which do not have sufficient resources requested by pod. It also filters out the nodes which are already under resource pressure (memory or disk).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*36jP5yLvNXi3eQmxi8g7Jg.png" /><figcaption>All Available Nodes in the cluster</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*LM6LYzdN8KUQ0QS9KQiG-g.png" /><figcaption>Feasible nodes after predicate function filtering</figcaption></figure><p><strong>Node prioritization</strong>: Once Scheduler has filtered out the nodes on which a pod can not be scheduled, it calculates the score for each remaining nodes through a set of <em>priority</em> functions. <em>Priority</em> functions are used to find the node that is the best fit to schedule the pod from all feasible nodes and each <em>priority</em> function is for specific criteria like spreading the pods across nodes, balance the requested resources on nodes while at the same time considering the pod’s pod and node prefer mode of affinity/anti-affinity requirement. See the <a href="https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes/">Advanced Scheduling in Kubernetes</a> blog post for more details.</p><p>Example scores for feasible nodes:</p><pre>Zone1 Node2 score - 15<br>Zone2 Node1 score - 12<br>Zone3 Node1 score - 16<br>Zone3 Node2 score - 18</pre><p><strong>Pod scheduling: </strong>In the final step Scheduler picks up the node with the highest score to schedule the pod. If multiple nodes have the same highest score, then scheduler picks one out of them randomly.</p><pre>Node with highest priority score is selected for scheduling the pod:- Zone3 Node2 </pre><blockquote><em>Set of predicates and priority functions to be used by the scheduler and the a weight-age associated with priority functions can be configured through the scheduler policy.</em></blockquote><p>The following are the default priority functions Scheduler applies to choose best fit node from all feasible nodes for scheduling.</p><ul><li><strong><em>Selector spread</em></strong><em>: Prioritize based on spreading the service replica/pods</em></li><li><strong><em>Least requested</em></strong><em>: Prioritize based on least requested resources on a node</em></li><li><strong><em>Balance resource:</em></strong><em> Prioritize to balance the resource utilization</em></li><li><strong><em>Inter pod affinity: </em></strong><em>Prioritize based on pod spec (Pod affinity with type</em><em>preferredDuringSchedulingIgnoredDuringExecution)</em></li><li><strong><em>Taint toleration</em></strong><em>: Priority based on least number intolerable taints on node</em></li><li><strong><em>Node affinity</em></strong><em>: Prioritized based on pod spec (Node affinity with type</em><em>preferredDuringSchedulingIgnoredDuringExecution )</em></li><li><strong><em>Node prefer avoid pod priority</em></strong><em>: Ignore pods owned by the controller other than the Replication controller</em></li></ul><p>Each priority function is scored between 0–10 where 10 being the <em>most preferred</em> and 0 being the <em>least preferred</em> score. Weightage is associated with all priority functions and the final score for each node is calculated as follows.</p><pre>Node score :- (weightage * priorityFunction 1) + (weightage * priorityFunction 2) …</pre><p>The default scheduler policy gives equal weightage 1 to all priority functions and by increasing the weightage on the “<strong><em>SelectorSpreadPriority</em></strong>” function which tries to evenly spread the pods across nodes and zones. You are more likely to achieve the even spreading of your pods across zones as long as there are enough resources in all the zones to run your pods. You can override the default scheduler policy through the policy file or through the configmaps.</p><p>Scheduler command to use if you override the default scheduling policy through policy file</p><pre>--use-legacy-policy-config=false<br>--policy-config-file=&lt;file path&gt;</pre><p>Scheduler command to use if you override the default scheduling policy configmaps</p><pre>--use-legacy-policy-config=false<br>--policy-configmap=scheduler-policy<br>--policy-configmap-namespace=kube-system</pre><p>Here is the sample policy where the weightage for <em>“</em><strong><em>SelectorSpreadPriority</em></strong><em>” </em>has been increased to 5 and kept at default values for all others. Once the weightage is increased, Scheduler will give more importance to spreading the pod across nodes and zones.</p><pre>&quot;priorities&quot;: [<br>    {<br>      &quot;name&quot;: &quot;SelectorSpreadPriority&quot;,<br>      &quot;weight&quot;: 5<br>    },<br>    {<br>      &quot;name&quot;: &quot;InterPodAffinityPriority&quot;,<br>      &quot;weight&quot;: 1<br>    },<br>    {<br>      &quot;name&quot;: &quot;LeastRequestedPriority&quot;,<br>      &quot;weight&quot;: 1<br>    },<br>    {<br>      &quot;name&quot;: &quot;BalancedResourceAllocation&quot;,<br>      &quot;weight&quot;: 1<br>    },<br>        {<br>      &quot;name&quot;: &quot;NodePreferAvoidPodsPriority&quot;,<br>      &quot;weight&quot;: 1<br>    },<br>        {<br>      &quot;name&quot;: &quot;TaintTolerationPriority&quot;,<br>      &quot;weight&quot;: 1<br>   },<br>   {<br>      &quot;name&quot;: &quot;NodeAffinityPriority&quot;,<br>      &quot;weight&quot;: 1<br>    }<br>  ]</pre><p>Once you increase the weightage on “<strong><em>SelectorSpreadPriority”</em></strong><em> </em>Scheduler will give more importance to evenly spreading the pods in your service to nodes and across zones.</p><p>You can also evenly spread the pods from a particular service across the group of nodes based on label values for a specific label attached to the group of nodes. Here is an example.</p><pre>        {<br>            &quot;argument&quot;: {<br>                &quot;serviceAntiAffinity&quot;: {<br>                    &quot;label&quot;: &quot;&lt;label_name&gt;&quot;<br>                }<br>            },<br>            &quot;name&quot;: &quot;&lt;priority-Name&gt;&quot;,<br>            &quot;weight&quot;: &lt;weight-age&gt;<br>        },</pre><p>It gives the same score to all nodes that have the same value for the specified label. It gives a higher score to nodes within a group with the least concentration of pods.</p><p>You can also run a secondary scheduler in your cluster with a different set of policy configs and call that scheduler in your service deployment YAML to meet different needs, see <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/">https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/</a></p><p>I hope this post has helped you to understand how the Kubernetes Scheduler works. The scheduler default configuration may work for your use case depending on your desired SLOs. If you have a more demanding SLO, you can override the scheduler policy to meet your requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfe719847b41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/how-to-keep-your-kubernetes-deployments-balanced-across-multiple-zones-dfe719847b41">How to Keep Your Kubernetes Deployments Balanced Across Multiple zones</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can Kubernetes evict your guaranteed pod ahead of other non-guaranteed pods?]]></title>
            <link>https://medium.com/expedia-group-tech/can-kubernetes-evict-your-guaranteed-pod-ahead-of-other-non-guaranteed-pods-ece18209d4bf?source=rss-2ddc680db047------2</link>
            <guid isPermaLink="false">https://medium.com/p/ece18209d4bf</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[RAHUL GUPTA]]></dc:creator>
            <pubDate>Tue, 10 Jul 2018 11:09:30 GMT</pubDate>
            <atom:updated>2019-08-27T08:51:17.520Z</atom:updated>
            <content:encoded><![CDATA[<h3>Can Kubernetes evict your <em>guaranteed</em> pod from a node ahead of other non-guaranteed pods?</h3><p>Yes! Read on to see when, why and how Kubernetes handles CPU, memory and disk pressure and how these affect your pods…</p><p>Let’s start with the resources that always matter: CPU, memory and disk I/O. These resources are categorized as compressible (CPU) and non-compressible (memory, disk I/O). Compressible resources don’t cause eviction; instead Kubernetes throttles the pod’s available CPU when the node comes under CPU pressure. Non-compressible resources cannot be shared in that way, so when these come under pressure, Kubernetes will evict one of the pods on the node to keep sufficient capability for that resource on the node.</p><p>Under non-compressible resource pressure, the pod chosen for eviction by the Kubelet agent is not random, but based on the logic described below. The Kubelet agent runs on every node monitoring the node’s resource usage by polling cAdvisor metrics. Note: <em>guaranteed</em> means <em>requests</em>=<em>limits</em>, <em>burstable</em> means <em>requests</em>&lt;<em>limits</em> and <em>best-effort</em> means <em>requests</em>=<em>limits</em>=0.</p><p><strong>Under Disk pressure</strong></p><p>The following is the default eviction threshold for disk pressure.</p><blockquote>nodefs.available&lt;10% (File system used by the container volumes)</blockquote><blockquote>imagefs.available&lt;15% (File system used by the container run-time images)</blockquote><ul><li>When <em>imagefs</em> usage threshold is exceed, Kubelet tries to clean up disk by deleting unused images.</li><li>When <em>nodefs</em> usage threshold is exceeded, Kubelet stops admitting any new pods on the node and registers a disk pressure event. Kubelet’s first attempt will be to reclaim space by removing dead pods and containers and if this still does not bring the usage below threshold, Kubelet initiates the process of evicting the pod. From Kubernetes 1.9, Kubelet does not consider the pod’s QoS for eviction; instead it simply ranks the pods based on the usage and the pod with the highest usage is evicted. So even a <em>guaranteed</em> pod will be evicted if it’s the biggest consumer of the file-system. As Kubelet treats daemonsets the same as pods for eviction, if a daemonset is evicted from a node, Kubelet will prevent it from being restarted until the resource usage has gone above the threshold. If the pod has multiple containers, Kubelet sums up the usage of all the containers inside pod for ranking as expressed below.</li></ul><blockquote>Sum of <strong>container_fs_usage_bytes </strong>for all containers inside a pod</blockquote><p>Consider the following example table where we have some pods running on a worker node with their mentioned QoS and <em>nodefs</em> usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nepeFD9h8PXwmej9kQQhDA.png" /></figure><p>When filesystem fs usage exceeds the eviction threshold criteria, Kubelet ranks the pods for eviction based on their <em>nodefs</em> usage, with highest consumer to be evicted first as shown below. As shown in this example, a <em>guaranteed</em> pod can be evicted first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSvdL-7vMgJca-cXS0pSjw.png" /></figure><p><strong>Under memory pressure</strong></p><p>The following is the default threshold for the memory pressure.</p><p><em>memory.available&lt;100Mi</em></p><p>Once the memory pressure event is registered as the usage crosses the threshold, Kubelet does not admit any new best effort pods and ranks pods for eviction first by:</p><ol><li>Whether the pod’s memory usage exceeds <em>requests</em>.</li><li>By priority (if configured).</li><li>By the consumption of the memory for a pod relative to that pod’s requested memory.</li></ol><p>From the above, it can be concluded that <em>guaranteed</em> pods will never be evicted before <em>best effort</em> pods, but can be evicted before <em>burstable</em> pods; similarly, <em>burstable</em> pods can be evicted before <em>best effort</em> pods. If the pod has multiple containers, it calculates the usage based on sum of usage from all containers relative to sum of requested memory for all containers as expressed below.</p><blockquote><em>Sum of (</em><strong><em>container_memory_usage_bytes</em></strong><em> relative </em><strong><em>to container_resource_requests_memory_bytes</em></strong><em>) for all the containers inside pod</em></blockquote><p>The above can be illustrated with the following example. Here we have a table representing a few pods on a worker node with their QoS, memory <em>requests</em>/<em>limits</em> and memory usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Td1UTtM3lo6AXtzckp3WrQ.png" /></figure><p>When the memory usage exceeds the eviction threshold criteria, Kubelet ranks the pods for eviction based on their memory usage relative to memory requested as shown below. As shown in this example, a guaranteed pod can be evicted first. Please note, on deciding eviction order memory limit is ignored, only <em>requests</em> and usage are considered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qylCwmAlXd9dnQYWaF6BHw.png" /></figure><p><strong>Node OOM (Out Of Memory)</strong></p><p>As Kubelet monitors cAdvisor data every 10 seconds, it’s quite possible that a sudden spike in usage would trigger the kernel OOM killer before Kubelet could catch or reclaim the memory by eviction. Under this situation, the kernel OOM killer decides which containers to terminate rather then Kubelet. However, Kubelet still controls this via <em>oom_adj_score</em> set for all the pods based on the pod’s QoS. This is set as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Io2_2O7xv-ImCLyE7d5ynw.png" /></figure><p>If the node experiences a system out of memory (OOM) event before Kubelet can reclaim memory by eviction, the kernel OOM killer takes action to relive the pressure which calculates the <em>oom_score</em> based on the following expression.</p><blockquote><em>% of node memory a container is using + </em><strong><em>oom_score_adj</em></strong><em> = </em><strong><em>oom_score</em></strong></blockquote><p>The OOM killer kills the container with the highest score. If multiple containers have the same score, OOM killer kills the container using more memory. As the OOM killer first looks at the pod’s QoS (instead of consumption of the starved resource), it’s possible that it could terminate the process using less memory and the node gets under pressure again almost instantly. This process could repeat, affecting multiple containers.</p><p>Consider the following example where we have 4 pods each having one container. Both burstable pods have requested 4Gi memory and total node capacity is 30Gi. Kubelet will assign the following <em>oom_adj</em> value to the pods as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mLM9b2ty8Uyn9kN0tWg4xg.png" /></figure><p>When the OOM killer is invoked, it first chooses the container with the highest score (1000). As we see two containers running with a score of 1000, OOM killer will pick the one using more memory out of the two.</p><p><strong>Note</strong></p><ul><li>As Kubelet polls cadvisor metrics every 10 seconds, it’s possible that Kubelet may evict multiple pods before realizing that usage has gone below eviction threshold.</li><li>When Kubelet evicts the pod, Kubernetes will reschedule the pod immediately on another node where enough resources are available. Scheduler may reschedule the pod again to the same node depending on restart policy and if the pod has been set with affinity to node or if pod is running as deamonset.</li></ul><p>As you are now hopefully aware of how and in which scenarios pods are evicted. If you set the right values for your component, know all the options available and if you understand your workload it makes it easier to balance your cluster as well as allow you to over commit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ece18209d4bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/expedia-group-tech/can-kubernetes-evict-your-guaranteed-pod-ahead-of-other-non-guaranteed-pods-ece18209d4bf">Can Kubernetes evict your guaranteed pod ahead of other non-guaranteed pods?</a> was originally published in <a href="https://medium.com/expedia-group-tech">Expedia Group Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>