<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdul Qadeer on Medium]]></title>
        <description><![CDATA[Stories by Abdul Qadeer on Medium]]></description>
        <link>https://medium.com/@abqadeer?source=rss-89878df718d2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BGjFJUvXSR6meSvjwSzNgw.jpeg</url>
            <title>Stories by Abdul Qadeer on Medium</title>
            <link>https://medium.com/@abqadeer?source=rss-89878df718d2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:15:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abqadeer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling Kubernetes to Over 4k Nodes and 200k Pods]]></title>
            <link>https://medium.com/paypal-tech/scaling-kubernetes-to-over-4k-nodes-and-200k-pods-29988fad6ed?source=rss-89878df718d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/29988fad6ed</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Abdul Qadeer]]></dc:creator>
            <pubDate>Thu, 20 Jan 2022 15:03:05 GMT</pubDate>
            <atom:updated>2022-01-20T17:55:43.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqY65sTiETTDZwRwIODJSw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@todd_diemer">Todd Diemer</a> on Unsplash</figcaption></figure><p>At PayPal, we recently started testing the waters with Kubernetes. A majority of our workloads run on Apache Mesos, and as part of this migration, we needed to understand several performance aspects of clusters running Kubernetes with a PayPal-specific control plane. Chief amongst these aspects is understanding the scalability of the platform as well as identifying opportunities for improvement through tuning the cluster.</p><p>Unlike Apache Mesos, which can scale up to 10,000 nodes out of the box, scaling Kubernetes is challenging. Kubernetes’ scalability is not just limited to the number of nodes and pods, but several aspects like the number of resources created, the number of containers per pod, the total number of services, and the pod deployment throughput. This post describes some challenges we faced when scaling and how we solved them.</p><h3>Cluster Topology</h3><p>We have clusters of varying sizes in production spanning thousands of nodes. Our setup consisted of three master nodes and an external three-node etcd cluster all running on Google Cloud Platform (GCP). The control plane was placed behind a load balancer and all data nodes belonged to the same zone as the control plane.</p><h3>Workload</h3><p>For the purpose of performance testing, we used an open-source workload generator, <a href="https://github.com/vmware-tanzu/k-bench">k-bench</a>, modified for our use cases. The resource objects we used were simple pods and deployments. We deployed them both in batches and sequentially with varying batch sizes and inter-deployment times.</p><h3>Scale</h3><p>We started with a low number of pods and a low number of nodes. Through stress testing, we found opportunities for improvement and continued to scale up the cluster as we observed improved performance. Each worker node had four CPU cores and could hold a maximum of 40 pods. We scaled to about 4,100 nodes. The application used for benchmarking was a stateless service running on 100 millicores of guaranteed Quality of Service (QoS).</p><p>We started with 2,000 pods on 1,000 nodes, followed by 16,000 pods, then 32,000 pods. After this, we jumped to 150,000 pods on 4100 nodes, followed by 200,000 pods. We had to increase the number of cores on each node to accommodate more pods.</p><h3>API Server</h3><p>The API server proved to be a bottleneck when several connections to the API server returned 504 gateway timeouts, in addition to local client level throttling (exponential backoff). These increased <strong>exponentially</strong> during ramp-ups:</p><pre>I0504 17:54:55.731559       1 request.go:655] Throttling request took 1.005397106s, request: POST:<a href="https://10.178.61.59/api/v1/namespaces/kbench-deployment-namespace-14/pods/kbench-deployment-oid-36-tid-0-856878759c-sdsxt/binding">https://&lt;&gt;:443/api/v1/namespaces/kbench-deployment-namespace-14/pods.</a>.<br>I0504 17:55:05.741655       1 request.go:655] Throttling request took 7.38390786s, request: POST:<a href="https://10.178.61.59/api/v1/namespaces/kbench-deployment-namespace-13/pods/kbench-deployment-oid-34-tid-0-646d9f4b79-fkgdm/binding">https://&lt;&gt;:443/api/v1/namespaces/kbench-deployment-namespace-13/pods.</a>.<br>I0504 17:55:15.749891       1 request.go:655] Throttling request took 13.522138087s, request: POST:<a href="https://10.178.61.59/api/v1/namespaces/kbench-deployment-namespace-13/pods/kbench-deployment-oid-33-tid-0-6f4756dc86-wftd2/binding">https://&lt;&gt;:443/api/v1/namespaces/kbench-deployment-namespace-13/pods.</a>.<br>I0504 17:55:25.759662       1 request.go:655] Throttling request took 19.202229311s, request: POST:<a href="https://10.178.61.59/api/v1/namespaces/kbench-deployment-namespace-20/pods/kbench-deployment-oid-50-tid-0-8549fb4dbf-f4czn/binding">https://&lt;&gt;:443/api/v1/namespaces/kbench-deployment-namespace-20/pods.</a>.<br>I0504 17:55:35.760088       1 request.go:655] Throttling request took 25.409325008s, request: POST:<a href="https://10.178.61.59/api/v1/namespaces/kbench-deployment-namespace-13/pods/kbench-deployment-oid-34-tid-0-646d9f4b79-z6hwv/binding">https://&lt;&gt;:443/api/v1/namespaces/kbench-deployment-namespace-13/pods.</a>.<br>I0504 17:55:45.769922       1 request.go:655] Throttling request took 31.613720059s, request: POST:<a href="https://10.178.61.59/api/v1/namespaces/kbench-deployment-namespace-6/pods/kbench-deployment-oid-15-tid-0-7898fd545d-tn2s4/binding">https://&lt;&gt;:443/api/v1/namespaces/kbench-deployment-namespace-6/pods.</a>.</pre><p>The size of the queue which governs rate-limiting at the API Server was updated via <em>max-mutating-requests-inflight</em> and <em>max-requests-inflight</em>. These two flags at the API server govern how the <a href="https://kubernetes.io/docs/concepts/cluster-administration/flow-control/">Priority and Fairness</a> feature introduced as Beta in 1.20 divides the total queue size amongst its queue classes. For example, requests for leader election get priority over pod requests. Within each priority, there is fairness with configurable queues. There is room for further fine-tuning in the future via <em>PriorityLevelConfiguration</em> &amp; <em>FlowSchema</em> API objects.</p><h3>Controller Manager</h3><p>Controller Manager is responsible for providing controllers for native resources like Replica Set, Namespaces, etc., with a high number of deployments (which are managed by replica sets). The rate at which the controller manager could sync its state with the API server was limited. Multiple knobs were used to tune this behavior:</p><ul><li><em>kube-api-qps — </em>The number of queries the controller manager could make to the API server in a given second.</li><li><em>kube-api-burst — </em>The controller manager burst, which is additional concurrent calls above <em>kube-api-qps.</em></li><li><em>concurrent-deployment-syncs — </em>Concurrency in the synchronization call for objects like deployment, replicaset, etc.</li></ul><h3>Scheduler</h3><p>When tested independently as an independent component, the scheduler can support a high throughput rate of up to 1,000 pods per second. However, upon deploying the scheduler to a live cluster, we noticed a real-world throughput reduction. A <strong>slow etcd instance caused increased binding latencies</strong> for the scheduler, leading to an increased pending queue size in the order of thousands of pods. The idea was to keep this number under 100 during test runs, as a higher count affects pod startup latencies. We also ended up tuning leader election parameters to be resilient against spurious restarts on short-lived network partitions or network congestion.</p><h3>etcd</h3><p>etcd is the single most critical part of a Kubernetes cluster. This is evident from the plethora of issues etcd can cause all across the cluster manifested in different ways. It needed careful investigation to surface the root causes and scale etcd to handle our intended scale.</p><p>When ramping up, a lot of Raft proposals started failing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*a2TwQWzWPnwRzdn0XEVj2A.png" /></figure><p>Through investigation and analysis, we determined that GCP throttled PD-SSD disk throughput to about 100MiB per second (as shown below) on our disk size of 100G. GCP doesn’t provide a way to increase the throughput limit — it only <a href="https://cloud.google.com/compute/docs/disks">increases with the size of the disk</a>. Even though etcd node only takes &lt; 10G of space, we first tried with 1TB PD-SSD. However, even the larger disk became a <strong>bottleneck</strong> when all 4k nodes joined the Kubernetes control plane at once. We decided to use local SSD, which has very high throughput at the cost of a slightly higher chance of data loss in case of failures as it is <a href="https://cloud.google.com/compute/docs/disks/local-ssd#data_persistence">not persistent</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4OuITPZunPJcFA3em5-jfQ.png" /></figure><p>After moving to local SSD, we didn’t see the expected performance out of the fastest SSD. Some benchmarks were done directly on the disk with FIO and the numbers there were expected. However, etcd benchmarks showed a different story for concurrent all member writes:</p><pre>LOCAL SSD<br>Summary:<br> Total: 8.1841 secs.<br> Slowest: 0.5171 secs.<br> Fastest: 0.0332 secs.<br> Average: 0.0815 secs.<br> Stddev: 0.0259 secs.<br> Requests/sec: 12218.8374<br><br>PD SSD<br>Summary:<br> Total: 4.6773 secs.<br> Slowest: 0.3412 secs.<br> Fastest: 0.0249 secs.<br> Average: 0.0464 secs.<br> Stddev: 0.0187 secs.<br> Requests/sec: 21379.7235</pre><p>The local SSD performed worse! After deeper investigation, this was attributed to <em>ext4</em> file system’s write barrier cache commits. Since etcd uses write-ahead logging and calls <em>fsync</em> every time it commits to the raft log, it’s okay to disable the write barrier. Additionally, we have DB backup jobs at the file system level and application level for DR. After this change, the numbers with local SSD improved comparable to PD-SSD:</p><pre>LOCAL SSD<br>Summary:<br>  Total: 4.1823 secs.<br>  Slowest: 0.2182 secs.<br>  Fastest: 0.0266 secs.<br>  Average: 0.0416 secs.<br>  Stddev: 0.0153 secs.<br>  Requests/sec: 23910.3658</pre><p>The effect of this improvement was seen in etcd’s Write-Ahead Logging (WAL) sync duration and backend commit latencies, which decreased by more than 90% around the 15:55 time mark as seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CONYmPoRi1KmszTZXjh6kw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BbuL6oW-a4SdZDTdfx0zBQ.png" /></figure><p>The default MVCC DB size in etcd is 2 GB. This was increased to a maximum of 8 GB upon setting off out of DB space alarms. With the utilization of about <strong>~60% of this DB</strong>, we were able to scale up to 200k stateless pods.</p><p>With all the above optimizations the cluster was much more stable at our intended scale, however, we were far behind in SLIs for API latencies.</p><p>The etcd server would still restart occasionally and just a single restart can spoil the benchmark results, especially the P99 numbers. A closer look revealed that there is a liveness probe <a href="https://github.com/kubernetes/kubernetes/pull/97034">bug in etcd</a> YAML for v1.20. A workaround was applied to fix this by increasing the failure threshold count.</p><p>After exhausting all avenues to scale etcd vertically, mainly resource-wise (CPU, memory, disk) we found that the etcd’s performance was affected with range queries. Etcd doesn’t perform well when there are many range queries and writes to the Raft log suffer, thereby slowing down the cluster’s latencies. The following are the number of range queries per Kubernetes resource that affect performance in one of the test runs:</p><pre>etcd$ sudo grep -ir &quot;events&quot; 0.log.20210525-035918 | wc -l<br>130830<br>etcd$ sudo grep -ir &quot;pods&quot; 0.log.20210525-035918 | wc -l<br>107737<br>etcd$ sudo grep -ir &quot;configmap&quot; 0.log.20210525-035918 | wc -l<br>86274<br>etcd$ sudo grep -ir &quot;deployments&quot; 0.log.20210525-035918 | wc -l<br>6755<br>etcd$ sudo grep -ir &quot;leases&quot; 0.log.20210525-035918 | wc -l<br>4853<br>etcd$ sudo grep -ir &quot;nodes&quot; 0.log.20210525-035918 | wc -l</pre><p>The etcd backend latencies were majorly impacted due to these time-consuming queries. After sharding the etcd server on the events resource, we saw an improvement in cluster stability when there is high contention of pods. In the future, there is room to further shard the etcd cluster on the pods resource. It is easy to configure the API server to contact the relevant etcd for interacting with a sharded resource.</p><h3>Results</h3><p>After optimizing and tuning various components of Kubernetes, we observed a huge improvement in latencies. The following charts demonstrate the performance gain achieved over time to meet SLOs. The workload here is 150k pods total with 250 replicas per deployment on 10 concurrent workers. As long as the P99 latencies for pod startup are within five seconds, per Kubernetes SLOs, we are good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QWE90InNMb6Tzq5qp9a3qA.png" /></figure><p>The following chart demonstrates the API call latencies well within <a href="https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md#steady-state-slisslos">SLOs</a> at 200k pods in the cluster.</p><blockquote>We also achieved P99 pod startup latencies of around <strong>five seconds</strong> for 200k pods at a much higher pod deployment rate than what K8s tests for 5k nodes claim at 3000 pods/min.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UIwqe64oosj7hMq8XNvqNA.png" /></figure><h3>Conclusion</h3><p>Kubernetes is a complex system and a deep understanding of the control plane is necessary to know how to scale each component. We have learned a lot through this exercise and will continue to optimize our clusters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29988fad6ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/paypal-tech/scaling-kubernetes-to-over-4k-nodes-and-200k-pods-29988fad6ed">Scaling Kubernetes to Over 4k Nodes and 200k Pods</a> was originally published in <a href="https://medium.com/paypal-tech">The PayPal Technology Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real Load Aware Scheduling in Kubernetes with Trimaran]]></title>
            <link>https://medium.com/paypal-tech/real-load-aware-scheduling-in-kubernetes-with-trimaran-a8efe14d51e2?source=rss-89878df718d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8efe14d51e2</guid>
            <category><![CDATA[trimaran]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[targetloadpacking]]></category>
            <category><![CDATA[kube-scheduler]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Abdul Qadeer]]></dc:creator>
            <pubDate>Tue, 16 Feb 2021 14:15:43 GMT</pubDate>
            <atom:updated>2021-09-16T07:55:31.965Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlJsVjnIxxbR4yXaLrgyFw.jpeg" /></figure><h4>Why is scheduling in Kubernetes inefficient?</h4><p>Native scheduling in Kubernetes is handled by the kube-scheduler service. Resource utilization of pods is defined via a declarative resource model and the kube-scheduler works with a kubelet service to provide pod <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#qos-classes">QoS guarantees</a>. This model can lead to low utilization and wastage of cluster resources, as live node resource utilization is not considered in scheduling decisions. Also, it is hard for users to predict the correct usage values of their pods when they define pod specs.</p><h4>Trimaran Scheduler</h4><p>As of Kubernetes 1.15, the scheduler has been made flexible for customizations with the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/">Scheduling Framework</a>. Our team at PayPal leveraged this to develop the <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/kep/61-Trimaran-real-load-aware-scheduling">Trimaran scheduler</a> which works on live node utilization values to efficiently utilize cluster resources and save costs. As part of this, we developed and contributed the <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/trimaran/targetloadpacking">TargetLoadPacking</a> plugin and <a href="https://github.com/paypal/load-watcher">Load Watcher</a> to the open-source community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x4tf12EtijD9y9M9PMRdCA.png" /><figcaption>Trimaran Architecture</figcaption></figure><h4>TargetLoadPacking Plugin</h4><p>There are multiple extension points in the Scheduling Framework that we can hook into for customization. TargetLoadPacking Plugin extends the Score extension point, which is responsible for scoring nodes to schedule pods in each <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#scheduling-cycle-binding-cycle">scheduling cycle</a>. Our algorithm is a hybrid variant of the standard bin pack algorithm that favors packing pods on nodes around a target utilization, by moving from best fit to least fit. In other words, given a target utilization of x%, the plugin favors nodes that are closer to x%. Currently, CPU Utilization is supported but it can be extended to multiple resources.</p><h4>Deployment Tutorial</h4><p>This tutorial will guide you to deploy the Trimaran scheduler in any K8s setup including Minikube, Kind, etc. Trimaran depends upon the <a href="https://github.com/paypal/load-watcher">load watcher</a> service to run, which in turn depends on a metrics provider to load metrics from. Currently, supported providers are Kubernetes Metrics Server (default) and SignalFx, with current ongoing work to add support for Prometheus. Make sure to deploy the metrics provider you want to use in your K8s cluster before you deploy the load watcher.</p><blockquote>Note that your Kubernetes version should be at least v1.19.0 as of this writing.</blockquote><p>As part of the Trimaran scheduler deployment, we will create Docker images for two services — kube-scheduler configured with TargetLoadPacking plugin (trimaran image) and load watcher service (load-watcher image), and deploy them as a single pod. It is also possible to use load watcher as a <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/trimaran#load-watcher-as-a-library">library</a> and run it in the same container as scheduler. However, following tutorial will focus on running it as a separate container.</p><p>Instructions to build a load-watcher Docker image can be found <a href="https://github.com/paypal/load-watcher#tutorial">here</a>.</p><p>To build atrimaran image, first build a kube-scheduler image as follows:</p><pre>git clone <a href="https://github.com/kubernetes-sigs/scheduler-plugins.git">https://github.com/kubernetes-sigs/scheduler-plugins.git</a><br>git checkout &lt;desired-release-branch&gt;<br>cd scheduler-plugins<br>env GOARCH=amd64 GOOS=linux go build -o kube-scheduler main.go</pre><p>Save the following Trimaran scheduler configuration in scheduler-config.yaml . Add pluginConfig args as needed depending on your <a href="https://github.com/paypal/load-watcher#metrics-provider-configuration">metrics provider</a> setup.</p><pre>apiVersion: kubescheduler.config.k8s.io/v1beta1<br>kind: KubeSchedulerConfiguration<br>leaderElection:<br>  leaderElect: false<br>profiles:<br>- schedulerName: trimaran<br>  plugins:<br>    score:<br>      disabled:<br>      - name: NodeResourcesBalancedAllocation<br>      - name: NodeResourcesLeastAllocated<br>      enabled:<br>      - name: TargetLoadPacking<br>  pluginConfig:<br>    - name: TargetLoadPacking<br>      args:<br>        watcherAddress: http://127.0.0.1:2020</pre><blockquote>It is strongly recommended to disable the two native plugins above to prevent scoring conflicts</blockquote><p>Create a Dockerfile with the following content in the root directory of the cloned repository above:</p><pre>FROM golang:1.5-alpine<br>ADD ./kube-scheduler /bin/kube-scheduler<br>ADD ./scheduler-config.yaml /home/scheduler-config.yaml</pre><pre>RUN [&quot;chmod&quot;, &quot;+x&quot;, &quot;/bin/kube-scheduler&quot;]<br>CMD [&quot;/bin/kube-scheduler&quot;]</pre><p>Build the Docker image as follows:</p><pre>docker build -t trimaran .<br>docker tag trimaran:latest &lt;your-docker-repo&gt;:latest<br>docker push &lt;your-docker-repo&gt;</pre><p>The following is a YAML spec for Trimaran K8s deployment:</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: trimaran<br>  namespace: kube-system<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: trimaran-as-kube-scheduler<br>subjects:<br>- kind: ServiceAccount<br>  name: trimaran<br>  namespace: kube-system<br>roleRef:<br>  kind: ClusterRole<br>  name: system:kube-scheduler<br>  apiGroup: rbac.authorization.k8s.io<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: trimaran-as-volume-scheduler<br>subjects:<br>- kind: ServiceAccount<br>  name: trimaran<br>  namespace: kube-system<br>roleRef:<br>  kind: ClusterRole<br>  name: system:volume-scheduler<br>  apiGroup: rbac.authorization.k8s.io<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: RoleBinding<br>metadata:<br>  name: trimaran-extension-apiserver<br>  namespace: kube-system<br>subjects:<br>- kind: ServiceAccount<br>  name: trimaran<br>  namespace: kube-system<br>roleRef:<br>  kind: Role<br>  name: extension-apiserver-authentication-reader<br>  apiGroup: rbac.authorization.k8s.io<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  labels:<br>    component: scheduler<br>    tier: control-plane<br>  name: trimaran<br>  namespace: kube-system<br>spec:<br>  selector:<br>    matchLabels:<br>      component: scheduler<br>      tier: control-plane<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        component: scheduler<br>        tier: control-plane<br>        version: second<br>    spec:<br>      serviceAccountName: trimaran<br>      hostNetwork: true<br>      containers:<br>      - name: trimaran<br>        command:<br>        - /bin/kube-scheduler<br>        - --address=0.0.0.0<br>        - --leader-elect=false<br>        - --scheduler-name=trimaran<br>        - --config=/home/scheduler-config.yaml<br>        - -v=6<br>        image: &lt;replace-me&gt;<br>        imagePullPolicy: Always<br>        resources:<br>          requests:<br>            cpu: &#39;0.1&#39;<br>        securityContext:<br>          privileged: false<br>        volumeMounts: <br>        - mountPath: /shared<br>          name: shared<br>      - name: load-watcher<br>        command:<br>        - /bin/load-watcher<br>        image: &lt;replace-me&gt;<br>        imagePullPolicy: Always<br>      volumes:<br>      - name: shared<br>        hostPath:<br>          path: /tmp<br>          type: Directory</pre><p>Save the above content in a file named “trimaran-scheduler.yaml” and deploy it using the command:</p><pre>kubectl apply -f trimaran-scheduler.yaml</pre><p>For any pods to be scheduled with Trimaran scheduler, schedulerName needs to be modified with value trimaran in a respective pod spec YAML file. An example pod spec is given below:</p><pre>apiVersion: v1<br>kind: Pod<br>...<br>spec:<br>  <strong>schedulerName: trimaran</strong><br>  containers:<br>  - name: pod-with-annotation-container<br>    ...</pre><p>Verify that the pod has been scheduled with the following command and status as Running.</p><pre>kubectl describe pod </pre><h4>Configuring TargetLoadPacking</h4><p>There are three configurable parameters other than watcherAddress that can be used to modify the behavior of the plugin according to your requirements:</p><ol><li>targetUtilization: CPU Utilization % target you would like to achieve in bin packing. It is recommended to keep this value ten less than what you desire. The default is 40.</li><li>defaultRequests: This configures CPU requests for containers without requests or limits i.e. Best Effort QoS. The default is one core. This is used for utilization prediction when scheduling.</li><li>defaultRequestsMultiplier: This configures the multiplier for containers without limits i.e. Burstable QoS. The default is 1.5 cores.</li></ol><p>These can be added in args under pluginConfig in scheduler-config.yaml. More details about the design can be found in <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/kep/61-Trimaran-real-load-aware-scheduling">KEP</a>.</p><h4>Contribution</h4><p>There are interesting areas where our work in Trimaran can be extended. For example, multidimensional bin packing with multiple resources (CPU, Memory, Network Bandwidth, etc.), ML/AI models for utilization prediction, etc. Readers are welcome to get in touch and contribute!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8efe14d51e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/paypal-tech/real-load-aware-scheduling-in-kubernetes-with-trimaran-a8efe14d51e2">Real Load Aware Scheduling in Kubernetes with Trimaran</a> was originally published in <a href="https://medium.com/paypal-tech">The PayPal Technology Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>