<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Julio Renner on Medium]]></title>
        <description><![CDATA[Stories by Julio Renner on Medium]]></description>
        <link>https://medium.com/@juliorenner123?source=rss-582581ec5a16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W1RT9u1e_aaOYbBwxA3aPg.jpeg</url>
            <title>Stories by Julio Renner on Medium</title>
            <link>https://medium.com/@juliorenner123?source=rss-582581ec5a16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:43:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@juliorenner123/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A journey into Monitoring as Code]]></title>
            <link>https://medium.com/@juliorenner123/a-journey-into-monitoring-as-code-d0e2fdfa8f42?source=rss-582581ec5a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0e2fdfa8f42</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Julio Renner]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 13:21:14 GMT</pubDate>
            <atom:updated>2026-07-02T21:56:57.173Z</atom:updated>
            <content:encoded><![CDATA[<p>Years ago, we started implementing and standardizing our practices around observability. We came from a background with little experience in both setting it up and using it, which led to several mistakes we had to live with and address over time. Nowadays we are definitely in a better place. For example, setup that used to take days now takes minutes. That said, there’s still a lot to improve. Our journey may be valuable to you, and your feedback may help us shape our next steps.</p><p>Before we dive into the details, here’s the whole journey at a glance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*43BuUgHm7PLvlUjcP8r1pw.png" /><figcaption>From fragmented, manual monitoring to monitoring as code at scale</figcaption></figure><h3>Our Background</h3><p>Five years ago, of the three pillars of observability (logs, metrics, and tracing), we had only logs, plus a few metrics derived from those logs at the application level. Our logging was spread across different tools such as Elastic, GCP Logging, Mezmo (LogDNA), and others. Our monitoring dashboards were scattered across different GCP projects in Stackdriver, none of them standardized. All that tooling definitely didn’t help when troubleshooting a distributed system.</p><p>We decided to adopt Grafana Cloud, using its hosted Prometheus for metrics and Loki for logs. The idea was to avoid the hell of jumping between different tools while troubleshooting, and to centralize our monitoring stack in a single place.</p><h3>The Beginning</h3><p>We started by manually creating dashboards in Grafana Cloud and instrumenting our applications with Prometheus clients, exposing metrics through the <em>/metric</em>s endpoint.</p><p>The goal was to use <a href="https://grafana.com/docs/grafana/latest/visualizations/dashboards/variables/">dashboard variables</a> to switch between environments. There would be one dashboard per service, and the environment (Development, Staging, or Production) would be selected from the variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*klcvEvjZ0dVkB_c6mMWPvQ.png" /><figcaption>Service dashboard with environment picked up from variable</figcaption></figure><p>With that pattern, we could reduce the maintenance for each application’s dashboard. Back then, though, Grafana alerts were bound to dashboard queries, which meant you couldn’t create alerts from panels that used variables. That was later solved by <a href="https://grafana.com/blog/the-new-unified-alerting-system-for-grafana-everything-you-need-to-know/">Unified Alerting</a>, launched in Grafana 8.</p><p>That single limitation forced us to copy/paste the dashboard JSON and replicate it per environment. To give you a clearer picture, it looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1STybjolUp-PjyGvkSrmg.png" /></figure><ol><li>Manually provisioned dashboards, one per environment</li><li>Manually created contact points, set with the proper severity and pointing to a Pager Duty service</li><li>Manually created alert rules</li><li>Manually configured Slack integration for notifications in Pager Duty, according to internal criteria</li></ol><p>In the same effort, we also configured Grafana Promtail, a log scraper to collect and forward logs to the Grafana Cloud Loki server. At this point, engineers had both application logs and dashboards in the same tool.</p><p>Spinning up monitoring for a single new service this way could take 2 to 3 days: building and validating the panels, replicating the dashboard across every environment, duplicating the alerting, and going through a review cycle each time.</p><p>The bright side is that we were finally moving toward a centralized place for monitoring and troubleshooting. The downside is that our lack of experience led to a few mistakes:</p><ul><li><strong>Manual configuration</strong>: since we maintained alerts and dashboards by hand for each environment, they drifted into a huge mess.</li><li><strong>Too many alerts:</strong> we weren’t sure what to monitor, so we erred on the side of more alerts.</li><li><strong>Lack of process for internal-environment monitoring:</strong> with no process to handle internal-environment alerts, they were simply ignored, instead of helping us validate alert configuration and catch bugs before they reached production.</li></ul><h3>Unified Alerting</h3><p>The release of Unified Alerting decoupled <a href="https://grafana.com/docs/grafana/latest/visualizations/panels-visualizations/">dashboard panels</a> from alerts. That single change let us keep one dashboard per service, with the environment selected as a variable, while still being able to create alerts from those panels.</p><p>The key change is that, before Unified Alerting, an alert was a second-class citizen of a panel, reusing its query and attached directly to it. Now the alert is fully independent: it has its own query, and you link it to a panel by referencing a dashboard ID and panel ID.</p><p>With that change, we could finally achieve what we had initially envisioned: a single dashboard with its alert rules per service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2MW3NFfZkzLBV3KNIX_Ohw.png" /></figure><p>The problem, once again, was having to change everything by hand. Even though it was now possible to consolidate alerts into a single dashboard, doing so meant refactoring every dashboard to support the environment variable and manually unifying the alerts from all the environments into a single dashboard.</p><p>We decided we couldn’t keep going without migrating our monitoring to code.</p><h3>Monitoring as Code</h3><p>The maintenance had become overwhelming, and alert and dashboard governance was a mess. When fixing issues with alerts and panels, only production alerts and dashboards were updated, while staging and development were often ignored or missed. And you couldn’t blame anyone: the burden was too high.</p><p>For monitoring as code, Grafana Cloud provides a few options:</p><ul><li>Terraform</li><li>Ansible</li><li>Grafana Operator</li><li>Crossplane</li><li>gcx (recently released)</li></ul><p>From the options we had, Terraform was a tool our engineers were already familiar with, and there didn’t seem to be any clear advantage to using a different one.</p><blockquote>Keep in mind that opting for a tool or language your team already knows means less adoption friction and a smaller learning curve. Don’t introduce a new language or tool just because it looks cool; it needs to solve a problem.</blockquote><p>We couldn’t just stop everything and do the migration all at once, so we started adopting monitoring as code gradually. Aligning this with the product team was essential, and we agreed on a rule: whenever monitoring was touched for any service, we would take the extra time to migrate it to monitoring as code. For new services it was straightforward: new dashboards would always be provisioned as code.</p><p>With Terraform, a dashboard definition looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsvdZ_iY9Bk95nuvbM91Yw.png" /></figure><p>Dashboard JSON files can run to thousands of lines, so reviewing them is not straightforward. I’d suggest reviewing the dashboard directly in Grafana, visually, before and after the <em>terraform apply</em>, rather than trying to go over the JSON itself.</p><p>Besides dashboards, we also defined <a href="https://grafana.com/docs/grafana/latest/alerting/configure-notifications/template-notifications/manage-notification-templates/">notification templates</a> as code, and created a reusable module, used by each service, that configures:</p><ul><li>Pager Duty service and its integrations (Slack, Jira, etc.)</li><li>Grafana contact points, linking Pager Duty and Grafana</li></ul><p>That ensures standardization and less maintenance, the same way packaging things into a library does for regular code. We also use AI agent reviews on PRs to enforce a set of practices each dashboard and alert should follow, for example, ensuring there is a link to a playbook the on-call engineer can act on. That’s another advantage of treating monitoring as code.</p><p>Improvements we saw:</p><ul><li>Faster setup of dashboards, Pager Duty services, and Grafana integration</li><li>Standardized setup across all services</li><li>Reduced maintenance time for the monitoring stack, with updates easily rolled out to all services</li></ul><p>Downsides of the approach:</p><ul><li>Single dashboard per service limits our ability to roll it out one environment at a time (development -&gt; staging -&gt; production)</li><li>JSON dashboards are hard to review</li></ul><p>A single dashboard per service is a tradeoff we accepted: it’s also what kills cross-environment drift and keeps rollout simple. Since a dashboard is only a view (alerts are independent), changing all environments at once is low-risk. So far, the lost per-environment promotion has cost us little, though there were times it would have been nice to have.</p><h3>Alerting Process</h3><p>Once we had things organized and most of our monitoring was provisioned as code, we were still faced with a huge number of alerts being ignored. Those were internal-environment alerts.</p><p>There were two main sources for them:</p><ul><li><strong>Old, manually-configured alerts:</strong> alerts created by hand a long time ago. Sometimes they were fixed in production but not in the internal environments.</li><li><strong>Bad configuration:</strong> new alerts firing because the internal environments had different configuration, with thresholds that weren’t set properly.</li></ul><p>The biggest cause of this problem, though, was a <strong>lack of ownership</strong>. No one really felt responsible for those alerts. <strong>If everyone is responsible, no one is.</strong></p><p>At this point, we still hadn’t taken the time to prune old, poorly configured alerts. They were thrown into a Slack channel that was just ignored.</p><p>To change that, the first action was to define clear ownership. We decided that each service’s alerting and monitoring would be the responsibility of that service’s developers. Once that was in place, we updated our monitoring as code with the proper contact points and defined per-team Slack channels to route the alerts.</p><p>Our alerts were classified into:</p><ul><li><strong>Internal:</strong> Development and Staging alerts, followed up by the service’s developers during working hours.</li><li><strong>Production Warning:</strong> production alerts that don’t require immediate triage or remediation, followed up by the on-call engineer during working hours to assess urgency, then handed to the service’s development team.</li><li><strong>Production:</strong> urgent alerts that require immediate investigation, followed up immediately by the on-call engineer, 24x7.</li></ul><p>With proper classification, each team knew which alerts fell in its realm.</p><p>We then applied the same process to internal environments, creating triage tickets for the engineering team to act on. That ensured the alerts were actually addressed. Through proper prioritization with the product team, we worked through and cleaned up the backlog. A key point: every alert deserves a follow-up, even a false positive. In that case, the action is to remove it if it’s unnecessary, or to fine-tune the metric or threshold.</p><p>Initially there were a lot of alerts to clean up, but after that, alerts in the internal environments became more effective at pinpointing issues before they reached production.</p><p>The added responsibility on engineers also forced everyone to become more familiar with how the monitoring stack works, which improved the quality of new alerts.</p><h3>Scaling Up Monitoring</h3><p>Now that we had things structured and defined through code, we started looking into ways to scale and simplify monitoring.</p><h4>Grafana Library Panels</h4><p>Grafana offers <a href="https://grafana.com/docs/grafana/latest/visualizations/dashboards/build-dashboards/manage-library-panels/">Library Panels</a>, which let you create a panel and reuse it across dashboards. Say you have a CPU monitoring panel: you can reuse it across different applications’ dashboards. Whenever you want to fix the query, you fix the library panel and the change propagates everywhere. Instead of updating N dashboards, you update a single library panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*d3Uay5193Q8RuvcfloroHg.png" /><figcaption>One CPU library panel, reused across every service dashboard.</figcaption></figure><p>Adopting Library Panels made an immediate impact: creating dashboards was much faster, especially when the panels you needed already existed. It also meant engineers who weren’t comfortable writing PromQL could use a panel with less chance of misconfiguration.</p><p>Still, whenever you investigated issues, each service had its own panel structure, based on what the developer who set it up considered relevant. There was still a lack of standardization, and creating new dashboards took manual, repetitive effort: selecting the library panels, organizing the dashboard, and creating new library panels when they were missing. So we still wanted to make the process easier.</p><h4><strong>Terraform Dashboard Module</strong></h4><p>As a next step to reduce the setup time of new dashboards and maintain consistency across applications, we decided to create a shared Terraform module for setting up dashboards.</p><p>From a developer’s perspective it’s quite straightforward: you import the shared module and choose which sections to enable. Sections are organized by component: the request-oriented ones (like HTTP) follow the <a href="https://grafana.com/blog/the-red-method-how-to-instrument-your-services/">RED method</a> (rate, errors, and duration), while the queue-consumer ones (like Kafka) track consumer lag and throughput. It looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*T37njcfFJM7AavYvW7jeFQ.png" /></figure><p>Each section you enable is added to the provisioned dashboard. The app_label variable filters the metrics to the right application in its Kubernetes namespace.</p><p>Maintaining application dashboards is now much simpler, since we only have to change the central module. Their layouts are also far more consistent with one another. What took 2 to 3 days in the early days now takes a couple of minutes: import the module, enable the sections you need, and apply.</p><p>Developers can also extend the default dashboard with extra panels by passing a library panel’s key and title through variables, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*sAQecdntkw5-sP8VrXWRqw.png" /></figure><p>This Terraform module for creating dashboards has already given us feedback such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tgg-4hWrWgHd8fiAS2vUtw.png" /></figure><p>Using the shared module isn’t mandatory. Engineers can still build their own independent dashboard if they need to, but the shared module is there to help.</p><h4>Alert as Code</h4><p>I haven’t mentioned alerts as code yet, but we also define them using the <em>grafana_rule_group</em> Terraform resource. Through code, we ensure there’s no drift in alert queries across environments, and we can customize them as needed. Here’s an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nv07tzEVYOrT63Qu6BODqA.png" /><figcaption>Alert as code example</figcaption></figure><p>You can also customize it, for example enabling it in internal environments only. There are many options, depending on each team’s needs.</p><h3><strong>Quick Tips</strong></h3><p>Before wrapping up, I’d like to share additional quick tips:</p><ol><li><strong>Test alerts in internal environments first.</strong> Before an alert reaches production, validate it where it’s cheap to get wrong: confirm it actually fires when it should, that the threshold makes sense, and that the notification lands in the right channel.</li><li><strong>Every false positive needs a follow-up action.</strong> A false positive isn’t something to mute, it’s a task. Either fine-tune the query or threshold so it’s meaningful, or delete it. Alerts you keep ignoring just train everyone to ignore alerts.</li><li><strong>Watch your metrics cost: cardinality is the lever.</strong> Metrics are billed by active series, and cardinality (the number of unique label-value combinations) is what drives that number up. A single high-cardinality label (a user ID, request ID, or full URL) can multiply your series count and your bill, so keep label values bounded. Check <a href="https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/metrics-costs/">these tips</a> from Grafana Cloud if needed.</li><li><strong>Alerts from logs can get expensive, so use them carefully.</strong> A log-based alert re-runs its query on every evaluation, scanning log volume each time, which adds up fast. Prefer alerting on metrics where you can, and reserve log alerts for the cases that genuinely need them. Check <a href="https://grafana.com/docs/grafana-cloud/cost-management-and-billing/analyze-costs/reduce-costs/logs-costs/">these tips</a> from Grafana Cloud if needed.</li></ol><h3>Wrap up and next steps</h3><p>Looking back, two things mattered more than any single tool.</p><p>The first was technical: treating monitoring as code and making it as simple as possible for engineers to adopt. Moving from manually maintained, drifting dashboards and alerts to a single source of truth in Terraform is what gave us consistency across environments, lower maintenance, and a baseline that every new service inherits for free. That frees up our time to focus on features that really impact our customers, without losing visibility.</p><p>The second was cultural: each team owns its services’ monitoring, alerts are classified by urgency, and every alert, even a false positive, is treated as something to act on rather than ignore.</p><p>As for what’s next:</p><ul><li><strong>Tracing.</strong> This is a bit of déjà vu. Just like our metrics and logs once lived scattered across separate tools, our tracing today sits in Sentry, outside the Grafana stack where everything else now lives. Traces are the missing third pillar, so the next step is the same move we already made for the other two: bring them into our own self-hosted Tempo, get all three pillars in one place, and finally correlate across them. Exemplars, for instance, let us jump from a metric spike straight to the trace behind it.</li><li><strong>Self-hosting Prometheus.</strong> Migrating from Grafana Cloud’s hosted Prometheus to our own instance, for more control over cost and retention.</li></ul><p>We’re not done, and that’s exactly the point. Monitoring is something you keep refining as your systems and your team grow. If any of this resonated, or you’ve tackled these problems differently, we’d genuinely like to hear about it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0e2fdfa8f42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K8S probes done wrong]]></title>
            <link>https://medium.com/@juliorenner123/k8s-probes-done-wrong-184d238b3883?source=rss-582581ec5a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/184d238b3883</guid>
            <category><![CDATA[kubernetes-pods]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Julio Renner]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 18:16:47 GMT</pubDate>
            <atom:updated>2024-03-25T18:16:47.309Z</atom:updated>
            <content:encoded><![CDATA[<p>Probes in K8S are health checks that let the system know if an instance of your application is working as expected. Based on the result of the check, K8S can intervene and take certain actions. However, improper use of probes can cause more harm than good.</p><p>Three probes may be configured in a K8S application:</p><ul><li>Startup Probe: Indicate whether the application within the container has started.</li><li>Readiness Probe: Determine if a container is ready to start accepting traffic.</li><li>Liveness Probe: Determine if a container is running properly.</li></ul><p>The probes are configured at the container level. This means that in a Pod with two containers, each container can have its own set of probes. By default, probes have a <em>failureThreshold </em>of <em>3</em>, meaning there is tolerance before any action is taken.</p><h3>Startup Probe</h3><p>If configured, a startup probe suspends liveness and readiness checks until successful, allowing sufficient time for the application to initiate before the other probes kick off. If the startup probe fails above a defined threshold, the kubelet kills the container, and the container is subjected to its <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy">restart policy</a>. If the startup probe is not configured, its state is set to Success by default.</p><p>When configured, the Startup Probe flow will look like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*MD1dsblrHbmYXjZTPzjIzg.png" /><figcaption>Startup Probe Flow</figcaption></figure><p>Configuring <strong>initialDelaySeconds </strong>for the readiness and liveness probes can get a similar result, though, startup probes are more precise since you do not have to guess a delay for the application to start up. For more details on the probes configuration, including <strong>initialDelaySeconds, </strong>check the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">K8S documentation</a>.</p><h3>Readiness Probe</h3><p>A readiness probe is used to know when a container is ready to accept traffic. When a Pod fails the readiness probe it is considered <em>not ready </em>and<em> </em>it is removed from service load balancers. Meaning that traffic is no longer forwarded to it. It is important to clarify that a container is not killed due to readiness probe failures.</p><p>The <em>Readiness Probe Flow</em> image describes the actions taken by K8S according to the probe result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*KPpkIZFZAJ3L1mahhxedBA.png" /><figcaption>Readiness Probe Flow</figcaption></figure><p>Now, in the image below you can see a demonstration of the <strong>Readiness Probe</strong> in action, the scenario demonstrated is the following:</p><ol><li>The probe is successful and the Service forwards requests to the application.</li><li>The probe fails, though, the failure is still in the accepted threshold. Requests are still forwarded to the instance and the probe is retried.</li><li>The probe fails above the accepted threshold. The instance no longer receives requests.</li><li>Once the probe succeeds the instance starts receiving traffic again.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*bgP94fctEhSEOx_nnYkqmg.gif" /></figure><h3><strong>Liveness Probe</strong></h3><p>Indicates whether the container is Running. If the liveness probe fails, the container is killed and is subjected to its <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy">restart policy</a>. If the process in the container is able to crash on its own whenever it encounters an issue or becomes unhealthy, a liveness probe may not be needed. An example of usage of a liveness probe is a deadlock, where the container enters a state where it is running but it&#39;s unable to respond to requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*JYY-3zsi0_wvMLoO1tzfjw.png" /><figcaption>Liveness Probe Flow</figcaption></figure><p>Let&#39;s check two different scenarios, focused on liveness and readiness probes.</p><h4>Scenario 1:</h4><ul><li>Application A is a web server that also consumes messages from a Kafka topic.</li><li>The <strong>Readiness probe</strong> of Application A takes into account the state of the Kafka connection.</li><li>Every time that a new consumer connects to a Kafka Topic, the consumer group is rebalanced, making the connection check temporarily unhealthy. A new consumer in this scenario is a new instance of <strong>Application A</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*TrXiB6FhCa_Woq4DB3IvNA.jpeg" /></figure><p>Since Application A is a web server and a Kafka consumer, every time the Kafka connection is unhealthy the web server portion of the application becomes unavailable. In each scale activity of the application (up or down), a new Kafka rebalance is triggered and the connection becomes unhealthy, meaning that the application is unable to receive requests, causing downtime. It eventually gets back, though, by then customers were already impacted.</p><p>In the image below you can see that:</p><ol><li>There are two instances of the application Ready and Receiving requests.</li><li>A new instance is added by K8S due to a scaling activity.</li><li>The existing instances <strong>Readiness Probe</strong> start failing due to the Kafka consumer group rebalance, since the failures didn&#39;t go above the threshold, requests are still forwarded to them.</li><li>All instances become <strong>NOT Ready </strong>and K8S stops forwarding traffic to the application.</li><li>Eventually, the Kafka consumer group rebalance is finished and <strong>Readiness Probe </strong>succeeds, traffic is then forwarded to the application again.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*Mw6NPlWQC0b9ZrMnXEo2Ww.gif" /></figure><p>One additional and relevant piece of information for this <em>hypothetical scenario </em>😜, is that the decision to monitor the Kafka connection state was made consciously to solve frequent issues during <a href="https://redpanda.com/guides/kafka-performance/kafka-rebalancing">Kafka consumer group rebalances</a>, where deadlocks were encountered and the application had to be restarted to solve it. However, the decision was made for the <strong>liveness probe, </strong>not <strong>readiness</strong>. Another mistake made here is that the same endpoint was reused for both readiness and liveness probes. Meaning that the check was propagated to the readiness probe unintentionally.</p><h4>Scenario 2:</h4><ul><li>Application B is a web server that communicates with an external dependency through RabbitMQ.</li><li>The liveness probe of Application B considers the queue size of RabbitMQ in its check. If above a threshold, it fails.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*TrXiB6FhCa_Woq4DB3IvNA.jpeg" /></figure><p>Once the queue size is above a threshold for any reason (let&#39;s say there&#39;s indeed an issue in the consumption of the messages), the application gets into a state of consecutive restarts. At some points all the instances could be unavailable, leading to sporadic downtime and performance degradation. Therefore, the liveness probe just made our existing issue worse.</p><p>In the image below you can see the instability in the application caused by improper usage of the liveness probe:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*-kI4zS0MXE8aM9258hMMgA.gif" /></figure><h3>Lessons learned</h3><ol><li>Understand properly how probes behave before setting them up. They can make your problems worse.</li><li>Avoid checking application dependencies in Liveness/Readiness probes. Instead, have specific monitoring for them.</li><li>Be careful reusing the same endpoints for probes. Liveness and Readiness probes behave differently, and developers may not realize the real impact when updating both when changing the endpoint.</li></ol><p>To identify what to check in each endpoint, ask yourself the following questions:</p><ul><li><strong>Startup Probe:</strong> What do I need to check to confirm the application initialization is finished?</li><li><strong>Readiness Probe:</strong> What do I need to check to confirm the application is ready to respond to requests?</li><li><strong>Liveness Probe:</strong> What do I need to check to confirm that the application needs to be restarted?</li></ul><p>Hope it helps, feedbacks are always welcomed :) Also, curious to hear about additional scenarios in which probes caused more harm than good.</p><h3>References</h3><ol><li><a href="https://www.padok.fr/en/blog/kubernetes-probes#Liveness_probe">https://www.padok.fr/en/blog/kubernetes-probes#Liveness_probe</a></li><li><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/">https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/</a></li><li><a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/">https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/</a></li><li><a href="https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html?source=post_page-----0f5302f2ff8b--------------------------------">https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html?source=post_page-----0f5302f2ff8b--------------------------------</a></li><li><a href="https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes?source=post_page-----0f5302f2ff8b--------------------------------">https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes?source=post_page-----0f5302f2ff8b--------------------------------</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=184d238b3883" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K8S- Creating a kube-scheduler plugin]]></title>
            <link>https://medium.com/@juliorenner123/k8s-creating-a-kube-scheduler-plugin-8a826c486a1?source=rss-582581ec5a16------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a826c486a1</guid>
            <category><![CDATA[kubernetes-administration]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kube-scheduler]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Julio Renner]]></dc:creator>
            <pubDate>Sun, 25 Jul 2021 20:49:51 GMT</pubDate>
            <atom:updated>2021-07-26T03:02:09.941Z</atom:updated>
            <content:encoded><![CDATA[<h3>K8S - Creating a kube-scheduler plugin</h3><p>Saying it in a few words, the K8S scheduler is responsible for assigning <em>Pods </em>to<em> Nodes. </em>Once a new pod is created it gets in the scheduling queue. The attempt to schedule a pod is split in two phases: the <em>Scheduling </em>and the <em>Binding cycle.</em></p><p>In the <em>Scheduling cycle</em> the nodes are filtered, removing those that don’t meet the pod requirements. Next, the <em>feasible nodes</em> (the remaining ones)<em>, </em>are ranked based on a given score. Finally, the node with highest score is chosen. These steps are called <em>Filtering </em>and <em>Scoring [1].</em></p><p>Once a node is chosen, the scheduler needs to make sure <em>kubelet</em> knows it needs to start the pod (containers) in the selected node. The step related to starting the pod into the selected node is called <em>Binding Cycle [2].</em></p><p>The <em>Scheduling </em>and<em> Binding cycle</em> are composed by stages that are executed sequentially to calculate the pod placement. These stages are called extension points and can be used to shape the placement behavior. <em>Scheduling Cycles</em> for different pods are run sequentially, meaning that the <em>Scheduling Cycle</em> steps will be executed for one pod at a time, whereas <em>Binding Cycles</em> for different pods may be executed concurrently.</p><p>The components that implement the extension points of kubernetes scheduler are called <em>Plugins. </em>The native scheduling behavior is implemented using the <em>Plugin </em>pattern as well, in the same way that custom extensions, making the core of the kube-scheduler lightweight as the main scheduling logic is placed in the plugins.</p><p>The extension points where the plugins can be applied are shown in <em>Figure 1. </em>A plugin can implement one or more of the extension points and a detailed description of each can be found in [4] (I won’t be copying stuff here, check it there before continuing 😃).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZwRBzbU4Bs3mE1tIRYPEg.png" /><figcaption>Figure 1</figcaption></figure><p>To configure the <em>Plugins </em>that should be executed in each extension point, and then change the scheduling behavior, kube-scheduler provides <em>Profiles </em>[3]<em>. </em>A scheduling <em>Profile </em>describes which plugins should be executed on each stage mentioned in [4]. It is possible to provide multiple profiles, which means that there’s no need to deploy multiple schedulers to have different scheduling behaviors [5].</p><h3>kube-scheduler</h3><p>The kube-scheduler is implemented in Golang and <em>Plugins </em>are included to it in compilation time. Therefore, if you want to have your own plugin, you will need to have your own scheduler image.</p><p>A new plugin needs to be registered and get configured to the plugin API. Also, it needs to implement the extension points interfaces that are defined in the <a href="https://github.com/kubernetes/kubernetes/blob/ed3e0d302fb546653b78df583569b0311687a7a8/pkg/scheduler/framework/interface.go#L268">kubernetes scheduler framework package</a>. Check out how it looks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c7a46ce88db72ba44e58b24d81b2dc9/href">https://medium.com/media/7c7a46ce88db72ba44e58b24d81b2dc9/href</a></iframe><p>The scheduler’s code allows to add new plugins without having to fork it. For that, developers just need to write their own main() wrapper around the scheduler. As plugins must be compiled with the scheduler, writing a wrapper allows to re-use the scheduler’s code in a clean way [7].</p><p>To do that, the main function will import the k8s.io/kubernetes/cmd/kube-scheduler/app and use the NewSchedulerCommand to register the custom plugins, providing the respective name and the constructor function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/273ed6b707fb4e7fa927e2b000505289/href">https://medium.com/media/273ed6b707fb4e7fa927e2b000505289/href</a></iframe><h4>Configuration</h4><p>The kube-scheduler configuration is where the profiles can be configured. Each profile allows plugins to be enabled, disabled and configured according to the configurations parameters defined by the plugin. Each profile configuration is separated into two parts [9]:</p><ol><li>A list of enabled plugins for each extension point and the order they should run. If one of the extension points list is omitted, the default list will be used.</li><li>An optional set of custom plugin arguments for each plugin. Omitting config args for a plugin is equivalent to using the default config for that plugin.</li></ol><p>Plugins that are enabled in different extension points must be configured explicitly in each of them.</p><p>The configuration is provided through the <a href="https://kubernetes.io/docs/reference/config-api/kube-scheduler-config.v1beta1/#kubescheduler-config-k8s-io-v1beta1-KubeSchedulerConfiguration">KubeSchedulerConfiguration</a> struct. To enable it, it needs to be written to a configuration file and its path provided as a command line argument to kube-scheduler. E.g.:</p><pre>kube-scheduler --config=/etc/kubernetes/networktraffic-config.yaml</pre><p>Below you can see an example configuration of the NetworkTraffic plugin. In the example, the clientConnection.kubeconfig points to the kubeconfig path used by the kube-scheduler, with its defined authorizations in the control plane nodes. The profiles section overwrites the default-scheduler score phase enabling the NetworkTraffic plugin and disabling the others defined by default. The pluginConfig sets the configuration of the plugin, that will be provided during its initialization [8].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0045789f9397f4527340fb7b1299da4/href">https://medium.com/media/e0045789f9397f4527340fb7b1299da4/href</a></iframe><p>PS: If you have HA with multiple control plane nodes, the configuration needs to be applied for each of them.</p><h3>Creating a custom plugin</h3><p>Now that we understand the basics of kube-scheduler, we can do what we came here for. As we’ve seen previously, adding a custom plugin requires to include our code during compilation time and we don&#39;t need to fork the scheduler code for that.</p><p>To proceed, we could create an empty repository and wrap the scheduler as described before, however, the project <a href="https://github.com/kubernetes-sigs/scheduler-plugins">scheduler-plugins</a> already does that and provides some custom plugins that are good examples to follow. So, we will just start from there.</p><p>Fork the <a href="https://github.com/kubernetes-sigs/scheduler-plugins">scheduler-plugins</a> repository and pull it into $GOPATH/src/sigs.k8s.io. With that done, we can start :)</p><p>To keep following the next steps, you need to:</p><ol><li>Have a K8S cluster (I am using a cluster created with <a href="https://github.com/kubernetes-sigs/kubespray">kubespray</a>).</li><li>Have prometheus configured with node-exporter. Check <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">kube-prometheus-stack</a>.</li></ol><h4>NetworkTraffic Plugin</h4><p>For this example, we are going to build a Score Plugin named &quot;NetworkTraffic&quot; that favors nodes with lower network traffic. To gather that information we will query prometheus.</p><p>To start, create the folder pkg/networktraffic and the files networktraffic.go and prometheus.go inside your fork of scheduler-plugins. The structure should look like this:</p><pre>|- pkg<br>|-- networktraffic<br>|--- networktraffic.go<br>|--- prometheus.go</pre><p>In the networktraffic.go we are going to have the implementation of the ScorePlugin interface and in the prometheus.go we will keep the logic to interact with prometheus.</p><h4>Prometheus communication</h4><p>In the prometheus.go we will start by declaring the struct used to interact with Prometheus. It will have the fields networkInterface and timeRange, which can be used to configure the query we will be executing. The field address points to the prometheus service on K8S and can also be configured. The field api will be used to store the prometheus client, which is created based on the address provided.</p><pre>type PrometheusHandle struct {<br>  networkInterface string<br>  timeRange        time.Duration<br>  address          string<br>  api              v1.API<br>}</pre><p>Now that we have the basic structure we can also implement the querying. We will be using the sum of the received bytes in a time range per node in a specific network interface. The kubernetes_node filter will query the metrics for the node provided, as described by the query below. The device filter will query the metrics on the provided network interface, and the last value between [%s] defines the time range taken into account. sum_over_time will sum all the values in the provided time range.</p><pre>sum_over_time(node_network_receive_bytes_total{kubernetes_node=\&quot;%s\&quot;,device=\&quot;%s\&quot;}[%s])</pre><p>At the end, the prometheus.go file will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bed130b841a179d2fb0386b1c6f26b1/href">https://medium.com/media/5bed130b841a179d2fb0386b1c6f26b1/href</a></iframe><h4>ScorePlugin interface</h4><p>Having the interaction with Prometheus done, we can move to the implementation of the Score Plugin. As mentioned, we will need to implement the Score Plugin Interface from the scheduler framework:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19c333384f20627841379639e0d929e2/href">https://medium.com/media/19c333384f20627841379639e0d929e2/href</a></iframe><p>The Score function is called for each node and returns whether it was successful and an integer indicating the rank of the node. At the end of the Score plugin execution, we should have a Score value in the range from 0 to 100. In some cases it could be difficult to have a value within that range without knowing the score of other nodes, for example. For those scenarios, we can use the NormalizeScore function implemented in the ScoreExtensions interface. The NormalizeScore function receives the result of all nodes and allows them to be changed.</p><p>Moreover, the ScorePlugin interface also have the Plugin interface as an <a href="https://travix.io/type-embedding-in-go-ba40dd4264df">embedded</a> field. So, we must implement its Name() string function.</p><p>Now that we understand the ScorePlugin interface, let&#39;s go to the networktraffic.go file. We will start by defining the NetworkTraffic struct:</p><pre>// NetworkTraffic is a score plugin that favors nodes based on their<br>// network traffic amount. Nodes with less traffic are favored.<br>// Implements framework.ScorePlugin<br>type NetworkTraffic struct {<br>  handle     framework.FrameworkHandle<br>  prometheus *PrometheusHandle<br>}</pre><p>With the structure defined, we can proceed with the Score function implementation. It will be straightforward. We will only call the GetNodeBandwidthMeasure function from our Prometheus structure providing the node name. The call will return a <a href="https://github.com/prometheus/common/blob/9e276a1555878cfdaeeea42127f40f2518861145/model/value.go#L113">Sample</a> which holds the value in the Value field. We will basically return it for each node.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9a853d3dc912aa09873dccb7274826a/href">https://medium.com/media/e9a853d3dc912aa09873dccb7274826a/href</a></iframe><p>Next, we will have returned the total bytes received by each node in a determined period of time. However, the scheduler framework expects a value from 0 to 100, thus, we still need to normalize the values to fulfill this requirement.</p><p>To do the normalization, we will implement the ScoreExtensions interface mentioned before. We will implement the interface embedded in the NetworkTraffic struct. In the ScoreExtensions function we will simply return the struct which implements the interface. The logic is placed under the NormalizeScore function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8adb746ecae00f504acda9fd5dfcb69b/href">https://medium.com/media/8adb746ecae00f504acda9fd5dfcb69b/href</a></iframe><p>The NormalizeScore basically will take the highest value returned by prometheus and use it as the highest possible value, corresponding to the framework.MaxNodeScore (100). The other values will be calculated relatively to the highest score using the rule of three.</p><p>Finally, we will have a list where the nodes with more network traffic have a greater score in the range of [0,100]. If we use it like it is, we would favor nodes that have higher traffic, so, we need to reverse the values. For that, we will simply replace the node score with the result of the rule of three, subtracted by the max score.</p><p>An example of the calculation which take as an example three nodes (<em>a, b </em>and <em>c</em>), the values are in bytes, is given below:</p><pre>a =&gt; 1000000   # 1MB<br>b =&gt; 1200000   # 1,2MB<br>c =&gt; 1400000   # 1,4MB</pre><pre>higherScore = 1400000</pre><pre>Y = (node.Score * framework.MaxNodeScore) / higherScore</pre><pre>Ya = 1000000 * 100 / 1400000<br>Yb = 1200000 * 100 / 1400000<br>Yc = 1400000 * 100 / 1400000</pre><pre>Ya = 71,42<br>Yb = 85,71<br>Yc = 100</pre><pre>Xa = 100 - Ya<br>Xb = 100 - Yb<br>Xc = 100 - Yc</pre><pre><strong>Xa = 28,58<br>Xb = 14,29<br>Xc = 0</strong></pre><p>With that explained, we have the main pieces of our plugin. However, that&#39;s not all. As mentioned before, the scheduler plugins can be configured, and there are three configurations we will allow in our Network Traffic plugin, which were already mentioned:</p><ul><li>Prometheus address</li><li>Prometheus query time range</li><li>Prometheus query node network interface</li></ul><p>Those values will be provided during the instantiation of the NetworkTraffic plugin by the scheduler framework, and we will need to declare a new struct called NetworkTrafficArgs that will be used to parse the configuration provided in the KubeSchedulerConfiguration. For that, we need to add a new function with the logic to create an instance of the NetworkTraffic plugin, described below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfbd6d4756def29acc02132f63c879cf/href">https://medium.com/media/cfbd6d4756def29acc02132f63c879cf/href</a></iframe><p>The New function follows the scheduler framework<a href="https://github.com/kubernetes/kubernetes/blob/4aae71695a8dd43918702fafd81e0401721d79d9/pkg/scheduler/framework/runtime/registry.go#L29">PluginFactory</a> interface.</p><p>We still haven&#39;t declared the NetworkTrafficArgsstruct, and that will come next. However, we have (almost) all we need for networktraffic.go:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52e4eb8ea2e3b995cbfb5e75415f1176/href">https://medium.com/media/52e4eb8ea2e3b995cbfb5e75415f1176/href</a></iframe><h4>Configuration</h4><p>The scheduler-plugins project holds the configurations under pkg/apis folder. So, we will have ours plugin config there as well.</p><p>We will add the configuration in two places: pkg/apis/config/types.go and pkg/apis/config/v1beta1/types.go. The config/types.go holds the struct we will use in the New function, while the v1beta1/types.go holds the struct used to parse the information from the KubeSchedulerConfiguration.</p><p>Also, the config struct must follow the name pattern &lt;Plugin Name&gt;Args, otherwise, it won&#39;t be properly decoded and you will face issues.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98b31ac6cd96a38309c4001ed030ef1e/href">https://medium.com/media/98b31ac6cd96a38309c4001ed030ef1e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/59c93ebebfaaa8d03d1d1f3c82e2efed/href">https://medium.com/media/59c93ebebfaaa8d03d1d1f3c82e2efed/href</a></iframe><p>With the structs added, we need to execute the hack/update-codegen.sh script. It will update the generated files with functions as DeepCopy for the added structures.</p><p>Furthermore, we will add a new function SetDefaultNetworkTrafficArgs in the config/v1beta1/defaults.go. The function will set the default values for the NetworkInterface and TimeRangeInMinutes values, but Address still needs to be provided.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e97a10aeb9b680b636081b673ea89130/href">https://medium.com/media/e97a10aeb9b680b636081b673ea89130/href</a></iframe><p>To finish the default values configuration, we need to make sure the function above is registered in the v1beta1 schema. Thus, make sure that it is registered in the file pkg/apis/config/v1beta1/zz_generated.defaults.go.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e8309b28b18a07b767da174ea510885/href">https://medium.com/media/9e8309b28b18a07b767da174ea510885/href</a></iframe><h4>Registering Plugin and Configuration</h4><p>Now that the arguments structure is defined, our Plugin is ready. However, we still need to register the plugin and the configuration in the scheduler framework.</p><p>The scheduler-plugins project already has a couple plugins registered which makes things a bit easier as we have examples. The registration for the plugin configuration is placed under pkg/apis/config. In the file register.go we need to add the NetworkTrafficArgs in the call to the AddKnownTypes function. The same needs to be done in the pkg/apis/config/v1beta1/register.go file. With both files changed, the configuration registration is done.</p><p>Next, we move to the plugin registration, which is done in the cmd/scheduler/main.go file. In the <em>main </em>function, the NetworkTraffic plugin name and constructor need to be provided as arguments to the NewSchedulerCommand. It should look like this:</p><pre>command := app.NewSchedulerCommand(<br>  app.WithPlugin(networktraffic.Name, networktraffic.New),<br>)</pre><p>Also, notice that in the in the <a href="https://github.com/juliorenner/scheduler-plugins/blob/921af74d339e99842d48bec88c27778749cff32b/cmd/scheduler/main.go#L29">main.go</a> file we have the import of sigs.k8s.io/scheduler-plugins/pkg/apis/config/scheme, which initializes the scheme with all configurations we have introduced in the pkg/apis/config files.</p><p>With that we are done from a code perspective. The full implementation can be found <a href="https://github.com/juliorenner/scheduler-plugins">here</a>, it also includes a couple of unit tests, so check it out!</p><h4>Deploying and using the Plugin</h4><p>Now that we have the plugin done, we can deploy it in our K8S cluster and start using it. In the scheduler-plugins repository, there is a documentation on how to do it, check it <a href="https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md">here</a>. We would basically need to adapt those steps with the Plugin we just implemented.</p><p>Nonetheless, before applying the changes to the cluster, make sure that you have build the scheduler container image and pushed it to a container registry which is accessible from your kubernetes. I won&#39;t go into the details as it will differ based to the environment used. You can check the Makefile as well, as there are some commands to build and push the image and also <a href="https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/develop.md#how-to-build">this development</a> doc may help you.</p><p>As our plugin doesn&#39;t introduce any CRD, a couple steps in the <a href="https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md">scheduler-plugins install doc</a> can be skipped. As I mentioned, I am using a cluster created with kubespray with HA. Therefore, I will need to repeat the following steps on each control plane node.</p><ol><li>Log into the control plane node.</li><li>Backup kube-scheduler.yaml</li></ol><pre>cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/kube-scheduler.yaml</pre><p>3. Create /etc/kubernetes/networktraffic-config.yaml and change the values according to your environment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0045789f9397f4527340fb7b1299da4/href">https://medium.com/media/e0045789f9397f4527340fb7b1299da4/href</a></iframe><p>4. Modify /etc/kubernetes/manifests/kube-scheduler.yaml to run scheduler-plugins with Network Traffic. The changes we have made are:</p><ul><li>Add the command arg --config=/etc/kubernetes/networktraffic-config.yaml.</li><li>Change the image name.</li><li>Add a volume pointing to the configuration absolute path.</li><li>Add a volumeMount to make the configuration available to the scheduler pod.</li></ul><p>Check the example below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f634d07f1a6797273b37fb5e5aa51d9/href">https://medium.com/media/7f634d07f1a6797273b37fb5e5aa51d9/href</a></iframe><p>Now, we can start taking advantage of our custom plugin. Once you check the logs of the running pod, you should see lines with the node bandwidth returned from prometheus and you can make sure the behavior is as expected. Below, we can see that node4 correctly has the higher score, as it is the node with less network traffic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*Tv2hNn0vY3a_Y_2byU1v0w.png" /></figure><p>Hope this post is useful to you and feel free to give feedbacks on the comments, they are very appreciated!</p><h3>References</h3><p>1: <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/">https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/</a></p><p>2: <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/">https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/</a></p><p>3: <a href="https://kubernetes.io/docs/reference/scheduling/config/#profiles">https://kubernetes.io/docs/reference/scheduling/config/#profiles</a></p><p>4: <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#extension-points">https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#extension-points</a></p><p>5: <a href="https://kubernetes.io/docs/reference/scheduling/config/#multiple-profiles">https://kubernetes.io/docs/reference/scheduling/config/#multiple-profiles</a></p><p>6: <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md">https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md</a></p><p>7: <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md#custom-scheduler-plugins-out-of-tree">https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md#custom-scheduler-plugins-out-of-tree</a></p><p>8: <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md#optional-args">https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md#optional-args</a></p><p>9: <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md#configuring-plugins">https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/624-scheduling-framework/README.md#configuring-plugins</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a826c486a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>