<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rodrigue Chakode on Medium]]></title>
        <description><![CDATA[Stories by Rodrigue Chakode on Medium]]></description>
        <link>https://medium.com/@rodrigue-chakode?source=rss-81362ccd540------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lIECXv3waUk_YlzI.jpg</url>
            <title>Stories by Rodrigue Chakode on Medium</title>
            <link>https://medium.com/@rodrigue-chakode?source=rss-81362ccd540------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:38:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rodrigue-chakode/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why the Bottleneck in Kubernetes FinOps Isn’t What You Think]]></title>
            <link>https://medium.com/swlh/why-the-bottleneck-in-kubernetes-finops-isnt-what-you-think-9c7c29c5eaad?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c7c29c5eaad</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Tue, 02 Jun 2026 12:39:17 GMT</pubDate>
            <atom:updated>2026-06-02T13:44:48.595Z</atom:updated>
            <content:encoded><![CDATA[<h3>A reading of the State of FinOps 2026 report, and what it means for the next layer of cloud-native cost intelligence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbWXq_RXgfnhrD_xaCJTUA.png" /><figcaption>A reading of the State of FinOps 2026 report, and what it means for the next layer of cloud-native cost intelligence</figcaption></figure><p>If you spent the last decade in cloud engineering, you have probably watched the FinOps discipline mature in front of you. From spreadsheet-driven cost reviews in 2015, to dedicated tooling in 2020, to executive-level visibility today. The discipline has grown up.</p><p>And yet, walk into any platform engineering team in 2026 and ask a simple question: <em>“Did you look at your cost dashboard this week?”</em></p><p>Most will hesitate. Some will admit they did not. A few will tell you the dashboard exists, somewhere, and that someone in the finance team probably looks at it.</p><p>The State of FinOps 2026 report, published by the FinOps Foundation a few weeks ago, captures this paradox better than any other industry document I have read this year. With 1,192 respondents representing over 83 billion dollars in annual cloud spend, it is the most authoritative snapshot of the discipline available.</p><p>And buried in its 60 pages is a sentence that stopped me cold:</p><blockquote>“Dashboards are table stakes of yesterday, reactive. You have to move to proactive, real-time, automation.”</blockquote><p>The quote comes from a practitioner. The Foundation chose to feature it in its editorial pull-quotes, which is its way of saying: <em>we agree</em>.</p><p>This article is a reading of what the report tells us, what it does not say out loud, and why I believe the next breakthrough in cloud-native cost intelligence will not come from better dashboards. It will come from a fundamentally different <strong>interface</strong>.</p><h3>The first paradox: cost is the number one Kubernetes challenge, but few teams act on it</h3><p>Let us start with the numbers everyone agrees on.</p><p>Cost is now the top challenge in Kubernetes operations. Spectro Cloud’s 2025 State of Production Kubernetes report, surveying 455 platform engineers, places cost as the number one operational concern, ahead of security and complexity, with 88 percent of organizations reporting year-over-year total cost of ownership growth.</p><p>The State of FinOps 2026 echoes this trajectory from a different angle: 98 percent of respondents now manage AI spend, up from 31 percent two years ago. Compute costs have exploded, GPU costs have exploded, and the discipline that was supposed to keep them in check is, by the practitioners’ own admission, struggling to scale.</p><p>But here is what nobody talks about in those numbers. The data exists. The metrics are collected. Prometheus has been scraping cluster telemetry for nearly a decade. Cloud providers have detailed billing APIs. Open-source projects like OpenCost, Kubecost, and <a href="https://github.com/realopslabs/kubeledger">KubeLedger</a> have been turning that raw telemetry into structured cost data for years.</p><p>The bottleneck is not data collection. The bottleneck is something else entirely.</p><h3>The second paradox: insights exist, but they do not reach the people who decide</h3><p>The Foundation’s report makes a quiet but striking observation about the organizational shift that happened in the last three years. In 2023, the typical FinOps team reported to the CFO. In 2026, 78 percent of FinOps teams report to the CTO or CIO. The CFO rattachement has collapsed to 8 percent.</p><p>This is not a cosmetic change. It signals that <strong>FinOps has stopped being a finance function and become a technical leadership function</strong>. The people responsible for cloud spend are now the same people responsible for architecture decisions, platform reliability, and engineering velocity.</p><p>Now consider what that means for tooling. The traditional FinOps dashboard was designed for a financial analyst reading a monthly report. Tabbed views, drill-down filters, exportable CSVs, monthly true-ups. Useful, but slow. A CTO does not have time to learn the dashboard’s UX every Monday morning. A Head of Platform Engineering does not want to click through six filter dropdowns to find out which team blew the budget last quarter.</p><p>So what do they do? They ask someone. They ping the FinOps lead on Slack. They wait. They get a screenshot. They forward it to their boss.</p><p>That round-trip is the bottleneck. Not the data. Not the dashboards. The <strong>interface</strong> between the data and the decision.</p><h3>The third paradox: AI is everywhere in FinOps, except where it would matter most</h3><p>This is where the 2026 report becomes really interesting. The Foundation lists, among the top emerging use cases for AI in FinOps:</p><ol><li>Anomaly detection and faster alerting</li><li>Automated right-sizing</li><li><strong>Natural language querying of cost data</strong></li><li>Automated procurement of discount instruments</li><li>Automated tagging to accelerate allocation</li></ol><p>The third item is the one that should make every FinOps tool builder pause.</p><p>For two years, the industry has been bolting AI features onto existing dashboards. A chatbot in the corner of the screen. An “AI insights” tab that summarizes what the human could read anyway. A recommendation engine that suggests right-sizing actions the team will probably ignore. All of these are AI <strong>added to the existing interface</strong>.</p><p>What the Foundation’s report identifies is something else. It is the recognition that <strong>the interface itself should change</strong>. That asking <em>“which namespaces drove our CPU growth last quarter?”</em> in plain English is not a feature, it is the new primitive. And that the entire workflow of cloud cost intelligence should be rebuilt around conversational, contextual, just-in-time queries, rather than around static dashboards that nobody opens.</p><p>This is not science fiction. The protocol layer to make this possible already exists. It is called the <strong>Model Context Protocol</strong>.</p><h3>Why MCP changes the equation</h3><p>The Model Context Protocol, introduced by Anthropic in late 2024 and adopted with surprising speed by the broader ecosystem, is a standard for connecting language models to data sources. It is to AI assistants what the Language Server Protocol was to code editors a decade ago: a way for any tool to talk to any client, without bespoke integration.</p><p>Concretely, an MCP server exposes a set of read-only tools that describe what the data is, how to query it, and what to do with the result. The language model becomes the orchestrator: it picks the right tool, parses the parameters, calls the server, and interprets the response in the conversation. The data layer stays clean, the narrative layer stays in the AI client, and the user never has to learn a query language.</p><p>For cloud cost data, this is transformative. Not because LLMs are magical, they are not, but because <strong>the interface friction disappears</strong>. When the cost data is sitting behind an MCP server, any tool that speaks MCP can query it. Claude Desktop, Cursor, Mistral’s Le Chat, custom internal assistants. The same data becomes accessible from wherever the engineer happens to be working.</p><p>This is not theory. This is what I just shipped with the <a href="https://github.com/realopslabs/kubeledger#mcp-integration-ai-assistant--requires-v26050">KubeLedger MCP server</a>.</p><h3>What this looks like in practice</h3><p>KubeLedger has been collecting and structuring Kubernetes cluster telemetry for years. The new MCP server adds a thin conversational layer on top of that data. Ten read-only tools exposing namespace usage, top consumers, breakdown with cluster overhead separation, efficiency ratios with descriptive classification, hourly trends, period comparisons, and group aggregation by glob pattern.</p><p>When you ask, in plain English: <em>“What’s the cost concentration across our cluster?”</em>, the language model decides to call the get_namespace_breakdown tool. The KubeLedger server returns structured data with the breakdown, the concentration metrics (top-3 share, top-10 share), and a separate cluster_overhead block for the non-allocatable capacity. The language model formats the answer in a readable summary, but the underlying data is auditable, traceable, and deterministic.</p><p>A few design choices deserve emphasis, because they matter for trust:</p><p><strong>Read-only by design</strong>. The MCP server never writes, never modifies, never takes action. It exposes data. The narrative intelligence lives in the AI client, where it belongs.</p><p><strong>No embedded LLM</strong>. KubeLedger does not bundle a language model. The server is a descriptive surface. The model is chosen by the user, and can be Claude, Mistral, a self-hosted Llama, or anything else that speaks MCP.</p><p><strong>Provenance for every response</strong>. Every tool call returns a metadata block with the cost model, the data window, the source file, and a freshness warning. Nothing is opaque. A skeptical engineer can verify exactly where every number came from.</p><p><strong>Descriptive, not prescriptive</strong>. The efficiency classification surfaces three labels (over-provisioned, balanced, under-provisioned) with documented thresholds. It never tells you what to do. The decision stays human.</p><p>These choices are not accidental. They are a deliberate counter-position to the wave of “AI-powered FinOps” products that promise actionable recommendations, automated remediation, and predictive savings. Those products may eventually deliver on their promises. But in 2026, what most platform teams need is not more automation. It is <strong>clearer access to the data they already collect</strong>.</p><h3>What the FinOps Foundation report does not say out loud</h3><p>Reading between the lines of the report, three observations emerge that I think will shape the next two years of FinOps tooling.</p><p><strong>First</strong>, pre-deployment cost estimation will become a battleground. The report identifies “pre-deployment architecture costing” as the number two most-requested capability missing from current tooling. Practitioners are building it themselves with home-grown scripts and adapted IaC scanners. None of the existing commercial platforms handle Kubernetes manifests natively. Whoever cracks this first, integrating real cluster utilization patterns into PR review comments, owns the next phase of shift-left FinOps.</p><p><strong>Second</strong>, the optimization era is over. As one advanced practitioner is quoted in the report: <em>“I haven’t been focused on optimization as priority one for a long time.”</em> The big rocks of waste have been picked. The remaining gains require disproportionate effort. The next wave of value comes from <strong>governance, allocation, and shaping spend before it happens</strong>, not from squeezing the last five percent out of existing workloads.</p><p><strong>Third</strong>, the Single Pane of Glass is finally a real demand, not just a marketing line. The report lists it as the number three most-wanted capability. Practitioners want one place to see cloud, SaaS, licensing, private cloud, and data center costs together. Vendor consolidation will accelerate. So will the temptation to lock teams into proprietary platforms. The open-source response to that consolidation matters.</p><h3>The deeper question: where does your cost data go?</h3><p>There is one dimension the Foundation’s report does not address, but that I find increasingly hard to ignore.</p><p>When you query your Kubernetes cost data in natural language through a commercial AI-powered FinOps platform, where does that telemetry actually go? Whose model processes it? Under which jurisdiction? With what guarantees about retention, training, and access?</p><p>For regulated sectors in Europe, finance, healthcare, defense, public sector, this question is not abstract. NIS2 demands traceable data flows. SecNumCloud requires data residency. The EU AI Act adds governance obligations on high-impact AI usage. DORA imposes operational resilience requirements on financial entities.</p><p>The conversational layer for FinOps cannot be a black box hosted in a foreign jurisdiction. It needs to be self-hostable, LLM-agnostic, and built on open standards. This is not a French paranoia, it is a structural requirement for any tool that aspires to enterprise adoption in Europe in 2026 and beyond.</p><p>This is why KubeLedger MCP is open-source, self-hostable, and designed to run inside your cluster, your VPC, your air-gapped environment. The data never leaves your perimeter unless you choose to send it somewhere. Pair it with Claude if you trust Anthropic. Pair it with Mistral if you prefer European sovereignty. Pair it with a self-hosted Llama if you need full isolation. The choice is yours, and the architecture supports all three equally.</p><h3>What I am betting on</h3><p>The State of FinOps 2026 report is, in my reading, the moment the industry collectively realized that the next phase of cloud-native cost intelligence will not come from better dashboards. The discipline has matured past that point.</p><p>What comes next is a layer of conversational, contextual, just-in-time access to cost data. Not as a feature, but as a primitive. Not as a chatbot bolted onto an existing platform, but as an open protocol that any client can speak. Not as a hosted SaaS that processes your telemetry in someone else’s cloud, but as a self-hostable component that runs where your data lives.</p><p>The KubeLedger MCP server is one bet on that future. It is open-source, it is live today, and it works on real production clusters. The first ten tools cover the descriptive layer: usage, concentration, efficiency, trends, comparisons, groupings. The next layer, GPU intelligence and pre-deployment cost estimation, is on the roadmap.</p><p>If you are running Kubernetes at scale, and you are tired of the dashboard-screenshot-Slack roundtrip, I would love to hear what your first question to your cluster would be.</p><p>You can find the project on GitHub, the spec is fully documented, and the MCP server runs in any Kubernes environment : <a href="https://github.com/realopslabs/kubeledger">https://github.com/realopslabs/kubeledger</a>.</p><p>The bottleneck is no longer the data. It is the interface.</p><p>It is time we built a better one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c7c29c5eaad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/why-the-bottleneck-in-kubernetes-finops-isnt-what-you-think-9c7c29c5eaad">Why the Bottleneck in Kubernetes FinOps Isn’t What You Think</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPU Costs Out of Control? Track Usage and Gain Visibility Across Kubernetes]]></title>
            <link>https://rodrigue-chakode.medium.com/gpu-costs-out-of-control-track-usage-and-gain-visibility-across-kubernetes-811b4eb7ca78?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/811b4eb7ca78</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[nvidia]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 09:19:33 GMT</pubDate>
            <atom:updated>2026-01-13T11:19:06.702Z</atom:updated>
            <content:encoded><![CDATA[<h4>Gain Insights into GPU Usage by Application Namespace</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAXXqC0SabNmwCmODuj7Ig.png" /><figcaption>GPU Usage Accounting — Hourly Trends · Daily Details · Montly Accouting</figcaption></figure><h3>Introduction</h3><p>Organizations running AI and ML workloads on Kubernetes are facing a significant challenge: GPU resources have become their largest infrastructure expense, yet they often lack the visibility needed to manage these costs effectively. With NVIDIA A100 GPUs costing over $ 30,000 per unit and cloud GPU instances reaching $2–4 per hour, understanding exactly which teams, applications, and namespaces consume these resources has become a business imperative.</p><p>While CPU and memory tracking has been standard practice for years, GPU monitoring remains a blind spot for many organizations. This gap leads to overprovisioned clusters, underutilized hardware, and finance teams requesting cost breakdowns that platform engineers simply cannot provide.</p><p>To illustrate the scale of this challenge, consider a typical ML training cluster where CPU and memory infrastructure costs around $5,000 per month, while GPU instances (8x A100) can reach $25,000 per month. Many organizations have detailed consumption breakdowns for traditional compute resources but lack equivalent visibility into their GPU spending, which often represents 80% or more of total infrastructure costs.</p><h3>Closing the GPU Cost Visibility Gap</h3><p>As described hereafter, recent versions of <a href="https://github.com/rchakode/kube-opex-analytics"><strong>kube-opex-analytics</strong></a> close this cost visibility gap. Extending its proven CPU and memory analytics capabilities to GPU workloads, it delivers the same namespace-level visibility and cost allocation features that teams already rely on for traditional compute resources. The dashboard that displays namespace-level CPU and Memory consumption also shows GPU utilization, memory bandwidth, and VRAM usage, all consolidated and ready for analysis over time.</p><h3>How GPU Tracking Works</h3><p>The solution integrates with NVIDIA’s DCGM (Data Center GPU Manager) Exporter, the industry standard for GPU telemetry in Kubernetes environments. This integration relies entirely on standard Prometheus metrics, requiring no custom agents or proprietary protocols.</p><h3>Metrics Collected</h3><p>For each GPU workload, kube-opex-analytics tracks:</p><ul><li><strong>GPU Utilization</strong> — Actual compute usage percentage</li><li><strong>Memory Bandwidth</strong> — Data transfer utilization</li><li><strong>VRAM Used</strong> — Framebuffer memory consumption</li><li><strong>VRAM Free</strong> — Available GPU memory</li></ul><p>These raw metrics are collected every 5 minutes and automatically attributed to pods and namespaces, providing the per-team visibility that finance departments require for accurate cost allocation.</p><h3>From Raw Metrics to Actionable Insights</h3><p>Raw metrics are useful, but what organizations really need is consolidated data that answers business questions:</p><ul><li><em>“How much GPU did the data-science team use last month?”</em></li><li><em>“Are our training jobs actually utilizing the GPUs we’re paying for?”</em></li><li><em>“Which namespaces should we charge back for GPU usage?”</em></li></ul><p>kube-opex-analytics transforms raw metrics into actionable insights through intelligent consolidation at three levels:</p><ul><li><strong>Hourly Trends</strong> enable teams to analyze GPU utilization patterns throughout several days, helping identify peak usage periods and optimize workload scheduling accordingly.</li><li><strong>Daily Accounting</strong> provides day-over-day consumption tracking by namespace, which is essential for capacity planning and anomaly detection.</li><li><strong>Monthly Reports</strong> deliver comprehensive chargeback data with namespace-level GPU consumption, giving finance departments the information they need without requiring manual spreadsheet work.</li></ul><h3>Lightweight, Self-Maintaining Storage</h3><p>A common concern with long-term metrics retention is storage growth. Prometheus databases can expand to hundreds of gigabytes over time, requiring constant maintenance and pruning policies.</p><p>kube-opex-analytics takes a different approach by using RRDtool (Round-Robin Database) for storage. This time-tested technology was designed specifically for time-series data with fixed storage requirements, offering several advantages:</p><ul><li><strong>Up to 1 year of historical data</strong> is retained automatically</li><li><strong>Fixed database size</strong> ensures storage requirements remain constant over time</li><li><strong>Self-maintaining architecture</strong> means old data is automatically consolidated without manual intervention</li><li><strong>Minimal footprint</strong> keeps the entire database typically under 100MB</li></ul><p>Once deployed, the system handles all retention and consolidation operations automatically, eliminating ongoing storage management overhead.</p><h3>Getting Started</h3><p>For organizations already using kube-opex-analytics for CPU and memory tracking, enabling GPU support requires just one configuration change:</p><pre># Set the DCGM Exporter endpoint<br>export KOA_NVIDIA_DCGM_ENDPOINT=http://dcgm-exporter.gpu-operator:9400/metrics</pre><p>Or with Helm:</p><pre>helm upgrade --install kube-opex-analytics ./manifests/helm \<br>  --set dcgm.enabled=true \<br>  --set dcgm.endpoint=http://dcgm-exporter.gpu-operator:9400/metrics</pre><p>Within 5–15 minutes of applying this configuration, GPU metrics will begin appearing alongside existing CPU and memory data.</p><h3>Prerequisites</h3><ol><li><strong>NVIDIA DCGM Exporter</strong> deployed in your cluster (standard component of NVIDIA GPU Operator)</li><li><strong>kube-opex-analytics</strong> v26.01.0-beta1 or later</li></ol><h3>The Complete Picture</h3><p>With GPU tracking enabled, kube-opex-analytics provides a unified view of all your compute resources:</p><ul><li><strong>CPU</strong> — sourced from Kubernetes Metrics Server, consolidated hourly, daily, and monthly</li><li><strong>Memory</strong> — sourced from Kubernetes Metrics Server, consolidated hourly, daily, and monthly</li><li><strong>GPU</strong> — sourced from NVIDIA DCGM Exporter, consolidated hourly, daily, and monthly</li></ul><p>All three resource types flow through the same processing pipeline, appear on the same dashboards, and export to the same Prometheus endpoint. This unified approach eliminates the need for context switching between tools or manual correlation of data from multiple sources.</p><h3>Conclusion</h3><p>As AI and ML workloads become increasingly central to business operations, GPU infrastructure represents both a strategic asset and a significant cost center. Organizations that can accurately track, allocate, and optimize GPU usage gain a meaningful advantage in managing their cloud and infrastructure spending.</p><p>As demonstrated in this story, kube-opex-analytics provides high-resolution visibility into how your GPUs are being utilized across your entire infrastructure. Whether you need to troubleshoot a spike or plan your quarterly budget, it delivers the data and insights you need. With minimal setup and zero ongoing maintenance, platform engineering teams can finally provide finance with the GPU cost data they have been requesting.</p><p><strong>Resources:</strong></p><ul><li><a href="https://github.com/rchakode/kube-opex-analytics">GitHub Repository</a></li><li><a href="https://github.com/rchakode/kube-opex-analytics/blob/main/docs/configuration-settings.md">Configuration Documentation</a></li><li><a href="https://github.com/NVIDIA/dcgm-exporter">NVIDIA DCGM Exporter</a></li></ul><p><em>kube-opex-analytics is open-source software licensed under Apache 2.0.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=811b4eb7ca78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenShift Lab: Virtualization, HCP, and Multi-cluster Testing Simplified]]></title>
            <link>https://rodrigue-chakode.medium.com/openshift-lab-virtualization-hcp-and-multi-cluster-testing-simplified-ca6f0301704f?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca6f0301704f</guid>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Sun, 23 Mar 2025 15:15:54 GMT</pubDate>
            <atom:updated>2025-06-16T16:02:45.510Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of platform architecture, a design that looks good on paper must prove its worth in practice. As an OpenShift Architect (my daily work), this means rigorous testing and validation. Forget relying solely on documentation; true confidence comes from hands-on experimentation. I’ve found that a well-designed lab is the key to unlocking the full potential of new technologies. This article showcases a flexible OpenShift lab, engineered to explore the latest advancements, including <strong>OpenShift Virtualization</strong>, <strong>Hosted Control Planes (HCP)</strong>, <strong>Red Hat Advanced Cluster Management for Kubernetes (RHACM</strong>), and <strong>Multicluster Observability (MCO)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIHybn1czVF1J4uBBIKK5g.png" /><figcaption>Architecture of our OpenShift lab</figcaption></figure><h3><strong>The Diagram: A Visual Guide to Build a Flexible Lab</strong></h3><p>The attached diagram provides a comprehensive overview of our lab’s architecture. It highlights the key components, their interactions, and the storage configurations. As illustrated in the diagram, the lab is structured around these two primary servers, with the <strong>Minisforum HM90</strong> handling control plane functions and the <strong>Dell Poweredge T640</strong> managing worker node workloads. The network configuration allows the communication between all the components.</p><h4><strong>Minisforum HM90</strong></h4><p>This compact yet powerful server, equipped with 8 cores (Hyper-Threading) and 32 GB RAM, functions as the agile management and control hub. It hosts the essential virtual machines required for cluster management:</p><ul><li>OpenShift control node: 12 vCPUs, 18 GB RAM</li><li>OpenShift bastion: 2 vCPUs, 4 GB RAM</li><li>Network services (DNS, NTP): 1 vCPU, 1 GB RAM</li></ul><h4><strong>Dell Poweredge T640</strong></h4><p>This high-performance server, featuring 2x20 cores (Hyper-Threading) and 128 GB RAM, serves as the dedicated OpenShift worker node. Its robust configuration provides the necessary horsepower to handle demanding lab workloads. This server is also equiped with SSD storage, to ensure the best performance.</p><p>It serves as the only OpenShift worker node and provide the necessary horsepower to handle the lab workloads:</p><ul><li><strong>OpenShift Virtualization:</strong> This component allows us to seamlessly integrate virtual machines into our OpenShift environment. By running <strong>HCP worker VMs</strong> on dedicated <strong>LVM storage</strong> backed by a separate <strong>SSD</strong>, we can explore the intricacies of running traditional workloads alongside containerized applications. The OpenShift Virtualization stack supports the virtual worker nodes of nested OCP clusters enabled through RHACM and the HCP features.</li><li><strong>Hosted Control Planes (HCP):</strong> HCP enables us to deploy and manage OpenShift control planes as applications within our existing cluster. This approach simplifies multi-cluster management and allows for more granular control over resource allocation. Our lab allows us to test HCP deployment, scaling, and lifecycle management within a controlled environment.</li><li><strong>RHACM (Red Hat Advanced Cluster Management):</strong> RHACM is pivotal for managing our multi-cluster environment. Its <strong>Multicluster Engine and HCP features</strong> streamline deployment and governance. We can test various RHACM functionalities, including application deployment across clusters, policy enforcement, and cluster lifecycle management.</li><li><strong>Multicluster Observability (MCO):</strong> MCO provides critical insights into the health and performance of our multi-cluster environment. By leveraging <strong>MinIO (S3 storage) for the MCO S3 Bucket</strong>, we can collect and analyze observability data, enabling us to identify performance bottlenecks, troubleshoot issues, and optimize resource utilization.</li></ul><h3>Highlights of the Lab</h3><ul><li><strong>Realistic Simulation:</strong> The bare metal worker and dedicated storage allow us to simulate production-like environments for testing advanced OpenShift features.</li><li><strong>Rapid Prototyping:</strong> OpenShift Virtualization and HCP enable us to quickly deploy and test new applications and configurations.</li><li><strong>Comprehensive Multi-Cluster Management:</strong> RHACM empowers us to manage multiple clusters effectively, facilitating the exploration of complex deployment scenarios.</li><li><strong>Observability:</strong> MCO provides valuable observability data, enabling us to optimize performance and troubleshoot issues proactively.</li></ul><h3><strong>Conclusion:</strong></h3><p>Our OpenShift lab is more than just a setup; it’s a dynamic and flexible platform for innovation. By leveraging OpenShift Virtualization, HCP, RHACM, and MCO, we can explore the full potential of the OpenShift stack and push the boundaries of what’s possible with OpenShift container orchestration. We encourage you to share your experiences and insights in the comments below!</p><p><strong>#OpenShift #Virtualization #HCP #RHACM #MCO #Kubernetes #CloudNative #DevOps #LabSetup #Innovation #Testing</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca6f0301704f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Operator for Multi-clusters Usage Tracking, Analytics and Accounting]]></title>
            <link>https://rodrigue-chakode.medium.com/kubernetes-operator-for-multi-clusters-usage-tracking-analytics-and-accounting-dd69c14bdd38?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd69c14bdd38</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[resource-management]]></category>
            <category><![CDATA[usage-analytics]]></category>
            <category><![CDATA[cost-management]]></category>
            <category><![CDATA[capacity-planning]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Fri, 09 Dec 2022 12:24:06 GMT</pubDate>
            <atom:updated>2023-01-03T09:19:53.762Z</atom:updated>
            <content:encoded><![CDATA[<h4>An Open Source Approach of Resource Usage Tracking</h4><p><strong>TL;NR.</strong> This story introduces <a href="https://github.com/2-alchemists/krossboard-kubernetes-operator">Krossboard Kubernetes Operator</a>, an open source Kubernetes operator to quickly setup and operate a system for multi-clusters, cross-distribution &amp; cross-site usage tracking, analytics and accounting (GKE, AKS, EKS, OpenShift, and other distributions; on-premises or in the cloud).</p><blockquote><a href="https://github.com/2-alchemists/krossboard-kubernetes-operator">GitHub Page</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ziqgab3Kcw02GuuaSkO5pw.png" /><figcaption>Krossboard — 5-minutes recent consolidated usage per cluster &amp; per resource type (CPU/RAM)</figcaption></figure><p>The rest of this story discusses the benefits and challenges of Multi-Kubernetes usage tracking, the proposal of Krossboard Operator with features and screenshots, as well as a short getting started procedure.</p><h3>Benefits &amp; Challenges for Multi-Kubernetes Usage Tracking</h3><p>No need to further demonstrate how relevant <em>usage tracking is a required functionality in an effective approach of Kubernetes adoption</em>. If that is still not evident for you, let’s highlight a few benefits of that: <em>cost effectiveness</em> by tracking how your Kubernetes clusters are used over time; <em>cost sharing</em><strong> </strong>by accounting how your different applications, business units or partners, are using your Kubernetes resources; <em>tracking resource scaling up/down </em>by analyzing how each Kubernetes cluster resources are consumed over time.</p><p>There are tools such as <a href="https://github.com/rchakode/kube-opex-analytics">kube-opex-analytics</a> that enable to track resource usage on a single Kubernetes cluster. Things get complex when we have several Kubernetes clusters in place. Indeed, navigating over the different clusters to get insights is fastidious; needless to talk about the difficulty to address cross-cluster reporting.</p><h3>Introducing Krossboard Kubernetes Operator</h3><p><a href="https://github.com/2-alchemists/krossboard-kubernetes-operator">Krossboard Kubernetes Operator</a> is an operator that enables to setup is less than 15 minutes a system (<em>Krossboard)</em> to track, consolidate and account the usage of your multiple Kubernetes clusters. Krossboard provides on top of the different clusters it manages a cross-distribution and cross-site analytics and accounting platform with the following benefits:</p><ul><li><strong><em>Long term (12+ months) analytics and accounting</em></strong> per namespace and per cluster, with hourly-consolidated trends, as well as automatic daily and monthly accounting.</li><li><strong><em>Per cluster cross-namespace consolidated usage tracking and accounting</em> </strong>with built-in charts and data export capabilities (CSV).</li><li><strong><em>Cross-cluster consolidated usage tracking and accounting</em></strong><em> </em>with built-in charts with built-in charts and data export capabilities (CSV).</li><li><strong><em>5-minutes recent consolidated usage per cluster and resource type</em></strong> (e.g. CPU, RAM).</li><li><strong><em>Cross-distribution and cross-site Kubernetes instances</em></strong> with continuous integration tests against GKE, AKS, EKS, OpenShift, Rancher and vanilla Kubernetes on premises and in the cloud.</li></ul><h3>Krossboard — Analytics &amp; Accounting Charts</h3><p>Krossboard features the following main analytics and accounting capabilities (checkout their description and screenshots next after):</p><ul><li>Per cluster 5-minutes recent consolidated utilization</li><li>Per cluster cross-namespace usage trends, analytics &amp; accounting</li><li>Per node 5-minutes recent occupation</li><li>Per cluster consolidated utilization trends</li></ul><h4>Per cluster 5-minutes recent consolidated utilization</h4><p><em>Update every 5 minutes per cluster and resource type (CPU, RAM)</em>, these analytics track the last five minutes consolidated utilization, free and reserved/<em>non-allocatable</em> capacities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ziqgab3Kcw02GuuaSkO5pw.png" /></figure><h4>Per cluster cross-namespace analytics</h4><p><em>Based on hourly trends along with daily and monthly accounting</em>, these analytics tracked at the scope of each Kubernetes cluster tracks the resource utilization by the different namespaces. The analytics data and charts also highlight reserved/<em>non-allocatable</em> capacities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lUWzhamfYcDVbLtSoeTKhg.png" /></figure><h4>Per node 5-minutes recent occupation</h4><p>For each cluster and resource type (CPU &amp; RAM), these analytics track resource utilization per node while highlighting resource consumption per pod as well as reserved/<em>non-allocatable</em> capacities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZzT9GG5NDnF7PdiJIH97Q.png" /></figure><h4>Per cluster consolidated utilization over time</h4><p>For each cluster and resource type (e.g. CPU and RAM), these analytics track over time (up to 12 months of history) the consolidated utilization, free and reserved capacities.</p><p>The Krossboard UI enables to set a data selection time frame (see <em>start date</em> and <em>end date</em> on the below screenshot).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m-ciRWhsMAy4RIIZ7zVktg.png" /></figure><h3><strong>Getting Started</strong></h3><p>Getting started with is easy as proceeding with the following three steps:</p><ul><li>Deploy the operator using this one-line command.</li></ul><pre>kubectl apply -f https://raw.githubusercontent.com/2-alchemists/krossboard-kubernetes-operator/main/config/releases/latest/krossboard/krossboard-kubernetes-operator.yaml</pre><ul><li>Create a secret containing a KUBECONFIG with permissions to access your Kubernetes clusters and <em>read</em> raw metrics (see how to <a href="https://github.com/2-alchemists/krossboard-kubernetes-operator/blob/main/docs/create-kubeconfig-with-minimal-permissions.md">setup a KUBECONFIG with minimal permissions for Krossboard</a>).</li></ul><pre># replace /path/to/kubeconfig with path towards your KUBECONFIG<br>kubectl -n krossboard \<br>    create secret --type=Opaque generic krossboard-secrets \<br>    --from-file=kubeconfig=/path/to/kubeconfig</pre><ul><li>Create a Krossboard CR (Custom Resource) Instance</li></ul><pre>kubectl -n krossboard apply -f https://raw.githubusercontent.com/2-alchemists/krossboard-kubernetes-operator/main/config/releases/latest/krossboard/krossboard-deployment.yaml</pre><p>For further details, checkout our official documentation at GitHub: <a href="https://github.com/2-alchemists/krossboard-kubernetes-operator">https://github.com/2-alchemists/krossboard-kubernetes-operator</a></p><h3>Conclusion</h3><p>This story introduced an open source Kubernetes operator that helps to address the need of usage tracking, analytics and accounting across multiple Kubernetes clusters. The Kubernetes clusters can be of any distributions (GKE, AKS, EKS, OpenShift, vanilla Kubernetes, or other) and located anywhere (on-premises or in the cloud) ; Krossboard handles the usage tracking, analytics and accounting across them in a central place and in a transparent way.</p><p>Give it a try and don’t hesitate to share your feedback or contribute by opening a pull request at GitHub: <a href="https://github.com/2-alchemists/krossboard-kubernetes-operator">https://github.com/2-alchemists/krossboard-kubernetes-operator</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd69c14bdd38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tracking the efficiency of resource requests on Kubernetes]]></title>
            <link>https://rodrigue-chakode.medium.com/tracking-the-efficiency-of-resource-requests-on-kubernetes-5a446e986aec?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a446e986aec</guid>
            <category><![CDATA[kubernetes-operations]]></category>
            <category><![CDATA[cloud-accounting]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[usage-analytics]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Wed, 09 Feb 2022 12:27:14 GMT</pubDate>
            <atom:updated>2022-02-10T08:49:01.537Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to set resource requests on your workloads efficiently?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zw1P4j653ZzChOWSsznSgw.png" /><figcaption>Tracing of Kubernetes resource requests efficiency by namespace using <a href="https://github.com/rchakode/kube-opex-analytics">kube-opex-analytics</a>.</figcaption></figure><h3>Problem Statement</h3><p>Best practices on Kubernetes recommend setting resource requests (and also limits?) on workloads. It’s even possible to set cluster-level policies to reject manifests without resource requests. However, one of the problems that this raises is the ability to suitably assess and set resource requests in a way that avoids wasting resources due to unused requested (i.e. reserved) capacities.</p><blockquote>So, how to measure the efficiency of resource requests set on your Kubernetes workloads to avoid wasting resources due to oversized requested (i.e. reserved) capacities?</blockquote><h3>Our Original Contribution</h3><p>Thanks to <a href="https://github.com/rchakode/kube-opex-analytics">kube-opex-analytics</a>, an open source Kubernetes usage analytics and accounting tool (read more about it at the end of this story), we address the problem of assessing the efficiency of resource requests set on Kubernetes workloads.</p><p>The release <a href="https://github.com/rchakode/kube-opex-analytics/releases/tag/v22.02.0">v22.02.0</a> introduces a new indicator called 𝐔𝐬𝐚𝐠𝐞/𝐑𝐞𝐪𝐮𝐞𝐬𝐭 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲. Conceptually, this indicator is defined as the ratio between the actual usage and the resource requests. The indicator is provided by namespace and, as all other kube-opex-analytics indicators, it’s computed based on hourly-consolidated Kubernetes metrics.</p><blockquote>By tracking and visualizing the usage/request efficiency indicator over time, Kubernetes administrators do have an additional tool to take appropriate cost-optimization decisions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9uUnSuQwlkfEljj4_TPiew.png" /><figcaption>kube-opex-analytics — Chart showing the usage/request efficiency for CPU (1) and memory (2) resources for a selectable period of time (4). At specific date time, a tooltip (3) provides details for each namespace. It’s possible to export (5) the chart as PNG image or as CSV and JSON data file.</figcaption></figure><p>The next section discusses possible values of this indicator.</p><h3>Understanding Usage/Request Efficiency</h3><p>At a given namespace and a given time, the value of this indicator can be in one of the following ranges:</p><ul><li>A <strong>value equal to 1</strong> means an effective match between the actual usage and requests; this is also the default value if no resource requests are set in the namespace.</li><li>A <strong>value between 0 (excluded) and 1</strong> does mean that the actual usage is lower than resource requests. A value close to zero typically means that requests are oversized, and that they need to be reviewed to avoid wasting resources.</li><li>A 𝐯𝐚𝐥𝐮𝐞 𝐠𝐫𝐞𝐚𝐭𝐞𝐫 𝐭𝐡𝐚𝐧 𝟏 means that the actual usage is greater than the requests. It’s likely an ideal situation to avoid wasting resources. However, in this case it’s likely necessary to have resource limits or namespace quotas set to be sure that the resource consumption will be under control.</li></ul><h3>About kube-opex-analytics</h3><p>kube-opex-analytics (literally <em>Kubernetes Opex Analytics</em>) is an open source Kubernetes usage analytics tool to help organizations track the resources being consumed by their Kubernetes clusters over time (hourly, daily, monthly). Doing so, kube-opex-analytics aims to prevent overpaying by enabling each organization to understand how their Kubernetes resources are consumed over time, and hence be able to make appropriate cost optimization decisions.</p><p>Some of its key features include:</p><ul><li>Usage accounting and trends per namespace.</li><li>Accounting of non-allocatable capacities(nodes and cluster levels).</li><li>Cluster usage accounting and capacity planning.</li><li>Usage/requests efficiency.</li><li>Cost allocation and chargeback analytics.</li><li>Insightful and extensible visualization.</li></ul><p>You can get additional information on kube-opex-analytics via its<a href="https://github.com/rchakode/kube-opex-analytics"> GitHub page</a>.</p><h3>Conclusion</h3><p>In this story we have presented an original indicator to track the efficiency of resource requests set on Kubernetes workloads. The aim of this indicator is to provide Kubernetes administrators a tool to make appropriate cost-optimization decisions.</p><p>Don’t hesitate to share your feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a446e986aec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RRDtool + Prometheus — Optimizing the storage and the query performance for aggregated metrics]]></title>
            <link>https://rodrigue-chakode.medium.com/rrdtool-prometheus-optimizing-the-storage-and-the-query-performance-for-aggregated-metrics-69986bb8cb1b?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/69986bb8cb1b</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[data-aggregation]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Sun, 17 Oct 2021 11:19:14 GMT</pubDate>
            <atom:updated>2021-10-21T20:14:18.315Z</atom:updated>
            <content:encoded><![CDATA[<h3>RRDtool + Prometheus — Optimizing performance and storage efficiency for aggregated metrics</h3><h4>Leveraging the old to make the modern better</h4><p><a href="https://oss.oetiker.ch/rrdtool/">RRDtool</a> is a pioneering time-series database and monitoring system. It has been around since 1999. However, with successive technology hypes in the monitoring ecosystem, the tool is unknown to many monitoring professionals and doesn’t have the attention it deserves despite its powerful capabilities that would be very helpful in today&#39;s Prometheus-driven observability world. In this article, we introduce a design approach where RRDtool is used to optimize the management of aggregated metrics for Prometheus. More specifically, we’ll show how this approach allows us to reduce the storage on the Prometheus server, while improving the performance and the scalability of queries prior to the visualization.</p><figure><img alt="Screenshot from Krossboard — a Kubernetes usage analytics tool built atop of the proposed concept." src="https://cdn-images-1.medium.com/max/1024/1*xBq9nzBnClEEHLsmzxYlNQ.png" /><figcaption>Screenshot from <a href="https://krossboard.app/">Krossboard</a> — a Kubernetes usage analytics tool that leverage the hereafter described concepts.</figcaption></figure><blockquote>This work is an experience sharing highlighting design decisions made when building a <a href="https://medium.com/swlh/bringing-prometheus-metrics-and-grafana-dashboard-for-cost-allocation-on-kubernetes-clusters-1ee7f68cd677">now popular open source Kubernetes usage analytics tool</a>.</blockquote><h3>Introduction</h3><p>In today&#39;s cloud-native computing world, <a href="https://radar.cncf.io/2020-09-observability">observability</a> is a fundamental building block to monitor the operations of applications and services in production. One of the core enablers for observability is the ability to collect relevant metrics concerning the applications and services that are monitored. In this context, <a href="https://prometheus.io/">Prometheus</a> has emerged as a de-facto standard component to expose, collect and aggregate metrics before visualization with tools like <a href="https://grafana.com/">Grafana</a>.</p><p>In a typical Prometheus architecture, each component to be monitored is backed by an <a href="https://www.metricfire.com/blog/first-contact-with-prometheus/">exporter</a> that exposes metrics to the Prometheus world. Then, the Prometheus server periodically goes through the different exporters to scrape exposed metrics. Sometimes, the scraped metrics may require some aggregation before visualization; this is for example the case when we have Prometheus exporters that get updated samples every couple of minutes while at the visualization side we just need hourly-aggregated metrics. Such a use case is typically applicable for analytics tools like <a href="https://rodrigue-chakode.medium.com/kubernetes-resource-usage-analytics-for-cost-allocation-and-capacity-planning-416800e85d16">kube-opex-analytics</a>, in which raw samples are collected every 5 minutes while the visualization is carried out on metrics consolidated on an hourly basis.</p><p>When aggregation is needed as discussed above, people typically rely on <a href="https://prometheus.io/docs/prometheus/latest/querying/operators/">PromQL</a> (<em>the Prometheus query language) </em>to perform the aggregation at the query time. For a given time range, the <a href="https://prometheus.io/docs/prometheus/latest/querying/operators/">PromQL operators</a> support various aggregation operations (sum, average, maximum, minimum, and more). While the use of these operators would be sufficient in many contexts, there are situations where such use may present some shortcomings.</p><p>Two aspects of optimization will be discussed: the storage of metrics samples and the performance and the scalability of the visualization.</p><h3>Shortcomings and optimization</h3><h4>Efficient storage : Avoid storing useless metrics</h4><p>For a better illustration, let‘s consider a use case where metrics are collected every 5 minutes and that, for consistency reasons, we only want to visualize hourly and daily consolidated metrics.</p><p>In such a context, if all the 5-minutes samples are exposed to the Prometheus world, then <em>we would need the following storage capacities for a single exporter: 12 samples every hour, 12 * 24 = 288 samples a day, and 288 * 30 = 8640 samples a month</em>. By taking into account the data size of each sample, the number of exporters, the data retention period, we’ll likely fall with a significant amount of data to store and process before visualization.</p><p>On the contrary, if only the hourly consolidated metrics are exposed in the Prometheus world, <em>we’ll need to store 1 sample each hour, 24 samples a day, and 24 * 30 = 720 samples a month</em>. This makes it evident that in such a context, storing all raw samples is not efficient. That said, as the raw samples are needed to compute the consolidated metrics over time, we should find a strategy to handle raw samples locally while exposing consolidated metrics to the Prometheus world.</p><p>That’s where RRDtool comes in. Indeed, the raw samples can be collected and injected into a RRD database specifically configured to internally compute and maintain the targeted coarse-grained metrics (e.g. hourly metrics in the use case described above).</p><p>Without diving into more RRDtool details, let’s take an example of implementation. As shown below, the RRD database settings can be configured to hold a data source called cpu_usage , which stores raw samples with a periodicity of 5-minutes (300 seconds) to produce hourly-consolidated metrics (average of 12 raw samples). This way, with an appropriate implementation of our Prometheus exporter, we can only expose the hourly metrics to the Prometheus exporter while maintaining the raw samples locally.</p><pre>DS:cpu_usage:GAUGE:300:U:U                         RRA:AVERAGE:0.5:12:1</pre><h4>Visualization performance and scalability: Avoid repetitive aggregations</h4><p>To dive into the performance and scalability aspects, let’s assume that once data is stored in Prometheus we’ll be using Grafana for the visualization. When aggregation is needed as introduced previously, we’ll be typically using <a href="https://prometheus.io/docs/prometheus/latest/querying/operators/">Prometheus operators</a> to compute aggregated metrics prior to the visualization. This means that each time the visualization is refreshed for a user-defined period of time (visualization window), samples are retrieved from Prometheus and processed on the fly to produce consolidated metrics.</p><p>For example, let’s reconsider our initial example where raw samples are collected every 5 minutes while the visualization is carried out on hourly-consolidated metrics (e.g. like <a href="https://github.com/rchakode/kube-opex-analytics">kube-opex-analytics</a> does). Let’s also assume that we have only one Prometheus exporter while visualization is handled on a 30-day sliding window with a 15-minutes refreshing period (i.e. refresh four times each hour).</p><p>In a first attempt, if all the 5-minutes samples are exposed to the Prometheus world, then <em>8640 samples will be queried each time the visualization is refreshed (i.e. four times every hour, 96 times in 24 hours or 2880 times in 30 days).</em></p><p>In a second attempt, by introducing an RRD database locally to maintain raw samples to just export hourly consolidated metrics to the Prometheus world, then only <em>720 samples will be queried at each refresh. </em>Doing so the performance and the scalability of the visualization will be significantly improved thanks to the <strong>reduced amount of samples queried and aggregated</strong> each time the visualization is refreshed.</p><h3>Conclusion</h3><p>In this article, we describe how RRDtool can be used to optimize the processing and the exposition of aggregated metrics for a Prometheus world. We showed how this approach allows us to reduce the storage on the Prometheus server, while improving the performance and the scalability of queries during the visualization.</p><p>This work is actually an experience sharing from lessons learned from designing and building <a href="https://github.com/rchakode/kube-opex-analytics">kube-opex-analytics</a>, a now popular open source Kubernetes usage analytics tool. Its original concepts, in part due to the power of RRDtool, makes the tool quickly adopted by the Kubernetes community. It’s currently totaling ~330+k download hits in less than 3 years. It’s even included in a commercial offer with <a href="https://krossboard.app/">Krossboard.app</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69986bb8cb1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitor and measure the availability of applications running on Kubernetes]]></title>
            <link>https://rodrigue-chakode.medium.com/monitor-and-measure-the-availability-of-applications-running-on-kubernetes-b286bf309b8c?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/b286bf309b8c</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Sun, 08 Nov 2020 14:59:02 GMT</pubDate>
            <atom:updated>2020-12-03T14:34:29.227Z</atom:updated>
            <content:encoded><![CDATA[<h4>Concepts and implementation to observe applications availability — A foundation to track SLA/SLO targets.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d_H6PVD9-0_38g0E-CpT0g.png" /><figcaption>Sample of Dashboard to observe incidents on probes over time</figcaption></figure><p>When applications go in production one of our main concerns is to ensure that they are properly monitored, notably with appropriate checks and suitable metrics to report about their availability over time. This article comes to tackle this concern. Particularly focused on applications running on Kubernetes, it sets up a standard to monitor, measure, and observe the availability of applications. <em>The goal being to help organizations to define Service Level Objectives (SLO) and/or Service Level Agreements (SLA) while be able to track them through factual KPIs over time.</em></p><p>This article is compose of two main sections. The first one is conceptual, introducing our fundamentals and assumptions to define, monitor, and measure application availability on Kubernetes. The second one is practical, demonstrating an implementation powered by <a href="https://realopinsight.com/">RealOpInsight</a> — an <a href="https://github.com/rchakode/realopinsight">open source</a> application operations monitoring framework designed to work atop of Kubernetes while leveraging the basic probe capabilities of the later.</p><h3>Statement of work</h3><p>Given one or more instances of Kubernetes, the goal is to be able to monitor, measure, and track the availability of applications as established by the following tenets:</p><ul><li><strong>Defining a Kubernetes application: </strong>As previously discussed <a href="https://rodrigue-chakode.medium.com/kubernetes-on-tv-watch-how-your-services-are-operating-in-production-100cd00a0d45">here</a>, we define an application in one of these two ways. On the one hand, a simple application can be modeled as a set of <a href="https://kubernetes.io/fr/docs/concepts/services-networking/service/">Kubernetes services</a> along with their <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">pods</a> within a single namespace. As already discussed <a href="https://rodrigue-chakode.medium.com/kubernetes-on-tv-watch-how-your-services-are-operating-in-production-100cd00a0d45">in a previous story</a>, the application in this case can be viewed as a dependency tree-like on the <em>Diagram 1.a</em> (below). Concretely, the dependency tree is a hierarchical composition of services along with the pods and containers that underlie them. Each container is associated to its pod, which in turn associated to its service, which finally is associated to the application at the top level. On the other hand, a complex application can be modeled as a composition of two or more namespace-scoped applications.</li><li><strong>Computing application status: </strong>Behind the scene the status of an application is computed by gathering the state of the underlying pods (along with their containers), then aggregate and propagate these low-level status throughout the dependency tree. By essence, probes at the pods and containers levels are achieved by Kubernetes. The status aggregation along the tree takes into account the type of resource controller that manages the different backing pods. Typically, pods in a ReplicatSet assume the same functionality (i.e. same importance) while it’s not always the case for pods in a StatefulSet. On the <em>Diagram 1.a</em> for example, we highlight a situation where a pod in a ReplicatSet is in error, but the service is just labeled with a warning state. The aim of such an advanced status aggregation is to help operations teams to trigger incident resolutions with appropriate priority.</li><li><strong>Defining application availability:</strong> At each time, the overall current status of an application is determined by status at the top level of the application dependency tree. This status gives quick insight on how the application is operating (e.g. green for healthy, red if critical problem, orange for non-critical problem). An application is meant available if its overall status is not critical.</li><li><strong>Observe application availability.</strong> At the top level of this approach (<em>Diagram 1.b</em>) we have at each time the healthiness status of each application. <em>Our standard design assumes that the status of each application is computed every five minutes (arbitrary periodicity) and logged for future availability analytics. </em>Not only the status of application is important to be logged, but also a summary of probes status (i.e. ratio of containers in failure and normal states).</li><li><strong>Defining SLA/SLO targets: </strong>On the basis of the above considerations, SLO/SLO targets for an application can be defined based on its availability for a given period of time set by a start and an end date. For example, considering a period of 7 days with five-minutes application availability checks (i.e. 1440 checks total), if the application was not available during 2 checks, then the availability during that period can be computed as follows : (1440 — 2)/1444 = 99.9986%. With this foundation, we open room to easily to track the operations of Kubernetes applications over time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K6ywJScQCxytAJpCeG4qGA.png" /><figcaption>Diagram 1: Conceptual view of Kubernetes applications availability monitoring — <strong>(a) </strong>Sample of a namespace-scoped application dependency tree per — <strong>(b) </strong>Sample of high-level dashboard showing the current consolidated status of a set of applications.</figcaption></figure><p>Keeping the above tenets in mind, still with the objective to be able to define and track SLA/SLOs targets for Kubernetes applications, the next section will focus on demonstrating an implementation (namely <a href="http://realopinsight.com/">RealOpInsight</a>) of the introduced concepts. We’ll present the architecture building blocks, demonstrate a quick deployment on a Kubernetes cluster. The deployment will be followed by a quick demo of different built-in dashboards and availability analytics features.</p><h3>RealOpInsight Design</h3><p>The <em>Diagram 2</em> demonstrates a simplified flow of RealOpInsight when integrated with Kubernetes. On the diagram, straight arrows indicate processing paths or internal data flows, while the dashed arrows highlight external data retrieval during processing steps. The following paragraphs describe the main bricks of this flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*TJ9nCNDgDSd4Vn1hZpEctA.png" /><figcaption>Diagram 2: RealOpInsight — Overview of Processing &amp; Data Flow</figcaption></figure><p><strong>Personas — </strong>The design of RealOpInsight introduces two personas (see <em>User </em>in the Diagram). On the one hand, <strong>Administrators </strong>are users responsible for configuration and various management tasks. In particular, their responsibilities include, to set up the integration with Kubernetes sources, to create and manage operator users along with information each of them is allowed to watch (see <em>Operations profile</em>s introduced hereafter). On the other hand, <strong>Operators </strong>are users whose the main responsibility is to watch information associated to their operations profiles. For each organization, it’s free to administrators to define how to define operations profiles per operator according to the organization’s monitoring and operations needs.</p><p><strong>Operations profile — </strong>This refers to the set of information an operator is allowed to watch. Operations profiles are created and managed by administrators. Concretely, an operations profile bound to an operator a set of applications that he’s allowed to watch. Upon a successful login an operator has access to a console from which he can only watch and navigate through the information associated applications bound to his operations profile.</p><p><strong>One or more Kubernetes sources — </strong>An instance of RealOpInsight can handle multiple Kubernetes instances. Each source is defined given the API endpoint URL with possibly a token to authenticate against it. The token is optional for a proxied access, or if the instance run as a pod with appropriate service account and RBAC permissions on Kubernetes.</p><p><strong>Discovery and composition of applications — </strong>When a Kubernetes source is added, RealOpInsight automatically triggers a discovery action consisting to query the Kubernetes API to retrieve all existing applications. The list of applications is stored internally and used when defining operations profiles. As mentioned previously, an application is basically bound to a namespace. However, thanks to a built-in Editor, administrators can also set up composite/enriched applications based on other applications.</p><p><strong>Availability monitoring &amp; Prometheus metrics — </strong>RealOpInsight features an internal component scheduled by a configurable timer (5 minutes by default) to compute applications availability and store them for future analytics. This component also enables a Prometheus exporter to expose current availability metrics to external third-party systems (e.g. a Prometheus/Grafana stack). RealOpInsight is released with built-in dashboards to handle availability analytics for each operations profile. The demo section (next of this one) provides a quick tour of this feature.</p><h3>Demo tour</h3><p>This section walks through the installation of RealOpInsight on Kubernetes and its use to observe applications availability as described above. We’ll see how easy it is, to set up a Kubernetes source, to create an operations profile, and to monitor and measure the availability of applications in production. In addition to the visualization through built-in dashboards, we’ll also show a Grafana dashboard backed by the Promeheus exporter.</p><h4>Installation of RealOpInsight on Kubernetes</h4><p>There is a <a href="https://github.com/rchakode/realopinsight/tree/master/helm">Helm chart</a> to ease the deployment of RealOpInsight on Kubernetes. The following suite of commands shall install the latest version of RealOpInsight in the namespace monitoring (the namespace must exist, or can be changed to any other existing namespace).</p><pre>git clone <a href="https://github.com/rchakode/realopinsight.git">https://github.com/rchakode/realopinsight.git</a> &amp;&amp; \<br>  cd realopinsight &amp;&amp; \<br>  helm upgrade \<br>  --namespace monitoring \<br>  --install realopinsight \<br>  helm/realopinsight/</pre><p>The deployment (see <a href="https://github.com/rchakode/realopinsight/tree/master/helm/realopinsight">Helm manifests</a>) also enables a ClusterIP service named realopinsiht exposing the UI on port 80 (in-cluster URL: <a href="http://realopinsight/ui">http://realopinsight.monitoring/ui</a>) and Prometheus metrics on port 4584(in-cluster URL: <a href="http://realopinsight.monitoring:4583/metrics">http://realopinsight.monitoring:4583/metrics</a>).</p><h4>Configuring a Kubernetes probe source</h4><p>This requires to log into RealOpInsight as administrator (the default username and password are adminand password).</p><ul><li>Select the menu Monitoring Sources to open the source configuration page (<em>Screenshot 1</em>).</li><li>Set the API Endpoint URL to <a href="https://kubernetes.defaulth/">https://kubernetes.default/</a> (in-cluster API URL) and , optionally if the cluster uses a self-signed certificate, check Don&#39;t verify SSL certificate.</li><li>Leave the field Auth String Token empty, meaning that RealOpInsight does authenticate against the Kubernetes API using its RBAC service account. That service account (named realopinsight) along with its RBAC permissions are created during the deployment with Helm.</li><li>Click Add as source and select an ID for the source when prompted.</li><li>Click on Apply to finish the operation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1uqbyriztiy8VKEreVaHQg.png" /><figcaption>Screenshot 1: RealOpInsight — Source configuration page.</figcaption></figure><h4>Availability metrics</h4><p>Once a source is created, the component responsible to process availability metrics starts working in background to compute the status of applications and exposes them as Prometheus metrics (<em>Screenshot 2</em>).</p><p>On these metrics the different applications are set by the scope label. The prefix Source0 is the ID of the Kubernetes source and allows to highlight discovered applications (i.e. bound to namespaces vs composite applications created through the built-in editor).</p><p>For each application, we have the following metrics:</p><ul><li>Overall aggregated status (see label status=&quot;summary&quot;) which the value can be one of the following: 0 →normal/healthy state; 1 → problem state with minor impact; 2 → problem state with major impact; 3 →problem state with critical impact; 4 →unknown state.</li><li>Ratio (%) of underlying containers in a given state (see label status=&quot;minor&quot; , status=&quot;major&quot;, status=&quot;critical&quot;, status=&quot;unknown&quot;).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*DIcIAynWDD77Nnq0TfBLmA.png" /><figcaption>Screenshot 2: RealOpInsight — Samples of exposed Prometheus metrics</figcaption></figure><h4>Configuring an operations profile</h4><p>The following steps demonstrate how to create an operator profile named itops to monitor the availability of applications in the namespaces kube-system , monitoring and redis in the Kubernetes cluster set on Source0.</p><ul><li>Login as administrator (default credentials:admin/password) and select the menu New User.</li><li>Fill in the required user information. For this example, set the username and the password to itopsand 123itsops respectively. Click Submit to confirm the creation.</li><li>Once the user created, select the menu Operations Profiles to configure its operations profile.</li><li>Select the useritops in the list of users (<em>Screenshot 3</em>), then select the items to associate to its profile at the right side.</li><li>Click on Assign and we’re done.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*3QitY8lIEzvJVSV7yvxo2w.png" /><figcaption>Screenshot 3: RealOpInsight — Defining operations profile for a user</figcaption></figure><h4>Watch applications availability</h4><p>To watch the availability of applications, it’s necessary to sign in as an operator user. In our example, it’s itops as created in the previous step.</p><p>The welcome page of an operations console shows a tactical overview on the health of applications. Each application is represented by a tile with a title and a summary of the current status of probes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HPffwfnKUxJf3toBgosqUA.png" /><figcaption>Screenshot 4: RealOpInsight — Tactical Overview</figcaption></figure><p>A click on a tile (or on the associated entry in the Active View field) opens the related application dashboard (<em>Screenshot 5</em>). This dashboard provides widgets to have more details on the current status of the application. At the upper part, a tree view (left) and a map (right) display the dependency among the application’s components. These two widgets also highlight how each incident impacts the overall status of the application. The bottom part of the dashboard is composed of a statistics panel (left) showing a consolidated view of probe status, and an event console (right) showing details on probes status. By default the event console only display problems, but the operator can opt to also view normal messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCj2_q7ce4rYF3Xw2psG6g.png" /><figcaption>Screenshot 5: RealOpInsight — Detailed operations view of an application showing dependency tree and in-depth root-cause exploration capabilities.</figcaption></figure><h4>Availability analytics dashboards</h4><p>Select Platform Availability Analytics from the Active View field to switch to the applications availability analysis dashboard (<em>Screenshot 7</em>).</p><p>At the left side, this dashboard shows for each application a graph of history of probes status during a selected period of time (30 days by default, see the date filter at the top of the page).</p><p>There is also a link to download the data associated to this graph in CSV format so to enable further analytics through 3-party analytics tools, e.g. Microsoft Excel, <a href="https://aws.amazon.com/athena">AWS Athena</a>, <a href="https://cloud.google.com/bigtable">Google Bigtable</a>, <a href="https://www.tableau.com/">Tableau</a>, etc.</p><p>At the right side, a pie chart provides an analytics summary of the application availability during the selected period. The availability ratio displayed is computed as the portion of time (out of the selected period of time) during the application was in a normal state.</p><p>Users can extend this native analytics dashboards by leveraging the built-in Prometheus exporter. For example, it possible to build equivalent, or even better, dashboards using 3-party tools like Grafana.shboard o observe incidents on p</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2nVFyoInCH1Ug3UCRLO_CQ.png" /><figcaption>Screenshot 6: RealOpInsight — Applications availability analytics dashboard.</figcaption></figure><h4>Custom Grafana dashboard</h4><p>The following dashboard built from Dashboard and the RealOpInsight’s Prometheus exporter shows another example to monitoring applications availability. For each application or set of applications, we can observe over time the proportion of probe incidents. This kind of indicator may be typically useful to set objectives on incidents and their resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d_H6PVD9-0_38g0E-CpT0g.png" /><figcaption>RealOpInsight — Sample of Grafana dashboard toobserve incidents on probes over time</figcaption></figure><h3>Conclusion</h3><p>In this article we discussed concepts and propose an approach to monitor and measure the availability of applications running Kubernetes. The goal being to be able to define and track SLA/SLOs targets over time. Beyond concepts, the second part of the article demonstrated how that works with an actual implementation, namely <a href="http://realopinsight.com/">RealOpInsight</a>. Prior to that, we presented the architecture building blocks, demonstrated a quick deployment on a Kubernetes cluster.</p><p>Before concluding this work, it may be worth to mention that, beyond Kubernetes, RealOpInsight also works with Zabbix and Nagios. Doing so, it supports a global strategy for applications operations monitoring by breaking silos with traditional monitoring tools. Last not the least, we recall that RealOpInsight is open source and available on <a href="https://github.com/rchakode/realopinsight">Github</a>. Don’t hesitate to give it a try.</p><p>Enjoy, and don’t hesitate to share your feedback !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b286bf309b8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Krossboard: A centralized usage analytics approach for multiple Kubernetes]]></title>
            <link>https://rodrigue-chakode.medium.com/in-search-of-converged-usage-analytics-for-multiple-managed-kubernetes-c5108cb7f0e1?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5108cb7f0e1</guid>
            <category><![CDATA[cloud-cost-analytics]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cost-allocation]]></category>
            <category><![CDATA[capacity-planning]]></category>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Sat, 05 Sep 2020 11:47:20 GMT</pubDate>
            <atom:updated>2020-11-12T06:58:06.936Z</atom:updated>
            <content:encoded><![CDATA[<h3>Krossboard: A centralized usage analytics for multiple Kubernetes</h3><h4>Any distributions — Amazon EKS, Google GKE, Microsoft AKS, Redhat OpenShift, Rancher RKE, and others.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*zcDlJoRR3B7QRpMjtkaf_w.png" /><figcaption><em>Conceptual integration with Amazon EKS, Microsoft AKS, Google GKE, Red Hat OpenShift and Rancher RKE</em></figcaption></figure><p>Today let’s introduce concepts and tools to address the use case to get a <strong><em>comprehensive visualization to analyse and understand resource usage on environments with many Kubernetes clusters</em></strong>. This approach aims at helping organizations to easily make cost allocation and capacity planning decisions. Called <a href="https://krossboard.app/">Krossboard</a>, our system is actively tested against <a href="https://aws.amazon.com/fr/eks/">Amazon EKS</a>,<a href="https://cloud.google.com/kubernetes-engine"> </a><a href="https://azure.microsoft.com/en-us/services/kubernetes-service/">Microsoft AKS</a>, <a href="https://cloud.google.com/kubernetes-engine">Google GKE</a>, <a href="https://www.openshift.com/">Red Hat OpenShift</a>, <a href="https://rancher.com/docs/rke/latest/">Rancher RKE</a>, and various other vanilla deployments. When deployed against managed Kubernetes, Krossboard is designed to <em>be able to automatically discover and track the usage of new and existing Kubernetes instances.</em> Its deployment and some screenshots will be demonstrated later in this story.</p><p><strong>Credit:</strong> The work is achieved in collaboration with <a href="https://www.linkedin.com/in/christophe-camel">Christophe Camel</a> with whom I co-developed Krossboard. It’s next in a series of stories (see<a href="https://medium.com/@rodrigue.chakode/kubernetes-resource-usage-analytics-for-cost-allocation-and-capacity-planning-416800e85d16"> here</a> and<a href="https://medium.com/swlh/bringing-prometheus-metrics-and-grafana-dashboard-for-cost-allocation-on-kubernetes-clusters-1ee7f68cd677"> there</a>) where I discuss practical concepts and tools to ease cost allocation and capacity planning on Kubernetes environments.</p><h3>Goals of this work</h3><p>This work raises from the need to provide a mean for organizations owning multiple Kubernetes clusters to easily tackle cost management and capacity planning decisions. The aim is to bring a global and consistent approach for resource usage analysis where metrics from all Kubernetes clusters are collected, processed, aggregated, visualized, all in one single place. The resulting analytics are intended to cover short, mid and long periods of accounting (hours, days, months, and even years), combined with fine-grained exploration capabilities.</p><blockquote>One of <strong>key point of the discussed approach </strong>is that we want to go beyond the traditional one consisting to deploy the analytics component inside the Kubernetes cluster itself. Indeed, this later approach consumes production resources and introduces noises on usage reports, not to mention the fact that the related tools are often mono cluster solutions.</blockquote><h3>Objectives &amp; benefits</h3><p>The core ambition of the current work aims to enable the following unique benefits:</p><ul><li><strong>Consolidated per-cluster and cross-cluster usage KPIs — </strong>We expect analytics that are based on consolidated metrics while being cross-cluster. On the one hand, following the approach introduced by <a href="https://medium.com/@rodrigue.chakode/kubernetes-resource-usage-analytics-for-cost-allocation-and-capacity-planning-416800e85d16">kube-opex-analytics</a>, all analytics are based on consolidated metrics in order to be consistent and relevant for business decisions. Our standard analytics do start with a minimum of 5-minutes consolidation interval for recent highlights, the main focus being hourly, daily and monthly reports. On the other hand, analytics shall be provided at namespace scope for in-cluster perspectives and/or at cluster scope for multi-cluster perspectives.</li><li><strong>Ease cost allocation decisions </strong>— We expect to provide detailed reports on resources consumed by each project (namespace) as well by each cluster during given accounting periods (e.g. hourly, daily, monthly). The intent is to help organizations to share infrastructure and operations costs among their projects or business units.</li><li><strong>Help anticipate costs </strong>— The idea is to feature dynamic analytics reports that show how resources are being consumed over time. This would help organizations to forecast resource scale up and scale down, there by anticipating the related costs.</li><li><strong>Enabler for cloud-cost reduction strategy</strong> — Last but not the least, our intent is to provide organizations with usage KPIs that enable factual insights to make cluster consolidation decisions. For instance, such insights can be used to motivate and drive consolidation decisions for clusters that are under utilized — hence leading to infrastructure and operations cost reduction.</li></ul><p>After having discussed the theoretical aspects of our vision, the next section presents Krossboard the result of our current implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ldy1AujNiKFJsyar" /><figcaption>Krossboard — A screenshoft showing consolidated usage of four GKE clusters</figcaption></figure><h3>Krossboard in a nutshell</h3><p>Krossboard is built atop tenets that make it pretty simple to understand and also easy to deploy and get started:</p><ul><li><strong>Cross-cloud, cross-distribution , easy to set up</strong>— Provided as ready-to-use cloud images for popular cloud platforms (currently Amazon EKS, Microsoft AKS and Google GKE), OVF virtual machine appliance, and as setup packages, an instance of Krossboard can be deployed in a couple of minutes. When deployed on a cloud environment, it’s designed to seamlessly discover and track the usage of all your managed clusters within a predefined scope (e.g. a region on AWS, a project on GCP, or an resource group in Azure).</li><li><strong>Consistent analytics </strong>— for each cluster, Krossboard computes resource usage for each individual namespace and aggregates them on an hourly basis (what we consider as a consistent and relevant cost allocation time unit). The resulting analytics are rendered with various perspectives that cover days, months, and up to a year.</li><li><strong>Central place of visualization</strong> — Beyond the advanced aggregation and consolidation capabilities that Krossboard features, it provides built-in charts for the resulting usage KPIs. The Krossboard UI actually works as a single panel of glass to quickly insights on the resources each namespace/project or cluster is spending over time (see a sample screenshot below).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nW7zGvXvMn9KGfrRfB5uKA.png" /><figcaption>Krossboard — sample of screenshot showing 5-minutes recent CPU and Memory usage per GKE cluster</figcaption></figure><h3>Integration with any Kubernetes</h3><p>A generic deployment of Krossboard requires a KUBECONFIG file. This will not be covered in this story, r<a href="http://with credentials to">ead the related documentation</a> for more details.</p><p>The rest of the article focuses on integration with managed Kubernetes, i.e. with automatic discovery and tracking.</p><h3>Specific integration with managed Kubernetes</h3><p>Krossboard is designed to have a simple integration with managed Kubernetes. Deployed as a virtual machine within the target cloud environment, each instance does discover and handle all managed clusters defined by a given scope.</p><p>In the current implementation, the default discovery scope are the following:</p><ul><li><strong>On Amazon Web Services (AWS)</strong>, an instance of Krossboard does automatically discover and handle all your EKS clusters belonging to the same<strong> AWS region</strong>.</li><li><strong>On Microsoft Azure,</strong> an instance of Krossboard does automatically discover and handle all AKS clusters belonging to the same<strong> Azure resource group</strong>.</li><li><strong>On Google Compute Platform (GCP),</strong> an instance of Krossboard does automatically discover and handle all GKE clusters belonging to the same<strong> GCP project</strong>.</li></ul><blockquote><strong>Security notes:</strong> Krossboard needs only read access to your managed clusters. On each cloud platform, suitable IAM policies are applied to the instance during the installation to ensure that it’ll be restricted to only the set of read-permissions needed for its proper operations.</blockquote><h3>Getting Started</h3><p>Thanks to the distribution setup scripts released with Krossboard, you shall set up an instance as quick (a few minutes) than easy on the supported platforms. <strong>This section shows how to set it up on Google GCP, Microsoft Azure and Amazon AWS in just a couple of minutes</strong>.</p><p><strong>On Google GCP</strong>, the following commands shall set up an instance of Krossboard with a g1-small GCE virtual machine located in the us-central1-azone (all GCP zones are supported). The instance does handle all GKE clusters belonging to the project having the ID my-gke-project.</p><blockquote>These parameters should be adapted according to your deployment environment.</blockquote><pre>export GCP_PROJECT=&quot;my-gke-project&quot;<br>export GCP_ZONE=&quot;us-central1-a&quot;<br>export GCP_INSTANCE_TYPE=&quot;g1-small&quot;<br>curl -so krossboard_gcp_install.sh \<br><a href="https://krossboard.app/artifacts/setup/krossboard_gcp_install.sh">https://krossboard.app/artifacts/setup/krossboard_gcp_install.sh</a> \<br>  &amp;&amp; bash ./krossboard_gcp_install.sh</pre><p><strong>On Amazon AWS</strong>, the following commands shall set up an instance of Krossboard with a t2.small EC2 virtual machine for the region eu-central-1(see the list of <a href="https://krossboard.app/docs/releases/">supported regions</a>). The key pair defined by the variable KB_AWS_KEY_PAIR must exist.</p><blockquote>These parameters should be adapted according to your deployment environment.</blockquote><pre>export KB_AWS_KEY_PAIR=&quot;MyKeyPair&quot;<br>export KB_AWS_REGION=&quot;eu-central-1&quot;<br>export KB_AWS_INSTANCE_TYPE=&quot;t2.small&quot;<br>curl -so krossboard_aws_install.sh \<br><a href="https://krossboard.app/artifacts/setup/krossboard_aws_install.sh">https://krossboard.app/artifacts/setup/krossboard_aws_install.sh</a> \<br>  &amp;&amp; bash ./krossboard_aws_install.sh</pre><p>On Microsoft Azure, the following commands shall set up an instance of Krossboard with a Standard_B1m1 Azure virtual machine located in centralus (see the list of <a href="https://krossboard.app/docs/releases/">supported locations</a>). The instance does handle all AKS clusters belonging to the resource group set by the variable AZURE_GROUP.</p><blockquote>These parameters should be adapted according to your deployment environment.</blockquote><pre>export AZURE_GROUP=&quot;YOUR_AZURE_GROUP_WITH_AKS&quot;<br>export KB_AZURE_VM_SIZE=&quot;Standard_B1ms&quot;<br>export KB_AZURE_LOCATION=&quot;centralus&quot;<br>curl -so krossboard_azure_install.sh \<br><a href="https://krossboard.app/artifacts/setup/krossboard_azure_install.sh">https://krossboard.app/artifacts/setup/krossboard_azure_install.sh</a> \<br>  &amp;&amp; bash ./krossboard_azure_install.sh</pre><h3>Sample of analytics charts</h3><p>At the end of the setup script, the URL of Krossboard UI shall be displayed in the terminal. Open it in a browser to get access to the analytics web interface. <strong><em>Note: </em></strong><em>You may need to wait almost 15 minutes to have the first consolidated analytics available.</em></p><blockquote><strong>Data Export:</strong> the data backing each report can be exported in CSV format. Doing so, Krossboard leaves freedom to users to unlock additional analytics capabilities using your favorite data analysis tools (<a href="https://cloud.google.com/bigquery">Google BigQuery</a>, <a href="https://aws.amazon.com/athena/">AWS Athena</a>, <a href="https://azure.microsoft.com/en-us/services/synapse-analytics/">Azure Synapse</a>, <a href="https://www.tableau.com/">Tableau</a>, <a href="https://www.microsoft.com/en-us/microsoft-365/excel#pivot-forPersonal">Microsoft Excel</a>, to list a few).</blockquote><h3>Recent consolidated usage</h3><p>For each cluster discovered and handled, this section displays pie charts showing the latest consolidated CPU and memory usage. Updated every 5 minutes, these reports highlight shares of used, available, and non-allocatable resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nW7zGvXvMn9KGfrRfB5uKA.png" /><figcaption>Krossboard — sample of screenshot showing 5-minutes recent consolidated usage for CPU and memory</figcaption></figure><h3>Per-cluster Usage Trends &amp; Accounting</h3><p>For each cluster (selected on-demand by the user), this section provides various reports showing hourly, daily and monthly usage analytics for CPU and memory resources.</p><p>The backing data can be exported in CSV using the links provided at the top of the charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Qh_JNXoVMJs_HeLvLJH5g.png" /><figcaption>Krossboard — Sample screenshot of per-cluster usage trends &amp; accounting (hourly, daily, monthly)</figcaption></figure><h3>Global Usage Trends &amp; History</h3><p>This section provides comprehensive usage reports covering all clusters for a user-defined period of time. The intent of these reports is to provide an at-a-glance visualization to compare the usage of different clusters for any period of time.</p><p>The backing data can be exported in CSV using the links provided at the top of the charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JiAARIrRLXfC5tIGmvSujA.png" /><figcaption>Krossboard — sample screenshot showing cluster-scoped global usage trends &amp; history</figcaption></figure><h3>Conclusion and next steps</h3><p>We introduced in this story original concepts to deal with resource usage analytics on multi-Kubernetes environments. With specific capabilities to automatically discover and track the usage of managed Kubernetes platforms, we described architectural aspects and showed how to quickly deploy it to handle Google GKE, Amazon EKS and Microsoft AKS clusters.</p><p>Krossboard is <a href="https://krossboard.app/docs">available and distributed at no cost</a>. Don’t hesitate to deploy it and let’s know your feedback. If you do encounter any problems or do have suggestions for enhancements, please open a request at the project’s<a href="https://github.com/2-alchemists/krossboard"> Github page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5108cb7f0e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A RESTful API for handling contact forms on Hugo static websites]]></title>
            <link>https://rodrigue-chakode.medium.com/a-generic-restful-handler-for-contact-forms-on-hugo-static-websites-3aec6b3466a0?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/3aec6b3466a0</guid>
            <category><![CDATA[gohugo]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[static-website-design]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-app-engine]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 20:13:30 GMT</pubDate>
            <atom:updated>2020-11-16T11:39:40.689Z</atom:updated>
            <content:encoded><![CDATA[<h4>Deploy in minutes on Google App Engine, Kubernetes or Docker</h4><p>This story introduces <a href="https://github.com/rchakode/hugo-mx-gateway">hugo-mx-gateway</a> a recently published open source (built in Go, Apache 2.0 License) project meaning to provide a generic yet effective RESTful API to handle contact/demo request forms on <a href="https://gohugo.io/">Hugo</a> static websites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6XWb6lCIw68jxRPxJDru0Q.png" /><figcaption>hugo-mx-gateway — Architecture and operations overview</figcaption></figure><p>This story first quickly explains the motivation of hugo-mx-gateway along with its key features and architecture. An in-depth analysis demonstrating how original it is, i.e. with regard to the state of the art, is provided later in this article. Before that, we’ll describe a sample of deployment using Docker and an integration with a static contact form built with Hugo.</p><h3>Introduction</h3><p>As a “Sunday” open source developer I often need to create basic websites to promote my projects. Though being a non web developer — I’m actually used to be better <a href="https://medium.com/@rodrigue.chakode">building cloud and monitoring things</a>, I would’t like to spend lots of time, efforts, nor money to develop complex web stacks — even a CMS with <a href="https://en.wikipedia.org/wiki/LAMP_(software_bundle)">LAMP</a> is not an option ☺. That’s why I’m a big fan of static websites. <a href="https://gohugo.io/">Hugo</a> being my favorite generator.</p><p>However, one of challenges of static websites is to effectively cope with the support of contact/demo request pages. See for example <a href="https://discourse.gohugo.io/t/is-it-possible-to-add-a-contact-form-to-a-site/1550/4">here</a> a related discussion in the Hugo’s forum. At this point our choice for static website appears to be very fragile, and we might be tempted to abandon that to opt for a dynamic website where contact requests seems to be easy to handle.</p><p>Didn’t you ever experience such a situation? If so, this story is for you! Let’s discover how <a href="https://github.com/rchakode/hugo-mx-gateway">hugo-mx-gateway</a> can make your journey easy to handle contact requests on your next or current static websites.</p><h3>Key features</h3><p>Here is a non-exhaustive list of hugo-mx-gateway features.</p><ul><li><strong>Deploy in minutes</strong> on <a href="https://cloud.google.com/appengine">Google App Engine</a>, <a href="https://kubernetes.io/">Kubernetes</a>, or <a href="https://www.docker.com/">Docker</a>: No hidden complexity, we just have to set the required <a href="https://github.com/rchakode/hugo-mx-gateway#configuration-variables">configuration parameters</a> and we’re done.</li><li><strong>Secured against spams and bots:</strong> each form submission is validated against a configurable list of allowed origin domains. To prevent bot submissions, it’s also possible to enable form validation with <a href="https://www.google.com/recaptcha/">Google reCAPTCHA</a> (v2).</li><li><strong>SMTP relay with authentication and TLS</strong>: hugo-mx-gateway acts as a SMTP client to relay contact requests. For each request, it automatically generates a <strong>templated summary email</strong> (based on <a href="https://golang.org/pkg/text/template/">Go HTML Template</a>) and sends it to the requester — for confirmation, while <strong>bbc</strong>ing a copy of that email to the site owner for tracking and follow up. For an enhanced security, it requires an authenticated SMTP account along with a server supporting TLS.</li><li><strong>Forward back to the origin page:</strong> upon the processing of each request (on success or failure), hugo-mx-gateway handles the reply to the calling <em>static</em> page by redirecting the browser with specific URL parameters describing the completion status. Hence, with a few lines of Javascript within the page, the reply parameters can be retrieved and displayed to the user — like on the below screenshot.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*qM5BuuKIs_SBSmQSVX_Oog.png" /><figcaption>hugo-mx-gateway — Sample of contact form submitted and successfully handled</figcaption></figure><h3>How does it work</h3><p>Once deployed, hugo-mx-gateway exposes a RESTful API <em>able to handle contact requests for one or many static websites.</em> It’s only needed to provided the list of supported sites as allowed origin domains for the service.</p><p>The architecture depicted on the diagram below adheres to the following operations principles:</p><ol><li>A instance of hugo-mx-gateway can be deployed either on <a href="https://github.com/rchakode/hugo-mx-gateway#deployment-on-google-app-engine">Google App Engine</a>, <a href="https://github.com/rchakode/hugo-mx-gateway#deployment-on-a-kubernetes-cluster">Kubernetes</a>, or <a href="https://github.com/rchakode/hugo-mx-gateway#deployment-on-docker">Docker</a>. Once deployed, it enables a RESTful POST service with a /sendmail endpoint handle contact form submissions. Then on each supported website (i.e. set as allowed origin), we only have to point the <a href="https://www.w3schools.com/tags/att_form_method.asp">action</a> towards this sendmail endpoint. These two points will be demonstrated in the next the section.</li><li>Upon a request, hugo-mx-gateway automatically retrieves information submitted by the user (email, subject, message details...), then <strong>generates and sends</strong> a <strong>templated email</strong> to the user-provided email address, while <strong>bcc</strong>ing a copy of that email to a configured site owner’s address for internal tracking and follow up. The email template is based on <a href="https://golang.org/pkg/text/template/">Go Template</a>.</li><li>Once the request is processed, the reply back towards the calling <em>static page</em> is addressed by redirecting the browser to the origin page with additional URL parameters describing the completion status (success/failure) along with a message for details. E.g. /contact.html?status=success&amp;message=request%20submitted. These parameters can then be retrieved (e.g. using Javascript) to be displayed to the requesting user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6XWb6lCIw68jxRPxJDru0Q.png" /><figcaption>hugo-mx-gateway — Architecture and operations overview</figcaption></figure><h3>Deploying hugo-mx-gateway on Docker</h3><p>A Docker instance of hugo-mx-gateway can be launched with the following command — Make sure to review the <a href="https://github.com/rchakode/hugo-mx-gateway#configuration-variables">configuration environment variables</a> and set them appropriately.</p><pre>docker run -d \<br>    --publish 8080:8080 \<br>    --name &#39;hugo-mx-gateway&#39; \<br>    -e SMTP_SERVER_ADDR=&quot;smtp.example.com:465&quot; \<br>    -e SMTP_VERITY_CERT=true \<br>    -e SMTP_CLIENT_USERNAME=&quot;<a href="/service/mailto:postmaster@example.com">postmaster@example.com</a>&quot; \<br>    -e SMTP_CLIENT_PASSWORD=&quot;postmasterSecretPassWord&quot; \<br>    -e CONTACT_REPLY_EMAIL=&quot;<a href="/service/mailto:noreply@example.com">noreply@example.com</a>&quot; \<br>    -e CONTACT_REPLY_BCC_EMAIL=&quot;<a href="/service/mailto:contact@example.com">contact@example.com</a>&quot; \<br>    -e DEMO_URL=&quot;<a href="https://demo.example.com/">https://demo.example.com/</a>&quot; \<br>    -e ALLOWED_ORIGINS=&quot;127.0.0.1,example.com&quot; \<br>    rchakode/hugo-mx-gateway</pre><p>The next command checks that the service is up and running.</p><pre>curl <a href="https://127.0.0.1:8080/healthz">https://127.0.0.1:8080/healthz</a></pre><h3>Create a Hugo form bound to hugo-mx-gateway</h3><p>The integration works as follows:</p><ul><li>A sample of Hugo contact form can be founded <a href="https://github.com/rchakode/hugo-mx-gateway/blob/master/samples/hugo-partial-contact-form.html">here</a>. The form must be copied in your <strong>Hugo HTML template</strong> file. Edit the HTML &lt;form&gt; tag to make the <strong>action</strong> pointing towards the URL of the sendmail endpoint of hugo-mx-gateway. With Docker, the endpoint shall looks like h<a href="http://docker-host:8080/sendmail">ttp://DOCKER-HOST:8080/sendmail</a> by replacing DOCKER-HOST with the address of the Docker machine.</li><li>In the <strong>Hugo Markdown content</strong> file, define a parameter named tags as an array containing the value contact as below. Indeed, according to the values included in this array the form can be rendered either as a contact or a demo request form. But let’s focus on the contact use case.</li></ul><pre>---<br>title: &quot;Contact Us&quot;<br>description: &quot;Contact request page&quot;<br>date: 2020-04-25T00:20:27+02:00<br>tags: [contact]<br>---</pre><h3>Why hugo-mx-gateway — state-of-the-art</h3><p>It’s essential to solve each problem with original solutions. This means not to reinvent the wheel and reuse what ever is possible to reuse, while taking into account your specific requirements.</p><p>Regarding the problem of handling contact forms on static websites, if we search on the Internet we’ll quickly find many people facing this problem. See <a href="https://discourse.gohugo.io/t/is-it-possible-to-add-a-contact-form-to-a-site/1550/4">this thread</a> for example. What are options to tackle that?</p><h4><strong>Managed form-handling platforms</strong></h4><p>We can opt for a managed form-handling platform, like <a href="https://getform.io/">Getform</a> and <a href="https://gsuite.google.com/products/forms/">Google Form</a>, which will be responsible to handle form submissions.</p><p>This choice means to accept to store our data on that managed platform — what is not always acceptable regarding our data governance strategies or regulatory laws (e.g GDPR in Europe). Further, we have to adapt our needs with the limited set of capabilities provided by the selected managed platform — what is not always convenient. For example, in many use cases a typical workflow is to automatically send a customized confirmation email to the requester while sending a summary of the request to a specific internal address for tracking and follow up. I didn&#39;t find a way to cope with these requirements using existing managed form handlers.</p><h4><strong>Implement a specific </strong>contact <strong>form handler</strong></h4><p>As alternative to a managed form-handling platform the second option is, as suggested <a href="https://discourse.gohugo.io/t/is-it-possible-to-add-a-contact-form-to-a-site/1550/4">here</a> for example, to build our own API to handle contact requests. So you have to do it for every single website you own. The required efforts may be heavy if you manage a several sites. I think that it would be very helpful to have a generic brick that can do this trick for any website — That’s why I started working on thehugo-mx-gateway project.</p><h4><strong>Towards a generic contact form-handling pattern</strong></h4><p>I took time to think how to avoid repetitive tasks and efforts to implement, setup and handle contact forms for static websites.</p><p>It appears to me that typical contact requests can be handled using a simple pattern or workflow in two components. One the one hand, the pattern defines a form with a common set of fields filled up by the requester (a name, an email, an organization, a subject, and a message). On the second hand, we have a common form-handling behavior consisting in sending a confirmation of the request to the sender while notifying the website’s owner (organization or individual) with a summary of the request — hence allowing the later to easily track and follow up requests. That’s where our proposal with <a href="https://github.com/rchakode/hugo-mx-gateway">hugo-mx-gateway</a> comes in.</p><h3>Conclusion</h3><p>In this story we introduced <a href="https://github.com/rchakode/hugo-mx-gateway">hugo-mx-gateway</a> a new open source software aiming at providing a generic contact form handler for <a href="https://gohugo.io/">Hugo</a> static websites. Its core features, its architecture as well as its operations principles have been described. We also demonstrated how it can be easily deployed on Docker and used to back contact forms from several static websites. To recall, hugo-mx-gateway works as a RESTful service deployed on <a href="https://github.com/rchakode/hugo-mx-gateway#deployment-on-docker">Docker</a>, <a href="https://github.com/rchakode/hugo-mx-gateway#deployment-on-a-kubernetes-cluster">Kubernetes</a>, or <a href="https://github.com/rchakode/hugo-mx-gateway#deployment-on-google-app-engine">Google App Engine</a> (GAE). E.g. Take a look <a href="http://here/">here</a> to see a form handled by an instance hosted on GAE. This project is open source (code in Go), shipped with sample resources to quickly get started. As an open source software, it’s open for contributions and any feedback that could help improving it. ;)</p><p><em>This story is about an open source software hosted at </em><a href="https://github.com/rchakode/hugo-mx-gateway"><em>https://github.com/</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3aec6b3466a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing Prometheus Metrics and Grafana Dashboard for Cost Allocation on Kubernetes Clusters]]></title>
            <link>https://rodrigue-chakode.medium.com/bringing-prometheus-metrics-and-grafana-dashboard-for-cost-allocation-on-kubernetes-clusters-1ee7f68cd677?source=rss-81362ccd540------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ee7f68cd677</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[capacity-planning]]></category>
            <category><![CDATA[cost-allocation]]></category>
            <dc:creator><![CDATA[Rodrigue Chakode]]></dc:creator>
            <pubDate>Sun, 30 Jun 2019 11:35:08 GMT</pubDate>
            <atom:updated>2020-10-24T08:37:15.017Z</atom:updated>
            <content:encoded><![CDATA[<p>This story introduces a Prometheus Exporter along with a Grafana Dashboard intending to provide cost-oriented consolidated resource usage analytics for Kubernetes clusters. Those analytics actually aim at highlighting factual metrics to help organizations easily make cost allocation and capacity planning decisions on short-, mid-, and long terms.</p><blockquote><strong>Sidenote:</strong> Readers may also be interested in this other <a href="https://medium.com/@rodrigue.chakode/kubernetes-resource-usage-analytics-for-cost-allocation-and-capacity-planning-416800e85d16">story</a>, which is related, but independent to the current one.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7rbOREwIUTJVnvpLImruUA.png" /><figcaption>A Grafana Screenshot of the Hereafter-described Analytics Dashboard for Kubernetes Cost Allocation and Capacity Planning</figcaption></figure><h3>Introduction</h3><p>A couple of months ago <a href="https://github.com/rchakode/kube-opex-analytics">Kubernetes Opex Analytics</a> has been introduced as an original open source resource usage analytics tool to make cost sharing and capacity planning decisions easy on Kubernetes clusters. Released under the terms of Apache 2.0 License the tool comes with built-in analytics charts covering various use cases. It received a lot of user feedback, some requesting the ability to expose its metrics to existing Prometheus environments.</p><p>A Prometheus exporter has been introduced in version 0.3.0, so in this story I’ll be pleased to introduce and describe it. This comes along with an integrated Grafana dashboard to let users easily getting started. All that comes in complement of the native Kubernetes Opex Analytics features and dashboard.</p><h4>What will be covered next</h4><p>The story will give details on the exposed metrics and provide steps to setup the Exporter and the provided Grafana dashboard. Before that we’ll review the core features of Kubernetes Opex Analytics for readers not familiar with it.</p><blockquote>If you’re already familiar with the concepts of Kubernetes Opex Analytics, you can skip the next section and move forward.</blockquote><h3>Kubernetes Opex Analytics in a Nutshell</h3><p>Kubernetes Opex Analytics is designed on the following core concepts:</p><ul><li><strong>Namespace-focused</strong>: Means that consolidated resource usage metrics consider individual namespaces as fundamental units for resource sharing. A special care is taken to also account and highlight non-allocatable resources .</li><li><strong>Hourly Usage &amp; Trends</strong>: Like on public clouds, resource use for each namespace is consolidated on a hourly-basic. This actually corresponds to the ratio (%) of resource used per namespace during each hour. It’s the foundation for cost calculation and also allows to get over time trends about resources being consuming per namespace and also at the Kubernetes cluster scale.</li><li><strong>Daily and Monthly Usage Costs:</strong> Provide for each period (daily/monthly), namespace, and resource type (CPU/memory), <em>consolidated cost </em>computed given one of the following ways: <em>(i)</em> accumulated hourly usage over the period; <em>(ii)</em> actual costs computed based on resource usage and a given hourly billing rate; <em>(iii)</em> normalized ratio of usage per namespace compared against the global cluster usage.</li><li><strong>Efficient Visualization:</strong> For metrics it generates, Kubernetes Opex Analytics provides dashboards with relevant charts covering as well the last couple of hours than the last 12 months (i.e. year) as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGfFeGTUezdDmo7_zGBEtg.png" /><figcaption>Kubernetes Opex Analytics — Screenshot of the Built-in Dashboard</figcaption></figure><h3>Installing Kubernetes Opex Analytics</h3><p>Next on we assume that the installation will be done on a Kubernetes cluster, as you can also install it on Docker as described <a href="https://github.com/rchakode/kube-opex-analytics#start-koa-on-docker">here</a>.</p><p>There is a <a href="https://github.com/rchakode/kube-opex-analytics/tree/master/helm/kube-opex-analytics">Helm chart</a> to ease the deployment on Kubernetes, either by using Helm Tiller or kubectl.</p><p>In both cases check the values.yaml file to modify the configuration options according to your needs (e.g. to have persistent volume for data storage).</p><p>Using Helm Tiller:</p><pre>helm upgrade \<br>  --install kube-opex-analytics \<br>  helm/kube-opex-analytics/</pre><p>Using kubectl:</p><pre>helm template \<br>  --name kube-opex-analytics \<br>  helm/kube-opex-analytics/ | kubectl apply -f -</pre><blockquote>This will enable the built-in dashboard via a HTTP service named kube-opex-analytics on port 80.</blockquote><h3>Prometheus Exporter</h3><p>Metrics are exposed for Prometheus via the /metrics endpoint.</p><h4>Exposed Metrics</h4><p>As shown on the sample on Figure below the exposed metrics are:</p><ul><li>koa_namespace_hourly_usage : exposes for each namespace its current hourly resource usage for both CPU and memory.</li><li>koa_namespace_daily_usage : exposes for each namespace and for the ongoing day, its current resource usage for both CPU and memory.</li><li>koa_namespace_monthly_usage : exposes for each namespace and for the ongoing month, its current resource usage for both CPU and memory.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*obCxbKRH3j_i4Wf4zvgD8Q.png" /><figcaption>Kubernetes Opex Analytics — Sample of Metrics Exposed for Prometheus</figcaption></figure><h4>Prometheus <strong>Scrape</strong> Job</h4><p>The job can be configured like below. An interval less than 5 minutes (i.e. 300s) is useless as no new metrics will be generated within the meantime.</p><pre>scrape_configs:<br>  - job_name: &#39;kube-opex-analytics&#39;<br>    scrape_interval: 300s<br>    static_configs:<br>      - targets: [&#39;kube-opex-analytics:80&#39;]</pre><blockquote>Recall that Kubernetes Opex Analytics works with hourly-consolidated metrics, so you may need to wait at least an hour to have all metrics available.</blockquote><h3>Grafana Dashboard</h3><p>Once metrics available in Prometheus, get <a href="https://grafana.com/dashboards/10282">this Grafana Dashboard</a> and import it into Grafana. The dashboard relies on a variable KOA_DS_PROMETHEUS that shall point to your Prometheus data source.</p><p>Once configured properly the dashboard shall just work out of the box to display charts as described hereafter.</p><h4>Hourly Usage</h4><p>There are two panels displaying respectively usage charts for CPU (left) and memory (right) over the selected interval (7 days by default). Series for the different namespaces are stacked. This makes usage comparison easy and also helps show how loaded the cluster is. On the below example we can see that during the last 5 days, global CPU and memory usage reached more than 90%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5bKOYCGVnlizBFeFEZMWRg.png" /><figcaption>Kubernetes Opex Analytics — Hourly Resource Usage</figcaption></figure><h4>Current Day’s Usage</h4><p>The two panels display respectively cost charts for CPU (left) and memory (right) over the ongoing day. Values are computed using cost algorithms described early in this story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSEQP-oQ5MoqGt97RDIQag.png" /><figcaption>Kubernetes Opex Analytics — Current’s Day Resource Usage</figcaption></figure><h4>Current Month’s Usage</h4><p>The two panels below<strong> </strong>display respectively cost charts for CPU (left) and memory (right) over the ongoing month. Values computed using cost algorithms described early in this story.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mU0qKLMq8iIQ1gnl_6jPgg.png" /><figcaption>Kubernetes Opex Analytics — Current’s Month Resource Usage</figcaption></figure><h3>Move forward</h3><p>In short we’ve introduced in this story a Prometheus exporter as well as a Grafana dashboard for <a href="https://github.com/rchakode/kube-opex-analytics">Kubernetes Opex Analytics</a>.</p><p>As you may have noticed charts of the described Grafana dashboard are less rich than the ones of the built-in dashboard of Kubernetes Opex Analytics. For instance daily and monthly usage are limited to the current day, respectively current month. This makes difficult to compare current usage with previous ones. <a href="https://community.grafana.com/t/display-stacked-series/3402/15">These are limitations inherent in how Grafana handles bar charts based on series names</a>. The current implementation hence leaves room for further improvements, any contributions will be really appreciated.</p><p>I want to recall that Kubernetes Opex Analytics is open source, it’s an open-to-contribution project. We’re always pleased to receive feedback and contributions on <a href="https://github.com/rchakode/kube-opex-analytics">Github</a>: submit an issue if you encounter problems or have some ideas of improvement; make a pull request; or give a star.</p><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ee7f68cd677" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>