<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dmytro Dyachuk on Medium]]></title>
        <description><![CDATA[Stories by Dmytro Dyachuk on Medium]]></description>
        <link>https://medium.com/@dyachuk?source=rss-c9d1152be9a7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*G_D2JXlRe2Fhrg4ktWRuHw.jpeg</url>
            <title>Stories by Dmytro Dyachuk on Medium</title>
            <link>https://medium.com/@dyachuk?source=rss-c9d1152be9a7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:29:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dyachuk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Economics of Self-Hosting LLMs (Part 1): Does the Cost Math Actually Work?]]></title>
            <link>https://medium.com/@dyachuk/the-economics-of-self-hosting-llms-part-1-does-the-cost-math-actually-work-752839783ff3?source=rss-c9d1152be9a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/752839783ff3</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Dmytro Dyachuk]]></dc:creator>
            <pubDate>Thu, 11 Jun 2026 20:27:17 GMT</pubDate>
            <atom:updated>2026-06-11T20:27:17.689Z</atom:updated>
            <content:encoded><![CDATA[<p>For many years, I’ve been dealing with capacity, cost, and performance of various systems, LAMP stacks, Kubernetes clusters, distributed databases and so on. Each of those technologies has its performance challenges and economics, and figuring those out can be quite interesting and useful.</p><p>Now there’s a new player in town: LLMs. Most people consume them through the UIs and APIs offered by OpenAI, Anthropic, and friends. Those providers run exceptionally large models, and they charge accordingly. And it works quite well, the experience is great and the friction is low.</p><p>But something else has been happening in parallel: an explosion of open-weight models-models that differ wildly in size and capability, and that you can download and run yourself. Self-hosting is starting to look genuinely tempting for a few reasons. First, data sovereignty (your data stays on your infrastructure); second, flexibility (you pick the model that fits the task); and finally, potentially cost (a smaller model might be perfectly sufficient for most day-to-day tasks, and running it yourself could be cheaper at scale). <em>Maybe.</em> That’s the part I want to actually measure rather than assume.</p><p>In this series, I’ll be looking at the tooling and the real-world experience of running open-weight LLMs — performance, cost to operate, and the non-obvious tradeoffs. No conclusions in advance; just experiments and observations as I go.</p><p>For the first one, I picked a small but capable model and pushed it to see what breaks and where the money goes.</p><h4>The Setup</h4><p>Obviously, it is not the exact number,I picked Qwen2.5–7B-Instruct — a 7B parameter model that comfortably fits on one of the smaller GPU instances AWS offers, with room to spare. Later in the series I’ll look at larger models on bigger hardware, but for a first experiment this keeps things clean: one GPU, one model, no multi-GPU complexity to reason about.</p><p>I ran a concurrency sweep on AWS g5.xlarge ($1.006/hr, NVIDIA A10G, 125 TFLOPS BF16 peak) serving Qwen2.5–7B-Instruct in BF16. The benchmark: 1024-token inputs, 128-token outputs, closed-loop concurrency from 1 to 64 using vLLM’s built-in bench tool.</p><p>The goal was to find the <strong>serving Pareto frontier</strong> — the curve that tells you exactly what latency you pay for a given throughput.</p><p>Two metrics drive everything here. <strong>Output throughput</strong> (tok/s) is how many tokens the server generates per second across all active requests, the raw capacity number. <strong>P99 TPOT</strong> (ms/token) is the 99th-percentile Time Per Output Token: the gap between consecutive tokens for the slowest 1% of requests. It’s the latency number that matters for real users — high TPOT means the response feels slow and stuttery, even if throughput looks fine in aggregate. The main findings are presented in the graph below, and summarized in the table which follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uH5LhEcy5Eji59z8.png" /></figure><pre>Concurrency │ Throughput │ P99 TPOT<br>────────────┼────────────┼──────────<br>          1 │   17 tok/s │    57 ms<br>          8 │  115 tok/s │    75 ms<br>         16 │  166 tok/s │    94 ms<br>         32 │  233 tok/s │   134 ms  ← knee<br>         64 │  275 tok/s │   228 ms</pre><p>From C=1 to C=32, throughput scales nearly linearly while latency rises gradually. Then the curve breaks. C=32→64 delivers only 18% more throughput at the cost of 70% more latency. <strong>C=32 is the knee</strong> — the point of maximum efficiency. Obviously, this is not the exact knee point, since I didn’t test every concurrency level between 32 and 64. Since the experiments were time-constrained, latency did not have enough time to climb significantly. With longer runs, the increase beyond the knee would likely be much more pronounced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5WQIhWWyBdWhM1ZFlalAPg.png" /></figure><p>The chart above shows that the Model Flop Utilization (MFU) went up to 12% at the knee, yet the GPU utilization observed via nvtop was reported to be almost 100%. The most likely explanation is that the GPU was busy fetching data from memory.</p><h3>What It Actually Costs Per Token</h3><p>Now let’s have a look at the cost numbers, at the knee (C=32, 233 tok/s), the math is straightforward:</p><p><strong>$1.006/hr ÷ 233 tok/s × (1,000,000 / 3,600) = $1.20 per million output tokens</strong></p><p>To put that in context against managed API pricing today. I’m comparing output tokens only, that’s where the cost is dominated in generation-heavy workloads; input is typically 3–5× cheaper across all providers.</p><pre>Model                              │ Output $/M tokens<br>───────────────────────────────────┼───────────────────<br>Self-hosted (Qwen2.5-7B, g5.xlarge)│   $1.20  ← this experiment<br>GPT-4o mini                        │   $0.60<br>Claude Haiku 4.5                   │   $5.00<br>GPT-4o                             │  $10.00<br>Claude Sonnet 4.6                  │  $15.00<br>Claude Opus 4.8                    │  $25.00</pre><p>A few things jump out. First, a self-hosted 7B model at optimal concurrency costs about as much as Haiku, while being a much smaller, fully on-prem model. Against GPT-4o or Sonnet the cost advantage is 8–12×. Second, and this is the part most teams miss, running at suboptimal concurrency (C=1 instead of C=32) inflates cost to <strong>$16.40/M tokens</strong>: 13× worse, on the same hardware, just because nobody benchmarked the curve.</p><p>The third thing: OpenAI can serve GPT-4o mini cheaper than I can self-host a 7B model. That’s economies of scale, not magic and it’s a useful anchor for thinking about when self-hosting actually wins.</p><h3>Can We Push it Further?</h3><p>Now let’s look at how close the hardware came to its theoretical capacity..</p><p>The instance’s 31.5 TFLOPS went quite unused. The GPU cores sat idle, waiting. Not because of bad software. Because LLM decode is pretty much memory-bandwidth-bound.</p><p>Every output token requires streaming the full model weights — 14 GB for a 7B BF16 model, from High Bandwidth Memory (HBM) to the compute units. This happens once per decode step, regardless of batch size. The A10G’s 600 GB/s HBM bandwidth is the real ceiling, not the compute.</p><p>The implication: TFLOPS can play a secondary role in inference. What you need to be watching is bandwidth and VRAM utilization.</p><p>This has direct consequences for instance selection and configuration:</p><ul><li><strong>Bigger instance ≠ better value.</strong> A g5.16xlarge has the same single GPU as a g5.xlarge but costs 4× as much. The extra CPU cores and RAM do nothing for inference throughput.</li><li><strong>FP8 quantization helps via bandwidth, not compute.</strong> Halving weight size (BF16→FP8) halves bytes streamed per token, directly increasing throughput, even though I never got close to the compute ceiling. This is why Ada and Hopper GPUs matter: FP8 halves the memory traffic.</li><li><strong>VRAM determines two things simultaneously:</strong> whether the model fits at all, and how much KV cache headroom is left for concurrent requests, which sets how far right the throughput curve can go before it bends.</li></ul><h3>In the Next Post</h3><p>I’ll instrument the same sweep more thoroughly to answer the question about the bottlenecks. That means sampling real HBM bandwidth during the run, pulling vLLM’s Prometheus metrics for KV cache utilization and queue behavior, and separating prefill latency from decode latency. TTFT (Time to First Token) tells a different story than TPOT and I haven’t looked at it yet.</p><p>I also want to test two things that should directly stress-test the bandwidth-bound hypothesis: varying context length (if KV cache bandwidth is the culprit, a shorter input should shift the knee rightward), and swapping BF16 weights for INT8 or FP8 (halving weight size should roughly double the throughput ceiling if memory is truly the bottleneck). If those experiments confirm the story, the next hardware question becomes obvious, but I’ll save that for when the data is in.</p><p>Years across different systems taught me the same lesson each time: the headline spec is never what actually constrains you. For little experiment showed that for LLM inference it can be memory bandwidth and the concurrency curve, not TFLOPS.</p><p>The cost math for self-hosting works, but it’s conditional. At the right operating point it’s $1.20/M tokens; at the wrong one, same hardware, it’s $16/M. The difference is measuring the curve. That’s what this series is about. Stay tuned.</p><p><em>Originally published at </em><a href="https://substack.com/home/post/p-201528071"><em>https://dmytrodyachuk.substack.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=752839783ff3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 reasons to do cost accounting for Kubernetes clusters]]></title>
            <link>https://medium.com/@dyachuk/4-reasons-to-do-cost-accounting-for-kubernetes-clusters-afc825d6d177?source=rss-c9d1152be9a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/afc825d6d177</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[capacity-planning]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Dmytro Dyachuk]]></dc:creator>
            <pubDate>Thu, 30 Aug 2018 17:03:54 GMT</pubDate>
            <atom:updated>2018-08-30T17:03:54.254Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine shopping in a store with no price tags, where all you can see is the total bill after you go through the check out. At some point in time, you might realize that you’ve spent too much, and then decide to spend less next time. You try buying other brands, smaller packages, but the final bill is just not getting much smaller.</p><p>This is what it’s like to manage Kubernetes clusters with no cost accounting. You keep adding/removing applications, optimizing various parameters, sometimes adding new tenants into the cluster and still, the surprise bills keep hitting you at the end of the month. It is time to cut back, but where? If you want to match what you need with the costs, you would need to put price tags on all items in your cluster. The process of assigning price tags to pods, deployments, services, and namespaces in a cluster is called cost accounting.</p><p>Back in the time of one to one deployments, a single application would run on a single server. The cost of that server would be effectively the cost of that application. Everything was simple. With the introduction of containers we are getting out of that era, and that’s a good thing. Running multiple pods/containers per server makes a lot of sense from an economic standpoint. It lowers the server fleet footprint, and thus improves the cost of running an application. Kubernetes orchestrates those containers into larger logical blocks and hides a lot of the low-level complexity of managing such an environment. It all sounds great, but you still have to manage it at a higher level. Now cost accounting is even more critical. Why? Well, here are four reasons.</p><h3>Understand what you’re buying</h3><p>Before you your financial department get surprised by the next bill from Amazon or Google, you need to understand the actual costs of cluster components. Or, put more simply, what did you spend our money on?</p><p>The spending categories can be viewed from two perspectives: from the perspective of the resources and from the perspective of the business function of a specific component. If the costs for each class of resources is usually given to you by the cloud providers, the split of those between the components is something you will need to work out yourself. In future posts I plan to describe some simple mechanisms you can use for doing that. So let’s define the categories. The charges for the consumed resources can be split into three different categories:</p><ul><li>compute, which consists of the charges for the nodes</li><li>storage, such as costs of the Persistent Disks or EBS volumes</li><li>networking, which will include external traffic charges, load balancing, cross AZ charges and so on</li></ul><p>Logical components of Kubernetes clusters can be divided into:</p><ul><li>admin, which includes etcd, API server, scheduler and everything you can find in the kube-system namespace</li><li>monitoring, which includes the metrics stack, e.g. Prometheus</li><li>logging, e.g. ELK</li><li>applications — all the pods which handling the user workload</li><li>idle — the resources, like CPU and memory which were provisioned but not allocated or used</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*CkIu8kiUP21ACTJ7" /></figure><p>Once you summarize the answers in a chart like the one above, for example using this <a href="https://docs.google.com/spreadsheets/d/1IOLV5Y2HAJwPGVCnlUdjDYN3qlZ79_yL-IIBnVjfpCU/edit#gid=0">template</a>, many things about the cluster become apparent right away. For instance, in this case, it is evident that a large chunk of money was spent on idle resources. Also, it is quite apparent that the cost of the storage for logging is on par with the cost of the storage for the user pods themselves. Perhaps running fewer nodes and trimming the logs a bit might reduce the monthly bill by a few thousand dollars.</p><h3>Produce realistic budgets</h3><p>For many years, I’ve watched people successfully applying the “coin toss”, also known as the “guesstimate” method, for estimating the budget of their clusters and applications. Even though this is a time-tested method which is well used across the industry, it works in only in one scenario — when no-one really cares about the budget. One legitimate example of this is organizations where the infrastructure cost is negligible when compared to the rest of the expenses. In other scenarios, this method has a number of shortcomings, i.e., it sucks! Here’s why.</p><p>There are two kinds of “guesstimate” budgets. Let’s call the first one the “bloated guesstimate.”, In this budget, costs are intentionally increased by some safety margin every year, to make sure you stay on the safe side and don’t run over budget. This works until the moment you realize that the money you’ve spent on that margin and did not use might have been better spent on something more valuable, like bonuses.</p><p>The other kind of budget is the “deflated guesstimate” budget. This is when the budgeted amount is lower than it should be. This makes you look good during annual budget planning meetings. When that happens, make sure you enjoy that moment, as for the rest of the year you will be trying to squeeze your actual costs into that budget.</p><p>In either case, “guesstimates” won’t make you look good in front of the rest of the organization in the long run.</p><p>Having realistic budgets for a cluster and its applications helps with directing the organization’s money where it is needed and makes everyone’s lives easier.</p><h3>Discourage overspending by teams</h3><p>Depending on the type of organization you are working for, there could be multiple teams using the clusters. Very often developers face tight deadlines and need to deliver the software in short time frames. With pressure like that, usually everyone focuses on the stability of that software, rather than optimizing the amounts of resources it uses. That’s understandable; business objectives always come first. But still, in situations like that, it is essential to understand the actual cost of such a scramble. The only way you can do that is by having fair and accurate cost accounting in place. By fair, I mean that every team sees the actual cost of running their services. Quite often, I’ve come across cases where the cost per team is calculated using guesses, while ensuring that their total always adds up to the total cost of the cluster. The guess numbers quite often are derived from previous guesses, gut feelings and a bit of randomization.</p><p>In any case, using guesswork to estimate the cost of the applications is most likely to make the resource-hogging applications look a lot better then they are in reality. Arbitrary division of the cluster costs forces the other tenants to absorb the cost of a new application hastily developed without regard for efficiency. As a result, the actual cost of rushing an app to production is hidden. It might not seem like a big deal at first, but as the time progresses, it can turn into a full-blown <a href="https://en.wikipedia.org/wiki/Tragedy_of_the_commons">tragedy of the commons</a>. Without seeing the actual monetary costs, teams are encouraged to keep sacrificing the efficiency of their applications for velocity. Once such an attitude starts propagating through the rest of the company, be prepared for a rapid increase in the clusters’ cost.</p><p>Once you start reporting the real costs per application, there is a high chance that some people will start scratching their heads, as the cost for some shortcuts they’ve taken turns out to be a lot greater than anyone has expected. Or it might be the other way around: the inefficiencies you have been worried about might not matter that much after all.</p><h3>Fix only what’s worth fixing</h3><p>Know the cost of each component and spend your time wisely; optimize only what is worth optimizing.</p><p>It is easy to get excited about improving the efficiency of the things you know you can improve. But is it worth it? Is this what you should really focus on right now? Well, it is hard to say. You need to do a cost-benefit analysis first. And it is impossible to analyze any improvement of the costs of the clusters unless you see a clear picture of the cost of the each of their components.</p><p>If you know the cost of the component, you can gauge the potential benefit of improving the cost of that component and can easily compare it to the cost of the required work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*QVHuyimqelijFfiU" /></figure><p>For example, let’s say you’ve read a great article about storage optimization in the new version of your metrics stack and cannot wait to roll it out. Apart from that, there is no other reason to switch to a newer version. You’ve estimated that the cost of the upgrade is 4 days, and it would halve the storage used by the metrics stack. Let’s also say that your time costs the company about 100$/hour. So the cost of the upgrade is 4 days x 8 hours/day x $100 = $3,200, while the potential savings are $100/month or $1,200/year. Clearly, in the presence of other tasks, this is not the best investment. At the same time, the cost of the idle resources is $5,000/month, and the fix is as simple as shutting down the few extra nodes — which won’t take even an hour.</p><p>Now it is pretty clear which tasks you might want to focus on first, isn’t it?</p><h3>What’s next?</h3><p>In the following post, I intend to share spreadsheets which you can use for basic cluster cost accounting in your organization. A spreadsheet like that would be sufficient for smaller organizations.</p><p>For larger setups, I would suggest checking out <a href="http://www.untab.io/">untab.io,</a> which I am working on right now. Untab can perform cost accounting for Kubernetes on a cluster/namespace/service/deployment/pod level, attribute costs to teams, and gauge the efficiency of the resource allocation and use.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ffast.wistia.net%2Fembed%2Fiframe%2Fy99pkwam37&amp;url=https%3A%2F%2Funtab.wistia.com%2Fmedias%2Fy99pkwam37&amp;image=https%3A%2F%2Fembed-ssl.wistia.com%2Fdeliveries%2Fb22966c2f7c85c89c6c7cf1f221ce173d78b8b5d.jpg%3Fimage_crop_resized%3D640x360&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=wistia" width="640" height="360" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1d198cb304434ae5a79a07c116690bb/href">https://medium.com/media/a1d198cb304434ae5a79a07c116690bb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afc825d6d177" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do Kubernetes clusters in AWS cost more than they should?]]></title>
            <link>https://medium.com/@dyachuk/why-do-kubernetes-clusters-in-aws-cost-more-than-they-should-fa510c1964c6?source=rss-c9d1152be9a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa510c1964c6</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Dmytro Dyachuk]]></dc:creator>
            <pubDate>Thu, 24 May 2018 16:01:01 GMT</pubDate>
            <atom:updated>2018-06-01T20:12:43.948Z</atom:updated>
            <content:encoded><![CDATA[<p>Congratulations, if you are reading this article this means that you made it to the point where the efficiency and cost your Kubernetes clusters became or about to become of some interest. Your journey probably started from a few fleet instances running in the cloud and now probably the costs for running the cluster amount to some significant number. Perhaps, you have already looked at your AWS bills and have already wondered if it can cost less. The truth is: yes it can. I spoke to a bunch of different companies running Kubernetes in the cloud and summarized the “tribal” knowledge about cost reduction. So, here is a list of the five issues which can contribute to cost overruns and easy remedies for fixing them.</p><h3>Over-sized pods</h3><p>It is common practice to use a standard template for limits and requests for pod provisioning. If requests describe the minimal requirement for the CPU and memory for a pod to be scheduled on a node, the limits describe the max amount of CPU and memory the pod can consume on that node. Very often engineers set the initial limits by “guesstimating”, doubling it just to be on the safe side, and then promising change it later, once more information is known. Usually, “later” never happens… As a result, the footprint of the entire cluster starts inflating over time, exceeding the demand for the actual services running inside the cluster. Just think about it, if every pod is using only 50% of the allocated capacities and the cluster is always is 80% full, that means that 40% of the cluster capacity is allocated but not used, or simply put — wasted. There are a few different ways of dealing with such scenarios.</p><p>The first and the most simple thing you can do is set up monitoring with<a href="https://grafana.com/"> Grafana</a> and<a href="https://prometheus.io/"> Prometheus</a> and periodically check the CPU and memory utilization by each pod. Then you can use the 99.9th percentile of the each to determine the new more optimal limits.</p><p>Employing the vertical pod<a href="https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler"> auto-scaler</a> (VPA) is another way of addressing this problem. The vertical pod auto-scaler effectively automates the promise the engineer made in the previous step. It monitors the CPU and memory utilization by the pods and adjusts the limits accordingly by terminating the pods and restarting them with the new limits. It works in both directions — if a pod is over-sized, the VPA reduces its size according to the observed utilization. If the pod is getting killed by OOM killer due the shortage of the allocated memory, it will be restarted with a larger memory limit. VPA handles CPU under-provisioning in a similar fashion and respects a disruption budget to avoid constant restarting.</p><p>Another way of dealing with the situation without resizing the pods is by using<a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/"> the horizontal auto-scaler (HPA</a>). If the demand for CPU and memory correlates with the traffic hitting those pods, rather than resizing the pods, it might be easier to increase their number. That’s exactly what HPA does. This can be triggered either by the CPU or memory utilization, or using a custom application metric. For specific information on setting up HPA with custom metrics it’s worth checking out<a href="https://medium.com/@marko.luksa/kubernetes-autoscaling-based-on-custom-metrics-without-using-a-host-port-b783ed6241ac"> this blog post</a> by Mark Luksa.</p><h3>Wrong node size and type</h3><p>Smaller nodes have higher relative OS footprint and increase management overhead. The smaller the node, the higher the amount of stranded resources. Stranded resources are CPU or memory which are idle, yet cannot be allocated to any of the pods, because the pods which are to be scheduled are too big to claim it. In my next post, I might dive a bit deeper into the subject of the maximum theoretical cluster utilization, but for now here is a simple graph showing the % of stranded resources as a function of node capacity. The server capacity is measured by the number of pods it can accommodate. As you can see, if the pods’ sizes are close to the size of the node (server) the percentage of the resources which are stranded gets higher. As a rule of thumb, if the node can accommodate 10 pods, the loss due to stranded resources is only 5%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*li9Op4-MrUhwNpSx." /></figure><p>Most of the companies I’ve spoken with mentioned that they run their workloads on M-type instances. Alternatively you can use C-type instances which have a higher ratio of CPU to memory. Which instance type is the right one to use? Well, it is a simple dynamic programming problem from a formal standpoint. I’ve put together a <a href="https://docs.google.com/spreadsheets/d/1cJe1aU5G8iFtAwqylKKAQ1hChO2DUzGHY5VAVF9Ix10/edit?usp=sharing">Google Sheet calculator</a> — please check the tab “Instance types”. It factors in the percentage of stranded resources and OS overhead, and estimates the monthly EC2 bill based on the pod size and the number of pods. You can clone the document, update the number of pods you plan to run and the average size of each pod, and it will find the cheapest instance type/size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*7dgELNbqYVvOnOhQ." /></figure><h3>Incorrect instance class</h3><p>On-demand, reserved, or spot instances — which ones are the best? On-demand is the most expensive option which permits the most flexibility and reliability. Reserved instances provide a good trade-off between pricing and flexibility. Spot instances are the trade-off between the reliability of the nodes and pricing. The math for the optimal selection of the instance type gets complicated pretty fast. As an alternative, the more straightforward rule is to periodically (e.g. quarterly) revise the instance allocation and switch from on-demand to reserved instances if your fleet size has not changed for some time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Y1PkzOs5VwTeV1ZF." /></figure><p>In the chart above, I’ve put together a visualization of this process with an example of a gradually growing yet fluctuating workload. This algorithm allows you to enjoy the savings from using reserved instances, yet preserves the flexibility of using on-demand instances.</p><h3>Over-provisioned storage</h3><p>Often little thought is given to the sizing of volumes attached to nodes. AWS offers two main types of storage: local, and EBS. The latter has different classes: GP, IO1, ST1 and SC1. While local storage comes for free with a selected list of instances, EBS storage can constitute<a href="https://www.paxautoma.com/blog/2017/11/14/buy-or-rent-the-cost-model-of-hosting-in-the-cloud-and-in-a-colo.html"> 20–40% of the final bill</a>. An audit of the allocated EBS volumes vs utilization often reveals that up to 50% of the volumes were substantially (&gt;60%) over-provisioned. The simplest remedy is as simple as checking the volume size, comparing it to the allocated size and then resizing it. The important step is to factor in the growth for the volumes and IOPS demand. I’ve put together a simple spreadsheet which shows how to do that [<a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vSho3u3KVf5_bbaOiDU1IQrbuYrwQOTG3vG1tqsYZx7u5VqIqt58Ega5JZsquL3F7-tZXWWk75SX2sg/pubhtml?gid=0&amp;single=true">Google doc</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*naxnOqud6V8LOow5." /></figure><p>Increasing the volume size can be done via the AWS API, yet downsizing it is not that straightforward. Recently, I came across this<a href="https://medium.com/@andtrott/how-to-downsize-a-root-ebs-volume-on-aws-ec2-amazon-linux-727c00148f61"> post</a> where Andrew Trott shows simple steps for downsizing EBS volumes. With a bit of scripting, this process can be fully automated.</p><p>If you want to eliminate the EBS costs completely, you might consider using<a href="https://github.com/rook/rook"> Rook</a>. Just pick some instances with which have local SSD storage and Rook will roll out Ceph. Ceph is a distributed storage system. It unites the local storage from multiple servers and makes it available to pods in the form of provisioned volumes.</p><h3>Lost and abandoned resources</h3><p>Bugs in provisioning scripts, misconfigurations and human errors can result in a significant amount of lost resources. This can include pretty much any type of resources used, e.g. instances which are not a part of any cluster, pods and deployments which are not part of any service, public IPs, ELBs, EBS volumes which are not attached, backups, and so on. Dealing with such cases is pretty straightforward, but it might require introducing some organizational rules. Here is an example of what such rules might look like.</p><ol><li>Each resource should have an owner; the owner can be declared using a tag. Some organizations really struggle with establishing the ownership of their cloud resources. My next suggestion might come across as a somewhat draconian rule, but you can introduce a script which deletes all the resources which do not have owners. Usually, if you implement such a rule, you will discover the owners of those resources right away.</li><li>Periodically perform utilization audits. The resources which have seen zero traffic automatically get on the eviction list.</li><li>Contact each of the owners which have items on the eviction list and clarify the business context of that resource. Sometimes, those resources might be a part of a critical system that is still in development.</li></ol><p>From my professional experience, the eviction lists can be as large as 30–40% of the total resources used, and 95% of the items on those lists do not bring any business value. So even though it might sound like a bit work, it is usually quite worth it in the end.</p><h3>Conclusion</h3><p>Kubernetes permits co-allocating the applications on the same nodes, which can dramatically reduce the AWS bill. Yet, incorrectly sized instances, volumes, instances and lost resources can lead to the inflation of the cost of Kubernetes clusters. Simple due diligence of collecting metrics and cloud bills, and some spreadsheet magic can help you to trim the total bill by 30–50%. I’ve shared a simple calculator which can help with finding the best fitting instance size, stranded resource overhead, and a template for doing IOPS and size projections using linear regression. For more complicated use cases of full automation of processes I described this post, you might want to check out<a href="http://www.untab.io/"> Untab</a>. Here is a short video explaining what you can do with it.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ffast.wistia.net%2Fembed%2Fiframe%2Fy99pkwam37&amp;url=https%3A%2F%2Funtab.wistia.com%2Fmedias%2Fy99pkwam37&amp;image=https%3A%2F%2Fembed-ssl.wistia.com%2Fdeliveries%2Fceee2e339ca4f4a1f3a86f0d0bef0312ea1bab11.jpg%3Fimage_crop_resized%3D640x360&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=wistia" width="640" height="360" frameborder="0" scrolling="no"><a href="https://medium.com/media/09f200e0405e4f2b6af30fd50d28526b/href">https://medium.com/media/09f200e0405e4f2b6af30fd50d28526b/href</a></iframe><p><em>Dmytro Dyachuk is a co-founder of </em><a href="http://www.untab.io/"><em>Untab.io</em></a><em> which audits Kubernetes clusters for inefficiencies, estimates the true cost of services and implements team billing for the use of infrastructure.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa510c1964c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>