<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Thiry on Medium]]></title>
        <description><![CDATA[Stories by Daniel Thiry on Medium]]></description>
        <link>https://medium.com/@danielthiry92?source=rss-9c5e70d02da0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*FeDrX4qzqDZql7SbrTgbLA.png</url>
            <title>Stories by Daniel Thiry on Medium</title>
            <link>https://medium.com/@danielthiry92?source=rss-9c5e70d02da0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:40:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielthiry92/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes: Why Boring Clusters Are Better Clusters]]></title>
            <link>https://faun.pub/kubernetes-why-boring-clusters-are-better-clusters-47c885d5712b?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/47c885d5712b</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Wed, 11 Nov 2020 16:53:18 GMT</pubDate>
            <atom:updated>2020-11-11T19:20:19.849Z</atom:updated>
            <content:encoded><![CDATA[<h4>Kubernetes and its massive ecosystem can easily overwhelm people but often, a simple Kubernetes cluster setup is already enough.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*aNi7ukPDKn7s_Z4m" /></figure><p>The Kubernetes and cloud-native ecosystem has become very complex over time. The best illustration for this is probably the <a href="https://landscape.cncf.io/">CNCF Landscape</a> that currently contains about 1,500 cards. This huge amount of different cloud-native projects and tools has led some people to complain about the overwhelming complexity of the ecosystem recently, while others took it with humor and started to create <a href="https://twitter.com/dastbe/status/1303858170155081728/photo/1">memes</a> about it.</p><p>However, there is not only the CNCF Landscape that can lead to some confusion, there are also <a href="https://www.cncf.io/sandbox-projects/">CNCF sandbox</a>, <a href="https://www.cncf.io/projects/">incubating, and graduated projects</a> as well as many additional startups and other vendors developing solutions without being part of the CNCF.</p><p>Therefore, you have almost endless options for tools, add-ons, and configurations and I would expect that there will be even more in the future. All of this can even make it difficult to get easy things done right and to keep focus, which was jokingly summarized in a <a href="https://twitter.com/kelseyhightower/status/1304045435971989505">tweet by Kelsey Hightower</a>:</p><p><em>“By 2025 it will take 64 CPUs and 1TB RAM to deploy a modern “Hello, World!” application.”</em></p><p>Still, I believe that only because there are many things you CAN do, this does not mean that you SHOULD do all of them. Often, it is better and sufficient to keep it simple, so you can focus on the things you actually need. For this, I will describe some advantages that simple Kubernetes cluster setups have compared to more elaborated ones.</p><h3>Easier Management</h3><p>The first advantage of a plain Kubernetes cluster is that it is easier to manage. This starts with the initial setup which is much faster and easier if you do not have to install many things.</p><p>However, it also continues during the normal operations of the cluster: If you have many tools and other things installed on a cluster, there also will be more upgrades for all these tools. In general, one could expect this effort to grow proportionally to the number of tools. Additionally, you also need to monitor the tools continuously. This does not only include monitoring that they are still working but also looking for critical upgrades, fixes, and potential bugs.</p><p>Finally, having a complex setup also means that there are many options for problems. For example, if you install one tool, it may require a certain configuration and then you need to make sure that this configuration change is not conflicting with any other tool and does not lead to any negative side-effect. The same goes for general Kubernetes version upgrades, which can be more complex if not every tool is compatible with it.</p><p><strong>Therefore, you should be careful that the tools that are supposed to help you to manage and control your cluster will not become a burden instead of providing actual help.</strong></p><h3>Easier Scalability</h3><p>One central advantage of Kubernetes is its scalability. Since it was originally made for this, “pure” and simple Kubernetes is already quite scalable. Of course, if you want to max out the full scalability of it, you may need some advanced Kubernetes configuration but for many standard scenarios, this may not be essential.</p><p>Here, it can be more of a problem if you have too many add-ons and a very complex configuration as the scalability is often associated with running many clusters. In this case, you either need to replicate your setup and install all tools on every cluster or you need to configure the tools in a way that they work across clusters. Both solutions can be challenging and may lead to more effort and thus cost than just adding another simple cluster.</p><p><strong>Hyper-complex cluster setups can destroy part of the core value of Kubernetes.</strong></p><h3>Smaller Attack Vector</h3><p>If your system is simpler, it may be more secure because the attack vector is smaller. This is because every tool you install may contain bugs or other vulnerabilities that could be exploited maliciously. In contrast to this, a simple setup with just basic and stable Kubernetes features offers attackers fewer options to bring down or break into your system. (Of course, there are also many security tools and add-ons that actually make your system more secure and are worth installing if needed.)</p><p>While the risk of bugs in the software you use naturally increases with the amount of software, the probability that you miss an upgrade or security patch is also higher if you have to monitor a lot of tools for such upgrades and patches at the same time.</p><p><strong>Running more tools and add-ons increase the attack surface of your system and thus the chance for vulnerabilities.</strong></p><h3>Fewer Human Errors</h3><p>Even if all the software you use is perfectly stable, the likelihood that you misconfigure something or that you miss an important update grows with the complexity of the Kubernetes cluster. It is just much more difficult for individuals to understand many tools, their configurations, and dependencies in-depth, especially if they are new or introduced at the same time.</p><p>Additionally, it is also much harder to trace an error and resolve an issue when there are endless options. Finally, also restoring a more complex system usually takes more time than restoring a simple system, which may be relevant in case of a problem.</p><p><strong>The more things you do, the more likely it is that you make mistakes and do things you do not fully understand.</strong></p><h3>What should you do now?</h3><p>While I try to make a case for simple Kubernetes clusters in this article, that does mean that you should use absolutely plain clusters. There are certainly many tools and even highly advanced configurations that can be very useful. I also think that the CNCF Landscape is a great concept as it helps to get an easy overview of available solutions for different categories and also provides smaller niche providers and startups the opportunity to get visibility so that (hopefully) the best solution wins. (I am even involved in two open-source projects listed on the Landscape: Kubernetes multi-tenancy extension <a href="https://github.com/kiosk-sh/kiosk">kiosk</a> and cloud-native development tool <a href="https://github.com/devspace-cloud/devspace">DevSpace</a>.)</p><p>So, what should you do in practice now?</p><p>I think the first step should be to <strong>evaluate what you really need before you start</strong>. This may even include the question <a href="https://loft.sh/blog/should-a-startup-use-kubernetes/">if you need Kubernetes at all</a>. Only when you know which problem you want to solve, you can start to look for potential solutions and at this point, the CNCF Landscape, <a href="https://loft.sh/blog/">Kubernetes blogs</a>, and many other resources will be very helpful. That means that you should <strong>only look for solutions if you have a problem</strong> and do not start to use tools because they are currently hyped or sound interesting and start to look for a “problem” they solve later on. (Of course, it may make sense to implement solutions for problems that you expect to have but do not have yet (e.g. scalability) if they are rather difficult to implement later.)</p><p>In general, you should <strong>only do things and use tools that you understand</strong>. Here, it often pays off to take some time to evaluate and learn about them first and then implement them correctly from the beginning. During this process, you may even learn that the solution you thought was perfect is in fact not right for your use case and you may avoid potentially costly mistakes.</p><p>Another way to approach the complexity challenge is to relocate the complexity: If you have a <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">multi-tenant Kubernetes system</a>, not everything that one tenant needs is relevant for the others and should be installed or configured for the whole cluster. Instead, you can enable tenants to have independent setups by encapsulating tenants and workloads in <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">virtual clusters</a>. This allows for different and complex configurations of the individual virtual clusters while the underlying physical cluster can remain relatively basic and simple, which combines the described advantages of simplicity with the diverse needs of the tenants.</p><h3>Conclusion</h3><p>The cloud-native ecosystem is certainly one of the most vibrant and exciting tech communities, but it may also easily overwhelm people because so many things happen at the same time and a lot of new tools emerge at an impressive pace. While this progression is generally a good thing, you need to make sure that you don’t get lost.</p><p>Therefore, it is often better and sufficient to start simple and progress with things that you really need so you can run an easily manageable, scalable, secure, and stable system.</p><p>Photo by <a href="https://unsplash.com/@benchaccounting?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Bench Accounting</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/kubernetes-simple-clusters-are-good-clusters/"><em>https://loft.sh</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p>👋 <a href="https://faun.dev/join"><strong>Join FAUN today and receive similar stories each week in your inbox!</strong></a><strong> </strong>️ <strong>Get your weekly dose of the must-read tech stories, news, and tutorials.</strong></p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47c885d5712b" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/kubernetes-why-boring-clusters-are-better-clusters-47c885d5712b">Kubernetes: Why Boring Clusters Are Better Clusters</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Everyone Builds Internal Kubernetes Platforms]]></title>
            <link>https://itnext.io/why-everyone-builds-internal-kubernetes-platforms-284c2cf76226?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/284c2cf76226</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Wed, 14 Oct 2020 14:46:35 GMT</pubDate>
            <atom:updated>2020-10-14T14:51:49.197Z</atom:updated>
            <content:encoded><![CDATA[<h4>Several factors drive more and more companies to develop and adopt internal Kubernetes platforms for their engineers.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Aexz6zIAKXvNNN7b" /></figure><p>Recently, you might have heard about “internal Kubernetes platforms” from many different sources: KubeCon talks, blog articles, or just colleagues and friends. Even if such platforms are not always called internal Kubernetes platforms, solutions that allow engineers to get a standardized and easy Kubernetes access in a cloud environment seem to become more common now.</p><p>In this article, I will describe what drives companies to build and adopt such platforms. For this, I will rely on three publicly described examples of popular tech companies, <a href="https://dev.to/ethanjjackson/why-eventbrite-runs-a-700-node-kube-cluster-just-for-development-2l9p">Eventbrite</a>, <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">Spotify</a>, and <a href="https://youtu.be/4YanEWCAPlk?t=544">Datadog</a> as well as share some of my own experience from working on an <a href="https://loft.sh/">out-of-the-box internal Kubernetes platform solution called Loft</a></p><h3>The Internal Kubernetes Platform</h3><p>I already defined <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform#what-is-an-internal-kubernetes-platform">what an internal Kubernetes platform is</a> in another article.</p><p>In short, an internal Kubernetes platform provides engineers with a direct access to a <a href="https://loft.sh/blog/kubernetes-development-environments-comparison/">Kubernetes dev environment</a> that they can work with during the pre-production stage. Such a platform usually allows the engineers to create <a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer">self-service namespaces</a>, but solutions with whole clusters or <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">virtual clusters (vClusters)</a> are also possible.</p><p>Although the term “platform” may suggest that it is just an infrastructure component, internal Kubernetes platforms often also comprise tools that make the use and management of the platform and the general <a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/">Kubernetes development workflows</a>, especially deployment workflows, easier for developers, even if they have no previous Kubernetes experience.</p><p>Therefore, if you hear about internal Kubernetes platforms, you will also often hear about developer experience and associated tools and not only about technical infrastructure decisions.</p><h3>What drives companies to build such platforms</h3><p>Since it covers so many aspects, <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform">building an internal Kubernetes platform for engineers</a> can be quite challenging. For example, it took Spotify 2 years to make its internal platform generally available. Given this extensive engineering effort and the considerable cloud computing cost to run the platform for large teams, the question comes up why companies undertake such an investment. There clearly must be a lot of value or even pressure to do this:</p><h4>1. Applications become too large for local environments</h4><p>So, let’s start exploring potential reasons for internal Kubernetes platforms by looking at the case of Eventbrite: <a href="https://dev.to/ethanjjackson/q-a-how-eventbrite-prioritizes-developer-productivity-33m0">In an interview</a>, Remy DeWolf, principal engineer on the DevTools team at Eventbrite, explained the reasoning why they moved away from <a href="https://medium.com/swlh/local-cluster-vs-remote-cluster-for-kubernetes-based-development-6efe2d9be202">local development to cloud-based development with Kubernetes</a>. As many companies do, Eventbrite also started out by building a monolith. This monolith became larger and larger over time, which is why it was partly transformed into microservices and new services were also added as microservices.</p><p>Since all of these are running in containers (at least for development), this is not a problem initially. However, as the software grows further, you will eventually reach a point when it is too big to run on local computers with local Kubernetes solutions such as <a href="https://github.com/kubernetes/minikube">Minikube</a> or with <a href="https://www.docker.com/products/docker-desktop">Docker Desktop</a>.</p><p>At this point, you can start to develop workarounds, e.g. just running some necessary parts of an application or running some parts locally and others remotely, to keep your existing local development workflows. This may be a suitable approach but because most companies and systems keep growing, it becomes harder over time to continue with local development. This will lead to reduced developer productivity as the developers are more and more struggling with their local setup and slowed-down laptops.</p><p>For this, you face a trade-off between investing in an internal platform (and the continuous cost for cloud resources) and wasted engineering time. Here, it becomes clear why many companies start with local Kubernetes solutions, but an increasing number now reaches the “tipping point” when it becomes worthwhile to move to a cloud-based Kubernetes environment.</p><p>At Eventbrite, they therefore also made the decision to adopt a cloud environment and built an internal tool called “yak”, which allows engineers to deploy and manage remote containers. So, every engineer now has an own namespace in a shared Kubernetes cluster and can easily run about 50 containers in it.</p><p>Such an evolution did not only happen at Eventbrite, but <a href="https://youtu.be/4YanEWCAPlk?t=544">Datadog</a> with about 800 developers seems to have made a similar decision. I can also support this with the experience from our Loft customers: At some point, their applications just become too complex to run locally, so the move to a cloud-based Kubernetes environment is a necessary step. Interestingly, this is not only true for large organizations but can also happen with fewer engineers if they develop generally resource-intense or complex applications, such as AI/ML software.</p><p><strong>In many cases, applications just become too large to run locally at some point. Then, it often makes sense to move from a local development environment to a Kubernetes dev environment in the cloud.</strong></p><h4>2. Combining autonomy with focus</h4><p>In his <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">KubeCon talk</a>, James Wen, Senior SRE at Spotify, did not explicitly share the reasons why they build an internal Kubernetes platform. However, one can still derive valuable insights that probably were drivers for this decision:</p><p>While he did no mention that local development was not possible for the engineers at Spotify, one can assume this given that they have a very large number of teams (280+), engineers (1,500+), and microservices (1,200+) and that they already had a non-Kubernetes cloud solution in place before.</p><p>Still, I think there are some even more interesting aspects that James Wen mentioned: Spotify uses an “Ops in Teams” model, which is an attractive mix of complete autonomy of the teams and completely centralized operations. The teams build and operate their own services independently, but they can rely on tooling and platforms provided by centralized ops teams.</p><p>I think this is a smart solution as it gives the teams the option to work autonomously and configure Kubernetes as they need it (they can even adopt it at their desired pace), but they do not have to care for and manage everything themselves and instead use a common platform that resolves all “standard” issues that otherwise had to be solved redundantly.</p><p>From an admin (DevTool team) perspective, such a solution is also attractive as the admins do not have to interact with all dev environments individually, which could be a huge problem, e.g. if they had to provision all environments manually. Instead, the admins can focus on providing the cluster and a great platform experience and on supporting the developers using it, e.g. by solving issues, which is also easier as it is possible to share a dev environment in the cloud.</p><p><strong>With an internal Kubernetes platform, developers can work freely and autonomously but at the same time do not have to care about operations tasks that can be better solved centrally. The admin teams benefit as they can focus on the platform and support the other engineers, but do not have to care about every individual environment.</strong></p><h4>3. Streamlining Kubernetes workflows</h4><p>When developers start to work with Kubernetes, it can be quite challenging. Using an internal platform can make the transition to Kubernetes easier as some tasks can be streamlined, and the developers do not have to set up Kubernetes themselves.</p><p>Spotify, for example, developed a comprehensive tutorial and documentation that enabled new developers to get started with the internal platform and thus Kubernetes easily and from Day 1. At Loft, we are providing the developers (users of our customers’ internal Kubernetes platform) with an <a href="https://loft.sh/docs/guides/onboarding">onboarding guide</a> that explains the basic concepts and all standard tasks to the developers.</p><p>This is possible because the work environments are always created in the same standardized way, which would not be possible on local computers due to different hardware, operating systems, system configurations, etc.. This is an advantage that also Eventbrite realized: They were able to reduce the “mean time to recovery”, i.e. the time to restore a clean state, which is necessary if a developer is stuck or made a mistake. So, an internal Kubernetes platform also reduces the risk of experimentation with Kubernetes making it more attractive for developers.</p><p>In general, many developers are interested in Kubernetes and want to learn and work with it, as the most recent <a href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-platforms-wanted5">Stack Overflow Developer Survey</a> found. With an internal platform, they can get in contact with Kubernetes easily and then develop their skills step by step.</p><p>Here, additional Kubernetes tools play an important role. Apparently, Spotify, Eventbrite, and Datadog have developed some kind of tools to make the <a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/">Kubernetes workflows</a> easier for engineers. At least Spotify’s “tugboat” and Eventbrite’s “yak” thereby do not only serve as platform management tools but also support deployment processes leading to a generally better developer experience with Kubernetes. At Loft, we also see that this is critical for our customers, which is why we have developed a direct <a href="https://github.com/loft-sh/loft-devspace-plugin">Loft integration</a> for <a href="https://github.com/devspace-cloud/devspace">DevSpace</a>, so that engineers only need one tool to start their work environment, develop in it, and deploy to it.</p><p><strong>An internal Kubernetes platform that also covers some dev tooling helps to standardize Kubernetes workflows and so enables even engineers without Kubernetes experience to work with it. This facilitates the adoption process of the platform and supports the developers in learning more about Kubernetes.</strong></p><h4>4. Saving cost</h4><p>A further driver for the adoption of an internal Kubernetes platform can be cost savings. Generally, <a href="https://medium.com/faun/kubernetes-cost-savings-by-reducing-the-number-of-clusters-336378680995">it is cheaper to share clusters</a> and a self-service namespace platform allows such cluster sharing. The cost savings are based on more efficient auto-scaling of computing resources and common sharing of resources and some central Kubernetes components (such as API servers). An internal platform can thus lead to cost savings compared to other cloud-based environments, such as individual clusters for developers/teams.</p><p>In addition to these direct cost savings, you can also save indirect cost due to the improved productivity when developers do not have to care about their Kubernetes environment themselves anymore and when they can use optimized workflows. As such, the previously mentioned reasons for internal platforms all can ultimately contribute to <a href="https://loft.sh/blog/reduce-kubernetes-cost/">reduced Kubernetes cost</a>.</p><p>Again, I can support this with the experience of our customers, which realized significant cost savings (also due to cost-saving features such as a <a href="https://loft.sh/docs/self-service/sleep-mode">sleep mode</a> that can be implemented in a common platform). However, the cost was also an important factor for engineering teams to move to their internal offering at Spotify, as James Wen noted in his talk.</p><p><strong>Internal Kubernetes platforms are a very cost-efficient solution to provide engineers with a Kubernetes environment because computing resources can be fully utilized and engineers can work more productively.</strong></p><h3>Important factors to provide an internal Kubernetes platform</h3><p>Now, you might want to start setting up your own internal Kubernetes platform. I already wrote a <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">guide on how to build an internal Kubernetes platform</a> and you certainly should watch James Wen’s talk to learn about Spotify’s experience.</p><p>However, before you start, there are some important factors that are decisive for the success of your platform that I want to highlight here:</p><p><strong>Provide a great developer experience:</strong> Even though your platform is for internal use, you have “customers” and these are the developers in your organization. To ensure that they will adopt and like your solution, you need to provide them a great developer experience. In the best case, that means that they can do everything with just one tool, including creating the environment, developing with it, and finally deploying to it. You will also need to have extensive documentation for the developers and this documentation should be written with the developer experience in mind (this can be challenging for the admins who write it). Overall, it is essential that your platform is the “best choice” for the developers, as James Wen described it.</p><p><strong>Measure engagement and communicate with users:</strong> It is very important to understand your users. Therefore, you have to continuously communicate with the developers, which often happens naturally as the platform providers support the developers in using it. Additionally, you should also implement engagement measures to see if your platform is actually used and to find out what might be missing.</p><p><strong>Measure cost and the performance of the platform:</strong> You should also measure if your platform works reliably and make sure that you could fix problems fast. Having backups and extensive testing in place are very helpful for this. Here, you should keep in mind that, even though the platform is used “only” internally, it becomes crucial for the productivity of your whole engineering department once everyone is working with it. To support and understand the business case for the platform, measuring the cost before and after the introduction of the platform is also important.</p><p><strong>Improve the platform and the dev experience:</strong> As you can see from the previous factors, building an internal platform is a continuous effort. Therefore, you always have to keep updating and improving both the stability and performance of the platform as well as the developer experience with it (, which also includes the documentation).</p><p><strong>Dev Experience is more important than technical details:</strong> For the developers, it matters most that they can efficiently use the platform. They usually do not really care how the underlying tech works exactly or where it runs. (Eventbrite uses EKS, started with one cluster, and now runs several clusters, while Spotify relies on many clusters on GKE and Datadog uses just one self-managed cluster. I believe this choice is not really decisive for the developers.) The team of engineers providing the platform should thus take on a very user-centric perspective. This means that it can be worthwhile to invest even in seemingly minor improvements, which make the life of developers much easier.</p><p><strong>Buying may be cheaper than building:</strong> As with most software, building it yourself is not always the best solution. Instead, you could simply buy an <a href="https://loft.sh/">internal Kubernetes platform solution such as Loft</a>, that is continuously developed by a dedicated team, which may make it better and cheaper for many use cases (especially without very special needs). And even if you decide to build it yourself, you should consider integrating already existing open-source solutions, for example <a href="https://github.com/kiosk-sh/kiosk">kiosk</a> can help you to implement multi-tenancy, and dev tools like <a href="https://github.com/GoogleContainerTools/skaffold">Skaffold</a> or <a href="https://github.com/devspace-cloud/devspace">DevSpace</a> may be used as development and deployment tools.</p><h3>Conclusion</h3><p>Internal Kubernetes platforms have become more common, even if they are not always called like this. With such a platform, developers get direct access to a Kubernetes environment in the cloud.</p><p>Software teams that work with Kubernetes and want to scale their technology should expect to reach a point when local development becomes infeasible or at least inefficient. Then, it will make sense to move to a cloud environment, which should be a Kubernetes environment if your production system is also running on Kubernetes.</p><p>Given the time and effort it takes to build such a platform, they should anticipate early on that they may need an internal development platform and start to continuously evaluate the introduction of such a technology, so they can find the right point in time when the investment will pay off.</p><p>Once the decision for an internal Kubernetes platform has been made, the focus should be on the developer experience to ensure real adoption and engagement because only then it is possible to realize the full benefits of such a platform.</p><p>Photo by <a href="https://unsplash.com/@inietop?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ivan Nieto</a> on <a href="https://unsplash.com/s/photos/platforms?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/why-internal-kubernetes-platform/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=284c2cf76226" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/why-everyone-builds-internal-kubernetes-platforms-284c2cf76226">Why Everyone Builds Internal Kubernetes Platforms</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Virtual Kubernetes Clusters In Production]]></title>
            <link>https://itnext.io/virtual-kubernetes-clusters-in-production-ccb09a0825fd?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccb09a0825fd</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Wed, 07 Oct 2020 14:51:49 GMT</pubDate>
            <atom:updated>2020-10-07T15:42:00.545Z</atom:updated>
            <content:encoded><![CDATA[<p>Virtual Kubernetes clusters cannot only be used for development settings but can also improve production systems in multiple ways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*DpypXXTwqSix39dR" /></figure><p>The idea of <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">virtual Kubernetes clusters (vClusters)</a> is to spin up a fully-functional cluster within another Kubernetes clusters to provide an efficient abstraction and direct Kubernetes access on top of a shared underlying cluster.</p><p>I have already described the <a href="https://loft.sh/blog/virtual-clusters-for-kubernetes-benefits-use-cases/">benefits and use cases use of such virtual clusters for development</a>, and specifically for <a href="https://loft.sh/blog/kubernetes-virtual-clusters-as-development-environments/">cloud-native development</a>, <a href="https://loft.sh/blog/kubernetes-virtual-clusters-for-ci-cd-testing/">CI/CD</a>, and <a href="https://loft.sh/blog/kubernetes-virtual-clusters-for-ai-ml-experiments/">ML/AI experimentation</a>. However, since vClusters are similarly flexible as regular Kubernetes clusters and namespaces, they can also be used in many situations apart from development.</p><p>In this article, I want to describe some of these production use cases that I believe virtual clusters are most valuable for.</p><h3>Replacing clusters by vClusters to save cost</h3><p><a href="https://loft.sh/blog/kubernetes-cost-savings/">Running fewer Kubernetes clusters is generally cheaper</a> because fewer clusters lead to less redundancy and higher utilization. Moreover, they are easier to manage by the cluster admins.</p><p>However, people are still not running everything on just one cluster for good reasons. Often, this reason is that you had to run everything in namespaces, which do not provide sufficient isolation in many situations. Compared to namespaces, vClusters improve the isolation of tenants and so facilitate the implementation of a harder form of <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">Kubernetes multi-tenancy</a>. This allows you to pool some parts of an application in one cluster that are currently separated due to isolation concerns. (Of course, this does not mean that you should run everything on just one cluster.)</p><p>Another reason to separate applications on different clusters is that they require individual configurations of the cluster. While this is not possible with namespaces, virtual clusters can be configured differently and flexibly, e.g. it is even possible to use different Kubernetes versions for the virtual clusters. This again lets you pool applications reducing the number of clusters and thus saves cost.</p><p>For this, a virtual cluster solution might also be interesting for hosting and cloud providers who want to provide their customers flexible Kubernetes environments but can also sell such an experience with vClusters at a lower price as not every customer will need an actual physical cluster but just full Kubernetes access.</p><blockquote><em>I wrote a dedicated article about further options to </em><a href="https://loft.sh/blog/reduce-kubernetes-cost/"><em>reduce your Kubernetes cost</em></a><em>.</em></blockquote><h3>Replacing namespaces by vClusters to improve stability</h3><p>When it is not strictly necessary to separate (parts of) applications on clusters, namespaces are often used as a cheaper and easier approach to run the software. In these situations, the namespaces could be replaced by virtual clusters because they help to improve the security of the system due to their better isolation and provide a more flexible configuration of the different parts of the application.</p><p>Virtual clusters Clusters thus do not only allow you to share clusters more often but also are useful to optimize your production system. Therefore, vClusters often are “the better namespace”.</p><h3>Reaching higher scalability</h3><p>If you run very large clusters with a lot of network traffic and load, you can reach a technical limit of scalability. Naturally, this mostly happens in production use cases with many users or customers.</p><p>In such a situation, you usually need additional clusters to handle the huge amounts of traffic. However, just because, for example, the Kubernetes API server or the etcd of your cluster reach their limit, it does not have to be optimal to run additional clusters. Instead, it could be more efficient to run multiple API servers or etcds to handle the load.</p><p>Since each virtual cluster has its own API server and etcd, it is possible to implement efficient <a href="https://loft.sh/use-cases/kubernetes-cluster-sharding">Kubernetes cluster sharding</a> with vClusters. This is possible as requests to a virtual cluster are generally handled by the virtual cluster itself, which reduces the load on the underlying cluster because only some requests are passed through and handled by it. You can so “spread” the load on different virtual clusters avoiding an early bottleneck issue from the physical cluster.</p><p>Therefore, it is possible to push the technical limits of how much traffic your cluster can handle, which allows you to save costs and optimize your system by only adding clusters if it is really sensible.</p><h3>Providing managed products easier and cheaper</h3><p>While the first production use cases for virtual clusters were rather general, there are also more specific scenarios they are very useful for. One of these is the <a href="https://loft.sh/use-cases/cloud-native-managed-products">provisioning of a managed product to your customers</a>.</p><p>There are generally two main technical approaches to provide your customers with a Software-as-a-Service (SaaS) experience: <a href="https://blog.scaleway.com/saas-multi-tenant-vs-multi-instance-architectures/">Multi-tenancy and multi-instances</a>.</p><p>If you provide your SaaS-product with a multi-tenancy approach, you run one instance of a system and a database that is shared by different customers. Here, vClusters can again help you by providing better tenant isolation and by allowing more scalability to improve the efficiency of your system.</p><p>With the multi-instance approach, every customer gets an own instance of your system and database. Virtual clusters allow you to do this quite easily as you can replicate your system in virtual clusters. So, every customer gets a dedicated instance of your software but all instances are still running on a single cluster, which reduces the management effort for your system. Since no new cluster has to be started for each customer, the setup of a new instance is also faster and the whole system runs more efficiently as the underlying computing resources can be shared and thus do not run idly.</p><p>Overall, vClusters are therefore useful to provide SaaS products to your customers no matter which architectural approach you want to use.</p><h3>Running interactive demos fast and easy</h3><p>A last production use case for virtual Kubernetes clusters is <a href="https://loft.sh/use-cases/live-demos">spinning up interactive demos for your product</a>. Even though demos are not really production systems, their use and functionality are closer to classical production scenarios than to development, which is why I want to mention them here.</p><p>If you sell an on-premise product, your customers (especially in B2B) usually want to see a demo before they decide to buy it or even before they set it up themselves on their infrastructure. Such demo versions of your product can either be used by salespeople or by the customers themselves.</p><p>In any case, it usually makes sense if the demo always starts in the same clean state and is available anytime, i.e. does not have to be provisioned by admins first. These requirements make them a very good use case for virtual clusters: vClusters can be started within just a few seconds from scratch instead of several minutes that “real” clusters need to be started in cloud environments. They are also always available if the underlying cluster is running.</p><p>Additionally, as there is only one underlying cluster needed for many vClusters, several salespeople (or customers) can start a demo independently on-demand so that there is not only one demo application that always has to be reset after each demonstration. When the demo is finished, the whole demo environment, i.e. the vCluster, can thus also be deleted, a process that even can be automized with an <a href="https://loft.sh/docs/self-service/sleep-mode">automatic delete setting</a>. Alternatively, it is also possible to keep the environment if it shall be used again to continue at the same stage.</p><p>In sum, running demos in virtual clusters is thus cheap due to the shared infrastructure and the automatic cleanup and easy due to the independent environments that are available on-demand. This makes the use of demos during your sales activities more attractive, which will ultimately help to drive sales and revenues.</p><h3>Conclusion</h3><p>Virtual clusters have many benefits that make them useful not only for development scenarios but also for various production settings. They can be used to improve your system technically by providing more isolation and thus stability or by increasing the limits to the scalability of your clusters. Additionally, virtual clusters are efficient in reducing your cost if you can replace whole clusters with vClusters. They can also support you in solving the practical problems of how to efficiently provide managed products and product demos to your customers.</p><p>Overall and given the abundance of Kubernetes applications, I believe there are also many more use cases for virtual clusters during production that even have not been explored yet.</p><p>Photo by <a href="https://www.pexels.com/@wolfgang-1002140?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Wolfgang</a> from <a href="https://www.pexels.com/photo/photo-of-people-watching-a-concert-2747449/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/virtual-kubernetes-clusters-in-production/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccb09a0825fd" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/virtual-kubernetes-clusters-in-production-ccb09a0825fd">Virtual Kubernetes Clusters In Production</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Cost Savings By Reducing The Number Of Clusters]]></title>
            <link>https://faun.pub/kubernetes-cost-savings-by-reducing-the-number-of-clusters-336378680995?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/336378680995</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[cloud-cost-management]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Mon, 05 Oct 2020 15:47:46 GMT</pubDate>
            <atom:updated>2020-10-15T15:04:09.131Z</atom:updated>
            <content:encoded><![CDATA[<h4>Reducing the number of clusters can save a lot of cost without affecting the stability of your system.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*dCnR0U-R0p1xdWvr" /></figure><p>When you are using Kubernetes at a larger scale and at different stages (development, testing, production), you will sooner or later face the question of how many clusters you should run. Finding the right answer to this question is as not easy as having many clusters has advantages and disadvantages compared to running only one or a few clusters, as discussed <a href="https://learnk8s.io/how-many-clusters">in this article</a>.</p><p>However, your answer to this question also determines how much you will have to pay for your Kubernetes system. In general, it is cheaper to run only a few clusters, which is why I will explain in this article how you can reduce the number of clusters and thus save Kubernetes cost without negatively impacting your system.</p><h3>Advantages of fewer clusters</h3><p><strong>No Redundancies:</strong> Running many clusters means that you also have many API servers, etcds and other parts of the control plane. This leads to a lot of redundancy and inefficiency as most of these components will be underutilized. However, if you only run a few clusters, all applications and users of the same cluster can share a control plane, which will drive utilization up and cost down significantly.</p><p><strong>No Cluster Management Fees:</strong> Some public cloud providers, including AWS and Google Cloud, charge their users a flat cluster management fee for every cluster. Naturally, if you have fewer clusters, you will pay less for the cluster management. The cluster management fee is particularly important for situations with many small clusters as the relative cost (about $70 per month per cluster) can represent more than 50% of the total cost here.</p><p><strong>Efficient Administration:</strong> In general, it is easier to manage and supervise a system with a limited number of clusters because you can get a much better overview of the system and do not have to repeat the same processes many times (e.g. updating every cluster). For this, reducing the number of clusters also reduces the admin effort for the clusters and will so ultimately lead to additional cost savings.</p><blockquote><em>Cluster sharing, which leads to fewer clusters, is also the basis for an </em><a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/"><em>internal Kubernetes platform</em></a><em> that provides the engineers with </em><a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/"><em>self-service namespaces</em></a><em> improving the </em><a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/"><em>Kubernetes development workflow</em></a><em>. This illustrates that a reduction of the number of clusters can also be a positive side-effect of improvements on other parts of your Kubernetes system.</em></blockquote><h3>Disadvantages of fewer clusters</h3><p><strong>Single Point of Failure:</strong> If more applications are running or more users are working on the same cluster, all of them will be impacted at the same time when the cluster crashes. This means that the stability of the cluster will be more important as the cluster becomes a “single” point of failure. Of course, “single” in single point of failure does not necessarily mean that there is only one cluster but the same goes analogously for a low number of clusters.</p><p><strong>Multi-Tenancy:</strong> If a cluster is shared by several users and applications, you need to implement an efficient <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">Kubernetes multi-tenancy</a>. This can be quite challenging because it does not only comprise the isolation of tenants but also user management and enforcement of limits. Especially hard multi-tenancy, i.e. a system that securely isolates tenants that are not trusted, is hard to establish with Kubernetes as some components are always shared within one cluster.</p><p><strong>Technical Limitations:</strong> While a single Kubernetes cluster is quite scalable, its scalability will be limited at some point. For example, <a href="https://kubernetes.io/docs/setup/best-practices/cluster-large/">Kubernetes v1.19 supports clusters with up to 5,000 nodes, 150,000 total pods, and 300,000 total containers</a>. These limits are of course pretty high, but you might already encounter scalability problems before reaching the limits, e.g. due to networking capacity. Therefore, it is also technically infeasible to run everything on just a single cluster if you reach a very large scale.</p><h3>Trade-Off: Efficiency vs. Stability</h3><p>Simply put, in determining the right number of clusters for your use case, you face a trade-off between (cost-)efficiency and stability of your system. Therefore, it is clear that your goal should not be to end up with just a single cluster and try to run everything there. To efficiently save Kubernetes cost, you should rather use the lowest reasonably possible number of clusters that still ensures the stability of your system. And this “right” number heavily depends on the size of your system, the number of users, and the nature of your software.</p><p>However, there are some approaches that can help you to drive down the number of required clusters and so will save you cost:</p><blockquote><em>Reducing the number of clusters is just one of several methods to save costs. In my </em><a href="https://loft.sh/blog/reduce-kubernetes-cost/"><em>guide to reducing Kubernetes cost</em></a><em>, I also explain some other approaches that make your Kubernetes system more cost-efficient.</em></blockquote><h3>How to Safely Reduce The Number of Required Clusters</h3><h4>Make Differentiated Decisions</h4><p>Since there is no one-size-fits-all answer to the questions of how many clusters you should run, you need to make an individual decision for your use case. However, it is not enough to make this decision once, but you should rather repeat it for every part of your application and for every group of users.</p><p>This means that you should not just adopt a dogmatic policy of “every engineer and every application get an own cluster” or “we are doing everything in one cluster”. You really need to evaluate if it makes sense in the situation at hand.</p><p>For example, there may be some applications, especially very critical applications running in production, that should get their own cluster so that these applications cannot be impacted by others. Other applications may share a cluster either because they are less important or are only used for developing and testing, which makes them less critical. However, also critical applications may run on a shared cluster, e.g. if they heavily depend on each other and would not work anyway if one of them failed.</p><p>You also need to make the cluster decision for your engineers: Usually, not every engineer needs an own cluster but namespaces are often enough, so you might provide them with <a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/">self-service namespaces</a> that run on just one cluster. Still, some engineers may have special requirements and need to work directly on the Kubernetes configuration. Since this is not possible with simple namespaces, these engineers should get individual clusters or alternatively virtual clusters, which I will describe next.</p><blockquote><em>If you want to learn more about the decision about clusters for development purposes, take a look at this </em><a href="https://loft.sh/blog/individual_kubernetes_clusters_vs-_shared_kubernetes_clusters_for_development/"><em>comparison of individual clusters and shared clusters for development</em></a><em>.</em></blockquote><h4>Use Virtual Clusters</h4><p>Another way to decrease the number of clusters is to use <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">virtual clusters (vClusters)</a>. Virtual clusters are very similar to “real” clusters but are virtualized and several virtual clusters can thus run on one physical cluster. Virtual clusters have some advantages that allow you to replace physical clusters with vClusters:</p><p>Virtual clusters solve the multi-tenancy issue as they provide better isolation of tenants than namespaces, which is why it is easier to implement hard multi-tenancy with vClusters. Additionally, the tenants can configure their vCluster freely and independently from others, so that <a href="https://loft.sh/blog/kubernetes-virtual-clusters-as-development-environments/">vClusters are very good development environments</a> even for engineers that need to configure Kubernetes or use a specific version of it. Finally, you can use <a href="https://loft.sh/use-cases/kubernetes-cluster-sharding">vClusters for cluster sharding</a>, so it is, for example, possible to handle more requests and thus push the feasible technical limits of Kubernetes.</p><p>Since virtual clusters are still running on a single physical cluster, this physical cluster remains a single point of failure. However, as the configurations and additional installations can be done on the vCluster level, the underlying cluster can be rather simple with only basic components, which makes this cluster less error-prone and thus more stable.</p><h4>Implement Effective Multi-Tenancy</h4><p>One of the reasons why some companies prefer to use many clusters is that they are not sure how to implement multi-tenancy and how much effort this is. If more organizations get the <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">best practices for Kubernetes multi-tenancy</a> right, they could reduce the number of clusters, especially for non-production use cases.</p><p>Fortunately, Kubernetes has some helpful features for multi-tenancy, such as Role-based access control (RBAC) and Resource Quotas. In general, setting up a good multi-tenant system is mostly a one-time effort and requires good user management and strictly enforced user limits. To get this, some companies such as <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">Spotify</a> started to build their own <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">internal Kubernetes platforms</a>, but it is also possible to buy out-of-the-box solutions for this, such as <a href="https://loft.sh/">Loft</a>.</p><h3>Conclusion</h3><p>Reducing the number of clusters can lead to significant cost savings. However, just using a single cluster is often not the right solution as this could negatively impact the stability of your system. Finding the right number of clusters for your use case is therefore not an easy task because there is no general rule of how many clusters are optimal.</p><p>It is rather a question you have to answer for every application and every group of engineers individually by assessing their situation and needs. However, it is possible to reduce the minimum number of necessary clusters by replacing some clusters with virtual clusters and by implementing efficient multi-tenancy.</p><p>This will allow you to improve the cost-efficiency of your Kubernetes system without a negative impact on its stability.</p><p>Photo by <a href="https://www.pexels.com/@karolina-grabowska?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Karolina Grabowska</a> from <a href="https://www.pexels.com/photo/composition-of-calculator-with-paper-money-and-notebook-with-pen-4386341/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/kubernetes-cost-savings/"><em>https://loft.sh</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p>👋 <a href="https://faun.dev/join"><strong>Join FAUN today and receive similar stories each week in your inbox!</strong></a><strong> </strong>️ <strong>Get your weekly dose of the must-read tech stories, news, and tutorials.</strong></p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=336378680995" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/kubernetes-cost-savings-by-reducing-the-number-of-clusters-336378680995">Kubernetes Cost Savings By Reducing The Number Of Clusters</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Sandboxes — Easy Development in a Realistic Environment]]></title>
            <link>https://codeburst.io/kubernetes-sandboxes-easy-development-in-a-realistic-environment-77efcba45d98?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/77efcba45d98</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Wed, 30 Sep 2020 15:37:41 GMT</pubDate>
            <atom:updated>2020-10-01T20:48:49.111Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Sandboxes — Easy Development in a Realistic Environment</h3><p>A Kubernetes sandbox is a great complement to cloud-native tools to complete an optimal developer experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*SzhYmZGKM2IxYd-K" /></figure><p>Using <a href="https://en.wikipedia.org/wiki/Sandbox_(software_development)">sandbox environments</a> is very common for software developers because it allows them to work, test, and experiment in an environment that is isolated from the production system but still provides a realistic experience. As a consequence of this early validation, the software quality improves and the number of bugs decreases. Because of this, “it-works-on-my-machine”-problems can be ruled out.</p><p>Now that the production environment is often Kubernetes, the engineers should therefore also start to work with Kubernetes. However, to establish efficient <a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/">development workflows with Kubernetes</a>, you need special development tools and you should also use a Kubernetes sandbox environment, which will be the focus of this article.</p><h3>Great Cloud-Native Dev Tools Already Exist</h3><p>The good news is that there are many open-source tools that solve the problem of how to interact with Kubernetes if you want to develop software for it. Examples for (mature) tools in this area include <a href="https://github.com/GoogleContainerTools/skaffold">Skaffold</a>, <a href="https://github.com/devspace-cloud/devspace">DevSpace</a>, <a href="https://github.com/tilt-dev/tilt">Tilt</a>, <a href="https://github.com/telepresenceio/telepresence">Telepresence</a>, and <a href="https://github.com/okteto/okteto">Okteto</a>. They are all open-source and have a slightly different approaches to solve a common problem: How to streamline the development and deployment processes with Kubernetes.</p><p>While it is worth looking at and comparing all of these Kubernetes tools, each of them will eventually allow developers to develop faster and more efficiently with Kubernetes without having to configure everything themselves.</p><blockquote><em>For a comparison, take a look at the </em><a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps#kubernetes-development-tools"><em>Kubernetes development tool section</em></a><em> of my article about </em><a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/"><em>Kubernetes development workflows</em></a><em>.</em></blockquote><p>No matter which tool you choose, you still face an essential challenge that none of the cloud-native tools addresses: how can developers easily get a Kubernetes sandbox to work in?</p><h3>Local Clusters as Kubernetes sandbox?</h3><p>One approach to get a Kubernetes sandbox environment is to use local clusters with tools such as <a href="https://github.com/kubernetes-sigs/kind">kind</a>, <a href="https://github.com/kubernetes/minikube">Minikube</a>, or <a href="https://github.com/rancher/k3s">k3s</a>. These are open-source tools that allow engineers to run Kubernetes on their local computer. This easy and cost-free setup makes local clusters a good solution to get started fast. The engineers do not even need to have a cloud platform access because everything happens locally.</p><p>However, this also means that the developers become admins of their cluster. Even if a local cluster obviously does not need to run securely (as only one developer is working with it), the effort associated with this should not be underestimated; especially if the developers have no admin and Kubernetes experience. Additionally, there are some differences between the different local Kubernetes solutions and between the local setups of the engineers (hardware, operating system, configurations), which makes it difficult to standardize the admin tasks completely. This is the reason why some developers struggle to work with local Kubernetes.</p><p>Another problem with such a solution is that local environments are not completely realistic copies of the production environment, which is an important element of good dev sandboxes. On the one hand, this is due to the special Kubernetes solutions used locally, and on the other hand this is due to missing scalability and other cloud features (especially of public cloud providers). The missing scalability of computing resources on local computers also means that the computers of the engineers are often under heavy load and reach their limits quite quickly, resulting in annoyingly loud fans (best case) or system crashes (worst case).</p><p>Therefore, local clusters are not really developer-friendly (for non-experts) and as such not perfectly suited as Kubernetes sandboxes. However, as I wrote in my article about the <a href="https://loft.sh/blog/kubernetes-development-environments-comparison/">comparison of Kubernetes development environments</a>, I still think that local Kubernetes solutions have a valid right to exist as they are very useful in some scenarios. For example, they are great to get a first experience with Kubernetes if you want to learn it or to try a new tool or setup without having to start a cluster in the cloud. Still, as Kubernetes sandbox, they are only suitable for more experienced engineers who are not working on very computing-intense applications.</p><h3>Shared Development Clusters as Kubernetes Sandboxes</h3><p>Another approach to provide Kubernetes sandboxes to engineers is to use shared development clusters. This approach is easier to scale and can be used quite easily by inexperienced engineers, which makes it more suitable for larger teams and teams with different technical backgrounds.</p><p>With shared clusters as sandboxes, the developers do not have to set up and manage their work environments anymore as this can be centrally done by admins who are more experienced with Kubernetes. This allows the developers to get started very fast and to get a real Kubernetes experience immediately.</p><p>Since the sandboxes are running in a scalable cloud environment, they have almost infinite computing resources available meaning that they can be used even for very complex applications. This allows engineers to work form weak laptops without long waiting times or the fear of crashing the environment. It is also possible to switch between devices for development as they only need to be connected to the sandbox and do not require much local configuration (which can be particularly useful in flexible work settings such as when working from home with a different computer than at the office).</p><p>Another advantage of cloud-based Kubernetes sandboxes is that they provide new opportunities for collaboration and sharing. The fact that Kubernetes is declarative and all sandboxes are very similar makes it easy to replicate a scenario and problem, so colleagues can help each other to solve a problem together. It is also possible to share access to the same environment, which allows for collaborative debugging. The easy replicability can also be useful if engineers have to repeat tests and experiments multiple times such as is often is the case for machine learning applications.</p><p>For this, I believe that shared development clusters are a perfect solution for Kubernetes sandboxes that can be used for any setup, team, and use case.</p><blockquote><em>If you want to learn more about the two different options for Kubernetes sandboxes, take a look at this article comparing </em><a href="https://loft.sh/blog/local-cluster-vs-remote-cluster-for-kubernetes-based-development/"><em>local clusters and remote cluster for Kubernetes-based development</em></a><em>.</em></blockquote><h3>How to Provide Cloud-Based Kubernetes Sandboxes to Engineers</h3><p>While they are a powerful solution, providing cloud-based sandboxes to engineers requires overcoming some technical challenges. These challenges are mostly about how to get <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">Kubernetes multi-tenancy</a> right:</p><p>At first, you need to ensure that the cluster is shared securely, i.e. that the tenants of your clusters are isolated. This is the basis to get actual sandboxes that engineers can work in without impacting their colleagues.</p><p>Secondly, you need to implement a user management system to determine who has the right to create and use the sandboxes and to assign limits to their usage. The enforcement of such limits on a per-user level is another challenge as you need to prevent one user to consume all available computing resources leaving nothing for the others. (This can be solved with the open-source Kubernetes multi-tenancy extension <a href="https://github.com/kiosk-sh/kiosk">kiosk</a>.)</p><p>Then, you need to provide the engineers with the sandbox environments. The most efficient way to do this is to allow them to create the sandboxes themselves, e.g. with <a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/">self-service-namespaces</a>, as this process can be easily integrated into the development workflows.</p><p>While namespaces are enough for many development use cases, you may alternatively use <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">Kubernetes virtual Clusters (vClusters)</a> that isolate users even better and provide them with more flexibility in terms of Kubernetes configuration. For this, using <a href="https://loft.sh/blog/kubernetes-virtual-clusters-as-development-environments/">virtual Clusters as development environment</a>s can even be used by Kubernetes experts who need access to more Kubernetes features such as CRDs, or who want to experiment with Kubernetes configuration.</p><p>Finally, you need to enable developers but also admins to work with and manage the sandboxes. This requires some form of user-interface, such as a GUI or a CLI. With this interface, engineers should not only be able to create the sandboxes, but they also need to manage their limits and admins need to manage the users. Since this interface is critical for the adoption and acceptance of Kubernetes in your teams, it should be very user-friendly and easy to understand.</p><p>Overall, you are essentially going to <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">build an internal Kubernetes platform</a> that companies such as <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">Spotify</a> and <a href="https://www.youtube.com/watch?v=4YanEWCAPlk&amp;feature=youtu.be&amp;t=544">Datadog</a> have already built for their engineers.</p><p>Since this requires some effort, not every company is willing to invest. Solutions such as <a href="https://loft.sh/">Loft</a> provide all the previously mentioned features out-of-the-box. With this solution, you only need to <a href="https://loft.sh/docs/quickstart">install the Loft Kubernetes extension</a> to your cluster and you can then let your engineers create their Kubernetes sandboxes (that run on your clusters) themselves.</p><h3>Conclusion</h3><p>Together with cloud-native tools, Kubernetes development sandboxes are a great way to enable engineers to work with Kubernetes directly and safely. Using such sandboxes can increase the quality and stability of your software as the target environment Kubernetes is already included throughout the development phase.</p><p>There are two options for Kubernetes sandboxes: They can either run on local clusters or on shared clusters in the cloud. While local clusters are a great solution for more experienced engineers or developers who want to learn more about Kubernetes, sandboxes in shared clusters are also appropriate for “average” engineers who want to keep their focus on software development and simply use Kubernetes without going into its details.</p><p>To provide developers with this experience, you need to offer them a simple way to create and manage Kubernetes sandboxes; which often results in an internal Kubernetes platform that some larger organizations have already built but that are now also available off-the-shelf.</p><p>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/kubernetes-sandbox/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77efcba45d98" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/kubernetes-sandboxes-easy-development-in-a-realistic-environment-77efcba45d98">Kubernetes Sandboxes — Easy Development in a Realistic Environment</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Reduce Your Kubernetes Cost]]></title>
            <link>https://itnext.io/how-to-reduce-your-kubernetes-cost-5b5486b10143?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b5486b10143</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-cost-management]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Mon, 28 Sep 2020 15:10:12 GMT</pubDate>
            <atom:updated>2020-10-15T15:08:48.159Z</atom:updated>
            <content:encoded><![CDATA[<p>Reducing the cost of running Kubernetes at a larger scale and with many users will become a priority for many companies soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*45oyqTpyA4yS3dqS" /></figure><p>Running Kubernetes can be very expensive, especially when it is done inefficiently. This is often the case when companies have just started to roll out Kubernetes in their organizations as then the same configuration and setup are often used that worked well for initial test projects or small applications. Such an initial unoptimized setup is probably normal. However, companies should start to also consider cost soon because this can save a lot of money and prevents the spread of bad practices.</p><p>In this post, I will describe some ways to control and reduce your Kubernetes cost that can be applied for very different Kubernetes use cases, from development and CI/CD to production.</p><h3>Cost Monitoring</h3><p>The first step to efficiently reduce your Kubernetes cost is to monitor these costs. This will not only help you to determine how much you saved eventually but also gives you an overview of what the most important cost drivers are.</p><p>Basic monitoring for your computing cost is provided by the public cloud providers as they are the ones ultimately charging you. Besides the total cost, you can also see in their billing overviews what exactly is running and what drives your cost, e.g. what proportion of your cost is attributed to computing, storage, and network traffic.</p><p>However, the overview of the cloud providers can only give you a basic understanding that is only limitedly helpful for multi-tenant Kubernetes clusters and of course is not available in private clouds. Therefore, it often makes sense to use additional tools to measure your Kubernetes usage and costs. Some useful tools in this area are <a href="https://github.com/prometheus/prometheus">Prometheus</a>, <a href="https://kubecost.com/">Kubecost</a>, and <a href="https://www.replex.io/">Replex</a>.</p><p>Once you have set up your Kubernetes cost monitoring and identified areas of improvement, you can start with the actual cost optimizations.</p><h3>Resource Limits</h3><p>A good first cost-saving step is to set limits for resource usage. Efficient resource limits ensure that no application or user of your Kubernetes system consumes excessive computing resources. Therefore, they prevent you from unpleasant surprises in form of sudden cost increases.</p><p>Especially if you give developers direct Kubernetes access, e.g. in form of a <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">self-service Kubernetes platform</a>, limits are critical as they enforce a fair sharing of available resources, which keeps the total cluster size smaller. Without limits, one user could consume all resources leaving others unable to work, which will make them require more computing resources in total.</p><p>This goes not only for engineers in a <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">Kubernetes multi-tenancy</a> environment but also for applications running on the same cluster. In any case, it is of course important to set the right limits: If they are too low, engineers and applications cannot work properly and if they are too high, they are mostly useless. Here, the previously mentioned Kubernetes monitoring tools can help you to determine the right limits for the different use cases.</p><p>To implement resource limits, you can configure them Kubernetes-natively with <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/">Resource Quotas</a> and <a href="https://kubernetes.io/docs/concepts/policy/limit-range/">Limit Ranges</a>.</p><h3>Autoscaling</h3><p>Now that you have cost monitoring in place and prevented unexpected cost explosions, you can focus on saving costs. The best way to do this is to only pay for what you actually need. For this, it is important to get the size of your clusters, <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">virtual clusters (vClusters)</a>, namespaces, etc. right.</p><p>Again, monitoring tools are helpful for this but manually observing and adjusting computing resources is not a very fast and flexible solution. To be able to respond to short-term fluctuations, you should therefore enable Kubernetes autoscaling.</p><p>Here, there are two types of autoscaling: Horizontal and vertical autoscaling. In short, horizontal autoscaling means adding and removing pods if the load is above or below a predefined threshold. With vertical autoscaling, the size of the individual pods is adjusted.</p><p>Both types of autoscaling are helpful to adjust your available computing resources automatically to your actual needs. However, this process is not always optimal because it may be too “conservative” or does not work for all use cases, e.g. there might be something running (requiring computing resources and thus not automatically downscaled) that is not used anymore. Therefore, some other measures should be implemented that will further drive your Kubernetes cost down.</p><h3>Discounted Computing Resources</h3><p>Another way to save computing cost is to use resources that have a discounted price. Of course, this only works in public clouds that offer such resources. However, all major cloud providers have some options for this:</p><p><a href="https://aws.amazon.com/ec2/spot/">AWS Spot Instances</a>, <a href="https://cloud.google.com/compute/docs/instances/preemptible">GCP Preemptible VMs</a>, and <a href="https://azure.microsoft.com/pricing/spot/">Azure Spot VMs</a> provide heavily discounted computing resources that are “excess capacity” of the cloud providers. The providers sell this computing capacity cheaper as it would otherwise be unused. However, this also means that prices for these computing resources fluctuate depending on the demand and that your instance and application may be stopped if the price exceeds your limit or if there is no capacity available anymore. For this, spot instances are mostly an option for applications and workloads that are not needed permanently, e.g. if you want to run a computing-intense experiment. However, if you can use this type of discounted computing, you can save a lot of costs — AWS claims that you may get “up to a 90% discount”.</p><p>If spot instances and preemptible VMs are not an option for your application, for example, because you always need to run it without any disruption, you may also get a discount for committing to use the resources permanently for a specific time period. By committing to a one- or three-year usage period, you can get a significant discount — 40%-60% according to AWS. Such a long-term discount is available for <a href="https://aws.amazon.com/ec2/pricing/reserved-instances/?nc1=h_ls">AWS</a> and <a href="https://azure.microsoft.com/pricing/reserved-vm-instances/">Azure as “Reserved Instances”</a>, and for <a href="https://cloud.google.com/docs/cuds">Google as “Committed Use Discount”</a>. This type of discount can make sense if you have a predictable and continuous need for computing resources over a longer period of time.</p><p>Overall, it depends on your type of application and computing needs, which type of discount is applicable. Here, it is also possible to use both types for different parts of your application or for different use cases. Therefore, even if the pricing of all public cloud providers is quite complicated, it can certainly pay off to take a closer look at the available discounts there so you get the same computing resources just at a lower price.</p><h3>Sleep Mode</h3><p>There are quite a few scenarios in which clusters, vClusters, and namespaces continue to run and create costs, although they are not needed anymore. These use cases often (but not only) happen when Kubernetes is used during development, testing, or CI/CD by engineers.</p><p>For example, if a developer is using a <a href="https://loft.sh/blog/kubernetes-development-environments-comparison/">Kubernetes development environment</a> in the cloud, they only need this environment during their work hours. If you assume that they work 40 hours per week with a Kubernetes environment (not accounting for meetings or holidays) and the environment is running all the time, you can save more than 75% (a week has 168 hours) by shutting off the environment when it is not needed.</p><p>The developers could of course shut down their environment themselves when they are done, but this is a process that is easily forgotten or ignored. It thus makes sense to automize it with a “ <a href="https://loft.sh/docs/self-service/sleep-mode">sleep mode</a> “, i.e. an automized process that scales down unused namespaces and virtual clusters. This ensures that the state of the environment is stored and that the environment can “wake up” again very fast and automatically when it is needed again so that the engineer’s workflow is not interrupted.</p><p>It is possible to implement such a sleep mode with scripts or by using tools that have a sleep mode inbuilt, such as <a href="https://loft.sh/">Loft</a>.</p><blockquote><em>For a more detailed description and calculation of the cost-saving potential of a sleep mode, you should look at my article about </em><a href="https://loft.sh/blog/how-to-save-more-than-2-3-of-engineers-kubernetes-cost#inefficiency-2-unused-resources"><em>saving Kubernetes cost for engineers</em></a><em>.</em></blockquote><h3>Cleanup</h3><p>Related to a sleep mode to scale down computing resources that are temporarily not needed, you should also clean up your Kubernetes system from time to time. Especially, if you allow engineers to <a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/">create namespaces on demand</a> or if you use Kubernetes for CI/CD, you may have a lot of unused namespaces or even clusters that still cost you money.</p><p>Even if you have a sleep mode in place that scales down the computing resources, the sleep mode is only intended for temporarily unused computing resources and thus preserves the state, including storage and configuration. However, especially to run CI/CD pipelines or tests, it is usually not necessary to preserve the state of the Kubernetes environment, so it is better to also delete it.</p><p>Again, this can be done with individual scripts or by using Loft, which provides such an <a href="https://loft.sh/docs/self-service/sleep-mode">auto-delete feature</a> out-of-the-box. As an additional positive side-effect, deleting some environments makes it easier for admins to oversee the system, which can also be seen as an indirect cost-saving factor.</p><h3>Cluster Sharing</h3><p>A final Kubernetes cost-saving approach is to reduce the number of clusters. This generally saves cost because the control plane and the computing resources can be shared by several users or applications in a <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">Kubernetes multi-tenancy setup</a>. If you use a public cloud that charges a cluster management fee, such as AWS or Google Cloud, you can also save this fee. Finally, managing many clusters is a complex task, so limiting the number of clusters also relieves the admins.</p><p>Especially during the pre-production stage, many companies have too many clusters, sometimes even for every individual developer. This is clearly inefficient as a whole cluster is rarely needed during development and even if developers want to experiment with cluster-wide configurations, they could use a <a href="https://loft.sh/blog/kubernetes-virtual-clusters-as-development-environments/">virtual cluster as dev environment</a> instead. Therefore, relying on a multi-tenant <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">internal Kubernetes platform</a> for all non-production use cases can save a lot of cost by eliminating unnecessary redundancies.</p><p>However, also during production, reducing the number of clusters can make sense with the same underlying idea of improving resource utilization, reducing redundancies, and avoiding cluster management fees. Of course, this does not mean that you should only have one cluster. You should rather generally evaluate what user groups and applications can share a cluster and when dedicated clusters make sense.</p><blockquote><em>For a more detailed discussion on this topic, also read my post about </em><a href="https://medium.com/faun/kubernetes-cost-savings-by-reducing-the-number-of-clusters-336378680995"><em>Kubernetes cost savings by reducing the number of clusters</em></a><em>.</em></blockquote><h3>Conclusion</h3><p>I believe that the cost of running Kubernetes at a larger scale and with many users will become an issue for more and more companies soon as some initial inefficiencies can be removed relatively easily but can still lead to huge cost savings.</p><p>A first step to manage your Kubernetes cost is to get an overview and to start monitoring the cost. Then, you should implement limits to prevent excessive computing resource consumption, which makes your cost more predictable. To reduce your cost, finding the right size of your clusters, virtual clusters, and namespaces is important and autoscaling can be very helpful for this. You should also take a look at the discount options of public cloud providers if they are an option for you. An automatic sleep mode and cleanup for unused Kubernetes namespaces and vClusters are further measures to eliminate idle resources. Finally, you should consider reducing the number of clusters because this improves resource utilization and reduces unnecessary redundancies, especially at the pre-production stage.</p><p>If you implement all these measures, your Kubernetes system is quite cost-optimized and thus much cheaper than any initial best guess setup. Then, you should be ready to tackle your next challenges in using Kubernetes without having to fear its cost.</p><p>Photo by <a href="https://unsplash.com/@alexandermils?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Alexander Mils</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/reduce-kubernetes-cost/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b5486b10143" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-to-reduce-your-kubernetes-cost-5b5486b10143">How To Reduce Your Kubernetes Cost</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Development Environments — A Comparison]]></title>
            <link>https://itnext.io/kubernetes-development-environments-a-comparison-4a2276254c98?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a2276254c98</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Wed, 23 Sep 2020 15:32:28 GMT</pubDate>
            <atom:updated>2020-09-23T15:46:16.238Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Development Environments — A Comparison</h3><p>There are several ways of providing developers with a Kubernetes work environment and all have advantages and disadvantages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C-HfP4M0mNE3I9ZMoPeVvQ.jpeg" /></figure><p>Kubernetes has <a href="https://loft.sh/blog/is-kubernetes-still-just-an-ops-topic/">left the state when it was mostly an ops technology behind</a> and now is also very relevant for many developers. As I wrote in my blog post about <a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/">the Kubernetes workflow</a>, the first step for every developer who starts to directly work with Kubernetes is to set up/get access to a Kubernetes development environment.</p><p>A Kubernetes work environment is not only the first step but also a basic requirement to be able to work with Kubernetes at all. Still, the access to such an environment is often a problem: A <a href="https://tanzu.s3.us-east-2.amazonaws.com/campaigns/pdfs/VMware_State_Of_Kubernetes_2020_eBook.pdf">VMware study</a> even found out that “access to infrastructure is the biggest impediment to developer productivity”. For this, Kubernetes development environments should have a high priority for every team that plans to use the technology.</p><p>In this article, I will describe and compare four different Kubernetes development environments and explain when to use which dev environment.</p><ol><li>Local Kubernetes Clusters</li><li>Individual Cloud-Based Clusters</li><li>Self-Service Namespaces</li><li>Self-Service Virtual Clusters</li></ol><h3>6 Evaluation Criteria For Dev Environments</h3><p>To make the different Kubernetes dev environments comparable, it makes sense to first define the evaluation criteria used. I will rate every environment using the following criteria:</p><p><strong>Developer Experience:</strong> How easy is it for developers to get started with and to use the environment? This includes factors such as the speed of setup, the ease of use, and the required knowledge by the developers.</p><p><strong>Admin Experience:</strong> How easy is it for admins to manage the environments and to manage the system? Here, I will consider the complexity of the system, the effort to manage it, and to add additional users.</p><p><strong>Flexibility/Realism:</strong> How realistic is the dev environment compared to the production environment and how flexible is it for different use cases? A good development environment should be very similar to the production environment to avoid “it works on my machine”-problems and it should also be freely configurable and useable for many different use cases (e.g. coding, testing,…).</p><p><strong>Scalability:</strong> How scalable is the environment itself and how scalable is the approach if many users are using the system? Especially for complex applications, a lot of computing resources are needed, so the dev environment should be able to provide them. Additionally, the general approach to provide this kind of environment to developers should be feasible also for large teams.</p><p><strong>Isolation/Stability:</strong> How are users isolated from each other and how vulnerable is the system? Developers should be able to work in parallel without interfering with each other and the system they use should be stable and secure to reduce inefficient outages.</p><p><strong>Cost:</strong> How expensive is this approach? This category should be quite self-explanatory but still is an important factor when choosing the right development environment for your team.</p><p>Now that the evaluation criteria are clear, we can start with the comparison of the Kubernetes development environments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-Se0ptM0FHjsB-phhIAow.png" /></figure><h3>1. Local Kubernetes Clusters</h3><p>Local Kubernetes clusters are clusters that are running on the individual computer of the developer. There are many tools that provide such an environment, such as <a href="https://github.com/kubernetes/minikube">Minikube</a>, <a href="https://github.com/ubuntu/microk8s">microk8s</a>, <a href="https://github.com/rancher/k3s">k3s</a>, or <a href="https://github.com/kubernetes-sigs/kind">kind</a>. While they are not all the same, their use as a development environment is quite comparable.</p><h4>Developer Experience: -</h4><p>Local development environments need to be set up by the developers themselves as they run on their computers. This can be quite challenging, especially as the local setup is always slightly different (different hardware, different operating systems, different configurations, etc.) which makes it harder to provide a very simple setup guide. After the setup is completed, the developers are also responsible to care and manage their environments themselves, which they are often not used to if they do not have previous Kubernetes experience.</p><p>Therefore, the general developer experience is relatively bad for developers (at least without Kubernetes knowledge).</p><h4>Admin Experience: o</h4><p>Admins are not involved in the setup and the management of local Kubernetes clusters. That means that they have no effort here. However, they also do not know if the developers are able to work with their clusters and are generally excluded from the setup and management of the clusters. Still, the admins probably have to support the developers in case of problems and questions.</p><p>Overall, the admin experience is mediocre because the admins do not face their typical challenges but rather have to educate and support the developers individually.</p><h4>Flexibility/Realism: o</h4><p>On the one hand, local clusters are always somewhat different from “real” clusters in a cloud environment. They are often pared-down Kubernetes versions that lack some features which cannot be replicated locally (and are often not needed locally). Exemplarily, this can be seen in the name “k3s”, which is an allusion to the original Kubernetes’ “k8s”. On the other hand, the engineers are able to do whatever they want with their local cluster, so they can also flexibly configure it.</p><p>In sum, local clusters score high in terms of flexible configuration but low on realism as they do not have all Kubernetes features and so cannot be used for any use case.</p><h4>Scalability: --</h4><p>Since local clusters can only access the computing resources that are available on the engineer’s computer, they reach their limit for complex applications relatively fast. Also, the approach to let engineers create their local clusters themselves is not really scalable as the same process has to be repeated for every engineer with little options for automation.</p><p>Scalability is thus a clear weakness of local Kubernetes clusters.</p><h4>Isolation/Stability: ++</h4><p>Every developer has a separate environment that is completely disconnected from any other environment. In theory, they can even be used without an internet connection. For this, the isolation of local clusters is perfect. This disconnection also ensures that only the individual environment can fail and never all environments at the same time, which minimizes the vulnerability of this approach to provide developers with a Kubernetes environment.</p><p>Isolation and security are definitely a strength of local clusters.</p><h4>Cost: ++</h4><p>Local Kubernetes clusters do not require sometimes costly cloud computing resources but only use the locally available computing resources. The different local Kubernetes solutions are all open-source and free to use.</p><p>Using the local Kubernetes cluster for development does not have any direct cost, so it is the cheapest solution possible.</p><h3>2. Individual Cloud-Based Clusters</h3><p>Individual clusters running in the cloud are the second type of Kubernetes dev environment. They can either be created by the admins who then give an individual access to the developers or the developers are enabled to create them themselves if they have their own account for the cloud provider.</p><h4>Developer Experience: o</h4><p>The developer experience can be very different and depends on the way the individual clusters are created: If developers have direct access to the cloud, e.g. with an elaborated Identity and Access Management (IAM), they can create their work environment on-demand and the setup is quite easy (especially in public clouds) as it is always the same. Still, they must do this themselves and might need some help for the management of the cluster.</p><p>If admins create the clusters and distribute the access to the developers, the dev experience can become quite bad. While the management of the cluster is now cared for, the admins become a bottleneck. Here, you will face the previously mentioned problem of waiting for central IT to provide the dev environments.</p><p>Overall, in the best case, the dev experience is sufficient if developers have direct cloud access.</p><h4>Admin Experience: --</h4><p>No matter in which way the developers get their cluster, the admin experience is always quite bad. If every developer has an own cloud account, the admins will have a hard time getting an overview of the whole system (What is still used? Who is using what?). In this case, they also have to support the developers in managing the clusters. Since the number of clusters grows proportionally with the number of engineers, the effort also grows with the team size.</p><p>In the case of a central creation and distribution of the clusters by the admins, the administrators will also have a lot of effort. They will have to answer all requests by developers for clusters and configuration changes and have to be always available for them because they are critical for the developers’ performance. In general, many clusters lead to more management effort for admins.</p><p>The individual cloud-based cluster approach is a bad solution from the admin’s perspective and necessarily leads to a lot of work on their side that can even become impossible for them to handle.</p><h4>Flexibility/Realism: ++</h4><p>Since the production systems usually also run in Kubernetes in the cloud, having such an environment for development is perfectly realistic. The individual environments can also be freely configured, so they exactly match the needs of the developers or are identical to the production system’s settings.</p><p>Individual cloud-based clusters are the best solution to get a highly realistic development environment.</p><h4>Scalability: o</h4><p>In terms of scalability, it is important that the clusters are running in a cloud environment, which allows you to scale them up almost infinitely. Still, the scalability criterion also includes the scalability of the general approach for larger teams and here, individual clusters can reach a limit as the admin effort grows with the team size.</p><p>Scalability in terms of computing resources is not a problem for individual clusters in the cloud but rolling out such a system in larger organizations will often be infeasible.</p><h4>Isolation/Stability: +</h4><p>Having an isolation of developers on a cluster level is very secure. If you are using a public cloud, the isolation of developers is almost the same as the isolation of different companies, which of course is a high priority for the cloud providers.</p><p>100% stability and isolation will probably never be reached in the cloud, but they are as good as possible with individual clusters.</p><h4>Cost: --</h4><p>Running many clusters is very expensive. This is due to several factors: At first, you will have a lot of redundancy because every cluster will have its own control plane. Secondly, having oversized or unused clusters is almost inevitable with this approach as either developers are responsible for right-sizing and shutting down clusters or admins have to do it centrally but they do not have the oversight and knowledge what is still used.</p><p>Additionally, dev environments are also only used if developers are working, so many clusters will probably run idle at night, during holidays and weekends. Finally, public cloud providers charge a cluster management fee that needs to be paid for every cluster, i.e. for every developer in this case.</p><p>Individual clusters for every engineer in the cloud are a very expensive approach to provide Kubernetes development environments.</p><h3>3. Self-Service Namespaces</h3><p>Instead of giving every developer a whole cluster, it is also possible to just give them Kubernetes namespaces. Again, these can either be created centrally by the admins or developers are provided with a tool to create <a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/">self-service namespaces</a> on-demand. Providing them centrally comes with many disadvantages I already managed for individual clusters, so I will focus on the self-service namespace approach here.</p><h4>Developer Experience: +</h4><p>As engineers can create the namespaces themselves, they are independent of the admins and never have to wait to get a Kubernetes development environment. At the same time, the namespaces are running on a cluster that is managed by admins, so the developers do not have to care for the management of the environment. Namespaces as constructs within clusters will often be enough for simpler development work, so developers will be able to do most standard tasks and are only limited in some situations, e.g. when they need CRDs or want to install Helm charts that use RBAC.</p><p>Therefore, the developer experience with self-service namespaces is very good for “standard” development tasks and developers without special Kubernetes configuration requirements.</p><h4>Admin Experience: +</h4><p>Admins need to <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">set up an internal, self-service Kubernetes platform</a> once, which may take some time if they want to build it from scratch, which companies such as <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">Spotify</a> did. Alternatively, it is also possible to buy solutions that add this self-service namespace feature to any cluster, such as <a href="https://loft.sh/features/self-service-kubernetes-namespaces">Loft</a>. In any case, the admins can focus on other tasks such as the security and stability of the underlying cluster once the system is properly set up. Additionally, it is relatively easy to get an overview of the whole system as everything is running in just one cluster.</p><p>Self-service namespaces are an admin-friendly solution that requires some initial setup effort.</p><h4>Flexibility/Realism: -</h4><p>Since namespaces are running on a shared Kubernetes cluster, it is not possible to configure everything individually by the developers. For example, all engineers have to use the same Kubernetes version and cannot modify cluster-wide resources. Still, namespaces are running in a cloud environment that resembles the production environment, which at least makes namespaces a relatively realistic work environment.</p><p>Overall, namespaces may restrict the flexibility of developers in some situations but are generally not an unrealistic dev environment.</p><h4>Scalability: ++</h4><p>The scalability of a self-service namespace system is very good in both aspects: It is possible to scale up the resources of the namespaces because they are running in the cloud (it is also possible to limit developers to prevent excessive usage, of course). At the same time, it is also no problem to add additional users to the system, especially if it provides a <a href="https://loft.sh/features/kubernetes-auth-sso">Single-Sign-On option</a>.</p><p>Namespaces are an efficient way of providing many developers with a Kubernetes environment that can be flexibly scaled up or down.</p><h4>Isolation/Stability: -</h4><p>Namespaces are a native solution for <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/">Kubernetes multi-tenancy</a> but the isolation is not perfect and rather a form of soft multi-tenancy. However, since the tenants (developers) are trusted, this is not necessarily a problem for development environments. Additionally, namespaces share the same underlying cluster, which means that all namespaces fail if the cluster is down, so the stability of the cluster is essential.</p><p>Namespaces are a Kubernetes-native isolation solution, but it is certainly not perfect. However, if the underlying cluster is running solidly, namespaces are still a good solution for trusted engineers within an organization.</p><h4>Cost: o</h4><p>To get the self-service experience, you might need to buy a self-service namespace software. Additionally, namespaces running in a cloud environment are not free as they also require cloud computing resources. However, the underlying cluster and its resources can be shared by many developers, which drives utilization up and prevents unnecessary redundancies. It is also easier to get a central overview of what is running idle, so these namespaces can be shut down. This process can even be automized by a <a href="https://loft.sh/docs/self-service/sleep-mode">sleep mode</a>.</p><p>Overall, namespaces are a very cost-efficient approach to provide developers with Kubernetes access.</p><h3>4. Self-Service Virtual Clusters</h3><p><a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">Virtual clusters (vClusters)</a> are a solution that lets you create Kubernetes clusters within a Kubernetes cluster. Like namespaces, virtual clusters run on a single physical cluster and can be created on-demand by developers if they have access to a <a href="https://loft.sh/features/virtual-kubernetes-clusters">vCluster platform</a>.</p><h4>Developer Experience: ++</h4><p>The developer experience with virtual clusters is similar to namespaces. Developers can easily create them on-demand and are so independent of central IT but still do not have to manage the underlying cluster themselves. At the same time, vClusters feel like “real” clusters for developers, so they will usually not be limited by them at all.</p><p>Therefore, the dev experience with vClusters is similarly good as with namespaces but even gives the developers more freedom to do and configure what they want.</p><h4>Admin Experience: ++</h4><p>Considering the admin experience, it is again very similar for self-service namespaces and vClusters. After the initial setup, the management effort for admins is very limited, so they can focus on other tasks again. However, compared to namespaces, vClusters isolate users better and so make it less likely that developers can get the underlying cluster to crash. Additionally, most of the Kubernetes configuration and installation can happen in the vCluster so that the underlying cluster can be very simple and just has to provide the basic features, which makes the admins’ job even easier.</p><p>A self-service vCluster platform thus also provides a very smooth admin experience once it has been set up properly.</p><h4>Flexibility/Realism: +</h4><p>Virtual Clusters run in the cloud, which makes them quite realistic dev environments, especially because the developers can configure them individually to fit their needs. However, vClusters are not exactly the same as real clusters, so the realism is not as perfect as with individual clusters.</p><p>Overall, vClusters can be flexibly configured to meet the requirements of different use cases. Since they are a virtual construct, they are still some minor differences to physical clusters.</p><h4>Scalability: ++</h4><p>The scalability of vClusters is as good as for namespaces. vClusters can have different and basically endless computing resources in the cloud. The self-service provisioning on a platform that runs on a single cluster also makes it possible to use vClusters with many engineers.</p><p>A self-service vCluster solution will fulfill all needs in terms of scalability for development environments.</p><h4>Isolation/Stability: o</h4><p>The isolation of virtual clusters is better than the isolation on a namespace-level, but vClusters are still a form of Kubernetes multi-tenancy and as such, the vClusters share a common physical cluster. A benefit of virtual clusters is that the underlying cluster can be very basic, which makes it easier to get it stable.</p><p>Overall, the isolation of vClusters is decent and the stability of the whole system can be quite good. However, a lot of the stability is determined by the stability of the underlying cluster.</p><h4>Cost: o</h4><p>A virtual cluster platform is not free because it requires cloud computing resources and a software for the platform. In this category, vClusters are again very similar to namespaces: The cluster sharing improves the utilization and makes it easier to get an overview and to shut down unused virtual clusters, which can again even be automized by a sleep mode.</p><p>A virtual cluster platform is as cost-efficient as a namespace platform, but all cloud-based solutions will necessarily not be completely free.</p><h3>When to use which dev environment</h3><p>After having described the four different types of Kubernetes development environments, the question remains which environment is right for your situation.</p><p>From my experience, many companies and engineers start with local dev environments. The fact that they are free and run on local computers reduces the initial hurdle as no complicated budget approvals are needed. Local environments are also a good solution for hobby developers and small applications but also for Kubernetes experts who know how to handle and set up these environments.</p><p>As organizations progress on <a href="https://loft.sh/blog/the-journey-of-adopting-cloud-native-development/">their cloud-native journey</a>, they want to roll out Kubernetes to more developers who do not have any experience with Kubernetes. These organizations often start with the “obvious” solution: Just give every developer an own cluster. After some time, they then often realize that this approach is very expensive and becomes more complex with a growing number of developers working with it. For this, the individual cloud-based cluster solution is often just a temporary solution unless the number of developers is relatively low and the cost so does not matter too much.</p><p>To avoid the high cost and the management effort for larger teams, many organizations want to provide developers with either namespaces or virtual clusters (virtual clusters are relatively new, so namespaces are still more common). However, as these companies have realized that scalability of the approach matters a lot, they want to do this in an automized fashion and therefore either start developing own internal Kubernetes platforms <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">as Spotify did</a> or just buy existing solutions, such as <a href="https://loft.sh/">Loft</a>. Thereby, it depends on the complexity of the application and the expertise and requirements of the developers if namespaces are sufficient or if virtual clusters are a better solution.</p><h3>Conclusion</h3><p>As more companies want their developers to work with Kubernetes, also more developers need to have access to a Kubernetes work environment. For this, there are several options that all have their strengths and weaknesses.</p><p>While local development clusters are a good and cheap starting point, they are often not the right solution for inexperienced developers or larger organizations.</p><p>These organizations then turn to the “obvious” solution of individual cloud-based clusters, which are unbeatable in terms of flexibility and realism but are also hard to manage for admins and can become very expensive.</p><p>Finally, shared clusters, which are the basis for either self-service namespaces or virtual clusters, are a solution that combines cost-efficiency with a good developer and admin experience. Although these solutions are not free and require some initial setup effort, they are a long-term solution even for larger companies.</p><p>Photo by <a href="https://unsplash.com/@rawfilm?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">RawFilm</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/kubernetes-development-environments-comparison/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a2276254c98" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/kubernetes-development-environments-a-comparison-4a2276254c98">Kubernetes Development Environments — A Comparison</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Multi-Tenancy — A Best Practices Guide]]></title>
            <link>https://faun.pub/kubernetes-multi-tenancy-a-best-practices-guide-88e37ef2b709?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/88e37ef2b709</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[multitenancy]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Mon, 21 Sep 2020 13:46:10 GMT</pubDate>
            <atom:updated>2020-09-21T20:06:03.714Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Multi-Tenancy — A Best Practices Guide</h3><p>A guide on how to tackle typical Kubernetes multi-tenancy challenges by implementing some best practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*f6fNZYp9diK0uuma" /></figure><p>Kubernetes multi-tenancy is a topic that more and more organizations are interested in as their Kubernetes usage spreads out. However, since Kubernetes is not a multi-tenant system per se, getting multi-tenancy right comes with some challenges.</p><p>In this article, I will describe these challenges and how to overcome them as well as some useful tools for Kubernetes multi-tenancy. Before, I will explain what Kubernetes multi-tenancy actually means, what the differences between soft and hard multi-tenancy are, and why this is such a relevant topic right now.</p><h3>What Is Kubernetes Multi-Tenancy</h3><p>Multi-tenancy in Kubernetes means that a cluster and its control plane are shared by multiple users, workloads, or applications. It is the opposite of single-tenancy, where only one user uses a whole Kubernetes cluster.</p><p>There are different types of multi-tenancy, ranging from soft multi-tenancy to hard multi-tenancy.</p><h4>Soft Multi-Tenancy</h4><p>Soft multi-tenancy is a form of multi-tenancy that does not have a strict isolation of the different users, workloads, or applications. It is thus an appropriate solution for trusted and known tenants, i.e. tenants that will not voluntarily abuse each other such as engineers within the same organization. The isolation between users is rather focused on preventing accidents and cannot prevent attacks on other tenants.</p><p>In terms of Kubernetes, soft multi-tenancy is typically associated with simple Kubernetes namespaces that the individual tenants are working in.</p><h4>Hard Multi-Tenancy</h4><p>Hard multi-tenancy enforces stricter isolation of tenants and so also prevents negative consequences of malicious behavior from other tenants. In addition to trusted tenants, it can thus also be used for tenants you do not trust such as many unconnected users and people from different organizations.</p><p>To implement hard multi-tenancy in Kubernetes, you need a more advanced configuration for namespaces or <a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/">virtual Clusters (vClusters)</a>.</p><h4>Soft vs. Hard Multi-Tenancy</h4><p>While it sounds like soft and hard multi-tenancy are distinct opposites, they are rather endpoints of a range of different implementations. Some even argue that perfect hard multi-tenancy in Kubernetes cannot exist because Kubernetes is designed in a way that there is only one central control plane that always has to be shared by tenants introducing a potential vulnerability.</p><p>In general, one can still say that hard multi-tenancy is generally “better” (as it can be used for trusted and non-trusted tenants), while soft multi-tenancy is easier to implement.</p><h3>Limitations of Multi-Tenant Kubernetes</h3><p>By implementing multi-tenancy, you automatically introduce “limitations” to your Kubernetes cluster because the tenants will be technically restricted compared to users of a single-tenant cluster and/or the tenants must consider the other tenants. An example of the limitations of a namespace-based multi-tenancy is that the tenants are not able to use CRDs, install Helm charts that use RBAC, or change cluster-wide settings such as the Kubernetes version.</p><p>Using virtual clusters mitigates these limitations as tenants have access to cluster-wide settings and resources in their vClusters. For this, virtual clusters feel much more like “real” clusters compared to namespaces, which is why vCluster-based multi-tenancy is a form of multi-tenancy that is relatively close to single-tenancy.</p><p>Nevertheless, introducing any form of multi-tenancy adds a layer of complexity to your system and comes with some restrictions for the tenants.</p><h3>Reasons For Kubernetes Multi-Tenancy</h3><p>You now might ask why Kubernetes multi-tenancy is relevant and why you could not just use many single-tenant clusters instead.</p><p>Theoretically, it is possible to use many single-tenant clusters instead of a shared multi-tenant cluster. Some companies actually do this at the moment, e.g. to provide developers access to Kubernetes. However, such a solution is very inefficient and thus expensive, especially at a larger scale.</p><p>This is something that many organizations realize now when their <a href="https://loft.sh/blog/why-adopting-kubernetes-is-not-the-solution/">adoption of Kubernetes spreads within their organization</a> and more engineers get Kubernetes access: While it is very simple and not so costly to create one cluster per tenant/user during the initial experimentation phase with Kubernetes, it becomes a huge problem if (almost) every developer in an organization gets an own cluster during the later stages of <a href="https://loft.sh/blog/the-journey-of-adopting-cloud-native-development/">the cloud-native adoption journey</a>. Suddenly, you end up with dozens, hundreds, or even thousands of clusters that all cost money (the cluster management fees of public cloud providers also start to matter a lot now) and need to be efficiently managed, which is far from trivial.</p><p>At this stage, the benefits of having a multi-tenant system outweigh the additional complications of it. It is simply much easier to manage a Kubernetes system with one or just a few clusters and sharing a cluster is also more efficient in terms of resource utilization as redundancies can be reduced.</p><blockquote><em>For more information about this topic, also take a look at my article about </em><a href="https://loft.sh/blog/individual_kubernetes_clusters_vs-_shared_kubernetes_clusters_for_development/"><em>a comparison of individual clusters and shared clusters</em></a><em>.</em></blockquote><h3>Kubernetes Multi-Tenancy Implementation Challenges</h3><p>There are three major challenges that you will face when implementing Kubernetes multi-tenancy:</p><h4>User Management</h4><p><strong>Challenge:</strong> Most companies already have a user management system for their engineers in place somewhere. This could be in GitHub, Microsoft, Google, or any other service. Since you do not want to manage all users twice or even more often, you need to enable Single-Sign-On (SSO) for your Kubernetes system.</p><p><strong>Solution:</strong> With the CNCF sandbox project <a href="https://github.com/dexidp/dex">dex</a>, you can provide the tenants with an SSO-option. Dex is an OpenID Connect and OAuth2 provider that supports various identity providers including LDAP and SAML and thus can be used with <a href="https://github.com/dexidp/dex#connectors">many user management systems</a>. Therefore, dex is a very good option to solve the user management challenge for your multi-tenancy system with Kubernetes.</p><h4>Fair Resource Sharing</h4><p><strong>Challenge:</strong> Since all tenants share the same underlying resources, i.e. network and computing resources, the second challenge is to ensure that the available resources are shared fairly between the tenants. This is important because you do not want to have one tenant consume all or an excessive amount of resources (accidentally or voluntarily) leaving the others unable to work. For this, you need to make sure that every tenant has appropriate usage limits.</p><p><strong>Solution:</strong> The resource consumption of tenants can be limited Kubernetes-natively with Resource Quotas. Since users must specify CPU and memory limits if quotas are enabled for these resources, it makes sense to also set smart defaults via LimitRanges.</p><h4>Isolation</h4><p><strong>Challenge:</strong> The third challenge is to isolate the different tenants from each other. This prevents tenants from interfering with each other. As described above, the degree of isolation is determining if you have <a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/#soft-multi-tenancy">soft or hard multi-tenancy</a> in place and if the system should only be used by trusted tenants or if it is also secured against voluntary attacks.</p><p><strong>Solution:</strong> Kubernetes namespaces serve as basic isolation for tenants. Alternatively, <a href="https://loft.sh/features/virtual-kubernetes-clusters">vClusters provide even more isolation than namespaces</a>, while they also give tenants more flexibility. Another factor that should be considered for isolating tenants is network traffic: Network policies should be configured in a way that by default all traffic is denied and only traffic within the same namespace, internet traffic for containers, and requests to the DNS are allowed. This makes it much harder for tenants to attack or interfere with each other.</p><h3>Available Multi-Tenancy Solutions</h3><p>Some useful solutions exist already that help you to implement multi-tenancy with Kubernetes. Besides the previously mentioned <a href="https://github.com/dexidp/dex">dex</a>, <a href="https://github.com/kiosk-sh/kiosk">kiosk</a> and <a href="https://loft.sh/features/kubernetes-multi-tenancy">loft</a> are worth mentioning here.</p><h4>kiosk</h4><p><a href="https://github.com/kiosk-sh/kiosk">kiosk</a> is an open-source multi-tenancy extension for Kubernetes. It is designed as a lightweight, pluggable, and customizable solution for any Kubernetes cluster and solves some of the multi-tenancy challenges in a simple way. This includes account separation, resource consumption limitation on a user level, and namespace templates for secure tenant isolation and self-service namespace initialization. While it does not provide an in-built user management, kiosk is a very good building block to develop an own multi-tenancy system.</p><p>Even though kiosk has been released only a few months ago, it is already included in the <a href="https://aws.github.io/aws-eks-best-practices/multitenancy">EKS Best Practices Guide for multi-tenancy by AWS</a>. You can find a <a href="https://aws.amazon.com/de/blogs/containers/set-up-soft-multi-tenancy-with-kiosk-on-amazon-elastic-kubernetes-service/">detailed guide to set up kiosk with EKS here</a>.</p><h4>Loft</h4><p><a href="https://loft.sh/">Loft</a> is based on kiosk and provides a comprehensive solution for a multi-tenancy platform. Loft can be installed into any Kubernetes cluster and then lets tenants create namespaces and virtual Clusters on-demand. It cares for the user management (including SSO) and user isolation and lets the cluster admins define usage limits, so all previously mentioned multi-tenancy problems are resolved. Loft provides some additional features such as a sleep mode that leads to reduced cloud computing costs by shutting down unused namespaces and vClusters.</p><p>Loft is a commercial offering with a free tier and is also included in the <a href="https://aws.github.io/aws-eks-best-practices/multitenancy">EKS Best Practices Guide for multi-tenancy</a>. While it was originally focused on multi-tenant development use cases, it can also be used for production use cases such as <a href="https://loft.sh/use-cases/kubernetes-cluster-sharding">cluster sharding</a> or to <a href="https://loft.sh/use-cases/cloud-native-managed-products">run multiple instances of a product in a shared cluster</a></p><h3>Conclusion</h3><p>Kubernetes multi-tenancy is one of the challenges that many organizations face at the moment as their Kubernetes adoption progresses and single-tenant solutions become increasingly infeasible.</p><p>When implementing multi-tenancy with Kubernetes, you need to decide if you need hard multi-tenancy or if soft multi-tenancy is enough. In any case, you need to solve three major problems: How to manage the users/tenants, how to limit their resource usage, and how to isolate them from each other. Here, several tools such as dex, kiosk, and loft can help you so you get multi-tenancy with Kubernetes right more easily.</p><p>Photo by <a href="https://unsplash.com/@jesuskiteque?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jesus Kiteque</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/kubernetes-multi-tenancy-best-practices-guide/"><em>https://loft.sh</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/0*Piks8Tu6xUYpF4DU" /></figure><p><strong>Join FAUN!</strong></p><p><strong>Subscribe to </strong><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumprebanner"><strong>FAUN topics</strong></a><strong> and get your weekly dose of the must-read tech stories, news, and tutorials </strong>🗞️</p><p><strong>Follow us on </strong><a href="https://twitter.com/joinfaun"><strong>Twitter</strong></a><strong> </strong>🐦<strong> and </strong><a href="https://www.facebook.com/faun.dev/"><strong>Facebook</strong></a><strong> </strong>👥<strong> and </strong><a href="https://instagram.com/fauncommunity/"><strong>Instagram</strong></a><strong> </strong>📷 <strong>and join our </strong><a href="https://www.facebook.com/groups/364904580892967/"><strong>Facebook</strong></a><strong> and </strong><a href="https://www.linkedin.com/company/faundev"><strong>Linkedin</strong></a><strong> Groups </strong>💬</p><figure><a href="https://www.faun.dev/join?utm_source=medium.com/faun&amp;utm_medium=medium&amp;utm_campaign=faunmediumbanner"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_cT0_laE4iPcqW1qrbstAg.gif" /></a></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88e37ef2b709" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/kubernetes-multi-tenancy-a-best-practices-guide-88e37ef2b709">Kubernetes Multi-Tenancy — A Best Practices Guide</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Service Kubernetes Namespaces Are A Game-Changer]]></title>
            <link>https://itnext.io/self-service-kubernetes-namespaces-are-a-game-changer-e356691528ab?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e356691528ab</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[namespaces]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 13:46:54 GMT</pubDate>
            <atom:updated>2020-09-16T13:52:41.810Z</atom:updated>
            <content:encoded><![CDATA[<p>Self-service for namespaces will lead to a broader use of Kubernetes by engineers in any organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*4q6BouLwdaejsMb8" /></figure><p>Many companies have adopted Kubernetes recently. However, most of them still do not realize its full potential because the actual Kubernetes usage in these organizations is very limited. Since Kubernetes has evolved dramatically, it is now <a href="https://loft.sh/blog/is-kubernetes-still-just-an-ops-topic/">not only a technology for operations anymore</a> but also non-ops engineers can work with it. For this, <a href="https://loft.sh/blog/why-adopting-kubernetes-is-not-the-solution/">Kubernetes adoption should not end here</a>, it rather just starts.</p><p>So, it now often makes sense to also include engineers in the Kubernetes adoption process and, as the latest <a href="https://insights.stackoverflow.com/survey/2020#technology-most-loved-dreaded-and-wanted-platforms-wanted5">Stack Overflow Developer Developer Survey</a> shows, engineers appreciate it as they both want to work with Kubernetes if they are currently not using it and also like working with after they have started.</p><p>An easy way to have more developers start working with Kubernetes is to provide them with self-service namespaces. In this article, I will describe what self-service namespaces are, why they are a game-changer for Kubernetes adoption, and how to get them.</p><h3>What are self-service Kubernetes namespaces?</h3><p>Self-service namespaces are Kubernetes namespaces that can be created by the users on-demand without the need to be an admin of the cluster the namespaces are running on. As such, self-service namespaces are running on a shared Kubernetes cluster and are created in a simple and standardized way by their users, e.g. via a UI of a <a href="https://loft.sh/features/self-service-kubernetes-namespaces">self-service namespace platform</a>.</p><p>Self-service namespaces so provide engineers an easy and always available access to Kubernetes, which is a huge advantage compared to possible alternatives: While local Kubernetes solutions such as <a href="https://github.com/kubernetes/minikube">minikube</a> always have to be set up and configured by the engineers themselves and so are never readily available, giving each the developer an own cluster in the cloud is very expensive. Individual clusters ars also often unfeasible due to restricted cloud access rights and unnecessary because simple namespaces are enough for most standard use cases.</p><p>Providing namespaces in a self-service fashion compared to letting admins create them manually is therefore a decisive feature as only this eliminates the <a href="https://tanzu.s3.us-east-2.amazonaws.com/campaigns/pdfs/VMware_State_Of_Kubernetes_2020_eBook.pdf">most important dev productivity impediment of “waiting for central IT to provide access to infrastructure”</a>.</p><p>Overall, self-service namespaces are therefore the easiest way of providing engineers with a readily available Kubernetes access.</p><h3>Benefits of self-service namespaces</h3><p>Providing a self-service namespace solution to users has advantages for both sides, the users (engineers) themselves and the admins.</p><h4>Benefits for namespace users:</h4><p><strong>1. Velocity:</strong> Self-service namespaces are always available and can be created fast and easy whenever they are needed by the users. This makes them a very useful solution for a variety of engineering tasks, ranging from <a href="https://loft.sh/use-cases/cloud-native-development">cloud-native development</a>, to <a href="https://loft.sh/use-cases/ci-cd-pipelines">CI/CD pipelines</a> and <a href="https://loft.sh/use-cases/ai-machine-learning-experiments">AI/ML experiments</a>.</p><p><strong>2. Independence:</strong> The self-service aspect enables engineers to work independently from admins as they do not have to wait for the admins to create a work environment before they can start.</p><p><strong>3. Easier Experimentation:</strong> This independence of the users also makes it possible to experiment more with namespaces as the namespaces can now be thrown away and recreated by the users themselves. The users so do not have to fear to break something and can eventually treat namespaces as “cattle” and not as “pet”.</p><blockquote><em>The independence of users can be further enhanced and the fear of breaking can be reduced by using self-service </em><a href="https://loft.sh/blog/introduction-into-virtual-clusters-in-kubernetes/"><em>virtual Clusters (vClusters)</em></a><em>, which are very similar to namespaces but provide harder isolation and give engineers even more freedom to configure Kubernetes.</em></blockquote><h4>Benefits for cluster admins:</h4><p><strong>1. Better Stability:</strong> Since all namespaces are created in the same standardized way by the users, there is little room for human error in the whole namespace creation process, which improves the stability of the underlying Kubernetes cluster. Additionally, the users are encapsulated in namespaces, which prevents that they interfere with each other.</p><p><strong>2. Less Effort and Pressure:</strong> The gained independence by the users reduces the pressure on the cluster admins. They do not have to be always available to provide work environments for the engineers and are so no longer a bottleneck for the <a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/">engineering workflows with Kubernetes</a>. Admins only have to set up the self-service platform in the first place and then ensure that it is available and that the underlying cluster is running.</p><p><strong>3. Focus on Stability and Security:</strong> As the admins are not needed in the creation process of every namespace anymore, they can now focus more on the stability and security of the underlying cluster.</p><blockquote><em>Providing </em><a href="https://loft.sh/features/virtual-kubernetes-clusters"><em>self-service virtual Clusters</em></a><em> can again improve the system, as vClusters provide an even stronger form of multi-tenancy and user isolation. They also allow the users to configure even more themselves in their vCluster, so that the underlying host cluster can be very rudimentary, which provides less attack surface and room for human error further improving stability and security.</em></blockquote><h3>How to get self-service Kubernetes namespaces</h3><h4>Underlying Cluster</h4><p>The first part you need for a self-service namespace system is an underlying Kubernetes cluster that the namespaces are supposed to run on. If the self-service namespaces will be used for development and testing processes, it makes sense to create a new cluster that is separate from the cluster you run production workloads on.</p><p>Since one of the benefits of a self-service namespace solution is that it can be used and shared by many users, the cluster needs to be a cloud-based cluster and cannot run locally (even though you may test your setup with a local cluster first and then start again with a “real” version in the cloud).</p><p>Here, it does not matter if it is a cluster running in a public cloud or private cloud and if it is self-managed or managed by the cloud provider. However, it often makes sense to use a cluster that is similar to your production cluster (e.g. use AWS if your production cluster is AWS) because this makes development, testing, and other processes you want to use the self-service namespaces for more realistic.</p><h4>User Management</h4><p>A second central component for a self-service namespace solution is permission and user management. This allows the admins to keep control of who is allowed to create namespaces and to overview who is using what.</p><p>Especially in larger teams, having a Single-Sign-On solution is helpful because admins do not have to manually add the users and the users can start immediately. If you build a self-service namespace system yourself, solutions such as <a href="https://github.com/dexidp/dex">dex</a> may be helpful for this task.</p><h4>User Limits</h4><p>While you want to enable the users to create namespaces on-demand, you also want to prevent excessive usage in terms of CPU, memory, and potentially other factors such as number of containers, services, or ingresses. Such a limitation is very helpful to control cost, but you need to be careful not to limit the users in their work. For this, it should be up to the users how they want to allocate their allowed resources.</p><p>Implementing efficient user limits is much easier with manually provisioned and statically assigned namespaces than with dynamic namespaces that are created by the users on-demand. This is due to the fact that Kubernetes limits in Resource Quotas are on a namespaces-basis and not on a user-basis.</p><p>However, you want to limit users and not namespaces, so you need to solve this problem to get sensible user limits. For this, you need to use aggregated resource quotas, which can be done with the open-source solution <a href="https://github.com/kiosk-sh/kiosk">kiosk</a>.</p><h4>Make vs. buy</h4><p>Now that you know the most essential components for a self-service namespace system, you need to decide if you want to build this system yourself or just buy an existing off-the-shelf solution.</p><p>Several large organizations have already built <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">an internal Kubernetes platform</a> for namespaces. A very good example of this is Spotify because there even was a public talk at <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">KubeCon North America 2019</a> about their platform, so you can learn from their experience. However, even when using some open-source components such as <a href="https://github.com/dexidp/dex">dex</a> or <a href="https://github.com/kiosk-sh/kiosk">kiosk</a>, building an own namespace self-service platform takes a lot of effort, which is probably the reason why mainly larger organizations or companies with very special needs go this way.</p><p>In contrast to this, buying an existing off-the-shelf solution is feasible for organizations of any size and has the advantage that you can get started very fast without a large upfront investment. Additionally, you get a specialized service that goes beyond the minimal needs that you would probably build on your own. One example ofsuch a ready-to-use solution is <a href="https://loft.sh/">loft</a>. loft is internally building on kiosk and, besides self-service namespaces on top of any connected cluster, it provides some useful additional features: It works with multiple clusters, has a GUI, a CLI, as well as a <a href="https://loft.sh/features/kubernetes-cost-management">sleep mode</a> to save cost, and it provides a virtual cluster technology that can be used to create self-service Kubernetes work environments that are even better isolated than namespaces.</p><h3>Conclusion</h3><p>If you enable your engineers to create namespaces independently and on-demand, this will change how Kubernetes is used in your organization. Especially if you have already adopted Kubernetes and now want to spread its usage among further people in your organization, a self-service namespace system is a very good solution. It answers the fundamental question of how to provide an easy and independent Kubernetes access to engineers, while it is still also admin-friendly because admins can easily manage it and so have more time to care for the underlying cluster’s stability.</p><p>To get a self-service namespace system, you need to decide if you want to make or buy it. Making it is the right solution for companies with very special needs, but even then, you can build upon already existing open-source components that will make your life much easier. For most companies, buying is still a more practical approach because you get a full solution from a specialized vendor without a huge upfront investment.</p><p>No matter how you decide, having a self-service namespace platform will help you to take the next step towards more effective use of Kubernetes at your organization.</p><p>Photo by <a href="https://www.pexels.com/@thiagopatrevita?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Thiago Patrevita</a> from <a href="https://www.pexels.com/photo/four-man-cooking-1121482/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></p><p><em>Originally published at </em><a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e356691528ab" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/self-service-kubernetes-namespaces-are-a-game-changer-e356691528ab">Self-Service Kubernetes Namespaces Are A Game-Changer</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Kubernetes Development Workflow — 3 Critical Steps]]></title>
            <link>https://levelup.gitconnected.com/the-kubernetes-development-workflow-3-critical-steps-ab456d0b9b44?source=rss-9c5e70d02da0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab456d0b9b44</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Daniel Thiry]]></dc:creator>
            <pubDate>Mon, 14 Sep 2020 14:59:31 GMT</pubDate>
            <atom:updated>2020-09-17T00:47:50.171Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Kubernetes Development Workflow — 3 Critical Steps</h3><p>Establishing an efficient Kubernetes development workflow is critical for the success and the acceptance of your Kubernetes adoption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*QaIdHkr4KozetNVZ" /></figure><p>More developers than ever before are now working with Kubernetes. This means that also their workflows have to change to account for this technology that originally was not made for developers. However, integrating Kubernetes into efficient development workflows is not easy and comprises several aspects that I will discuss in this article.</p><blockquote><em>Not all companies are using Kubernetes in the same way and to the same extent. To get an overview of the different stages of Kubernetes adoption and cloud-native maturity, read my article about </em><a href="https://loft.sh/blog/the-journey-of-adopting-cloud-native-development/"><em>the journey of adopting cloud-native development</em></a></blockquote><h3>1. Setting Up Kubernetes Work Environments</h3><p>The first step to set up an efficient Kubernetes development workflow is to decide which kind of work environment shall be used. Here, the question is not only about which cloud environment or managed Kubernetes service to use, but also if one should use a cloud environment at all. Compared to production systems, it is also possible to only work with local Kubernetes environments for development.</p><h4>Local Kubernetes or cloud-based Kubernetes</h4><p>Of course, local and cloud-based work environments have advantages and disadvantages: While local environments such as <a href="https://github.com/kubernetes/minikube">Minikube</a> are free to use, cloud environments cost money. Local environments are also available offline and are fully independent of other developers and other infrastructure.</p><p>Cloud environments have the advantage that they provide more computing resources, run “standard” Kubernetes (not only versions made to run on local computers), and are easier to start. The provisioning of such environments can even be automized with <a href="https://loft.sh/blog/building-an-internal-kubernetes-platform/">internal Kubernetes platforms</a>, so they do not require any effort or knowledge on the developer’s side.</p><blockquote><em>For more information about the pros and cons of local and remote clusters, take a look at this article: </em><a href="https://loft.sh/blog/local-cluster-vs-remote-cluster-for-kubernetes-based-development/"><em>Local Cluster vs. Remote Cluster for Kubernetes-Based Development</em></a></blockquote><h4>Setup process for Kubernetes work environments</h4><p>From a workflow perspective, it is important to establish a standardized way of setting up the work environments for developers. This of course depends heavily on the type of work environment you use. Local environments need to be set up individually by every developer because they are only running on local computers, which prevents a central setup. This is why you should provide detailed instructions on how to start the local environment. It also makes sense to determine which local Kubernetes solution to use.</p><p>If you use a remote Kubernetes environment, you face a wholly different challenge: While it is easy to create cloud-based work environments, you need to determine how the developers get access to them. Some companies may decide to let admins centrally create the environments and individually give the developers access to them. However, this turns the creation process into a bottleneck that can slow down the whole development workflow. Therefore, it is more efficient to let developers create these environments on-demand. Companies such as <a href="https://www.youtube.com/watch?v=vLrxOhZ6Wrg">Spotify</a> have built internal <a href="https://loft.sh/blog/self-service-kubernetes-namespaces-are-a-game-changer/">self-service namespace </a>platforms for exactly this use case. However, there are also off-the-shelf software solutions such as <a href="https://loft.sh/">loft</a> that provide this functionality for any Kubernetes cluster.</p><h3>2. Development</h3><p>After the developers have access to a Kubernetes work environment, the actual development phase needs to be figured out. With development phase, I refer to what is sometimes described as <a href="https://mitchdenny.com/the-inner-loop/">the “inner loop” of software engineering</a>, i.e. coding, building, and observation/testing of results.</p><h4>How to interact with Kubernetes</h4><p>While most engineers have no experience in setting up a Kubernetes environment (<a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/#1-setting-up-kubernetes-work-environments">Step 1</a>), they are very familiar with the software development phase. Still, their workflows may significantly change compared to what they are used to when Kubernetes is introduced.</p><p>Developers normally know a lot about the peculiarities of the programming languages, frameworks, and tools they use, but Kubernetes brings in some new challenges that are mostly independent of the actual software: How can the software be containerized? How can the containers be built and started in Kubernetes? How can code changes be deployed into the containers, so developers see their changes? How can the software be debugged?</p><p>All of these questions need to be solved and all the solutions should become standardized workflows. For this task, it often makes sense that not every developer is involved in solving the initial one-time problems (such as initial containerization) and that only one engineer or a small team sets up the new workflows. (In another blog post, I describe this role of a <a href="https://loft.sh/blog/why-every-software-team-should-have-a-developer-experience-owner-dxo/">Developer Experience Owner (DXO)</a> in more detail.)</p><h4>Kubernetes development tools</h4><p>Since many companies face the same issues by introducing Kubernetes into the development phase, several open-source tools have been developed that address the issues in this area. Examples for this are <a href="https://github.com/devspace-cloud/devspace">DevSpace</a>, <a href="https://github.com/GoogleContainerTools/skaffold">Skaffold</a>, <a href="https://github.com/tilt-dev/tilt">Tilt</a>, and <a href="https://github.com/telepresenceio/telepresence">Telepresence</a>.</p><p>All of them solve similar problems but with slightly different conceptual and technical approaches: DevSpace provides a bi-directional, real-time code synchronization that allows for hot reloading of containers, so that code changes are visible without restarting the containers. (The other tools are partly offering this now, too.) It is also focused on development scenarios in cloud-based Kubernetes environments. In contrast to this, Tilt is rather specialized on development with local Kubernetes environments, while Telepresence enables the development of a locally running application that can interact with other parts running remotely. Finally, Skaffold is similar to DevSpace and Tilt but has a bit more focus on fast deployment workflows.</p><p>In general, all of these tools serve a similar purpose and are relatively versatile (e.g. Tilt can also be used with remote environments and DevSpace also works with local environments or in CI/CD pipelines). To decide which solution is best for your situation, you should look at all of them yourself as the decision really depends on your preferences and needs.</p><p>In any case, you should have a common configuration for the tool you want to use in your team so that it is very easy for developers to use. For example, that means that a developer should only have to use very few commands such as devspace dev or skaffold debug and directly can start to work with Kubernetes efficiently. Of course, this requires some initial configuration and documentation effort, but this effort will pay off very fast.</p><h3>3. Deployment</h3><h4>How to deploy to a Kubernetes system</h4><p>The last step that is very relevant for the development workflow with Kubernetes is deployment. This means that the developers need to have an easy opportunity to push their code to a staging or testing environment and eventually to production.</p><p>This challenge in the Kubernetes workflow should be relatively easy to solve as most developers and companies are used to this and already have solutions in place. Still, the process in this phase should be easy and fast for developers, so that they are encouraged to deploy their applications when it is appropriate.</p><h4>Kubernetes deployment tools</h4><p>Possible solutions for deployment to Kubernetes are the previously mentioned development tools, most notably Skaffold and DevSpace, which can also be integrated into more complex CI/CD pipelines. DevSpace, for example, can be configured in a way that a developer only has to use the command devspace deploy and their code will be deployed to a pre-specified Kubernetes cluster where it will be executed. This can be very practical for fast deployments, e.g. to run tests.</p><p>To deploy an application to production, more complex continuous integration and deployment solutions exist. Since Kubernetes is so common now that almost all CI/CD tools support it, it does not really matter if these solutions are particularly specialized on Kubernetes or not. You should rather compare different solutions again and see which best fits your needs. A good starting point are these tools: <a href="https://www.jenkins.io/">Jenkins</a>, <a href="https://codefresh.io/">Codefresh</a>, <a href="https://travis-ci.org/">Travis CI</a>, and <a href="https://circleci.com/">Circle CI</a>.</p><h3>Conclusion</h3><p>To establish an efficient Kubernetes development workflow, several workflow steps need to be defined and facilitated. The first is to provide the developers with a Kubernetes work environment, which can either run locally or in the cloud. Then, they need easy-to-use Kubernetes dev tools that support the “inner loop” of development, i.e. coding, quick deploying, and debugging. Finally, developers must have an easy way to deploy their developed code to a production environment.</p><p>All these steps have in common that they should be standardized and easy for the developers so that the adoption of Kubernetes becomes as smooth as possible. In doing so, you should never underestimate how complicated Kubernetes can be if you have never used it before. Therefore, documentation and support is critical throughout the whole process within your organization.</p><p><em>Originally published at </em><a href="https://loft.sh/blog/kubernetes-development-workflow-3-critical-steps/"><em>https://loft.sh</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab456d0b9b44" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/the-kubernetes-development-workflow-3-critical-steps-ab456d0b9b44">The Kubernetes Development Workflow — 3 Critical Steps</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>