<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[The Spot to be in the Cloud - Medium]]></title>
        <description><![CDATA[If you want to be part of how companies are transforming for and being transformed by the cloud, you’re in the right Spot. We hope you’ll enjoy our content on best practices, processes and tools for continuously optimizing cloud infrastructure management and cost. - Medium]]></description>
        <link>https://medium.com/application-driven-infrastructure?source=rss----a784a32a40f5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>The Spot to be in the Cloud - Medium</title>
            <link>https://medium.com/application-driven-infrastructure?source=rss----a784a32a40f5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:34:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/application-driven-infrastructure" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Best Practices for Understanding Kubernetes Costs]]></title>
            <link>https://medium.com/application-driven-infrastructure/best-practices-for-understanding-kubernetes-costs-f3c58a5e1ebf?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/f3c58a5e1ebf</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[cloud-finops]]></category>
            <dc:creator><![CDATA[Eric Knauer]]></dc:creator>
            <pubDate>Thu, 01 Sep 2022 00:45:07 GMT</pubDate>
            <atom:updated>2022-09-01T00:45:07.501Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwAm5lzqKNdJ_az995VaGQ.jpeg" /></figure><p>Kubernetes is the go-to orchestration platform for running large-scale systems of containerized workloads. However, understanding your Kubernetes costs — particularly in a cloud environment — is complex.</p><p>Kubernetes does not directly expose the cost of the infrastructure it utilizes but instead exposes the various resources it allocates and uses. Gathering and tracking the costs mapped to these resources is a time-consuming extra step, often delegated to a DevOps engineer (who has many other things to do). In addition, it’s not only the DevOps team that needs this cost information. Providing useful and actionable cost information to teams outside of DevOps makes the task even more burdensome.</p><p>Before you can optimize your Kubernetes costs, you’ll need a solid understanding of where those costs come from. You can also take some concrete steps to organize your Kubernetes setup to make cost visibility easier for you and your organization.</p><p>In this article, we’ll lay out the relationship in Kubernetes between costs and resource usage. We’ll look at the main areas of Kubernetes that incur costs, examining where those costs come from. Then, we’ll tie it all together with a list of best practices to put in place for ensuring cost visibility.</p><p>Let’s start by looking at the key areas in Kubernetes that incur costs.</p><h3>Understanding cluster costs</h3><p>Even an empty Kubernetes cluster has costs. The control plane typically runs on multiple nodes for high availability and manages an etcd data store for tracking the cluster’s desired state. Google Kubernetes Engine (GKE) and Amazon’s Elastic Kubernetes Service (EKS) charge $0.10 per hour per cluster which comes to about $70 per month. Azure Kubernetes Service (AKS) currently doesn’t charge separately for the control plane. If you are not using a managed Kubernetes service, then you will be responsible for the costs associated with the control plane nodes.</p><p>The control plane cost is fixed and reasonable unless your development process involves creating many clusters (for example, for testing) and leaving them running for extended periods of time.</p><h3>Understanding compute costs</h3><p>Compute costs are the costs of the worker nodes of your cluster. This is by far the most complicated and dominant aspect of your cloud costs.</p><p><strong>Node costs</strong></p><p>There are multiple factors that impact the cost of a running node.</p><p><strong>Instance types</strong></p><p>Cloud providers offer many <a href="https://spot.io/blog/the-complete-guide-to-ec2-instance-pricing/">instance types</a> with different profiles based on the ratio of performance to price. You need to understand the requirements of your workloads to choose the correct instance types you will use for different workloads. Cloud providers frequently upgrade and update their instance types. This is not a one-time decision. You need to monitor the offerings and be ready to move to better instance types when they become available.</p><p><strong>On-demand instances</strong></p><p>On-demand (or pay-as-you-go) instances are the typical instance type used for workloads requiring stable nodes that need to run continuously. Note that Kubernetes is designed for situations in which nodes come and go, and it will automatically move your pods to other nodes if a node becomes unavailable or unhealthy. These nodes come in a variety of sizes for each instance type. The price per unit of CPU or memory is the same. Different sizes are just multiples of the base unit. These nodes are scheduled.</p><p><strong>Reserved instances</strong></p><p>If you commit to purchasing a certain number of instances for a long term (one to three years), you can receive significantly lower prices (60%-80%). One caveat is that you may be stuck with large quantities of outdated instance types. Make sure you can change the instance types of your <a href="https://spot.io/blog/reserved-instances-the-complete-guide/">reserved instances</a>.</p><p><strong>Spot/preemptible instances</strong></p><p>Many stateless workloads can operate within their SLA even if their nodes disappear every now and then. This is a great opportunity to benefit from <a href="https://spot.io/blog/amazon-ec2-spot-vs-azure-lpvms-vs-google-pvms-vs-ibm-transient-servers/">spot</a> or preemptible instances. These instances have lower priority than on-demand instances and may be evicted or reclaimed by the cloud provider at any time to accommodate their resource needs. However, this type of instance is significantly less expensive (60%-90%).</p><p><strong>Considering discounts</strong></p><p>Cloud providers offer various discounts. It’s important to understand how these discounts will impact your costs when planning your cloud spend strategy. If your organization plans to spend millions of dollars in the cloud, then you should be able to negotiate special terms and discounts.</p><p><strong>Region pricing</strong></p><p>The cloud is not completely consistent and uniform. Different geographic regions may have significant differences in pricing. For example, instances in the GKE us-west2 region are 14%-20% more expensive than those in the us-central1 region.</p><p><strong>Node Daemonsets</strong></p><p>Kubernetes itself needs to run some software components on each node. These can be deployed as a native Kubernetes object via DaemonSet. Whenever a new node joins the cluster, the DaemonSet pod will be scheduled there. This means that a certain portion of a node’s resources is always taken by the DaemonSets before workloads can utilize it.</p><p><strong>Autoscaling</strong></p><p>The most prominent aspect of cloud computing is its elasticity. From a user perspective, the cloud has infinite capacity, and you pay for what you request. Kubernetes supports pod autoscaling, which means that if a workload operates at capacity, Kubernetes can create more pods to handle the demand. This requires that the cluster has nodes with available resources to schedule the new pods. If there are no suitable nodes, then the open-source <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler">Cluster Autoscaler</a> can add nodes to your cluster.</p><p>Of course, autoscaling incurs costs, and you should make sure that you understand the autoscaling behavior of your workloads. You can configure autoscaling at the node pool level, which is a pool of nodes of the same instance type. You can configure the minimum and maximum number of instances. The minimum number of nodes will be always allocated, and you pay for them even if they are empty. The maximum number of nodes will not be exceeded, which is a good way to prevent runaway costs.</p><p><strong>Bin packing</strong></p><p><a href="https://en.wikipedia.org/wiki/Bin_packing_problem">Bin packing</a> is the practice of packing pods into nodes. If your nodes are half-empty, then you are paying for resources that you don’t need. There are different mechanisms that can lead to inefficient bin-packing, such as misconfigured scheduling criteria or autoscaling minimums.</p><p><strong>Utilization</strong></p><p>Utilization is the ratio of resources in use to the total amount of requested resources. For example, let’s consider a pod that requests four CPU cores while only using two CPU cores at runtime. Bin packing can be 100% efficient, with Kubernetes allocating the pod to a node with four CPU cores. However, you pay for four CPUs when you need only two. If the pod requested only two CPUs, then you could fit two pods into each of the four CPU nodes and save 50%.</p><p><strong>Node-workload shape</strong></p><p>Node-workload shape is the ratio between CPU and memory requests. For example, if a node has two CPUs and 40 GiB memory, then it can fit exactly one pod that requests two CPUs and 10 GiB memory. In this case, there is a misalignment in the shape and you pay for 30Gibs of unused memory. There are different strategies to minimize the impact of shape misalignment from multi-tenancy to dedicated node pools.</p><h3>Understanding Networking Costs</h3><p>Large-scale systems are often deployed across multiple Kubernetes clusters, geographical regions, and availability zones (AZ). The workloads running in these various locations often need to communicate with one another and to other systems, as well as expose APIs and access data. There are multiple types of costs associated with cloud networking.</p><p><strong>Cross-AZ and cross-region costs</strong></p><p>There are different ways to connect networks. Transferring data within the same data center (that is, in the same AZ) is typically free, but cross-AZ and cross-region traffic within the same cloud will cost you. Traffic between different regions can have different costs. It’s important to understand these costs in order to design the network topology of your system.</p><p><strong>Ingress costs in the cloud</strong></p><p>Kubernetes clusters in the cloud interact with other systems, including Kubernetes clusters on other clouds as well as end users. Ingress (getting data into the cloud) is free because cloud providers prefer that you manage your data in their cloud. However, there are costs associated with network components that process ingress traffic. These components include the NAT gateway, VPN gateways, and load balancers.</p><p><strong>Egress costs in the cloud</strong></p><p>Egress traffic (sending data out of the cloud) will cost you a lot. This is by design. More recently, cloud providers are being pressured to pass on some savings to customers. AWS <a href="https://aws.amazon.com/blogs/aws/aws-free-tier-data-transfer-expansion-100-gb-from-regions-and-1-tb-from-amazon-cloudfront-per-month/">announced recently</a> that the first 100GB of egress per month is free. This may be significant for smaller companies. For organizations that use the cloud at an enterprise scale, this might not make a difference.</p><p>There are different solutions with different pricing profiles to consider for getting a lot of data out of the cloud from CDNs to storage appliances.</p><p><strong>Service mesh costs</strong></p><p>The major networking cost drivers are typically cross-region and egress traffic. However, if you use a <a href="https://www.cncf.io/blog/2017/04/26/service-mesh-critical-component-cloud-native-stack/">service mesh</a> for connectivity within your Kubernetes cluster or among a set of clusters, then there are costs you should be mindful of.</p><p>Service meshes usually attach a sidecar container to every pod. These sidecar containers know about all the other pods in the mesh. Each sidecar container requires CPU and memory, which will drive up your costs. The sidecar container uses the memory to store the global mesh configuration (all of the other sidecar containers in the mesh) to know where to send traffic. Whenever new pods come and go, all the service mesh sidecar containers need to be updated. The resources needed to handle these updates — especially for large meshes — are significant, yielding additional costs. Of course, when using small pods that use little CPU and memory, the resources used by the sidecar containers can be dominant.</p><p><strong>Network policy costs</strong></p><p>Kubernetes has a lot of optional components that an administrator can choose to enable or disable. When network policies are enabled, Kubernetes installs an agent on every node via DaemonSet, and that agent monitors the network traffic to enforce policies at the network level. Each agent, therefore, requires resources (CPU and memory). The additional cost of these resources is proportional to the cluster size. In large clusters with many small nodes, the network policy agent can consume a significant portion of the node resources, leaving fewer resources available to the application and its workloads.</p><h3>Understanding storage costs</h3><p>Storage is another core piece of infrastructure. Kubernetes provides abstraction layers in the form of storage classes, persistent volumes (PV), and persistent volume claims (PVC). However, as far as costs go, there isn’t much that impacts the cost at the Kubernetes level.</p><p>Kubernetes stores its own state in etcd. Developers may be tempted to use etcd to store some of their application data alongside the cluster state, but this is a dangerous anti-pattern. When provisioning nodes, you should also consider the size and type of the local storage (local spinning disks, local SSDs, and so on).</p><p>In general, storage is relatively inexpensive when compared to compute or networking. You should consider the spectrum of storage options and find the mix that works for you from expensive in-memory caches all the way to inexpensive slow archives.</p><h3>Best practices: concrete steps</h3><p>Understanding and managing costs on Kubernetes is not trivial. First, Kubernetes doesn’t directly capture the costs of the system. Second, Kubernetes is designed to support autoscaling and auto-healing dynamic systems, which makes it difficult to control costs. Let’s look at some best practices that can help.</p><p><strong>Cost allocation and namespaces mapped to teams</strong></p><p>Ensure you can associate costs with specific teams. This is critical for cost visibility and for working with teams to manage the costs they are responsible for.</p><p>Kubernetes supports namespaces that contain resources. Administrators can control resource usage in a cluster by defining resource quotas on a per-namespace basis. By assigning teams to namespaces, it’s easy to map costs per namespaces to the corresponding team. If namespaces alone are not sufficient, then use labels in conjunction.</p><p><strong>Resource labeling</strong></p><p>Every Kubernetes resource has a metadata section that includes labels, which are key-value pairs. Assign labels for accounting purposes to keep track of changes in costs that are related to changes in resources.</p><p><strong>Resource requests and limits</strong></p><p>Containers in pods can specify requests and limits for resources such as CPU and memory. A pod is scheduled on a node only if the total of its container requests can be satisfied, making the requested resources available for that pod.</p><p>The limit is a promise the container makes as to the quantity of resources it will use. If a container exceeds its CPU limit, then it may be throttled. If it <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-out-of-memory-behavior">exceeds its memory limits</a>, then it may be terminated by the out of memory (OOM) killer.</p><p><strong>Utilize cloud provider cost tracking and reporting</strong></p><p>Although Kubernetes itself is not cost-aware, cloud providers offer excellent cost tracking and reporting facilities that allow you to slice and dice your cost data. When you practice proper resource labeling, you can match changes in Kubernetes resources to cost at a very granular level (for example, by region, environment, team, day of the week, or even time of day).</p><p><strong>Efficiency calculation</strong></p><p>Calculate the efficiency of the resources you use. Empty, idle, or underutilized nodes and other unused resources are low-hanging fruit that you can detect and eliminate.</p><p><strong>Anomaly detection</strong></p><p>Kubernetes makes it easy to scale your resources. Make sure to have proper controls in the form of resource quotas, limits, and alerts that notify you before you run into a wall. This is important both for general operation as well as for preventing spikes in cost.</p><p><strong>Infrastructure as code</strong></p><p>Among the many benefits that come with infrastructure as code is the window of time you have to review the infrastructure changes that can impact code. Make cost-awareness part of the review process to catch early, unjustified infrastructure changes that can significantly increase your cost.</p><p><strong>Governance and audit</strong></p><p>Carefully consider who can make changes to your infrastructure that can impact the bottom line. Minimizing access to modify infrastructure will help you to avoid runaway costs as well as narrow down the culprit when investigating unexpected changes. Make sure to enable and tune Kubernetes auditing. Your cloud provider may provide additional tooling to make this easier, or even have it configured by default.</p><h3>Conclusion</h3><p>Kubernetes is a sophisticated orchestration platform. It is very suitable for large-scale distributed systems, possibly across multiple cloud providers, regions, and clusters. It is not simple to understand the considerable costs that such systems accrue. It takes expertise, experience, and attention to detail to make correct tradeoffs and ensure you don’t leave a lot of money on the table.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3c58a5e1ebf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/best-practices-for-understanding-kubernetes-costs-f3c58a5e1ebf">Best Practices for Understanding Kubernetes Costs</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[QA best practices for cloud and K8s solutions]]></title>
            <link>https://medium.com/application-driven-infrastructure/qa-best-practices-for-cloud-and-k8s-solutions-e680c95f70ae?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/e680c95f70ae</guid>
            <category><![CDATA[cloudops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[qa]]></category>
            <dc:creator><![CDATA[Vladyslav Zagorodnyi]]></dc:creator>
            <pubDate>Tue, 13 Jul 2021 13:22:34 GMT</pubDate>
            <atom:updated>2021-07-14T05:44:43.730Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*STrvjUFZAADmmxfcYMTb8g.jpeg" /></figure><p>Delivering innovative and reliable solutions in any industry is complex and time-consuming. At my current company, <a href="https://spot.io/">Spot by NetApp</a>, we help our customers (usually DevOps engineers) reduce the time and cost involved in managing their cloud compute infrastructure — whether in AWS, Azure or Google Cloud.</p><p>Behind our platform there is a lot of cutting-edge code and machine learning algorithms. Of course, it’s impossible to develop without any errors or bugs and that is where my team, the QA (Quality Assurance) group comes in. Here at Spot the QA team is completely integrated into the development lifecycle as customer obsession and customer satisfaction is baked into everything we build and do.</p><p>In this post I’ll share some of the best practices we employ to ensure our products bring our customers the value they expect.</p><h3>Manual and automated QA — complementary functions</h3><p>As always, establishing processes and standards is key to success in any project, and most certainly in QA.</p><p>At Spot by Netapp, QA consists of 2 segments: QA automation and manual QA.</p><p><strong>QA automation</strong> works to deliver information about software quality on a regular basis. We receive functional suitability reports 24/7/365 days from various automation tools (e.g. <strong>Selenium webdriver together with Jenkins</strong>) regarding regression of our UI and APIs. In case of failures there exists a notification mechanism and escalation flow, so we discover the problems before our customers experience any issue, so we can perform remediation immediately.</p><p><strong>Manual QA</strong> is indispensable when working on new product or feature releases. This is where it makes sense to invest human resources into activities that are not applicable for automation, such as:</p><ul><li>Unstable versions</li><li>Retests</li><li>Edge cases</li><li>One time tests</li><li>Urgent tests</li><li>User experience evaluation</li></ul><p>In short, we use automated and manual approaches wherever they are best suited. During planning, we evaluate what tasks are applicable for human-led QA and which are best for machine processing. Our QA team is always aligned with our development teams, with jointly established KPIs to ensure we increase coverage of all products and features in a timely manner.</p><h3>QA requirement planning</h3><p>When a task gets assigned to a QA engineer the requirement analysis starts. This is the most sensitive part of a QA job, because the QA engineer must understand all details and propose a foolproof solution in text, tabular or graph representation. Here at Spot we pay attention to coverage, depth of testing, scope, consistency, preconditions and possible risks.</p><p>We use the “Test Decomposition” methodology for verifying product requirements (typically based on the PRD) and transforming it into these documents:</p><ul><li>Test Plans — this is a general structure of the indicated level of testing, any required test design techniques and any estimations</li><li>Test Cases — this provides details of what will be tested and what are the expected results</li><li>Test Suits — the list of well-built sequence of test scenarios (e.g. first test login and then test logout)</li></ul><p>By detecting issues with the testing process at an early stage, we gain lots of benefits. In development practice it’s easier, faster and even cheaper to fix issues at the very beginning than after official production release. So we value a chance to work with product owners even during the early stage of establishing feature functionality and benefits or gathering requirements stages of the software development lifecycle.</p><h3>Where QA automation shines</h3><p>In general software products have a great variety of use cases, different flows and millions of applicable values. This can make manual testing almost impossible. Say we add a new feature to our Spot Console which consists of: <br>- 3 mandatory drop-downs<br>- each dropdown has 100 values, all applicable for selection<br>- “confirm” button<br>- result window (assume result always “true”)</p><p>Straightforward calculation shows us:<br>All possible test scenarios = 1 million possibilities (100 (values of 1st dropdown) x 100 (values of 2st dropdown) x 100 (values of 3st dropdown) = 1,000,000).</p><p>It’s exhaustive to test all of them manually, time consuming and risky due to human error. Fortunately in manual QA theory there exist test design techniques. If you want to know how to decrease the list of all possible combinations for 60+% without negative impact on coverage — read my upcoming article on Pairwise test design technique, that is being used here at Spot by NetApp.</p><h3>Hand-made excellence with manual QA</h3><p>To reach maximum quality QA engineers fully use manual stack capabilities of these Test Design techniques:</p><ul><li>Equivalence classes</li><li>Boundary values</li><li>Decision Table</li><li>Pairwise</li><li>State Transition</li><li>Domain Analysis</li><li>Use case</li></ul><p>In general, test design techniques allow testers to be more efficient, decreasing the amount of tests but increasing coverage, so we can perform more testing activities and deliver more customer satisfaction.</p><h3>Evaluate and escalate</h3><p>The most interesting things happen when QA detects an issue as that is the moment we start to make our product really better. The bug report is a result of professional test design and it’s not a secret that bug reports must be organised professionally as well. Bug reports in Spot consist of:</p><p>- Clear title<br>- Consistent steps to reproduce<br>- Clear difference between actual and expected results<br>- Environment<br>- Handling person</p><p>But one of the most important is to prioritise issues, to ensure that if it’s urgent it will be fixed immediately, as it blocks our clients. If not, it will wait in the queue and won’t block urgent activities. The way to indicate those rules are attributes such as <strong>severity</strong> which denotes the technical impact of bugs on a system and <strong>priority</strong> which denotes business impact of bug.</p><p>After the issue is fixed we perform a retest and also verify backward compatibility to ensure that fix activities didn’t affect or damage other functionality.</p><p><strong>Stay tuned for our next post on QA best practices!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e680c95f70ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/qa-best-practices-for-cloud-and-k8s-solutions-e680c95f70ae">QA best practices for cloud and K8s solutions</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Fargate Pricing: Optimize Billing and Cut Costs]]></title>
            <link>https://medium.com/application-driven-infrastructure/aws-fargate-pricing-optimize-billing-and-cut-costs-59cc3c89ada2?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/59cc3c89ada2</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[fargate]]></category>
            <category><![CDATA[elastic-container-service]]></category>
            <dc:creator><![CDATA[Melissa Cheng]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 17:58:59 GMT</pubDate>
            <atom:updated>2020-11-05T17:58:34.333Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*txAbQ0i5EqJV-IpG.jpg" /></figure><p>AWS Fargate is a managed service that enables you to run containers in Amazon Elastic Kubernetes Service (EKS) or Elastic Container Service (ECS). Since Fargate is serverless, you don’t need to provision or manage servers or clusters. However, you do need to set up package containers, define resource requirements, and configure permissions and networking policies.</p><p>Fargate pricing is determined according to virtual CPUs (vCPUs) requirements, as well as usage of GBs of RAM for running services. You can choose between on-demand instances pricing and spot instances pricing. There are also options for cost reduction when using AWS Saving Plans.</p><p><strong>In this article, you will learn:</strong></p><h3>What is AWS Fargate and how does it work?</h3><p>AWS Fargate is a managed, serverless compute engine that you can use to run containers in Amazon Elastic Kubernetes Service (EKS) or Elastic Container Service (ECS). It eliminates the need to provision or manage servers or clusters of instances and doesn’t require you to select specific server types, optimize cluster packing, or manually <a href="https://platform9.com/blog/kubernetes-cluster-sizing-how-large-should-a-kubernetes-cluster-be/">size and scale kubernetes clusters </a>.</p><p>When running containers, you can use a Fargate launch type or capacity provider. With this option, you are responsible for packaging containers, defining resource requirements, setting permissions and networking policies, and launching your application. This service eliminates the need to define or deploy EC2 instances, provision or manage resources, or isolate applications.</p><p>Benefits of Fargate include:</p><ul><li>No need to deploy, manage, or maintain infrastructure</li><li>Increased isolation for greater security</li><li>Built-in integrations for observability</li></ul><p>Fargate also has several limitations you should be aware of:</p><ul><li>Up to three times more expensive than on-demand EC2 instances</li><li>Only available in limited regions</li><li>No SLA for spot instances</li><li>No support for GPUs</li></ul><p>See more details in our section about <a href="https://spot.io/blog/aws-fargate-pricing-how-to-optimize-billing-and-save-costs/#header4">Fargate limitations</a> below.</p><h3>AWS Fargate pricing</h3><p>Pricing in Fargate is determined by how many virtual CPUs (vCPUs) your services require. It is also affected by how many GBs of RAM you use while running services, priced per hour. Below you can find example pricing for the US East region. This gives you a general idea of cost but for more detailed options and up-to-date information, you can see the official AWS Fargate <a href="https://aws.amazon.com/fargate/pricing/">pricing page </a>.</p><p>There is also a Compute Savings Plan available that you can use to save on costs. You can learn more about this plan in the <a href="https://spot.io/blog/aws-fargate-pricing-how-to-optimize-billing-and-save-costs/#header5">utilization best practices</a> section below.</p><h3>AWS Fargate vs Lambda vs EC2: pricing comparison</h3><p>When moving workloads to AWS, it helps to understand your options before making a decision. The most common services you should consider are Fargate, Lambda, EC2, and our very own . All four can be used to run workloads and host services although the methods these services support differ.</p><p><a href="https://lumigo.io/learn/">AWS Lambda</a> is a serverless functions service that you can use as an application or service backend or to run individual workload processes. EC2 is a virtual machine service that enables you to create instances as standalone machines or as hosts for containers. It is also used in the background by AWS to provide Lambda and Fargate services.</p><p>Below, you can see a breakdown of how these services differ in terms of pricing:</p><p><strong>AWS Fargate Pricing</strong></p><p>Fargate instances are billed on CPU and memory use per hour. The amount of resources available depends on the amount you have configured and any unused resources you have reserved are wasted. Billing is not dependent on the number of requests or run length. This makes it easier to predict boundaries for use and price range.</p><p>You can use AWS’s <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html">Container Insights </a>feature to help estimate your resource needs. You may also consider mixing services, using EC2 for more predictable or higher scale workloads. If you move these processes to EC2 you can save costs through spot or reserved instances.</p><p><strong>AWS Lambda Pricing</strong></p><p>Lambda is billed based on the number of requests, execution time, and resources reserved. Like with Fargate, any resources you reserve but don’t use are still billed for the duration of your function execution. When configuring functions in Lambda you can choose from a variety of memory tiers to right-size your resources.</p><p>To ensure that resources are properly distributed you can use CloudWatch Billing Alarms and Lambda Function logs. Both can provide you with metrics about your functions and how efficiently functions are operating. You can also configure events and reserved instances to reduce the number of function invocations, reducing billed time.</p><p><strong>EC2 Pricing</strong></p><p>In EC2, billing is based on the instances types you choose, the resources consumed, and how long instances are retained. You have the choice of using either on-demand or reserved instances. The former is more flexible but more expensive while the latter is more reliable and may be cheaper.</p><p>When using EC2, it is your responsibility to efficiently host your containers. Alternatively, you can use a combination of Fargate and EC2 with the Fargate Launch Type. This type enables you to access billing per second and only charges you for the instance resources you use.</p><p><strong>Ocean by Spot Pricing</strong></p><p>Ocean by Spot is a cloud managed service for running and managing container infrastructure in the cloud. With a pay-as-you-save pricing model, Ocean utilizes EC2 spot instances to maximize customer savings. Users are still billed by their cloud provider for the instances they use, and when spare capacity is not available to run applications, Ocean will automatically fallback workloads to on-demand.</p><p>To minimize idle resources and over provisioned nodes, Ocean chooses instance type and size based on real time container and application resource utilization (CPU, memory, disk, network) and auto scales infrastructure to meet those requirements.</p><h3>Fargate limitations to be aware of</h3><p>Despite its benefits over self managed container deployments, Fargate has limitations you should be aware of.</p><ul><li>Fargate is up to three times more expensive than on-demand EC2 instances, and Fargate with spot instances can be significantly more expensive than using EC2 spot instances</li><li>You can only use Fargate in limited regions, depending on whether you are using it with ECS or EKS.</li><li>There is no service level agreement (SLA) for availability when using spot instances with Fargate.</li><li>Your resource requirements are rounded up, meaning you may still pay for unused resources.</li><li>There is currently no support for stateful services in EKS. With ECS, you are limited to using volumes for persistent storage.</li><li>You cannot currently use GPUs with Fargate.</li><li>Fargate operates as a “black-box” making it difficult to diagnose performance issues or debug. This includes lack of visibility into cluster utilization or infrastructure.</li></ul><p>Many additional limitations exist when using Fargate with EKS. These include:</p><ul><li>You can only use AWS’s Application Load Balancer to distribute your workloads.</li><li>Your pods must be scheduled with a Fargate profile to ensure smooth operation. However, this limits each pod to 4 vCPU and 30Gb memory per pod.</li><li>You are unable to run privileged containers, daemonsets, or pods with specified HostNetwork or HostPort.</li><li>Only private subnets are supported.</li></ul><h3>Best practices: how to reduce AWS Fargate costs and improve utilization</h3><p>To get around the above limitations and ensure that you are getting the greatest ROI on your Fargate implementation, you can employ the following best practices. These practices can help you reduce your overall costs by maximizing pricing plans, eliminating wasted resources, and easing workload management.</p><p><strong>AWS Savings Plans</strong></p><p><a href="https://aws.amazon.com/savingsplans/">AWS Savings Plan </a>is a pricing option that you can use to save up to 72% of the cost of EC2 instances. It applies to any instance type, including those used for standalone EC2, Lambda, and Fargate.</p><p>The savings plan works by recommending an amount of resources based on your previous use and defining a recommended amount you can commit to using in a given period. These resources are then billed like Reserved Instances in one or three year terms. Any resources you use within the savings plan limit is automatically deducted. If you go over your reservations, any additional resources are billed at the regular, On-Demand rate.</p><p><strong>Terminate idle pods</strong></p><p>If you have unpredictable workloads with traffic that fluctuates, you may have idle pods that cost money without contributing to performance. Right-sizing the number of active pods you have at any given time can help you avoid these costs and simplify your deployment.</p><p>To manage pods, the easiest way is to use the Kubernetes horizontal pod autoscaler feature. This feature enables you to control how pods scale and to control scaling in response to utilization metrics.</p><p><strong>Resource tagging</strong></p><p>Tagging resources can grant more granular control over your resources and increase visibility. It is an especially helpful practice when managing multiple environments, including test, staging, and production. Tagging can help you quickly identify which pods are in use and help you ensure that pods are not orphaned or consuming more resources than expected.</p><p>AWS has a built-in tagging mechanism you can use or you can use Kubernetes features. However, if you use the AWS option, you can also monitor tagged resources through the AWS Billing and Cost Management console. This can help you better manage costs and evaluate the breakdown of your resources.</p><h3>Conclusion</h3><p>AWS Fargate eliminates the need to deploy and manage infrastructure. This managed service can increase isolation, promoting greater security. You can also use Fargate built-in integrations for observability. Fargate instances are billed on CPU and memory use per hour. If you continuously terminate idle pods and utilize resource tagging for monitoring, you can significantly reduce costs. You can also leverage AWS Saving Plans for further cost reduction.</p><p>Where you would find the most savings, however, Fargate falls short. Spot instances come at an enormous discount, but with Fargate, a spot instance can actually cost you eight times more than with EC2. In contrast, <a href="https://spot.io/products/ocean/">Ocean by Spot</a> offers an enterprise grade SLA for workloads running on EC2 spot instances, saving you up to 90% of what you would have spent on on-demand, and ten times less than what you would have spent on spot instances with Fargate.</p><p>Ocean optimizes cluster resources via auto scaling and right sizing, both driven by container resource requirements, resulting in highly utilized infrastructure with minimal waste. With detailed visibility into resource allocation and utilization, Ocean opens up the black box of infrastructure and provides cost breakdowns per node, deployment, and pod, right down to the container level. You can maintain granular control and full infrastructure access with features such as custom tagging per launch specification and detailed dashboards.</p><p><em>Originally published at </em><a href="https://spot.io/blog/aws-fargate-pricing-how-to-optimize-billing-and-save-costs/"><em>https://spot.io</em></a><em> on July 21, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59cc3c89ada2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/aws-fargate-pricing-optimize-billing-and-cut-costs-59cc3c89ada2">AWS Fargate Pricing: Optimize Billing and Cut Costs</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon ECS Vs EKS: Container Orchestration Simplified]]></title>
            <link>https://medium.com/application-driven-infrastructure/amazon-ecs-vs-eks-container-orchestration-simplified-36de33d32695?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/36de33d32695</guid>
            <category><![CDATA[ek]]></category>
            <category><![CDATA[elastic-container-service]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Melissa Cheng]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 17:46:16 GMT</pubDate>
            <atom:updated>2020-11-05T17:44:14.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fwBqPWRUtd-8Hd6q.png" /></figure><p>Ever since Docker containers were first introduced in 2013, the transition to microservice, container-based architecture, has been trending as the most popular industry topic.</p><p>Organizations that leverage Containers technology to run their mission-critical applications are doing so in order to adopt complex micro-serviced architectures, which increase speed, reliability, and availability.</p><p>When talking about containerized applications, the leading technologies which will always come up during the conversation are Kubernetes and Amazon ECS (Elastic Container Service).</p><p>While Kubernetes is an open-sourced container orchestration platform that was originally developed by Google, Amazon ECS is AWS’ proprietary, managed container orchestration service.</p><p>Amazon ECS is a provider-specific solution for running and managing docker containers in the public cloud. During re-invent 2017, AWS also announced support for Kubernetes with their EKS offering (Elastic Kubernetes Service). Amazon EKS is AWS’ managed service for Kubernetes, where users can easily deploy, a high-available, scalable and fully managed Kubernetes control-plane on AWS.</p><p>With both services offering similar yet distinct computing concepts for running and managing containers on AWS, how can engineering teams decide if Amazon ECS or Amazon EKS is the most suitable selection for their workloads?</p><p>In this blog post, we’ll share the differences between both platforms and highlight the advantages of each one, so you will be fully equipped to choose the best option for you.</p><h3>Why use a Container Orchestrator?</h3><p>In the past few years, containers have dramatically changed the way organizations develop, package and deploy applications.</p><p>Running applications in containers rather than traditional VM’s brings great value due to the fact that they are easily scalable and <strong>ephemeral</strong>.</p><p>Container orchestration is all about managing the<strong> lifecycle of containers</strong> in large environments, and that includes various tasks such as:</p><ul><li>Provisioning and deployment of containers on instances</li><li>Redundancy and availability of containers</li><li>Scaling up\down based on a given load</li><li>Resource allocation</li><li>Health monitoring of containers and hosts</li><li>Seamless deployment of new application versions</li></ul><h3>Key Differentiators Between ECS and EKS</h3><p>Being able to assign a network card directly to a task\pod might not seem like that big of a deal at first sight, however, this effectively means a higher level of security. This way the user is able to assign a Security Group dedicated for that individual task\pod, rather than simply opening all network ports for the hosting EC2 instance. (Regardless, some might argue that using a LB will eliminate this challenge as the user will need to open a certain security group to the LB alone.)</p><p>With <strong>ECS</strong>, you have the option to associate an ENI directly with a task, by choosing to launch a task in “awsvpc” mode. However, the maximum number of ENIs (read Virtual NICs) that can be assigned to each EC2 instance varies according to the EC2 type, and that ranges from 8–15 ENI’s per EC2 instance, potentially not enough to support all the containers we wish to run on that particular instance.</p><p>Lately, AWS has increased its support for ECS clusters running in “awsvpc” mode, and now users are able to assign 3 to 8 times more ENI’s than the previous limit (depending on the instance type), therefore increasing elasticity and enhancing container security.</p><p>With <strong>EKS</strong>, the user has the option to assign a dedicated network interface to a Pod. This practically means that all containers inside that pod will share the same internal network and public IP. On top of that, with EKS it’s also possible to share an ENI between several pods, thus enabling the user to place many more Pods per instance (EKS allows up to 750 Pods, depending on instance size,), significantly more than the capacity of ECS, which accommodates only to a maximum of 120 tasks per instance.</p><p>Generally speaking, both ECS and EKS clusters running on EC2 instances are debited to the same compute costs, based on the instance type they are using and the running time of that instance.</p><p>The only cost difference between <strong>ECS</strong> and <strong>EKS</strong> is that EKS is subjected to an additional cost of running the master nodes (across 3 availability zones) on On-Demand instances (this is due to the fact that the Kubernetes <strong>control plane</strong> must always be available and redundant, as EKS manages the master nodes separate from the worker nodes).</p><p>The <strong>service cost</strong> of EKS is 0.2$ an hour, which amounts to 144$ per month per K8s cluster.</p><p>As mentioned in the intro to this blog, both Kubernetes and ECS are quite similar in their orchestration concepts, however, another key differentiator between the two is the ease of deployment.</p><p>While is considered an <strong>ECS”Out of the box”</strong> solution for container orchestration due to its deployment <strong>simplicity</strong>, deploying Kubernetes clusters on EKS is a bit trickier and requires a more complex deployment configuration and expertise.</p><p>To summarize, orchestrating containers via Kubernetes requires more expertise from DevOps engineers, whereas in ECS it’s considered to be easier.</p><p>Securing cloud infrastructure has always been a challenge for engineering teams, especially securing containers.</p><p>Both ECS and EKS have their docker container images stored securely in ECR (Elastic Container Registry), AWS’ service for storing docker images. Every time a container spins up, it securely pulls its container image directly from ECR.</p><p>The main security differentiator between ECS and EKS is the fact that ECS supports <strong>IAM roles per task</strong>, whereas IAM roles are not supported in EKS at the moment.</p><p>The capability to assign an IAM role per task\container provides an additional thickened layer of security by specifically <strong>granting containers access</strong> to various AWS services such as S3, DynamoDB, Redshift, SQS and more.</p><h3>Multi-Cloud Compatibility</h3><p>As the cloud-compute world evolves, more and more organizations are <strong>decentralizing</strong> their workloads across multiple cloud providers, thereby benefiting from the different services and pricing each cloud offers.</p><p>ECS is an AWS-native service, meaning that it is only possible to use on AWS infrastructure, causing a <strong>vendor lock-in</strong>.</p><p>On the other hand, EKS is based on Kubernetes, an <strong>open-source</strong> project which is available to users running on <strong>multi-cloud</strong> ( <strong>AWS</strong>, <strong>GCP</strong>, <strong>Azure</strong>) and even On-Premise. This factor provides extra <strong>flexibility</strong> and allows users to be more <strong>cloud-agnostic</strong> when it comes to where they are running their workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/0*nfsTSsFzB8WOXyyS.png" /></figure><h3>Bottom line</h3><p>As discussed in this blog, having a <strong>container orchestrator</strong> is not an add-on, it’s a real necessity when running production workloads in the public cloud.</p><p>Well, there is no right or wrong when selecting a container orchestration platform, as each of them has pros and cons.</p><p>If you are new to containers and are looking for a simple way to set up and deploy clusters, perhaps ECS is the easier choice.</p><p>On the other hand, if you are experienced and are looking for a better way to scale your clusters and avoid vendor lock-in, perhaps EKS is the solution for you.</p><p>When you’re looking for a solution that combines <strong>simplicity</strong> and <strong>availability</strong>, and you want to have <strong>advanced control</strong> over your infrastructure, then ECS is the right choice for you.</p><h3>Spot Ocean: Abstracting Containers from the Infrastructure</h3><p><a href="https://spot.io/products/ocean/">Spot Ocean</a> is our serverless compute-engine that provides data-plane (worker nodes) end to end management, and it abstracts Kubernetes Pods and Amazon ECS tasks from the underlying VMs / infrastructure. Spotinst Ocean relieves engineering teams from the overhead of managing the most complicated part of Kubernetes or ECS clusters by dynamically provisioning, scaling and managing the data-plane component (worker nodes and EC2 instances).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ok0-MgIRfRa0PXs6.png" /></figure><h3>Advanced Automation</h3><p>Spot Ocean takes advantage of multiple compute purchasing options like Reserved and Spot instances and uses On-Demand only as a fall-back, providing an 80% reduction in cloud infrastructure costs while maintaining high-availability for production and mission-critical applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OXyiwK-eytK9rTLL.png" /></figure><p>Please check out this explaining the capabilities you can achieve with Spotinst Ocean and why it is considered as the go-to product to run container workloads in the public cloud, while significantly lowering cloud-compute costs.</p><p>Are you already running containers and are looking to automate your workloads at the lowest costs and gain deeper visibility into your clusters?</p><p><em>Originally published at </em><a href="https://spot.io/blog/amazon-ecs-vs-eks-container-orchestration-simplified/"><em>https://spot.io</em></a><em> on August 25, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36de33d32695" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/amazon-ecs-vs-eks-container-orchestration-simplified-36de33d32695">Amazon ECS Vs EKS: Container Orchestration Simplified</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Log aggregation and the journey to optimized logs]]></title>
            <link>https://medium.com/application-driven-infrastructure/log-aggregation-and-the-journey-to-optimized-logs-ea17797969a0?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea17797969a0</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Liran Haimovitch]]></dc:creator>
            <pubDate>Thu, 15 Oct 2020 16:42:45 GMT</pubDate>
            <atom:updated>2020-10-15T16:42:45.622Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FgH4MO9Sv4ppdU2d.png" /></figure><p>Ever experienced bad logging- whether it’s the wrong log, the wrong information, or a multitude of other logging woes? We aren’t able to count the number of times anymore that we’ve happily gone and set log lines, only to find out that it was all for naught. The frustrations are endless. What is meant to be magic for your code, the ultimate savior when debugging, has become the ultimate frustration. The pitfalls of logging are many, but there is a way to optimize them to your benefit. Let us take a look at what makes logging so challenging and how you can ultimately optimize your logs to your maximum benefit.</p><h3>The logging rabbit hole</h3><p>It seems that log aggregation systems aren’t the end all and be all that we make them out to be. Yes, they’re awesome, but as developers we often find ourselves dealing with a phenomenon known as “Logging FOMO”. This is the fear of missing out on logs that could, perhaps maybe, might be needed at some point in the future, driving devs to unnecessarily add more and more logs. Many of these log lines are unnecessary, causing high overhead fees and wasting a lot of time that developers could be using for other projects. So how do we make sure we’re not logging needlessly?</p><h3>The 3 stages of logging</h3><p>Developers go through three stages when logging. They begin at the ‘no logging’ stage. Here, as you may have accurately guessed, they write no log lines. The common mentality is that “logging is for losers” and unnecessary because they know their code best. Unfortunately (or not, depending on how much you enjoy watching people learn life lessons), these devs soon find out just how wrong they were. This is when, at some future point, they wake up in the middle of the night to find that something broke, and worse they have no log lines to give them the data to fix it.</p><p>After that painful experience, the dev reaches the ‘bad logging’ stage of their career. While the quality of their logs is nothing to write home about, it’s definitely better than writing no logs at all.</p><p>Once the dev has learned to write logs, albeit not great ones, comes the final ‘good logging’ stage. Here, they have finally learned not only the importance that quality logs give in terms of productivity and efficiency, but more importantly as a source of data when looking to understand their code.</p><h3>Maintaining lessons learned</h3><p>Staying in the effective logging stage is no easy feat, so here are a few tricks that can help. First, remember to write the logs not for yourself to read, but for the person who has never encountered your code before. This lets them understand it, even out of context. Next, keep the surprises in your code to a bare minimum. Log the unexpected so you don’t have to be THAT person, pretending you like surprises when everyone knows you almost ran out of the room at your last surprise party.</p><p>After surprises, try to contain as much relevant data as possible. There’s no such thing as too many relevant details. Every piece of data is critical when it comes to problem solving. The last trick is to remember that it’s okay to add and remove logs. Logging is dynamic, and if you want to minimize the noise and increase visibility, it’s often necessary to do just this. In order to facilitate adjusting what and when to log, application logging frameworks have been developed such as log4j for example. These logging frameworks let you define log severity levels for your logs so that you can turn down logging severity levels during normal execution of your application and turn them up if you need more information or to debug an issue.</p><p>Although keeping as much relevant data as possible is extra helpful when debugging challenging issues, the sheer volume of logging data collected can grow exponentially. This leads to increased management overhead as well as costs associated with managing that data. We’ll address this later on in the blog.</p><h3>Logging money pits</h3><p>Despite the importance and value of good logging, the actual cost can be prohibitive at times. There are a variety of ways in which you can aggregate your logs to make them easier to manage. If using a SaaS offering, such as Logz.io, Splunk, or Datadog, you can often find yourself shelling out anywhere from hundreds of dollars to hundreds of thousands — and that’s just per month.</p><p>If you’re using an on-prem version of the above solutions, whether in your data center or the cloud, you might find that while your licensing is lower, ultimately your Total Cost of Ownership might be higher.</p><p>Logs require a lot of storage, as they bring in a large amount of data about your applications. Not only that, but we need to use fast storage formats like SSDs to run fast queries, and look for additional storage capacity for the indices. Then there is the ingestion for building those indices since developers will be wanting the available data as fast as possible. With all of these, you’ll find yourself requiring a lot of computing power. You will also find that processing these queries and storing some of these indices requires a significant amount of memory and CPU resources. Tally all of these up and it comes together to create a black hole of logs of which valuable resources are thrown into.</p><h3>Manage your logs, manage the money</h3><p>You now find yourself at a fork in the road. How do you get a grasp on the sheer number of logs and the money they’re draining without getting rid of important data? One way could be by reducing logging volume. In essence, this method figures out which logs are taking up the most space and removes them.</p><h3>Choosing your path</h3><p>Once you pinpoint the logs that are taking up too much space, it’s time to decide what to do with them. If it’s for logs created by your own applications, there are a few options. Large variables, such as buffers and collections don’t need to be logged in their entirety, rather just the information that may be most relevant to your debugging workflow. Alternatively, a log that is too detailed can be replaced with one that is less so and more to the point. Take advantage of logging verbosity levels (INFO, ERROR, DEBUG, etc) and ensure you are logging at the right level. If it’s logs created by 3rd party applications, check out their logging configuration (whether you can define verbosity or define the events you need), as well as use drop filters in your log aggregator or log processor to remove excess logs.</p><h3>Moving forward</h3><p>If you are using <a href="https://www.rookout.com/blog/logging-in-the-cloud-machines-first-humans-second">structured-logging </a>, keep in mind it will increase your overall logging volume. Enriched logs are going to make your life so much better, but if you aren’t careful they can easily balloon every single record by a factor of ten or more. A surprising amount of volume can be saved by culling out insignificant or rarely used fields in your metadata.</p><p>Once you’ve done this, check to ensure that the important metadata fields are efficiently represented within whichever format you’re using. Do this by using short-forms of values, optimizing field names, and avoiding unnecessary padding. All of these will help optimize your logs moving forward.</p><p>Take one step further down your logging path, and you’ll find the ability to archive. While many logs become less useful as time passes (older data simply tends to be less relevant) you may need to keep them for security, compliance, or regulatory purposes. If so, rather than throwing your money away with high-cost log aggregation services, archive your logs into cold storage services like Amazon S3 Glacier that allow you to load them back up when you need them.</p><h3>Logging Optimized</h3><p>Simply put, logging can be challenging. What happens if you don’t have any logs in an area of code that breaks? What if you discover a defect in your application, but your logs don’t give you any useful information? Luckily, there are tools that help you avoid these possibilities by giving you the ability to augment your existing logging data with additional data collected on demand from any place in your application without stopping or restarting it.</p><p>lets you set Non-Breaking Breakpoints in your production code and let’s you get any data you need, all with just one click, in seconds. There’s no need to write extra code, restart or redeploy your app and it works across all cloud providers and platforms. It also <a href="https://www.rookout.com/blog/is-it-suffocating-your-organization-heres-how-to-get-your-contextual-data-pipelines-right">lets you send your data </a>to any 3rd party logging or monitoring system you might be using. Also, any personally identifiable information (PII) in your data can be easily redacted to ensure no sensitive data leaves your network.</p><p>This enables developers to change the way they think about and implement logging. It allows developers to take a second look at their logs and decide which logs are necessary as well as adjust what they are logging.</p><p>Don’t let your log be the straw that broke the camel’s back. Optimize your logs for maximum efficiency and see how much time you can save yourself. On that note, did someone say it’s time for another relaxing cup of coffee?</p><p><em>Originally published at </em><a href="https://spot.io/blog/log-aggregation-and-the-journey-to-optimized-logs/"><em>https://spot.io</em></a><em> on July 10, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea17797969a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/log-aggregation-and-the-journey-to-optimized-logs-ea17797969a0">Log aggregation and the journey to optimized logs</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Container Security]]></title>
            <link>https://medium.com/application-driven-infrastructure/kubernetes-container-security-1bec6df8d1b2?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/1bec6df8d1b2</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Robert Haynes]]></dc:creator>
            <pubDate>Thu, 15 Oct 2020 16:15:30 GMT</pubDate>
            <atom:updated>2020-10-15T16:15:30.308Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iWU9ep2zAqLCcSoM.jpg" /></figure><p>The National Institute of Standards and Technology (NIST) <a href="https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf">Special Publication on Container Security </a>provides a comprehensive review of the major risks for core components of a container system. One of the most obvious objects of concern (alongside a host of other things to keep a good security professional up at night) is, of course, the containers running on your platform.</p><p>Container security is obviously critical to your applications and data. It’s a complex topic, but one way to think about it is to break it down into three component parts, and a simple mantra:</p><p><em>Build secure containers, pull them from a trusted registry and deploy with ‘least privilege’, then protect them at runtime.</em></p><p>While this might be a gross oversimplification (as befits a 1,000-word blog), it does provide us with a starting framework to assess the controls and assurance we have in place when it comes to container images.</p><h3>Building secure containers</h3><p>Container images are built from layers, with a readable/writable layer containing specific code or configuration (your HTML assets, custom binaries, python scripts etc) sitting on top of a stack of read-only-layers containing the base image, software tools, and libraries.</p><h3>Monitoring for vulnerabilities</h3><p>Any one of these layers — from the base image to the libraries and packages you add, can contain vulnerabilities. Some might be trivial, some might be critical. When we analyze containers using <a href="https://www.paloaltonetworks.com/prisma/cloud">Prisma Cloud </a>, it’s not uncommon to find multiple vulnerabilities in a single container image.</p><p>There are obviously other misconfigurations you need to be aware of, like storing private keys or other secrets in the container image, or running an SSH daemon.</p><p>The good news is that container images are scannable, therefore the layer manifest can be parsed and the components assessed against known vulnerabilities. This process should be part of all container build CI/CD pipelines, so that containers stored in registries have at least a known vulnerability state.</p><p>Since new vulnerabilities in old software are routinely found, scanning needs to be an ongoing process with routine reassessment of container images against the latest known vulnerabilities.</p><h3>Infrastructure as Code</h3><p>It’s also important to assess how a container is going to be deployed at runtime (a topic we will touch on later in this blog). You don’t want your containers running as root, sharing networks with hosts, or making insecure file system mounts, for example. These attributes are defined in Infrastructure as Code (IaC) files, like a Kubernetes YAML file or an AWS Cloud Formation Template.</p><p>Since IaC files are increasingly written by developers who also write the application code, giving them to scan their IaC field before committing them makes a lot of sense, but don’t stop scanning them as part of your CI/CD build workflow.</p><p>The end result of your CI/CD pipeline should be a container that, as well as performing the required function, has a level of security inspection and testing such that it can be, to an extent, trusted.</p><h3>Trusted registries and secure deployments</h3><p>NIST SP 800–190 states:</p><p><em>“Organizations should maintain a set of trusted images and registries and ensure that only images from this set are allowed to run in their environment, thus mitigating the risk of untrusted or malicious components being deployed.”</em></p><p>Kubernetes will, however, generally run any container image, from any accessible container registry. So merely establishing an organization-specific container registry isn’t going to be sufficient.</p><h3>Trusted Containers and Registries</h3><p>You need to establish mechanisms to prevent untrusted containers being run within your cluster, by specifying trusted registries, or even trusted images. This can be done using an admission controller and associated rules, or as part of a more comprehensive container security runtime defense solution like <a href="https://www.paloaltonetworks.com/prisma/cloud">Prisma Cloud </a>.</p><p>Stopping untrusted containers at runtime, rather than just during the build process, is important. A common component of an attack is to compromise a running container, then use the pod’s service account and API server access to start additional malicious pods within the cluster.</p><p>This concept of identity and access leads into the next key area.</p><h3>Secure deployments</h3><p>There are a number of aspects to secure container deployments, which could be generalized as how a container interacts with the local runtime — the container engine, underlying compute node and network, and how it interacts with the Kubernetes control plane — via the API.</p><p>Since configuring how a container interacts with the runtime is configured largely via the API there is something of a circular argument here, but as ever in security, we are aiming to practice ‘defense in depth’.</p><p>In our build phase, if we have properly inspected the container image and IaC files, and then configured our container platform to only run images from trusted registries, then a lot of threats should be mitigated, but that’s no replacement for real-time defense.</p><h3>Additional controls</h3><p>Since not everyone will be able to enforce such strict rules (although, really they should), following good ‘ ‘ practices, run-time defense for your container platforms should still be a key part of your armour. Prisma Cloud, for instance, deploys specialized pods onto cluster nodes which scan and analyze the composition and configuration of a container on a node at startup, and alert when policies are violated.</p><p>So now that we have used controls to ensure container security during the build and deployment phases, it’s time to think about what happens when they are running in production.</p><h3>Protecting against application and network attacks</h3><p>Any service connected to a network is likely to be attacked, irrespective of whether it’s internet-facing or not. Plan to protect running Kubernetes pods from network and application layer attacks by deploying container-level monitoring to detect anomalies such as new connections, processes, or file system access.</p><p>Deploying a mix of <a href="https://blog.paloaltonetworks.com/2020/06/network-cn-series/">next generation firewalls </a>and <a href="https://www.paloaltonetworks.com/prisma/cloud">a cloud native security platform </a>to protect and monitor your Kubernetes nodes and pods gives you real-time detection and blocking capabilities for a range of malware and application layer threats.</p><p>One final point is to properly <a href="https://blog.paloaltonetworks.com/prisma-cloud/kubernetes-infrastructure-security/">secure the underlying infrastructure of your Kubernetes deployment</a>.</p><p>With your container’s journey from code to production protected and secured, you can feel more confident that your applications and data are safe (ok, well maybe less at-risk).</p><p><em>Originally published at </em><a href="https://spot.io/blog/kubernetes-container-security/"><em>https://spot.io</em></a><em> on August 18, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bec6df8d1b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/kubernetes-container-security-1bec6df8d1b2">Kubernetes Container Security</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging cloud computing pricing models for greater cost efficiency]]></title>
            <link>https://medium.com/application-driven-infrastructure/leveraging-cloud-computing-pricing-models-for-greater-cost-efficiency-e9450f14524e?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/e9450f14524e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-cost-management]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Zev Schonberg]]></dc:creator>
            <pubDate>Sun, 11 Oct 2020 15:28:08 GMT</pubDate>
            <atom:updated>2020-10-11T15:20:14.390Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GwZuXU-AC6IqBK0d.png" /></figure><p>As we discussed in our <a href="https://spot.io/blog/cloud-cost-optimization-with-2020-hindsight/">previous post on cloud cost analysis and optimization</a>, with over 60% of all cloud costs attributable to compute, focusing on compute infrastructure spend should be of the utmost priority.</p><p>In this post we will focus on the public cloud vendors’ pricing models and how to best leverage them all for maximum cost optimization.</p><h3>Cloud compute pricing strategies — discovering the inherent efficiencies</h3><p>When properly leveraged, especially together, the various pricing models offered by cloud providers, actually hold the key to the most potential cost savings, and in some cases can reach up to 90% cost reductions. Moreover, these savings can be achieved without any need to tinker with your architecture or applications.</p><p>The challenge however, is avoiding the built-in tradeoffs inherent to each pricing plan so you can fully benefit from the cost savings without giving up any ground on flexibility and availability. Let’s take a look at how each pricing model works, understand its constraints, and how to best leverage each one.</p><h3>On-demand instances</h3><p>This option, also referred to as pay-as-you-go, allows you to spin up and down compute instances at will, provided that the instance type you want is available when you want it. While allowing the greatest flexibility, on-demand is also the most expensive option and should be used only in the event that no other options are available.</p><h3>Spot instances</h3><p>This option, also referred to as Preemptible VMs in GCP, offers the greatest cloud compute cost efficiency with potential of up to 90% savings in AWS and <a href="https://azure.microsoft.com/en-us/pricing/spot/#overview">Azure</a> and 79% in GCP. Spot instances represent cloud providers’ excess capacity which they offer at massive discounts to drive consumption and offset the cost of idle resources. However, the caveat that cloud providers may “pull the plug” and terminate these instances with minimal warning can result in data loss, performance degradation, unavailable services and the like.</p><p>Enjoying the upside of spot instances without experiencing negative impact on availability can be realistically achieved even for mission-critical workloads if users can tackle the complexities of the spot instance market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HvKyMGAPAGU1M3R2.png" /></figure><p>To illustrate, the above chart shows the availability levels across Amazon’s EC2 spot instance markets in us-east-1. Globally AWS has approximately 15,000 spot instance markets and each is uniquely defined by its region/availability zone, instance type and size. Availability is constantly changing, so determining which spot instances will have greater longevity and which are about to be terminated requires ongoing monitoring along with an automated mechanism for gracefully draining at-risk instances and seamlessly moving their workloads to either new spot instances, available reserved capacity or if need be, on-demand instances.</p><p>Doing this successfully requires either expert, do-it-yourself DevOps configuration and management or a turn-key 3rd party solution that can handle all this heavy-lifting while providing an SLA for high availability.</p><p>about spot instances, surprising use cases and tips for DIY.</p><h3>Reserved instances, Savings Plans and committed use discounts</h3><p>For those with predictable, steady state workloads, the cloud providers all offer pricing options with an upfront commitment (1 or 3 years; with/out upfront payment) that can deliver cost savings anywhere in the range of 50–70%.</p><p>To support a broad range of customer needs, the cloud providers offer many flavors of committed usage. For example, AWS <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/reserved-instances-types.html">standard reservations </a>can be resold on the AWS Marketplace (providing an out if your capacity needs change) but cannot be applied to a different instance family type than the one originally specified. On the other hand, <a href="https://docs.aws.amazon.com/whitepapers/latest/cost-optimization-reservation-models/standard-vs.-convertible-offering-classes.html">convertible reservations </a>cannot be resold, but they can be modified and applied to different family types.</p><p>Both standard and convertible reservations can be further fine-tuned with <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/reserved-instances-scope.html">regional and zonal reservations </a>. The latter provides guaranteed capacity but zero flexibility in terms of modifying the reserved instance’s zone, family and size, and the former provides modification flexibility but no guarantees on capacity availability.</p><p>In 2019 AWS introduced Savings Plans where customers could commit to spend a desired amount per hour, e.g. $35/hour, for either 1 or 3 years. In this example, anything spent up to $35 would be charged in accordance with Savings Plans rates (anywhere from 26–72% savings depending on the term, instance type, etc. ). Any spend above the committed amount would be charged at on-demand rates. Similarly, Google Cloud offers their <a href="https://cloud.google.com/compute/docs/instances/signing-up-committed-use-discounts">committed use discount </a>where you commit to a specific amount of usage (e.g. vCPUs, memory, GPUs, local SSDs, etc.) for 1 or 3 years.</p><p>Google also offers a unique pricing model called to the potential savings that can be achieved by simply leveraging a <a href="https://cloud.google.com/compute/docs/sustained-use-discounts#sud_table">sustained use </a>where you can receive automatic discounts of up to 30% without any need for prior commitment, whenever you run your workloads anywhere from 25%-100% of the month. However, the savings achieved by running your compute resources for sustained periods <a href="https://www.parkmycloud.com/blog/google-sustained-use-discounts/">need to be compared </a><a href="https://help.spot.io/elastigroup-for-aws/concepts/general-concepts/scheduling/">scheduling tool</a> that will instead turn instances on and off as needed.</p><h3>Reserved capacity savings versus flexibility</h3><p>However, even with all the flexibility cloud providers have added to these pricing models, you still are to a large degree financially locked-in for the term of your commitment. If your project suddenly ends or resource requirements change, you need to pay for the compute capacity you signed up for or figure out how to best recycle the committed capacity.</p><p>This necessitates very careful planning and ongoing assessment of actual cloud compute usage across your environments to evaluate if any committed capacity needs to be modified and recycled internally or offloaded when possible (i.e. standard reservations can be sold to other customers in the AWS marketplace).</p><p>Due to this complexity it is not uncommon for companies to involve multiple teams, including engineering, DevOps and finance to project planned compute requirements and evaluate if upfront commitment is warranted from an ROI perspective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_-mUKxiBLapst5oX.png" /></figure><p>Creating a well balanced reserved capacity portfolio, such as the one illustrated above for AWS reservations and Savings Plans, requires continuous oversight that can be handled either by an expert, dedicated in-house team or 3rd-party tools for managing and optimizing committed capacity.</p><p>about the different types of reserved capacity pricing models and how to identify the commitment plan that works for you.</p><h3>Leveraging all cloud compute pricing models</h3><p>While each of the pricing models can be used separately, more often than not, companies will employ a mix of on-demand, spot instances and reserved capacity. In this situation consideration should be given to the possibility of committed to, yet unused reserved capacity.</p><p>As we mentioned earlier this reserved capacity needs to be paid for no matter what. Therefore, to maximize cost savings, every effort should be made to ensure that available reserved capacity is fully used, <strong>before</strong> using new spot instances.</p><p>In the event that spot instances or reserved capacity is unavailable, on-demand instances play a crucial role in ensuring high availability for business continuity.</p><p>This can all be handled with careful DIY configuration as well as with <a href="https://spot.io/blog/aws-spot-instances-and-diy/">turn-key cost management and workload automation solutions </a>.</p><h3>Overcoming cloud pricing model paralysis</h3><p>With a plethora of cloud compute pricing models, the process of selecting the optimal path to cloud cost efficiency is not straightforward. Even a large, expert DevOps team may be challenged in creating and managing a balanced portfolio of cloud computing resources that delivers maximum cost efficiency without losing any availability and flexibility.</p><p>This reality has spawned a whole new slew of job titles such as Infrastructure Financial Engineer, <a href="https://spot.io/blog/introducing-the-finops-engineer/">FinOps Manager</a>, Director of Cloud Capacity Manager and the like, as companies need to manage their cost more than ever. As you consider your options for optimizing your cloud, investing in cloud cost experts (whether in-house or as external consultants) as well as in 3rd party software solutions, will no doubt net you rich dividends.</p><p><em>Originally published at </em><a href="https://spot.io/blog/leveraging-cloud-computing-pricing-models-for-greater-cost-efficiency/"><em>https://spot.io</em></a><em> on July 2, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9450f14524e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/leveraging-cloud-computing-pricing-models-for-greater-cost-efficiency-e9450f14524e">Leveraging cloud computing pricing models for greater cost efficiency</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EKS simplified on EC2 Graviton2 instances]]></title>
            <link>https://medium.com/application-driven-infrastructure/eks-simplified-on-ec2-graviton2-instances-3df73254224b?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/3df73254224b</guid>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[arm-processors]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ec2]]></category>
            <dc:creator><![CDATA[Tsahi Duek]]></dc:creator>
            <pubDate>Wed, 08 Jul 2020 08:53:31 GMT</pubDate>
            <atom:updated>2020-07-08T08:53:31.322Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JBwrceUMFB54Qicm.png" /></figure><p>Amazon Web Services (AWS) and Kubernetes (K8s) have proven to be a powerful combination, with more K8s users also using Amazon Elastic Kubernetes Service (EKS) <a href="https://www.cncf.io/wp-content/uploads/2020/03/CNCF_Survey_Report.pdf">more than any other Kubernetes managed service </a>. With EKS, users are able to benefit from the agility and scalability of Kubernetes on AWS without having to manage the K8s control plane. Paired with Ocean by Spot, users are able to abstract away infrastructure management as well. Ocean provides a full data plane management platform creating a serverless container experience.</p><p>Powering all this is the EC2 instances provisioned by AWS, who are gradually making available a new generation of EC2 instances, which allow for more applications to run without the need for more CPU cores and memory, providing an ideal environment for fast-scaling applications.</p><h3>EC2 Graviton</h3><p>Custom built by AWS, and exclusively for the cloud, these EC2 instances run on an Arm architecture processor (as opposed to X86 architecture). Arm architecture is gaining popularity with many enterprises, including <a href="https://www.theverge.com/2020/4/23/21232441/apple-arm-mac-2020-5nm-processor-12-cores">Apple who has plans to adopt it for its next generation of Macbooks </a>. The first generation of these Arm-based instances-the A1 instance family-was introduced two years ago and AWS Graviton2 will now power M6g, C6g, and R6g instance family types. These instances promise better price/performance compared with the M5 instance types, and <a href="https://www.anandtech.com/show/15578/cloud-clash-amazon-graviton2-arm-against-intel-and-amd">when tested against CPU intensive workloads did result in faster performance and less costs </a>. With the addition of these compute choices, users are better able to utilize their resources more efficiently and match the right type of compute to the right workload, making these instances especially beneficial to Kubernetes and EKS users. In fact, <a href="https://docs.aws.amazon.com/eks/latest/userguide/arm-support.html">EKS provides support for Arm instances </a>with modifications of AWS-VPC-CNI, CoreDNS, and kube-proxy components.</p><h3>Building multi-architecture Docker images</h3><p>Once EC2 Arm instances have been added to the EKS cluster, some modifications have to be made to the docker build process in order to utilize the worker nodes running on these new instances. When building the docker image, an image should be built and tagged for every hardware architecture that is being run (e.g. my-docker-image:arm-v1, my-docker-image:x86-v1, my-docker-image:arm-v1.1). For Ocean users, that means that all of their Docker images (alongside with Ocean K8s controller) need to be configured to run on the Gravition Arm-based instance.</p><p>Multiple deployment files per architecture also need to be created to run the “right” image (built for the right architecture) on the target instance. Luckily, Docker released a new set of build tools that enable users to <a href="https://www.docker.com/blog/multi-arch-images/">create a single docker image with multiple architectures </a>in it. When the Docker daemon runs a container from that image, the daemon will pull only the needed layers of that docker image according to the instance architecture.</p><h3>Ocean by Spot</h3><p>A fully automated solution for managing the Kubernetes data plane (i.e. Kubernetes worker nodes), Ocean by Spot implements and automates the concept of container-driven auto-scaling, where infrastructure is scaled to meet pod requirements. Via its K8s controller, Ocean continuously monitors and reports pod metrics for resource requests and allocations, and subsequently scales infrastructure to the right size and configuration.</p><p>To enable Ocean users to leverage the new EC2 Graviton instances within their EKS clusters, Spot has put together the guidelines below.</p><ol><li>Build an EKS cluster with one node as described in the following guide: <a href="https://docs.aws.amazon.com/eks/latest/userguide/arm-support.html">https://docs.aws.amazon.com/eks/latest/userguide/arm-support.html</a></li></ol><ul><li>Make sure you’ve launched an instance and that it is connected to your EKS cluster, and that instance is of arm64 architecture.</li><li>Use the following command which should return the value of “arm64” specifying that the instance is arm64 based: kubectl get node -o jsonpath=&#39;{.items[*].status.nodeInfo.architecture}&#39;</li></ul><ol><li>Sample application with a Multi-Arch image. A simple python flask HTTP responder is used and a multi-architecture docker image is created from it. See <a href="https://github.com/tsahiduek/flask-http">https://github.com/tsahiduek/flask-http </a>for more information</li><li><a href="https://help.spot.io/getting-started-connecting-your-aws-account/">Connect your AWS </a>account to Spot SaaS platform</li></ol><h3>Connecting Ocean to the EKS cluster</h3><p>In order for the K8s controller in Ocean to run on a Graviton arm-based instance, the `docker buildx` tool is used to generate the multi-architecture docker image needed. The docker image can be found on tduek/kubernetes-cluster-controller:multiarch in Docker hub.</p><p><strong>To create your own Ocean cluster for EC2 Graviton instances:</strong></p><p>Create Ocean Log in to your Spot console, move over to the Ocean → Cloud Clusters tab and press the Create Cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*AXO7vSJ9a5G287m6.png" /></figure><p>Choose the “Migrate Worker Node” option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*kII0dNOTsq14wDLP.png" /></figure><p>Name the cluster `arm-cluster`, as well as the cluster identifier field.</p><p>Choose the Region where your cluster had been created.</p><p>In the `Import From` field, choose `Instance` and type the instance ID which had been created as part of the EKS creation process described above.</p><p>In the next part, we need to choose only EC2 Graviton instances. Click on the `Customize` button under `Machine Types` and un-select all instance types except A1 instances.</p><p>All other configurations (VPC, subnets, AMI, UserData, etc…) are automatically imported from the running instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*9IC7OxiwHkmTCwUN.png" /></figure><p>The next step is to install Ocean-Kubernetes-Controller as described above. Instead of using the script that is shown in the UI, use the following custom script. Replace the placeholders with appropriate values (Spot token and Spot account — this can be taken from the UI). The reason for this modification is because of the arm supported deployment image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*KQ9AxhkwY9mD9Rop.png" /></figure><p>The output of this script should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Z-AL8FgRG-8lfO-8.png" /></figure><p>After a few seconds, you can click on the `Test Connectivity` button and verify that the controller is running properly and all configurations successfully created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*cUyK_2EU311F5SB4.png" /></figure><p>Click on the create button and verify that the cluster created successfully</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*g9bn0oHLalRaZb3_.png" /></figure><p>After about a minute, you can see that a new instance had been created by Ocean, which is of the lifecycle of Spot and is part of the A1 instance-type family.</p><p>From here, you can successfully drain the single on-demand node, and let Ocean manage all infrastructure (data plane) of your arm-based EKS cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*IwovQ5StT8nfGZVz.png" /></figure><h3>Deploy sample application</h3><p>As described above, a super simple flask python application is used. The docker image is built using the `docker buildx` new tool (see in that repo). Use the following command file to deploy this application in your cluster:</p><pre>kubectl apply -f <a href="https://raw.githubusercontent.com/tsahiduek/flask-http/master/K8s/deployment.yaml">https://raw.githubusercontent.com/tsahiduek/flask-http/master/K8s/deployment.yaml</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*koRrVMxwFQAlHBUa.png" /></figure><p>As you can see, we have one pod in a Running state, with one node in the cluster.</p><p>Let’s simulate a scale of our application. Use the following command to simulate a scale for our deployment</p><pre>kubectl scale deployment flask-http --replicas=4</pre><p>Running kubectl get pods shows that there are pods in a Pending state because there is insufficient Cpu in the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*DMNLHaaHnzFG5K7Y.png" /></figure><p>Ocean instantaneously monitors the pending pods and reacts with automatic scaling events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Bp8ZAnyL3P7e12I3.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*h8zt1NTk4jP6P4QR.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*JdQstU0ZOeo8ZJiU.png" /></figure><p>After about 45 seconds, the new worker nodes are launched and registered to the cluster, which can now take advantage of the newly improved Graviton instances while users continue to focus on their applications while Ocean manages infrastructure.</p><p>Feedback is welcome, contact me at:</p><p><a href="https://www.linkedin.com/in/tsahiduek/">https://www.linkedin.com/in/tsahiduek/ </a><a href="https://twitter.com/tsahiduek">https://twitter.com/tsahiduek</a></p><p><em>Originally published at </em><a href="https://spot.io/blog/eks-simplified-on-ec2-graviton2-instances/"><em>https://spot.io</em></a><em> on June 22, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3df73254224b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/eks-simplified-on-ec2-graviton2-instances-3df73254224b">EKS simplified on EC2 Graviton2 instances</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud cost optimization with 2020 hindsight]]></title>
            <link>https://medium.com/application-driven-infrastructure/cloud-cost-optimization-with-2020-hindsight-a5d0c4fbfdbe?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/a5d0c4fbfdbe</guid>
            <category><![CDATA[cloud-cost-optimization]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[cost-management]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[finops]]></category>
            <dc:creator><![CDATA[Zev Schonberg]]></dc:creator>
            <pubDate>Wed, 08 Jul 2020 08:52:10 GMT</pubDate>
            <atom:updated>2020-07-08T08:51:04.084Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O5NUq3Arx2HJC1nD.jpg" /></figure><p>The first half of 2020 has turned “business as usual” on its head, underscoring the criticality of both the cloud and cloud cost management in our digital era.</p><p>For the last 16 years, the world of IT has been gradually moving from on-premise data centers to the public cloud, with SMBs and enterprises alike, drawn to the promise of scalability and potential cost-savings. But the accelerated rate of consumption and pace of adoption in the first half of 2020 is primarily due to the volatility COVID-19 engendered.</p><p>One of the most prominent examples of increased demand for cloud is Zoom. With lockdowns across the globe, schools, businesses and other groups drove a <strong>2900%</strong> increase in daily Zoom usage in just a couple of months . Thousands of other businesses and services in industries such as entertainment, health, finance, education and others, have also experienced massive and unexpected surges in online demand due to shelter-in-place and other quarantine efforts . Many, if not all, major online services are leveraging public cloud resources through providers such as AWS, Azure and GCP, to power their offerings. To support their customer’s needs during the pandemic, vendors scaled up the usage of cloud services quickly.</p><p>In terms of business continuity, companies with traditional data centers, even if they didn’t actually experience outages or delays during the pandemic, are realizing that owning and maintaining physical hardware can be a liability, with lockdowns preventing key IT staff getting to work. Industry experts predict this will drive increased adoption of the cloud .</p><p>With sustained spikes in usage, there are no doubt also sustained spikes in cloud cost. In theory, the pay-as-you-go model that public cloud providers typically offer can help keep costs low, however the reality is that most companies grossly overspend and many organizations that are already using the cloud still see cost management as a significant pain point .</p><h3>The cloud bill is delivered to all</h3><p>While one might think that increased customer demand for online services completely offsets increased cloud costs, the fact is greater user consumption does not always translate into increased profitability. Case in point is Zoom. Thanks to 290 million additional daily users, Zoom reported a 169% increase in Q1 revenue (in comparison to Q1 2019), yet saw their margins shrink by 14% due to increased spending on AWS and other cloud infrastructure to support free users .</p><p>On the flip side, online businesses who saw decreased demand, traffic and revenue during COVID-19, no doubt are viewing cloud cost management, not as a nice-to-have but as critical in ensuring their long-term viability.</p><p>Whether the business volatility of the last six months has been good or bad for your business, it’s clear that cloud computing is becoming ubiquitous, and while it plays a major role in delivering services to customers, the cloud can still come with a significant price tag if resources aren’t managed properly. As such, it’s imperative for any company using the public cloud to properly plan and prioritize cloud spend optimization as part of their digital transformation strategies to ensure continued business success.</p><h3>Getting started — know thyself, know thy cloud</h3><p>“If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.” <strong>Sun Tzu, The Art of War</strong></p><p>Successful cloud cost management starts with an understanding of your organization’s needs. Before you can appropriately apply cloud resources to different areas of your business, you first need to answer questions like what applications you expect to scale, or where are there gaps in your workflows. From there, it’s about mapping the right resources to the right places, creating an implementation plan and sticking to it. As you continue to expand cloud operations into other areas of your business, this process is repeated and continuously improved.</p><p>In practical terms, this would entail some, if not all of the following activities: :</p><ul><li>Interviewing all product and engineering teams to understand their current, near-term and future projects to anticipate their needs for cloud resources</li><li>Reviewing engineering’s projects with your DevOps, SREs, TechOps or FinOps teams to verify requirements for cloud architecture, associated infrastructure and budget.</li><li>Verifying with the Finance team that the requested cloud spend is in line with your company’s budget.</li><li>Ensuring that measures for governance are established as an integral part of any technical roll-out. For example, open source tools such as <a href="https://cloudcustodian.io/">Cloud Custodian</a> offer a comprehensive <a href="https://cloudcustodian.io/docs/aws/examples/index.html">library of scripts</a> for reporting on and enforcing compliance with fundamental aspects of cloud management such as new resource labeling or tagging .</li><li>Gaining comprehensive visibility of all cloud spend by leveraging cloud providers’ native tools such as <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/">AWS Cost Explorer</a>, <a href="https://cloud.google.com/billing/docs/how-to/reports">Google Cloud Billing Report</a> and <a href="https://azure.microsoft.com/en-us/services/cost-management/">Azure Cost Management</a>. For multi-cloud cost visibility, 3rd party solutions such as Spot’s <a href="https://spot.io/products/cloud-analyzer/">Cloud Analyzer</a> offer a single pane of glass view for multiple cloud providers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fw1boxtoeDA-b5VT.png" /></figure><h3>Prioritizing your cloud cost management focus</h3><p>In this blog series we will be focusing on four major areas of cloud resources that can be optimized:</p><p>All of these are important and depending on your particular cloud environment, some areas may be more relevant than others. However, with over 60% of all cloud costs attributable to compute , we have decided to start with how to optimize compute infrastructure spend. Of equal significance, optimizing compute infrastructure spend has the potential to yield the greatest savings in comparison to all other cloud resources.</p><p>In the <a href="https://spot.io/blog/leveraging-cloud-computing-pricing-models-for-greater-cost-efficiency/">next blog post</a> of this series, we will focus on how to leverage public cloud pricing models for maximum cost savings.</p><p><em>Originally published at </em><a href="https://spot.io/blog/cloud-cost-optimization-with-2020-hindsight/"><em>https://spot.io</em></a><em> on June 12, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5d0c4fbfdbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/cloud-cost-optimization-with-2020-hindsight-a5d0c4fbfdbe">Cloud cost optimization with 2020 hindsight</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the FinOps engineer — The Spot.io Blog]]></title>
            <link>https://medium.com/application-driven-infrastructure/introducing-the-finops-engineer-the-spot-io-blog-b555b8c7aef2?source=rss----a784a32a40f5---4</link>
            <guid isPermaLink="false">https://medium.com/p/b555b8c7aef2</guid>
            <category><![CDATA[cloudfinancialengineering]]></category>
            <category><![CDATA[wix]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[cloudfinancialmanagement]]></category>
            <dc:creator><![CDATA[Dvir Mizrahi]]></dc:creator>
            <pubDate>Thu, 11 Jun 2020 11:47:32 GMT</pubDate>
            <atom:updated>2020-06-11T11:47:32.817Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing the FinOps engineer — The Spot.io Blog</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MxS2vKYGavSzpRP3.jpg" /></figure><p>It’s 2020 and running infrastructure on the cloud has become quite standard for many companies. This is underscored by a <a href="https://www.canalys.com/newsroom/canalys-worldwide-cloud-infrastructure-Q4-2019-and-full-year-2019">Canalys report </a>that shows 2019 global cloud spend at $107 billion. This spend on cloud services is spread across AWS with 32.4% of the market, Azure at 17.6%, Google Cloud at 6%, Alibaba Cloud at 5.4%, and other smaller cloud players with a combined 38.5% of market share.</p><p>Yet despite all the growth, there are many challenges deploying infrastructure on the cloud, of which perhaps the biggest (after typically receiving increasingly large cloud bills) is the financial aspect.</p><p>In this post, based on our experience at <a href="http://wix.com/">Wix</a>, we will introduce the role of FinOps engineers along with some of the technical and financial approaches they employ for successful cloud cost management.</p><h3>What really is driving your cloud bill sky-high</h3><p>It has been said that the cost of your cloud infrastructure is not a derivative of the number of customers you have, but rather the number of engineers. The mere introduction of a new feature, even without any additional customers, can significantly increase costs simply because of its architectural design, resource utilization and the like.</p><p>To ensure that rapid innovation and increasing scale remains within budget, a new role has been born, that of the FinOps engineer. The FinOps engineer combines intimate knowledge of software development requirements combined with an expertise in finding optimal and cost-efficient solutions for deployment in the cloud.</p><p>As this field is growing and starting to better define itself, we can point to the main areas of responsibility of the FinOps engineer:</p><ol><li>Creating full visibility for any financial aspect related to the company’s hosting solutions</li><li>Creating governance on spend on a daily basis</li><li>Reviewing architectures and deployment topologies</li><li>Creating optimization tasks and data driven recommendations</li><li>Working closely with the finance team for budgeting and forecasting activities</li><li>Establish and maintain the connection between the operations, procurement and finance teams</li></ol><p>These FinOps activities require companies to culturally adapt as they have in the past for DevOps and agile development methodologies.</p><p>The goal for FinOps engineers is to create visibility in the organization for our cloud spend, lead the technical optimization to improve performance and efficiency and reduce the cost while scaling activity up. For that FinOps engineers need to work on two fronts:</p><ul><li>Architecture and deployment topology</li><li>Financial tools and reports</li></ul><p>Let’s explore each one.</p><h3>Cloud architecture, deployment topology and how money makes all the decisions</h3><p>When we approach the exciting assignment of deploying (or reviewing) our infrastructure in the cloud, we often face a great number of options. With the phrase “anything is technically possible” often thrown around, it is sometimes very hard to understand and choose the right architecture for your organization. Common practice in the cloud is to choose between dozens of instance types and sizes, several storage options, different network architectures based on regions and availability zones, and much more (e.g. managed services, licenses, marketplace offerings etc.).</p><p>Surprisingly, when we add cost as a KPI for cloud architectural excellence, things become a lot clearer when choosing which tools we want to use. Let’s look at some examples.</p><h3>Choosing the right EBS volumes</h3><p>When you need fast storage for your instance, many folks will tell you to use provisioned IOPS SSD (io1) volumes. If you require only the IOPs and can live with less throughput, deploying a larger GP2 volume that supports the IOPs you need, will always be more affordable.</p><h3>Choosing the right instance types</h3><p>Sometimes with the huge selection of instances, we can get lost in choosing the right one for us. Things get more complicated when we’re not just looking at the general information of the instance (number of cores, RAM, IO) but rather the actual hardware and performance metrics to check how well the instance performs.</p><p>But with patience and research, you can find the computing power you need, at the right price. For example, when AWS released the c5 family we got really excited as it was not only 15% less expensive than the c4 family, it also came with Skylake CPUs, more memory compared and better IO. In practical terms for one thousand c5.2xlarge hosts deployed, you could save in comparison to the older c4.2xlarge over on-demand prices over $40,000 monthly. And that’s without even talking about reserved instances.</p><p>Another incredible example we can talk about is the release of the c5n type that can reach up to 25 Gbps, something that was barely possible on the biggest sizes of previous generations.</p><p>For simulations, in-memory caches, data lakes, and other communication-intensive applications the savings are absolutely astronomical. For example, you can enjoy even better performance with the c5n.large than you would with the older c4.8xlarge workloads AND you would pay 94% less, with the c5n.large costing only around $79/month and the c4.8xlarge still super pricey at around $1,164/month (in US East-1).</p><h3>Choosing the right locations</h3><p>Here things get a bit trickier, as it’s no longer only dependent on technical and financial factors. Things like proximity to your users, GDPR regulations, privacy, security, on-premise locations that need a connection to the cloud and other issues, all need to be considered.</p><p>While cloud providers offer several “regions” where you can deploy, each region typically has different pricing. Sometimes deploying the exact same infrastructure in a specific region can translate into thousands of dollars in monthly savings. Within regions themselves, there are usually multiple availability zones, each representing a different AWS data center.</p><p>Based on this, there are several architectures you can use, such as “single availability zone — multiple regions”, “multiple availability zone — single region”, “multiple availability zone — multiple regions” and the like. When you’re deploying your infrastructure based on these architectures you need to also consider the availability of your services, how fast you can shift traffic, the hardware capacity of the desired availability zone and more.</p><p>For example, running on a single availability zone might damage your HA architecture as you might face a capacity limit or availability zone downtime, but the financial benefits can be really great as traffic in the same availability is free — definitely something to consider.</p><p>Let’s now take a look at some available cloud tools and reports for better managing your cloud costs.</p><h3>Cloud knowledge is cloud power</h3><p>To gain visibility into your cloud spend you can use several tools and reports to investigate. I’ll be focusing on AWS and GCP as Wix is mainly working with these two vendors.</p><h3>AWS Tools</h3><p>You have “out-of-the-box” tools such as Cost Explorer to show graphs and trends, Trusted Advisor which provides right-sizing recommendations and other cost recommendations such as purchasing plans (reservations, Saving Plans, spot instances, etc.), budgets alerts and more.</p><p>The most powerful tool you can use for cost management are tags. Cost allocation tags can create complete visibility on your infrastructure, separating a single account activity into business units, service descriptions, agenda owners, and more. You can enable the tags that are interesting for you in the cost reports thereby gaining huge insights into your organization’s cloud activities.</p><p>For more technical engineers that want to automate things, you have the cost explorer API, and SDK libraries supporting Go, Python, PHP, Ruby, .NET, Java and more to connect to your EC2 activities, CloudWatch metrics, etc.</p><h3>AWS Reports</h3><ol><li>DBR or Detailed billing report — this is the legacy report that contains your usage information</li><li>CUR or Cost and allocation report — this is the main billing report that contains not only usage and cost information, but also technical information as well (from the CPU clock speed and the number of virtual cores), as well as details on any existing reservations and Saving Plans.</li></ol><p>Both reports are exported to S3 and can be used as a data source for your own dashboarding tool.</p><h3>GCP Tools</h3><p>Google has a “Compute Engine rightsizing recommendation” out of the box and a “Report” page with a utilization graph where you can see your activities. It also has different purchasing plans like committed usage (similar to AWS’s reservations and Savings Plans) and Preemptible machines (like AWS’s spot instances). You can manage your budget and alerts with automated budget actions, billing export and billing APIs.</p><p>Google also supports tags (known as labels in GCP), but has somewhat limited capabilities as Keys and values can contain only lowercase letters, numeric characters, underscores, and dashes. All characters must use UTF-8 encoding, etc.</p><p>Google also has client libraries integrated with the major services of GCP supporting Go, Java, Node.js, Python, Ruby and more.</p><h3>GCP Reports</h3><p>GCP doesn’t support a traditional report object, and the billing information is written to BigQuery among several billing tables that are connected. If you want to use all the detailed information on your billing you’ll have to query BQ to get the relevant billing information.</p><h3>FinOps is forever</h3><p>FinOps (or how we call it at Wix — Cloud Financial Engineering) is not a one-time effort, but an ongoing, challenging, day-to-day effort.</p><p>We have just done a basic overview of some the most common questions you need to ask when evaluating your deployment architecture. As we have shown, it’s not enough just knowing what will solve your problem technically. You really need to think about what is the most cost-effective solution available before making a decision. Likewise, we have covered some of the more basic tools and reports available that you need to be familiar with to understand your cloud spend.</p><p>But this is just the beginning. The value that a FinOps engineer brings to a company is huge and plays an integral part of either a CCOE (cloud center of excellence) or any other team dedicated to monitor, govern and set KPIs for organizations’ cloud expenses.</p><p>The best part is that you don’t need to know it all or do it all yourself.</p><p>Many companies, such as <a href="http://spot.io/">Spot.io</a> can provide the knowledge and automations to help you succeed in this journey, from creating deep cost visibility to actually running your infrastructure for you in the most optimal way, so you can enjoy the savings without having to worry about availability or needing to get involved in managing all the underlying infrastructure yourself.</p><p>If you have read this post until here, you have already taken your first step in your journey to FinOps excellence. Safe travels!</p><p><em>Originally published at </em><a href="https://spot.io/blog/introducing-the-finops-engineer/"><em>https://spot.io</em></a><em> on May 24, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b555b8c7aef2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/application-driven-infrastructure/introducing-the-finops-engineer-the-spot-io-blog-b555b8c7aef2">Introducing the FinOps engineer — The Spot.io Blog</a> was originally published in <a href="https://medium.com/application-driven-infrastructure">The Spot to be in the Cloud</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>