<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Remy DeWolf on Medium]]></title>
        <description><![CDATA[Stories by Remy DeWolf on Medium]]></description>
        <link>https://medium.com/@remy-dewolf?source=rss-4a3788ab7bf2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1OEkSXYsoLdISrKv_yjtxg.jpeg</url>
            <title>Stories by Remy DeWolf on Medium</title>
            <link>https://medium.com/@remy-dewolf?source=rss-4a3788ab7bf2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:48:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@remy-dewolf/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Speed, Security, and Scalability: Pick Only 2!]]></title>
            <link>https://medium.com/better-programming/speed-security-and-scalability-pick-only-2-5e61c637b08e?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e61c637b08e</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Wed, 06 Apr 2022 17:16:06 GMT</pubDate>
            <atom:updated>2022-04-06T17:16:06.106Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How do you measure the efficiency of an engineering organization?</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KgIa2qVX5zZTQNoQ4Nla3Q.jpeg" /><figcaption>Photo credit <a href="https://www.instagram.com/bolovtsova/">bolovtsova</a></figcaption></figure><p>This article will break down this question into three attributes: Speed, Security, and Scalability (and call them the 3S).</p><p>First, we will describe some of the challenges engineering teams face when they simultaneously try to be great at the 3S.</p><p>Next, we will review how companies of all sizes might change their strategy to adapt to their business needs. And finally, we will look at a few interesting trends that could help some engineering teams in their journey.</p><h3>What is 3S?</h3><h4><strong>Speed</strong>: The time it takes for an idea to go to market</h4><p>Metrics: Time to market, number of features shipped.</p><p>Traits observed:</p><ul><li>Individual contributors have a lot of autonomy and can move fast</li><li>Fewer processes and checks when making changes</li><li>Innovation is a differentiator.</li><li>It can lead to feature creep and tech debt.</li></ul><h4><strong>Security: T</strong>he process of incorporating security controls into the information system</h4><p>Metrics: Number of security incidents, weighted by severity, Compliance achievements.</p><p>Traits observed:</p><ul><li>Individual contributors have shared ownership and keep each other accountable</li><li>Many processes and checks when making changes</li><li>Innovation could be seen as a risk</li><li>Compliance and security requirements become key projects, taking over the product roadmap at times</li></ul><h4><strong>Scalability: </strong>The ability of a system to change in size or volume</h4><p>Metrics: Performance and Uptime metrics</p><p>Traits observed:</p><ul><li>Individual contributors rely on tools and automation. Setting up this automation can be complex and time-consuming in the initial stages</li><li>Increased automation</li><li>Infrastructure as code</li><li>Multi-region, Multi-cloud</li><li>Tech debt can impair scalability</li></ul><h3>Why trying to excel at the 3S can be counterproductive?</h3><p>While these three attributes are well-intentioned, they might sometimes act as opposing forces. Let’s provide a few examples.</p><p><strong>Speed working against Security:<em> </em></strong><em>To prevent developers from being blocked, one might give them full access to some systems. But that would also go against the security requirements.</em></p><p><strong>Scalability against Speed:<em> </em></strong><em>One scalability goal could be to use multiple regions, possibly multi-cloud providers. But that would make the release and deployment process much more complicated for developers. This also creates a new problem; the application data is now stored in many places.</em></p><p><strong>Security against Scalability:<em> </em></strong><em>To improve security, every new system or library might require an approval process before using it. But that could mean introducing a new tool or technology to improve scalability (ex: Kubernetes) is slower. Furthermore, it can create deadlocks where innovation is a risk in the worst case.</em></p><p>We could provide more arguments on where to put the cursor and lead to emotional discussions for all of these examples. The answer might be different for each company. But the point is made, these can be opposing forces. If a business has limited resources, it might want to make an educated decision about where it would like to put its focus to reduce these friction points.</p><p>Introducing unnecessary complexity when the business doesn’t need it can be damaging. Maybe the current team does not have all the skills required, so they will need to ramp up. It might slow down the ability to go to market. Some teams would work on some projects that turned out to be not required. Having opposite forces might lead to heated discussions and delay decisions. Ultimately it could negatively impact morale and lead to attrition.</p><p>We can parallel the <a href="https://en.wikipedia.org/wiki/CAP_theorem">CAP theorem</a> and state that an engineering team cannot simultaneously excel at Speed, Scalability, and Security to push this even further.</p><blockquote>The <strong>CAP</strong> theorem states that a distributed system cannot simultaneously be <strong>C</strong>onsistent, <strong>A</strong>vailable, and <strong>P</strong>artition tolerant.</blockquote><p>Big disclaimer: it’s not about dropping one attribute but accepting that the focus should be on the two other attributes at the current stage. The engineering team will be more focused by clearly stating what the company needs now and communicating about the accepted tradeoffs. A great leader would anticipate a step further in their message and explain how they would address today’s tradeoff in the future. And prepare accordingly when the time approaches.</p><h3>Which 3S to pick?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-FBHoCtyPzk0VeifmQhQQ.png" /></figure><p>Early-stage startups value <em>Speed</em> before anything else. They want to find their market and keep their customers happy by shipping new product features at a high pace.</p><p>Some industries such as Finance or Healthcare might dictate that <em>Security</em> alongside <em>Speed</em> is the top attribute to focus on.</p><p>More prominent startups experiencing high growth would usually favor <em>Scalability and Speed</em>. If they compete in a winner-takes-all market, that strategy will allow them to execute on their product roadmap and reach out to new markets faster. If they are preparing for IPO or expanding globally, their compliance requirements will make them pivot to focus on <em>Security and Scalability</em>.</p><p>Finally, big tech companies with years of experience had time to learn and went through the many stages themselves. As a result, they have some of the best and largest engineering teams; they excel at the 3S.</p><p>This is also not uncommon for large companies to spin up a unique project into an independent entity to enable it to move faster, sometimes also referred to as an Internal Startup.</p><h3><strong>What’s next for the 3S?</strong></h3><p>Nobody can predict the future, but we can observe some trends in the market and report some of the feedback we hear from our users.</p><h3><em>Speed: hiring is the bottleneck, a new wave of startups coming in</em></h3><p>Early-stage funding is at an all-time high, and it’s a race for these startups to find their market. The number one impairment to speed this day seems to be tech hiring. Because hiring is slow, they want to do more with fewer people. A new wave of PaaS and software simplifies everyday engineering tasks.</p><p>Simplifying cloud computing is our mission for <a href="https://github.com/run-x/opta">Opta</a>, but there are many other new startups whose goal is to help developers. We have a few exciting examples among our users: <a href="https://blog.runx.dev/how-the-canvas-team-uses-opta-to-make-data-easier-to-explore-f5615647cc43">Canvas</a>, which brings SQL to non-technical teams, and <a href="https://union.ai/">Union.ai</a>, which simplifies machine learning.</p><h3><em>Scalability: Kubernetes adoption, multi-cloud, and demand for opinionated frameworks</em></h3><p>There is a big industry trend to move to Kubernetes. But it comes with <a href="https://blog.runx.dev/3-things-i-hate-about-kubernetes-49f1656baeaa">some challenges</a> and frequent upgrades (<a href="https://blog.runx.dev/will-that-kubernetes-v1-22-upgrade-break-my-application-cc339dc2e2c7">not all backward compatible</a>). Using Kubernetes is a journey, not a destination. This complexity is sometimes misunderstood and can negatively impact the product roadmap by holding engineering resources.</p><p>The latest <a href="https://www.hashicorp.com/state-of-the-cloud">HashiCorp State of Cloud</a> survey reported that multi-cloud is now the new normal. Cloud providers are very different, and supporting many will create much more work for the engineering team. Our customers want Kubernetes on their cloud provider but don’t necessarily have a strong opinion on how they should configure it. They want best practices, such as application monitoring and Infrastructure-as-Code.</p><h3><em>Security: help needed with Kubernetes networking and IAM</em></h3><p>Here again, moving to Kubernetes creates more complexity. Most cloud providers provide a service to run a cluster, but it’s up to the user to set their network (ingress, service mesh) and IAM for their cluster. Some companies start with Kubernetes; they don’t want to start from a blank state. They appreciate using products that can provide an opinionated way to configure their networking stack, from their Load Balancer to the application pods running on their cluster.</p><h3>What we have seen</h3><p>This article looked at engineering efficiency by considering the 3S: Speed, Security, and Scalability. For many companies, trying to excel on all 3S too early can create unnecessary complexity, increasing the tech debt. This could ultimately create frustration for the engineering team and lead to attrition.</p><p>Depending on the stage and the market where the business operates, it might make sense to focus on one or two attributes simultaneously. Communicating clearly about the current strategy and the tradeoffs will help the engineering team stay focused.</p><p>The latest trends, such as the tech hiring crisis and the popularity of Kubernetes, have left some companies turning to new products which simplify their engineering tasks so they can execute on their roadmap with fewer hires.</p><h3>How to Manage your Team’s Infrastructure Easily</h3><p><a href="https://github.com/run-x/opta">Opta</a>, our free, open-source tool, reduces most of the complexity of cloud infrastructure and pipelines for your team. It is an Infrastructure-as-Code framework built on top of Terraform that lets you work with high-level constructs to set up custom cloud infrastructure quickly.</p><pre><strong>Want to Connect?</strong></pre><pre>If you run into any issues in the installation stage, you can reach out to the contributors in <a href="https://slack.opta.dev/">our slack channel here</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e61c637b08e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/speed-security-and-scalability-pick-only-2-5e61c637b08e">Speed, Security, and Scalability: Pick Only 2!</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Golden Path for Kubernetes Networking]]></title>
            <link>https://blog.runx.dev/the-golden-path-for-kubernetes-networking-66c21a308c7d?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/66c21a308c7d</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[linkerd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[service-mesh]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Thu, 31 Mar 2022 17:22:14 GMT</pubDate>
            <atom:updated>2022-04-08T23:02:45.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WIPPxNJnY4apl3ry7nK-bA.jpeg" /><figcaption>photo credit <a href="https://www.instagram.com/withluke/">withluke</a></figcaption></figure><p><em>TL;DR: this post simplifies Kubernetes networking by presenting significant concepts and providing a recommended architecture that we call the Golden Path at the </em><a href="https://github.com/run-x/opta"><em>Opta</em></a><em> team.</em></p><h4>HTTP Request Journey</h4><p>Let’s take the typical example of a person clicking on their profile page on a website to dive in; we assume that Kubernetes is being used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-TVRAgkZEX7BClZ2mF_nA.png" /><figcaption>HTTP request journey</figcaption></figure><p>In this diagram, the request gets sent through the Internet to a Load Balancer <strong>(1)</strong>, then to an ingress controller pod running on a Kubernetes cluster <strong>(2)</strong>. Finally, one container running a web app receives the request <strong>(3)</strong> and is tasked to respond. To build the response, that container reaches out to an API service <strong>(4)</strong>, which connects to a database not hosted in Kubernetes <strong>(5)</strong>. The web application also publishes some metrics to a web analytics service that provides a public API on the Internet <strong>(6, 7)</strong>.</p><p>While this example is a widespread use case, it involves many different network connections with other network properties. Let’s review them in the following table.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a336b65198e4ded14772c7e925a21449/href">https://medium.com/media/a336b65198e4ded14772c7e925a21449/href</a></iframe><p>Now that we understand the various types of connections let’s list the components used and their function.</p><h3><strong>What You Need</strong></h3><ul><li><strong>Cloud Provider</strong>: A third-party company offering a cloud-based platform, infrastructure, application, or storage services. You pay only for the number of cloud services you use.</li><li><strong>Load Balancer</strong>: This is a specialized cloud service to prevent server overload by distributing the traffic across multiple servers.</li><li><strong>Kubernetes</strong>: The leading platform for managing containers.</li><li><strong>Ingress Controller</strong>: This is another load balancer that distributes the traffic inside the Kubernetes cluster. We will review this one later.</li><li><strong>Kubernetes Service</strong>: A logical abstraction for a deployed group of containers.</li><li><strong>Service Mesh</strong>: A software architectural pattern used for micro services deployments to enable secure, fast, and reliable service-to-service communications. We will come back on this one as well.</li><li><strong>Other Cloud Services: </strong>Most likely, you will need services not hosted in Kubernetes. For example, a database. This is relevant when we look at networking configuration later.</li></ul><p>Let’s review some of the most confusing parts of this architecture.</p><h3>1. What is the difference between the Load Balancer and the Ingress Controller?</h3><p>These two components serve a similar purpose, distributing the traffic, so why do we need both?</p><p>Let’s imagine for a second that we skip the first Load Balancer:</p><ul><li>All public traffic will come directly to the Kubernetes cluster, and the Ingress Controller pod will need to have a public IP, which means your cluster is now public.</li><li>The IP of an Ingress Controller pod can change over time (when the worker node is replaced), meaning that the public DNS would need to be updated when it happens.</li><li>Auto-scaling / High-Availability: By default, a cloud-native Load Balancer will scale according to traffic and use multiple availability zones. While it’s recommended that you configure your Ingress Controller to achieve this, it requires some work.</li></ul><p>So by acting as a buffer between the Internet and your Kubernetes cluster, the Load Balancer improves the security of your cluster by keeping it private and provides some advanced features that make it easy to operate, such as public DNS/IP, Auto-scaling, and High-Availability.</p><h3><strong>2. Do I need a Load Balancer for every service to be exposed or one per k8s cluster?</strong></h3><p>This is another super frequent question and a confusing one. My take is that the cloud-native Load Balancers (LB) can scale very well (it’s automatic!) so having one public Load Balancer works for most cases. Also, spawning an LB for each service in a micro services architecture would be expensive. So unless you have a specific infrastructure or security requirement, my recommendation is to keep it simple with one LB per cluster.</p><h3>3. Why do I need a Service Mesh?</h3><p>Again, I would like to take the angle of what happens if you don’t have a service mesh.</p><ul><li>Each service needs to be coded with some logic to manage service-to-service communication. This will include features like <strong>retry, timeout</strong>.</li><li>Providing <strong>consistent network metrics</strong> for the service to service might be challenging. From a network perspective, what happens inside your cluster might be hard to expose if the services manage all the communications directly. You can hack this by adding HTTP headers, but this is extra work.</li><li>If you want <strong>TLS between two services</strong>, you are now in charge of generating, deploying the TLS certificates, updating the code to use TLS, and importing the self-signed certificates. What about rotating the certificates?</li></ul><p>These are some of the most common problems that a service mesh solves. Configuring a service mesh is <em>not</em> easy, but it is done in one place. Your application code doesn’t need to be aware of these networking features. You might be fine without a service mesh if you have a few or some uniform services (one programing language — with a standard network library). But sooner or later, you might want something from the <a href="https://servicemesh.es/">long list of features</a> that a service mesh provides.</p><h3>What is in the Golden Path?</h3><p>Now that we have covered the basic networking architecture for Kubernetes, and brought some clarity about the different components and their function, let’s make a recommendation. But, again, take this with a grain of salt. Please feel free to pick and choose what works for you.</p><p>Let’s define the goals that we want to achieve with this solution:</p><ul><li>First, implement network best practices for security, resilience, and scalability.</li><li>It can be deployed on major clouds.</li><li>Favor use vs. build — to get started fast.</li><li>Favor simplicity over complexity — to lower operational cost.</li></ul><p>Ok, so now, let’s see what the <a href="http://github.com/run-x/opta">Opta</a> team has selected for the Golden Path.</p><ul><li><strong>Cloud Provider</strong>: Most engineers can’t pick the cloud provider, and it’s not easy to change. At this time, most cloud providers provide similar services. Thankfully, Kubernetes provides some abstraction over the cloud provider, so we can make a recommendation that works on any major cloud provider. This also means that we won’t recommend a component that would only work on a specific cloud.</li><li><strong>Load Balancer:</strong> Don’t reinvent the wheel; use a cloud-native Load Balancer service. You will get all the main features from day one, such as auto-scaling and high availability.</li><li><strong>Kubernetes</strong>: Again, for 99% of users, you don’t want to manage the Kubernetes cluster yourself; use a managed service like EKS, GKE, AKS. These services come with some components to control the worker nodes as well. The more you can use these managed services, the less work for yourself. Most of the cost of operating a Kubernetes cluster is the compute cost(VMs). Using these managed services is a marginal cost in comparison. And it will save you a lot of time.</li><li><strong>Service Mesh</strong>: Here is the first extensive choice. Let’s proceed by elimination. There is some service mesh developed by some cloud providers (Open Service Mesh and AWS App Mesh) that we eliminated since we don’t want to make a recommendation specific to a cloud provider. As a security best practice, we want mTLS support; therefore, Traefik mesh was not considered. From this point, we have two categories of service mesh. On one side, we have Istio and Consul providing many advanced features, which result in more complexity and require more compute resources to operate. On the other side, Linkerd has fewer features but is optimized for performance and usability. Linkerd has some missing parts; it can’t work with the non-Kubernetes workload, Header-based routing and Delay Injection are not supported. The first two are coming in <a href="https://linkerd.io/2021/12/29/the-service-mesh-in-2022/">2022</a>. These are arguably advanced features that many users won’t use. However, it does everything else (see this <a href="https://servicemesh.es/">comparison table</a>). Linkerd <a href="https://linkerd.io/design-principles/">design principle</a>s to minimize resource requirements work very well when you think that a service mesh relies on deploying a proxy alongside every application container. Our pick is <strong>Linkerd</strong>.</li><li><strong>Ingress Controller</strong>: Linkerd doesn’t come with an Ingress Controller. It supports many; see the list <a href="https://linkerd.io/2.10/tasks/using-ingress/">here</a>. Our pick is <strong>Nginx</strong>: it’s pretty easy to use and is <a href="https://www.nginx.com/blog/inside-nginx-how-we-designed-for-performance-scale/">very performant</a>. There is no wrong choice here, but Nginx, the most popular, made sense for us to select it.</li></ul><p>Other notable choices for the Golden Path:</p><ul><li><strong>Subnets: </strong>The subnets are<strong> split into Public and Private</strong>. Only the Load Balancer and the Internet Gateway require a public IP; everything else (Kubernetes, Databases) is private. In a micro services architecture in Kubernetes, every service will have many pods. Each Pod uses an IP address. This translates into a high number of IPs required. <strong>Sizing your subnets for growth</strong> will prevent your subnets from running out of IP addresses. Use CIDR prefix /20 and lower. (see <a href="https://kb.wisc.edu/ns/page.php?id=3493">table</a>)</li><li><strong>High Availability: </strong>There are many strategies to achieve HA: Multi-cloud, Multi-region, Multi-zones. Most cloud providers guarantee availability for 99.99% in any given region — roughly 5 min of downtime per month. This should work for most users, so we went with <strong>Multi-zones</strong>, the easiest and cheapest to achieve. We run Nginx and Linkerd in high availability (HA) mode. For every Kubernetes workload, we ensure a minimum of 3 pods to cover each zone. We also configure an Ingress Gateway in each zone.</li><li><strong>Infrastructure as Code</strong>: Use Terraform or a Terraform compliant tool. This one is pretty straightforward; the Golden Path should be easily duplicated and distributed. We use <a href="https://github.com/run-x/opta">Opta</a>, a Terraform compliant tool, which implements this network architecture natively (see <a href="https://docs.opta.dev/features/networking/network_overview/">documentation</a>). Opta can be used to provision this infra stack on the cloud provider or generate the terraform files.</li></ul><p>Here is the same diagram we had earlier but it now includes the choices listed in the Golden Path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzpCSSvRoXTYNpjZ7SSIDQ.png" /><figcaption>The Golden Path for Kubernetes Networking</figcaption></figure><p>The main features are:</p><ul><li>Public and private subnets to ensure isolation.</li><li>High availability to provide resiliency in case of an availability zone outage. 99.99% SLA, guaranteed by the major cloud providers.</li><li>Service mesh features are Automatic mTLS, Load Balancing, Retries and Timeouts, Traffic Split, etc.</li><li>Simplicity and Resilience: This solution is pretty straightforward to operate. The cloud provider manages the Load Balancer, the Database, and the Kubernetes cluster. Both Nginx and Linkerd are optimized for performance and usability.</li></ul><h3>What’s missing?</h3><p>You might also consider a few add-ons such as:</p><ul><li><strong>A content delivery network (CDN)</strong> to serve assets needed for loading Internet content, including HTML pages, javascript files, stylesheets, images, and videos. Each cloud provider would have such a service, and they are cheaper to operate than going to your application to serve these static assets.</li><li>Configure <strong>Network Policies</strong> within the Kubernetes cluster: By default, Linkerd allows all traffic to transit in the mesh, but this can also be configured; see this <a href="https://linkerd.io/2.11/features/server-policy/">guide</a>.</li></ul><h3>Finally, what would you have done differently?</h3><p>Now that we have presented the Golden Path for Kubernetes Networking. <strong>What else do you think should be considered?</strong> <strong>Do you have some experience that you would like to share?</strong></p><p>Please add your comments below. In addition, other folks from the Opta team or I will be happy to discuss these topics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66c21a308c7d" width="1" height="1" alt=""><hr><p><a href="https://blog.runx.dev/the-golden-path-for-kubernetes-networking-66c21a308c7d">The Golden Path for Kubernetes Networking</a> was originally published in <a href="https://blog.runx.dev/">Run[X]</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Generate Terraform Code with Opta]]></title>
            <link>https://blog.runx.dev/how-to-generate-terraform-code-with-opta-f255f71c73d3?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f255f71c73d3</guid>
            <category><![CDATA[opta]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 16:33:04 GMT</pubDate>
            <atom:updated>2022-03-18T16:33:04.450Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*49DiVE-Zh2ke3NQY1QSuNg.jpeg" /></figure><p><em>TL;DR: In this article, we will review how you can use this feature, look at the generated files and run through an example.</em></p><h3>What is Opta?</h3><p>Opta is an Infrastructure-As-Code framework where you work with high-level constructs instead of getting lost in the low-level cloud configuration. Opta gives you a vast library of modules that you can connect to build your ideal Infrastructure stack. Best of all, Opta uses Terraform under the hood — so you’re never locked in. You can always write custom Terraform or even take the Opta-generated Terraform and go your own way!</p><p>The Opta team recently released a new feature to generate Terraform code. Today, we will look at two ways this can be very beneficial to you.</p><h3>Use case 1: Rapidly Speed up Terraform work.</h3><p>Most users familiar with Terraform already know this; working with Terraform can be difficult. To understand this complexity, let’s look at the process that an engineer would follow when setting up new infrastructure using Terraform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QYWA73IgvkicPyF2" /></figure><ol><li><strong>Define the problem statement</strong> — We can assume this would be an infrastructure requirement such as hosting a static website, creating a database, or configuring a Kubernetes cluster.</li><li><strong>Define the target architecture</strong> — Terraform manages low-level components (<a href="https://www.terraform.io/intro">official definition</a>). Therefore, you can’t just go from the technical requirement to write the Terraform files; you will need to understand the target architecture. For example, you will need to know what needs to be created in detail, how these resources will be connected (network, firewall rules), and which permissions will be required (IAM).</li><li><strong>Identify the Terraform resources/modules — </strong>You have a few options. You can use plain Terraform resources, use modules from the <a href="https://registry.terraform.io/">official registry</a> (8,000 modules) or build your own Terraform modules. Terraform modules can have input, output, and dependencies on other modules. You will need to understand these relationships.</li><li><strong>Try it out, find issues</strong> — At this point, you should be able to provision the bulk of your infrastructure. But once you connect them and deploy them in your cloud, you might find some issues. The most common issues are module dependencies, misconfigured networks, or missing permissions.</li><li><strong>Update the Terraform files</strong> — Eventually, you will come to a working state after updating the Terraform files. You might also want to refactor your Terraform at this stage, possibly creating some logical abstraction by creating some modules.</li><li><strong>Maintain the Terraform files </strong>— Your infrastructure is like a living plant, so are your Terraform files. Every resource or module used is subject to change, new release, new functionality. So you will need to scale it, secure it, reproduce it, refactor it.</li></ol><p>Now, let’s use Opta to accelerate this process. Instead of figuring out everything from scratch, you can use the generated files to provision your infrastructure with Terraform. As a result, you will get a faster working state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KQWNc8rt2ssDhnuYR8PsSA.png" /></figure><ol><li><strong>Define the problem statement</strong> — same as above</li><li><strong>Define Opta modules</strong> — Opta provides high-level components so you can map your requirements directly to some Opta <a href="https://docs.opta.dev/reference/aws/modules/">modules</a>. Examples: host a static website: use the Opta modules <em>s3 and CloudFront</em>, create a database: use module <em>MySQL</em>, configure a Kubernetes cluster: use module <em>k8s-cluster and k8s-base</em>.</li><li><strong>Generate the Terraform files</strong> — Run <em>opta generate-terraform.</em> This will give you some Terraform files to get to a working state faster. While there might be thousands of ways you can provision your infrastructure with Terraform, Opta provides one recommendation on how to use Terraform based on the Opta modules defined. Because all the Opta users use the Opta modules, the generated Terraform works out of the box.</li><li><strong>Maintain the Opta/Terraform files </strong>— You are always free to diverge and modify the Terraform files. However, you can also use the Opta files as the source of truth and generate the Terraform files after each change.</li></ol><p>A good analogy for this use case would be to compare writing assembly language vs. using a higher-level language to generate a similar binary. We understand that it might not work for every user, but we aim to support the most common use cases.</p><h3>How are Opta Modules different from Terraform Modules?</h3><p>While they both have the same goal of providing a logical abstraction for some infrastructure components, Opta modules offer a higher level of abstraction. As a result, Opta users don’t need to be Terraform experts to build their infrastructure. In addition, all the Opta modules are supported by the same team, integrating well together.</p><h3><strong>Use case 2: Generate the Terraform files for your Opta infrastructure.</strong></h3><p>When we created <a href="https://github.com/run-x/opta">Opta</a>, we decided to adopt the open-source model. Our users can see the code base, roadmap, the team behind, and the daily flow of pull requests going in the project. Opta simplifies the path to infrastructure as code, but it’s not reinventing the wheel when provisioning the infrastructure. Opta uses Terraform, and as such, the user can see the Terraform files for their infrastructure. Generating the Terraform files can be used to migrate from Opta to Terraform and integrate with other great tools such as <a href="https://aquasecurity.github.io/tfsec/">tfsec</a> or <a href="https://terragrunt.gruntwork.io/">terragrunt</a>.</p><p>Here is a diagram showing the generated Terraform files for a typical Kubernetes stack with an HTTP service (our <a href="https://docs.opta.dev/getting-started/">getting started guide</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*voCgQ3rHHqwcEHWiMK1Nmg.png" /><figcaption>Left side: Opta files, Right side: Output of generating terraform command</figcaption></figure><p>A few observations:</p><ul><li>We see a lot of Terraform code generated! Opta provides high-level components, which require minimum user input. Terraform is a declarative language, so each resource describes what needs to be created.</li></ul><blockquote>To create a Kubernetes cluster (including networking) with a deployed service, the Opta file is 20 lines long. The Terraform files are 3,500 lines and an additional 1,500 lines for Helm charts.</blockquote><ul><li>The generated files are organized into Terraform modules to group the resources used together and prevent duplication.</li><li>The generated files also include all the needed files to run Terraform. For example, Terraform deploys a helm release; the helm files are also generated.</li><li>For each Opta file generated, there is a readme file to document how to use the Terraform files.</li></ul><p>Bottom line, we recommend that our users know the Terraform code that provision their infrastructure stack. Still, they should not have to deal with the complexity of writing and maintaining these files. Of course, if they want to start using Terraform instead of Opta, this feature will allow them to switch.</p><h3><strong>Demo: Generate Terraform files for a static website</strong></h3><p>For this section, we will refer to one of our previous blog posts, the <a href="https://blog.runx.dev/fastest-way-to-deploy-a-static-web-application-to-aws-e2c760c512ae">Fastest way to deploy a Static Web Application to AWS</a>, but this time we will use Terraform to provision the infrastructure.</p><ol><li>Define the Opta file</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/602638daab85d290535c1d4cd3135564/href">https://medium.com/media/602638daab85d290535c1d4cd3135564/href</a></iframe><p>2. Generate the Terraform files</p><pre><strong>opta generate-terraform -c opta.yaml</strong></pre><pre># List the generated files<br>tree ./generated-terraform<br>.<br>├── data.tf.json<br>├── module-awss3.tf.json<br>├── module-cloudfrontdistribution.tf.json<br>├── modules<br>│   ├── aws_s3<br>│   │   └── tf_module<br>│   │       ├── bucket.tf<br>│   │       ├── files.tf<br>│   │       ├── mime.json<br>│   │       ├── outputs.tf<br>│   │       ├── replication.tf<br>│   │       └── variables.tf<br>│   └── cloudfront_distribution<br>│       └── tf_module<br>│           ├── main.tf<br>│           ├── outputs.tf<br>│           └── variables.tf<br>├── output.tf.json<br>├── provider.tf.json<br>├── readme-static-app.html<br>├── terraform.tf.json</pre><p>2. Check out the 2048 project</p><pre>cd ./generated-terraform</pre><pre>git clone <a href="https://github.com/gabrielecirulli/2048.git">https://github.com/gabrielecirulli/2048.git</a></pre><p>3. Run Terraform — These commands are documented in the generated readme file.</p><pre><strong>➜ terraform init</strong></pre><pre><strong>➜ terraform plan -compact-warnings -lock=false -input=false -out=tf.plan -target=module.awss3 -target=module.cloudfrontdistribution</strong></pre><pre><strong>Plan:</strong> 65 to add, 0 to change, 0 to destroy.</pre><pre><strong>➜ terraform apply -compact-warnings -auto-approve tf.plan</strong></pre><pre><strong>Apply complete! Resources: 65 added, 0 changed, 0 destroyed.</strong></pre><pre>Outputs:</pre><pre>bucket_arn = &quot;arn:aws:s3:::runx2048&quot;<br>bucket_id = &quot;runx2048&quot;<br><strong>cloudfront_domain = &quot;d2ckxsjau98ix.ckoudfront.net&quot;</strong></pre><p>3. That’s it! You can use the Cloudfront domain to access the web app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ealeQfYU6l0J0oZJ.png" /></figure><h3>Wrapping up</h3><p>You, too, can <a href="https://github.com/run-x/opta">start using Opta</a> today to generate Terraform code, you can <a href="https://github.com/run-x/opta">get started here</a>, and If you run into any issues or errors at this stage, you can reach out to the contributors in <a href="https://slack.opta.dev/">our slack channel here</a> who are waiting to help.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f255f71c73d3" width="1" height="1" alt=""><hr><p><a href="https://blog.runx.dev/how-to-generate-terraform-code-with-opta-f255f71c73d3">How to Generate Terraform Code with Opta</a> was originally published in <a href="https://blog.runx.dev/">Run[X]</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I joined RunX? — The issue with hiring DevOps teams]]></title>
            <link>https://blog.runx.dev/why-i-joined-runx-the-issue-with-hiring-devops-teams-d5b3a39477da?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5b3a39477da</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Tue, 04 Jan 2022 18:26:46 GMT</pubDate>
            <atom:updated>2022-01-04T18:26:46.523Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why I joined RunX? — The issue with hiring DevOps teams</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*b0p9Ckqpm_XJGJhkiSyJCw.png" /></figure><p>Recently, I started a new job at RunX, a startup which provides tools to automate cloud infrastructure. I thought it would be an interesting post to write about my motivations and why I decided to join this team.</p><ol><li>What are my motivations?</li><li>What is the main issue with the DevOps market?</li><li>Use Lego blocks vs do-it-yourself</li><li>Why do I believe RunX has a bright future?</li></ol><h3>What are my motivations?</h3><p>Every engineer would have their own criteria when evaluating the fit with a potential employer.</p><p>Here is my list:</p><ul><li>Leadership</li><li>Engineering team</li><li>Company mission</li><li>Company culture</li><li>Interest &amp; Impact of the work</li><li>Growth opportunities</li><li>Work flexibility</li><li>Total compensation</li></ul><p>The importance of each one will depend where you are in your career and what are your goals.</p><p>One common mistake is to not understand the interdependencies. For example, a great leadership team will attract talent and this will result in interesting work and deliver a great impact. This would eventually lead to growth opportunities and higher compensation.</p><p>It is also not uncommon for DevOps engineers to have some relevant experience in one specific area which becomes their differential skill. They end up taking on similar challenges over and over, but at higher scale each time:</p><ul><li>They have solved it for X, now they are looking to solve it again for 2X or 10X.</li></ul><p>In my case, I have worked on various projects to improve Developer Productivity for <em>hundreds</em> of engineers, and my new personal goal is to have an impact for <em>thousands</em> of engineers.</p><h3>What is the main issue with the DevOps market?</h3><p>On one side, you have some DevOps engineers that have really specific aspirations and on the other side there are many companies in various stages that are competing for these engineers. To stress it even further, the cloud computing market is growing rapidly and the pandemic has only accelerated the trend.</p><p>What nobody will tell you is that having each company to build their own DevOps engineering team is not scalable. There are just not enough engineers.</p><p>To make the problem worse, some small or medium sized companies might not be able to attract these DevOps engineers for various reasons.</p><ul><li><strong>Engineering team:</strong> They might not have a DevOps team yet, or their existing team is subject to attrition. It makes joining their team a challenge.</li><li><strong>Interest / Impact of the work, Solve it again for 2X or 10X</strong>: Due to their size, they might not appeal to these engineers that want to experience <em>the next level.</em></li><li><strong>Total remuneration</strong>: It could be hard to compete with Tier 1 startup or FAANG.</li></ul><p>You could consider becoming the first DevOps engineer at a new startup, but unless it comes with a plan to grow the team quickly, it might create new issues.</p><ul><li>There are many specialities involved in Cloud Computing: Networking, Security, Scalability, Storage, Big Data… Even the smartest individual will have to make some choices on where to spend their time, and expose the business to some risks.</li><li>For the business itself, the dependency on one single individual might become a risk, especially if the plan is to target a multi-million dollar market.</li></ul><p>So to recap, we have:</p><ul><li>High competition to hire DevOps engineers.</li><li>The cloud computing market growing very rapidly.</li><li>Some risks associated with having a small DevOps engineering team.</li></ul><h3>Use Lego blocks vs do-it-yourself</h3><p>Instead of having each company building their own DevOps team to solve similar problems, RunX’s goal is to synthesize years of expertise in DevOps teams and package it as a service. RunX has already open sourced <a href="https://github.com/run-x/opta">Opta</a>, an Infrastructure-As-Code framework which provides high-level constructs to manage cloud infrastructure.</p><blockquote>Think of it as a Lego block pattern. A business can configure their cloud infrastructure to their needs, without having to mold each block and wonder if they would fit together.</blockquote><p>Being a DevOps engineer, I have experienced the complexity of working in the DevOps space. I wanted to answer the question: Would I use <a href="https://github.com/run-x/opta/">Opta</a> if I would start my own business?</p><ul><li><strong>Option 1: Do not use Opta:</strong> I am confident that I could maintain a Kubernetes cluster, configure it following best practices and scale it to a certain level. But I am not as confident with Networking and Security. Having worked in bigger companies, there are already some existing teams that would set the network stack for us and maintain it. Because these areas are critical, they can’t be ignored. I would need to spend a lot of time to figure this out. I will create some tech debt, make some errors and learn from them. Also my experience is mostly with AWS, if I would have to use a different cloud provider, there will be a learning curve.</li></ul><blockquote><em>Pros: Learn a lot.</em></blockquote><blockquote><em>Cons: Slow rollout, expose the business to new risks, create some tech debt.</em></blockquote><ul><li><strong>Option 2, Use Opta</strong>: By selecting the modules I need, I can get a quick head start to setup my cloud infrastructure. It comes with best practices, such as proper networking configuration and is Infrastructure as Code (IaC). I don’t need to become an expert in each area. I have more time to focus on the technical challenges that are unique for the business. The first time I tried Opta it took me 15 min to have a VPC with a functional Kubernetes cluster and 5 min to deploy my first app (see <a href="https://docs.opta.dev/getting-started/aws/">Opta documentation</a>). Just recruiting a DevOps engineer would take months, and probably a few weeks worth of work to have a similar infrastructure.</li></ul><blockquote><em>Pros: Fast rollout, follow best practices.</em></blockquote><blockquote><em>Cons: Use a new tool (Opta), which can be seen as a risk.</em></blockquote><p>The advantages are evident, but I would also need to address the part that Opta is a new tool and here are my findings:</p><ul><li>Opta can output Terraform code, the leading tool for Infrastructure-As-Code.</li><li>Opta is fully open source: the code, the tests, the documentation, the contributors can be seen on <a href="https://github.com/run-x/opta">Github</a>.</li></ul><h3>Why do I believe RunX has a bright future?</h3><p>At this point, I am convinced that the companies using RunX will greatly benefit from it. But coming back to my original list, how does working there would align with my motivations?</p><ul><li><strong>Leadership &amp; Engineering team</strong>: I am lucky enough to have worked with some members of the RunX team before. I have seen their leadership and engineering skills in action and the huge impact that they have.</li><li><strong>Company Mission/Culture/Growth opportunities:</strong> RunX is hyper focused on their mission and their customers. The engineers have a lot of autonomy and work on things that they enjoy. The culture promotes inclusiveness over intolerance. With these values, we, engineers, can unlock our full potential and grow with the company.</li><li><strong>Interest / Impact of the work</strong>: In a traditional DevOps role, the impact is internal. You build a new tool or fix a bug for the in-house engineering team. But at RunX, we provide open source tools, every new feature or bug fix is available to the community. As RunX will grow, so will the community of users. It will reach a point where it would just make sense to use RunX for managing your cloud infrastructure. That’s how impactful this will be. This aligns well with my personal goal of improving Developer Productivity for thousands of engineers.</li></ul><p>There are many more reasons why RunX is a company to watch for in this space. If you are curious about joining or want to learn about our values, check the <a href="https://www.runx.dev/careers">careers page</a>. If you want to try it yourself, have a look at <a href="https://github.com/run-x/opta">Opta</a>, the Infrastructure-As-Code framework.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5b3a39477da" width="1" height="1" alt=""><hr><p><a href="https://blog.runx.dev/why-i-joined-runx-the-issue-with-hiring-devops-teams-d5b3a39477da">Why I joined RunX? — The issue with hiring DevOps teams</a> was originally published in <a href="https://blog.runx.dev/">Run[X]</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Eventbrite runs a 700 node Kube cluster just for development]]></title>
            <link>https://remy-dewolf.medium.com/why-eventbrite-runs-a-700-node-kube-cluster-just-for-development-b0b84463a6e1?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0b84463a6e1</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Thu, 28 Jan 2021 21:33:57 GMT</pubDate>
            <atom:updated>2021-01-28T21:33:57.040Z</atom:updated>
            <content:encoded><![CDATA[<p>Originally published on Aug 19, 2020 in the Kelda blog: <a href="https://blimpup.io/blog/eventbrite-interview-part-2/">link</a>.</p><p>In <a href="https://remy-dewolf.medium.com/why-managing-dev-environments-is-a-full-time-job-at-eventbrite-b2bc94b0ae95">Part 1</a> of this interview with Remy DeWolf, a principal engineer on the DevTools team at Eventbrite, we discussed what information factored into Eventbrite’s decision to move their development environment to the cloud.</p><p>The DevTools team at Eventbrite set out to build yak because they had too many services to run locally. They knew they wanted to <strong>move their development environment to the cloud</strong>, but it ended up yak had additional benefits that ranged from <strong>sharing environments</strong> to <strong>transitioning to working remotely due to COVID</strong>.</p><p>In this post, we dig into <strong>how </strong><strong>yak works</strong>, <strong>what it’s like for devs to use it</strong>, and <strong>how it’s been received</strong>.</p><h3>Q&amp;A</h3><h3>How is the Eventbrite application architected?</h3><p>This is a common story that you will find in a lot of startups. The founding engineers built a monolith and the strategy was to build features fast and capture the market. It was a very successful approach.</p><p>As the company grew over time, having a large team working on the monolith became challenging. When reaching a certain size, it was also harder to keep scaling vertically.</p><p>Over time, some of the monolith was migrated over to microservices. Now, new services are generally containerized, and the monolith is containerized in dev but not in production.</p><h3>What prompted you to rehaul your dev envs and what problems did you set out to solve?</h3><p>See <a href="https://blimpup.io/blog/eventbrite-interview/#how-did-you-decide-it-was-time-to-build-yak"><strong>How did you decide it was time to build yak?</strong></a> from Part 1 of the interview.</p><h3>How did you convince your company?</h3><p>In the beginning, we partnered with developers to help us focus on the most important features and also to keep them excited about the technology. Specifically, there was a great interest to learn more about Kubernetes. We also added <a href="https://blimpup.io/blog/eventbrite-interview/#how-did-you-collect-feedback-at-eventbrite"><strong>instrumentation</strong></a> to our developer tools so we could measure how much time developers were wasting.</p><p>Once we understood how much time the developers spent waiting or dealing with issues, <strong>we had to make a call between spending money on cloud computing or wasting engineer time</strong>. We presented our plan to the CTO and we got the green light to move forward.</p><h3>What’s the developer workflow like with yak?</h3><p>Every morning, a developer has two options:</p><ol><li>Reconnect to their previous session: this takes a few seconds and they can resume their work from where they left it the previous day.</li><li>Update their local branch and update their remote docker images: this takes 5–7 minutes to get the environment updated.</li></ol><p>From this point, all their containers are running remotely and they can work through the day. Here are some of the common operations:</p><ul><li><strong>Change code locally:</strong> Changed files are automatically synced over to their remote containers. It usually takes a few seconds for the changes to be available. We use rsync for this, which is very efficient. To keep it simple, we do a one-way sync (from laptop to remote container).</li></ul><p>Note: this flow is much faster than the standard flow of building/pushing/deploying images, which in practice is hard to get under a minute for large applications.</p><ul><li><strong>Debug code:</strong> Developers can add breakpoints in their code and attach to a running container to get a live debugging session. We provided a command that wrapped kubectl attach under the hood.</li><li><strong>Run tests:</strong> Developers can run unit tests locally, but any tests that require dependencies (such as a DB or Redis) can be run remotely in a pod. For integration tests, they can run tests in a specific pod and connect to the other services directly.</li></ul><p>Most of the frontend changes are done locally and don’t require the cloud. <strong>The cloud is very useful for backend development and running various tests.</strong></p><h3>How does it work?</h3><p>Every developer has their own namespace where they manage their remote developer environment. Kubernetes does the heavy lifting and yak simplify the management of their containers.</p><h3>How does the DevTool team interact with the development environments?</h3><p>Usually, we don’t need to interact with the dev env but only focus on the big picture, such as managing the clusters, and adding more features to the tools.</p><p>We do support the developers, so sometimes we would directly connect to a namespace to troubleshoot some issues, via standard Kubernetes commands like kubectl logs.</p><h3>What’s the ongoing maintenance burden?</h3><p>In the beginning, it was a lot of work because we had a few issues where we didn’t understand the root cause, and we were also weak with the user documentation.</p><p>Over time, we got this under control. The documentation was revamped and we built a good knowledge base about the most common errors.</p><h3>How has the environment changed from when you first designed it?</h3><p>Our approach was always to deliver incremental value by first focusing on a minimum viable product (MVP) and adding features over time. For these reasons, we made many changes from the original design.</p><p>Here are a few interesting changes:</p><ul><li>Our infrastructure was running on one EKS cluster originally. At one point, we had <strong>700 worker nodes, and 14,000 pods running</strong>. We ran into performance and rate-limiting issues that made us reconsider this single-cluster approach. Over time, we switched to a <strong>multi-cluster architecture</strong> where each cluster had no more than 200 nodes.</li><li>Syncing the code directly into running containers could sometimes cause the container to crash if the changes made the application fail the probe check. After iterating a few times on how to solve this problem, we decided to set up a <strong>sidecar container</strong> that is responsible for syncing the code.</li><li>To persist data over time (for example to save the MySQL database files of a developer) we use Statefulsets backed by EBS volumes. However, AWS has a limitation around EBS volumes — an application running in a pod on EKS must be on a node in the same availability zone (AZ) as the EBS volume. To solve this problem, we partitioned our EKS nodes <strong>per availability zone</strong> and we used taints to make sure that our Statefulset would be in the same AZ.</li></ul><h3>Have there been any unexpected benefits?</h3><p>Yes,</p><ol><li><strong>Sharing environments</strong>: Have you ever heard a developer say <em>“but it worked for me locally when I ran the tests?”</em> Consistency improves by running in the cloud. The ability to share developer environments proved to be very helpful when trying to understand test failures or work on issues that were hard to reproduce.</li><li><strong>Working globally</strong>: We have a globally distributed team but most of the test/QA infrastructure is in the US. Simple operations like resolving the application dependencies or downloading a Docker image requires a lot of networking round trips. If the network latency is poor, these operations are slow.</li><li>By running on the cloud, the developer opens a connection to some container (with port forwarding or by getting a shell) and then they can run their commands from the same AWS region where the rest of the infrastructure is located. For our engineers based outside of the US, being able to develop on the cloud has been a huge improvement.</li><li><strong>Transitioning during COVID</strong>: When COVID happened, all developers switched to working remotely. For some, this meant sharing home internet connections with other households or moving back to their family. It would have been extremely difficult or impossible for some of them to run the developer environment locally. Operations such as pulling Docker images or resolving application dependencies would require gigabit of data on a daily basis. By developing on the cloud, the transition to remote work was fairly seamless and the developers were able to continue their work from home.</li></ol><p>See <a href="https://remy-dewolf.medium.com/why-managing-dev-environments-is-a-full-time-job-at-eventbrite-b2bc94b0ae95">Part 1</a> of this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0b84463a6e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why managing dev environments is a full time job at Eventbrite]]></title>
            <link>https://remy-dewolf.medium.com/why-managing-dev-environments-is-a-full-time-job-at-eventbrite-b2bc94b0ae95?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2bc94b0ae95</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[okr]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Thu, 28 Jan 2021 21:29:52 GMT</pubDate>
            <atom:updated>2021-01-28T21:35:16.388Z</atom:updated>
            <content:encoded><![CDATA[<p>Originally published on Aug 11, 2020 in the Kelda blog: <a href="https://blimpup.io/blog/eventbrite-interview/">link</a>.</p><p>Deciding when to invest in developer productivity improvements is hard. If you’re on the ops side of things, you’re usually concerned about production and releases. If you’re a developer, you’re concerned about getting new features out as quickly as possible.</p><p>Usually, teams make development productivity improvements in two situations. Either the fix is so small that you can just do it in addition to your other work, or development is so painful that making changes has ground to a halt.</p><p>However, there’s still a large murky middle ground: how do you decide that it’s worth investing in a <strong>large change</strong> to your development workflow <strong>before development has ground to a halt?</strong></p><p><a href="https://medium.com/@remy.dewolf"><strong>Remy DeWolf</strong></a> spent three years making these sorts of decisions as a <strong>principal engineer on the DevTools team at Eventbrite</strong>. He was part of the decision to build yak, which moved Eventbrite’s development environment into the cloud. This was a highly calculated decision since it cost a few EC2 instances per engineer and yak was built from scratch.</p><p>In this first post, we’ll dig into how Remy made this tough decision, and got buy-in from the rest of the company. In our next post, we’ll get into the nitty-gritty on how their remote development environment works, and what it’s been like for developers.</p><p>Read <a href="https://blimpup.io/blog/eventbrite-interview-part-2/"><strong>part 2 of this interview</strong></a> about Eventbrite’s specific setup and 3 unexpected benefits of remote dev environments.</p><h3>Q&amp;A</h3><h3>How is the Eventbrite application architected?</h3><p>This is a common story that you will find in a lot of startups. The founding engineers built a monolith and the strategy was to build features fast and capture the market. It was a very successful approach.</p><p>As the company grew over time, having a large team working on the monolith became challenging. And after a certain size, it was also harder to keep scaling the monolith vertically.</p><p>Over time, some of the monolith was migrated over to microservices. New services are generally containerized, and the monolith is containerized in development but not in production.</p><h3>What’s your development environment setup now?</h3><p>Every engineer runs <strong>~50 containers</strong> which corresponds to the monolith, the microservices, the data stores (MySQL, Redis, Kafka…) and various tools (logging, monitoring).</p><p>Developers use yak (which we built internally) to deploy and manage their remote containers.</p><p>We use AWS EKS for the Kubernetes clusters, in which every developer has their own namespace. We have hundreds of developers and many EKS clusters.</p><p>yak is very similar to <a href="https://blimpup.io/"><strong>blimp</strong></a> since it enables the engineers to manage their remote containers without exposing them to the complexity of Kubernetes.</p><h3>How did you decide it was time to build yak?</h3><p>Before yak, each developer ran their development environment locally on their laptop. However, the development environment became so big that it slowed down developer laptops.</p><p>The main issue was that you might not realize that this was an issue because it was creeping one service at a time.</p><p>Once we added instrumentalization to our tools, we started to understand the scale of the problems. Moving to the cloud is expensive but when we were able to put it side by side with the wasted engineering time, the decision was easy for us.</p><p>Another goal of yak was to make Kubernetes easy for developers. We kept it as minimal as possible and the configuration files are plain Kubernetes manifest files. The intent was to feed developer curiosity so they learn more about Kubernetes over time.</p><h3>What areas do you recommend tracking regarding developer productivity?</h3><p>Whenever possible, align the developer productivity goals with the business. Every DevTool team should understand how they contribute to the company goals and vice versa. If this is unclear, I would start with that.</p><p>Next, make sure that developer productivity is part of the plan, not an afterthought. For example, some engineering teams move to microservices and only track the number of services and the uptime in production. These are great metrics, but they’re incomplete. They will generate inconsistency and the developer experience will suffer over time.</p><p>In terms of which metrics to pick, there is no general recommendation. It’s important to understand how developers work, understand how frequently they perform critical tasks, and instrument the tools that they use. With this data, you will be able to identify the most important areas to invest and track the progress over time.</p><p>I would also recommend having a metric about mean time to recovery (MTTR). If a developer is completely stuck, how would you bring them back to a clean state so they can resume their work? For this one, if you run the developer environment locally, you will have many different combinations of OS/tools/versions resulting in many different issues. If you are on the cloud and use a generic solution (e.g. Docker + Kubernetes), this problem will be much easier to solve.</p><h3>How did you collect feedback at Eventbrite?</h3><p>We had many channels:</p><ul><li>Instrumentation into the tools. Every time a developer would build, run, or deploy docker images we would send metrics. Similarly, every CI job would do the same. Then we would generate some dashboards for the metrics to track and measure the progress over time. If you are using a tool like Sumologic or Datadog, it’s very easy to send custom metrics and build dashboards.</li><li>Quarterly engagement surveys.</li><li>Demos: invite other engineers to show them the progress and engage with them.</li><li>New hires: these new employees bring a fresh perspective and they are not afraid to ask questions and challenge the status quo.</li><li>Networking: build relationships with other developers (coffee breaks, office visits, lunches, etc..)</li></ul><h3>Can you give some examples of developer productivity OKRs (objectives and key results)?</h3><ol><li><strong>Time to start the developer environment is under x min: </strong>This time is usually wasted time, so it’s important to track it and improve it. If the dev stack is unreliable or slow, it would be captured in this OKR.</li><li><strong>Engagement is over x%: </strong>If you send an engagement survey every quarter, you can have an OKR to make sure the trend is upward. Seeing a drop would mean that the team might not be working on the most relevant projects.</li><li><strong>Average time from commit to QA/Prod: </strong>This one will capture the CI/CD pipeline effectiveness. If you experience some flaky tests or deployment errors in the pipeline, it would negatively impact the key results.</li></ol><p>Over time, some OKRs will be exhausted, so consider renewing them over time. For example, if your survey always has the same questions, developers will eventually stop responding. Also if an OKR has been greatly improved, it’s a good time to shift priorities.</p><p>In my personal experience, I would focus on a few OKRs instead of having too many. Sometimes by trying to please everybody, you will not have a big impact. Some projects might require the full team focus, which can temporarily impact other OKRs. This would be a calculated strategy as these projects would bring huge improvements when delivered.</p><h3>Are there any warning signs people should look out for in order to know their developer productivity is suffering?</h3><p>This is where it’s important to have good metrics and monitor them over time. You should be able to feel the pulse of your developers by looking at different data points. Ideally, you would tie these to your OKRs and review the progress every sprint and make adjustments.</p><p>If you don’t have this data there are still warning signs that productivity is suffering:</p><ul><li>Increase in support cases and/or requests for help. If developers need external help to do their work, this is a sign that a process is too hard to use or not well documented.</li><li>On the other hand, I’d be worried if you find out that some processes aren’t working properly but nobody reported them to your team. You want developers to be always looking for improvements and not accepting a broken process.</li></ul><p>See <a href="https://remy-dewolf.medium.com/why-eventbrite-runs-a-700-node-kube-cluster-just-for-development-b0b84463a6e1">part 2</a> of this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2bc94b0ae95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Rainbow: Compare the Performance of Different Programming Languages]]></title>
            <link>https://medium.com/better-programming/introducing-rainbow-compare-the-performance-of-different-programming-languages-f08a67453cd4?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f08a67453cd4</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Mon, 31 Aug 2020 14:33:11 GMT</pubDate>
            <atom:updated>2020-09-03T14:42:29.360Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Speed is everything</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89LE-gMoKJBeP0EMeVQ3oA.png" /><figcaption>Design by <a href="http://www.baileykass.com/">Bailey K</a></figcaption></figure><p>Recently, I was on the job market. For a software engineer, that means reviewing algorithms and deciding which programming language(s) to focus on for your interviews. I used a platform called <a href="https://leetcode.com/">LeetCode</a> to practice my rusty coding skills.</p><p>How does LeetCode work? For every question, it allows you to code a solution and submit it. It then runs it against a set of test cases to verify that the implementation is correct. It also gives you some information about runtime and memory and how your solution compares against others. These are very popular metrics, and a lot of users mention the stats of their algorithms in the LeetCode discussions.</p><p><em>“Python easy solution, speed beats 99.32% and memory beats 99.78%.”</em></p><p>If you happen to solve the exercises using different languages, you might be surprised by some outputs.</p><p>Let’s take a popular exercise: <a href="https://leetcode.com/problems/two-sum/">two-sum</a>. It can be solved efficiently by using a hash table. Here is a valid solution in Go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*vkuN7qY8-RDKh0ud9yG1lw.png" /><figcaption>Two-sum solution using Go</figcaption></figure><p>Now let’s submit this solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZqlHqgDqgOaOI52dhoQig.png" /><figcaption>Two-sum output using Go</figcaption></figure><p>Sounds fast! Let’s write a similar implementation in Python3 and submit it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*HRYQBWAYYs_oZzcCCDJP8A.png" /></figure><p>A bit slower but someway expected, as Python is an interpreted language.</p><p>Let’s try Java next:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*xac4ZBAddDfgB8VDe-jkvw.png" /><figcaption>two-sum output using Java</figcaption></figure><p>OMG! Java is so fast!</p><p>Let’s recap the results:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/643a5c00520289f10b7ce1cc92c308fa/href">https://medium.com/media/643a5c00520289f10b7ce1cc92c308fa/href</a></iframe><p>To summarize:</p><ul><li>Java is 7x faster than Go and 10x faster than Python.</li><li>Java uses 3x more memory than Python and 6x more than Go.</li></ul><p>Is Java really that fast? The raw results could be a bit surprising, no? We are getting into interesting territory: How do programming languages perform against each other?</p><p>Let’s take a step back. Imagine you have a PaaS to compute two-sum and you spend millions every year on cloud computing and you run your services on Python. Would that mean that you could lower your bill by 10x if you switch to Java? Well, things like storage or network would be the same, but you could save on the computing cost (AWS EC2, GCE, Azure VM). If you try out different languages for your computing needs, you might be able to make a choice that would significantly reduce your VM costs.</p><p>But how would you collect the data to make that decision?</p><h3>Programming Language and Containers</h3><p>Comparing programming language performance is nothing new. But the common practice is not to use Docker containers, to not impact the results due to containerization.</p><p>But what if a language is exceptionally fast and robust but slow and unreliable when running in a container? What if the target infrastructure is Kubernetes?</p><p>There are a lot of parameters that can impact performance when running an application in a container:</p><ul><li><strong>Startup time</strong>: The number of replicas can be scaled up or down to adjust based on traffic. If a container is slow to start, it would impact the throughput.</li><li><strong>Image size</strong>: A new node would need to download the image at least once for every version. The time to download the image is not spent processing requests.</li><li><strong>Resiliency</strong>: If a container crashes, a new one would need to be created to replace it.</li><li><strong>CPU/Memory footprint</strong>: The lighter the container is, the more that can potentially be executed on the same node.</li><li><strong>Language type</strong>: compiled or interpreted. Running machine code is faster than running through a program line by line.</li></ul><p>This list is not exhaustive which makes assessing programming language performance in a container a very hard question to answer.</p><h3><strong>Introducing rainbow</strong></h3><p>If every characteristic of a containerized application is defined by a color, the idea of <a href="https://github.com/RemyDeWolf/rainbow">rainbow</a> is to look at the full spectrum and to only compare the output. rainbow is an extensible framework to test and compare similar implementations running on containers. These implementations can be written in different languages. rainbow runs the containers on similar infrastructure and only focuses on the throughput.</p><p><a href="https://github.com/RemyDeWolf/rainbow">rainbow</a> is an open source project hosted on GitHub under the MIT license.</p><h3>How Does rainbow Work?</h3><p>The idea is simple. For each language, there is a main class that calls the function to test in an infinite loop. A batch size X is set, and on every X call, a Redis counter is incremented by X to keep track of the number of computations done by each language.</p><p>Every image is deployed on a similar node (same VM type) and can run as many replicas as needed to maximize the throughput.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*0fRMeceeWAfXunlbuIXmpw.png" /><figcaption>Image source: Author</figcaption></figure><p>This architecture provides a fair ground and rewards a language that can maximize the properties of containerization.</p><p>For more info about the testing methodology, check the project <a href="https://github.com/RemyDeWolf/rainbow#how-does-it-work">readme</a>.</p><h3>Assumptions</h3><p>Let’s come back to the three programming languages that we used for two-sum.</p><ul><li><strong>Java</strong>: This was the fastest language on LeetCode. Because it requires a Java Virtual Machine (JVM), the Docker image should be bigger. This might impact performance.</li><li><strong>Go</strong>: This language produces binary executables, so we might be able to minimize the Docker image. Being light might reduce the gap with Java.</li><li><strong>Python</strong>: Python in Docker is notoriously large. The default image uses a Linux buster (Debian). Also, it is an interpreted language, so most likely it will perform poorly against the competition. Anything else would be a surprise.</li></ul><h3>Implement Two-Sum in rainbow</h3><p>Since we want to test the performance of the two-sum algorithm, how can we write a program that would maximize the runtime doing that computation?</p><p>Let’s keep it simple:</p><ol><li>Implement the two-sum function exactly like it was defined in LeetCode.</li><li>Call this function with the same test case for each language. We can reuse one of the most complex test cases used by LeetCode.</li></ol><p>See the two-sum code for <a href="https://github.com/RemyDeWolf/rainbow/blob/master/base/go/twosum.go">Go</a>, <a href="https://github.com/RemyDeWolf/rainbow/blob/master/base/python/twosum.py">Python</a>, <a href="https://github.com/RemyDeWolf/rainbow/blob/master/base/java/src/main/java/compute/twosum.java">Java</a>.</p><h3>Building the Docker Images</h3><p>Now that we have the source code, let’s package them in a Docker image and build them. Let’s take a moment to look at the image generated.</p><ul><li><strong>Go image: 4.49 MB </strong>Since Go produces a binary we can use a multistage build and copy the binary to an alpine image. (see <a href="https://github.com/RemyDeWolf/rainbow/blob/master/base/go/Dockerfile">Dockerfile</a>)</li><li><strong>Java image: 64.6 MB</strong> The JDK takes 61 MB to install on Alpine. (see <a href="https://github.com/RemyDeWolf/rainbow/blob/master/base/java/Dockerfile">Dockerfile</a>)</li><li><strong>Python image: 325.04 MB </strong>The default Python base image is 320 MB. There is an Alpine image, but it’s not recommended as it can lead to <a href="https://pythonspeed.com/articles/base-image-python-docker-images/">weird production bugs</a>. (see <a href="https://github.com/RemyDeWolf/rainbow/blob/master/base/python/Dockerfile">Dockerfile</a>)</li></ul><h3>Benchmarking in Kubernetes</h3><p>Now that we have our Docker images, we can deploy them to a Kubernetes cluster. For simplicity and affordability, we will use Google Kubernetes Engine (GKE) and create three nodes using the same machine type: e2-standard-2 (2xCPU, 8GB). Each pod is assigned to a node (using node labels).</p><p>Let’s try multiple combinations of workers/replicas and for each run the benchmark for 10 minutes. <em>Count</em> is the number of times the two-sum method was called.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a84b66f1a4027b37182b6af8f225ff66/href">https://medium.com/media/a84b66f1a4027b37182b6af8f225ff66/href</a></iframe><p>A few comments:</p><ul><li>This is clearly a CPU-intensive operation. Memory does not impact the overall throughput.</li><li>Python is limited by the <a href="https://realpython.com/python-gil/">Python Global Interpreter Lock which prevents a container from using more than 1 CPU</a>. When running on two replicas, it used more CPU but the throughput was not improved, unfortunately.</li><li>Go had the best results when using workers. This might be explained by the lower overhead of the Goroutines (lightweight threads).</li><li>Java performed slightly better when using two replicas.</li></ul><h3>Comparing With LeetCode Results</h3><p>Java performance when running on LeetCode seems a bit inflated, but overall, the trend is verified. For a computation like two-sum, Java greatly outperforms Go and Python. We can note that Go reduced the gap with Java, possibly because of the lighter Docker image and the use of Goroutines.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43d3e21b70ec171700080919853252dc/href">https://medium.com/media/43d3e21b70ec171700080919853252dc/href</a></iframe><h3>Is Java Really That Fast?</h3><p>Not so … fast. As we have seen in this exercise, the memory footprint of Go is much lower than Java. If memory was the limiting factor, we would have been able to run more Go containers on the same node and potentially outperform Java. According to <a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html">The Computer Language Benchmarks Game</a>, the fastest languages are usually C++ and Rust.</p><p>Using rainbow, I ran some benchmarks to compare a few languages (Go, Java, Python, Ruby, Rust) on generic implementations such as Hashtable, File IO, Sort, and N-square and got various results. Rust and Java performed very well. Go did well but had a few weaknesses. The compiled languages outperformed the interpreted languages each time. All the results can be seen in the <a href="https://github.com/RemyDeWolf/rainbow#implementations">readme</a>.</p><p>This framework is extensible. If you have a specific implementation and would like to compare the performance of different languages, feel free to implement the compute function and have fun with it. See the <a href="https://github.com/RemyDeWolf/rainbow#how-to-add-an-implementation">instructions here</a>.</p><p>Further readings:</p><ul><li><a href="https://github.com/RemyDeWolf/rainbow">RemyDeWolf/rainbow</a></li><li><a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html">Which programming language is fastest?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f08a67453cd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/introducing-rainbow-compare-the-performance-of-different-programming-languages-f08a67453cd4">Introducing Rainbow: Compare the Performance of Different Programming Languages</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experiment: Start Jenkins servers recursively with Docker]]></title>
            <link>https://remy-dewolf.medium.com/experiment-start-jenkins-servers-recursively-with-docker-f16984a94284?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f16984a94284</guid>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[experiment]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Tue, 19 Dec 2017 14:21:02 GMT</pubDate>
            <atom:updated>2017-12-19T16:04:12.299Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*fC7v_SN1bGNkAbfUFPS2Vw.png" /></figure><p>Have you ever heard of <strong>running Docker within Docker</strong>?<br>Now, imagine you have a <strong>Jenkins container which can run Docker</strong>, <strong>would that image be able to start itself?</strong> We could repeat the process and have the new container to start another instance, and repeat…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*wJkMGmt19gL4-7IubHfM5A.png" /><figcaption>What we want to achieve: running nested Jenkins servers using docker</figcaption></figure><h3>1. Configure Docker in Docker for the Jenkins image</h3><p>First let’s modify the official <a href="https://github.com/jenkinsci/docker/blob/master/Dockerfile">Jenkins Dockerfile</a> to install docker. This image extends a Linux Debian distribution, we can configure the Debian repo and use <strong>apt-get</strong> to install docker.</p><p>Here are the instructions added to the Dockerfile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bd0b2112faec659859004eca0dfd79e/href">https://medium.com/media/5bd0b2112faec659859004eca0dfd79e/href</a></iframe><h3>2. Allow Jenkins to start another Jenkins instance</h3><p>Now that Jenkins is able to run a Docker command, let’s configure a job to start another instance of Jenkins. The requirements are:</p><ul><li>Always run the same docker command to start Jenkins, so it can be repeated.</li><li>The TCP server port has to be different for each new server.</li></ul><p>Here is a simple shell script that would do the job. An environment variable is used to store the TCP port, it is incremented when running the new container.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b81e77b19d21cdcbb2bd87cf95fd29a/href">https://medium.com/media/3b81e77b19d21cdcbb2bd87cf95fd29a/href</a></iframe><h3>3. Expose the Docker socket to your CI container</h3><p>We want the container to be able to start container, the easiest way to achieve this is to mount to the Docker socket using a volume.</p><p>Start the Jenkins container with the param-v/var/run/docker.sock:/var/run/docker.sock</p><h3>4. Run the experiment</h3><p>This Docker image was uploaded to dockerhub and can be started from command line.</p><pre>$ docker run -p 8080:8080 -e JENKINS_PORT=8080 --rm -v /var/run/docker.sock:/var/run/docker.sock --name jenkins remydewolf/jenkins-in-jenkins</pre><p>This will start Jenkins at <a href="http://localhost:8080/">http://localhost:8080</a></p><p>Once Jenkins is up, we can start another Jenkins instance by building <strong>start-new-jenkins</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XaRJsoX4iYD5G61rint2HA.png" /><figcaption>Build the job to start another Jenkins server</figcaption></figure><p>Look for the “Jenkins is fully up and running” message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYVg_pdMwqHAeTe0kvvi6A.png" /><figcaption>It takes a few seconds to start a new Jenkins</figcaption></figure><p>There is now, a second instance of Jenkins up at port 8081.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PI4GbEeDtllzjBotB7LrSw.png" /><figcaption>Running the job will now start a Jenkins server on port 8082</figcaption></figure><p>To run the experiment, we can repeat this process a few times.</p><h3>How does it work?</h3><p>When starting the container from Jenkins, it is actually started on the host daemon. We can confirm this by running <strong>docker ps</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1akdyofzqpi2VxUX5J-_Ww.png" /><figcaption>Running <strong>docker ps</strong> on the host shows all the containers started.</figcaption></figure><p>All the docker daemons are actually referring to the host, to confirm this we can run docker ps in one of the container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r0rau_PJZRNs-B-HqH9OYg.png" /><figcaption>Running <strong>docker ps</strong> in one of the instance gives the same output</figcaption></figure><p>Under the hood, the containers were started on the host.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*T-kRDhzUIOyhWj17yNyBLQ.png" /><figcaption>All containers were started in the same Docker daemon, on the host.</figcaption></figure><p>In conclusion, here is a quote from <a href="https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/">Jérôme Petazzoni</a> about this solution:</p><blockquote>This looks like Docker-in-Docker, feels like Docker-in-Docker, but it’s not Docker-in-Docker: when this container will create more containers, those containers will be created in the top-level Docker. You will not experience nesting side effects, and the build cache will be shared across multiple invocations.</blockquote><p>For more info on this topic, check out these great resources:</p><ul><li><a href="https://tripdubroot.com/jenkins-docker-in-docker-dind-2040cc90eeab">Jenkins Docker in Docker (DinD)</a></li><li><a href="https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/">Using Docker-in-Docker for your CI or testing environment? Think twice.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f16984a94284" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Fargate: First hands-on experience and review]]></title>
            <link>https://remy-dewolf.medium.com/aws-fargate-first-hands-on-experience-and-review-1b52fca2148e?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b52fca2148e</guid>
            <category><![CDATA[ces]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[fargate]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Tue, 05 Dec 2017 17:21:49 GMT</pubDate>
            <atom:updated>2017-12-07T00:00:55.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CM0d2xaW6GIMO1H2lBcIFw.jpeg" /></figure><p>On November 29, 2017, AWS announced <a href="https://aws.amazon.com/fargate/">Fargate</a> with the following promise:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9mQ41JFGgD3DCaMktDgRA.png" /></figure><p>This article will guide you in setting up a simple HTTP server with AWS Fargate. From this point, the infrastructure will be scaled using Auto-Scaling.</p><p>Based on this initial experience, we will assess the delivery of this technology’s original promise and reflect on potential use cases going forward.</p><h3>Deploy an application using Fargate</h3><p>The fastest way to get you started is to go to AWS Console and run the <a href="https://console.aws.amazon.com/ecs/home?region=us-east-1#/firstRun">new first-run wizard</a> for Fargate. Make sure you are in the <strong>us-east-1</strong> (N. Virginia) Region. Fargate is only available for that region at this time.</p><p>The first screen comes up with a pre-configured sample application based on the http Docker image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOC7k60hVDc0iXIO9-FOMA.png" /><figcaption>Select the sample-app and click next</figcaption></figure><p>The next step is to decide if we want to use an Application Load Balancer. With an ALB, we could register multiple container instances for the service, therefore setting us up for Auto-Scaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2PHWguY29PIx5w3RtBqyA.png" /><figcaption>Select Application Load Balancer and click Next.</figcaption></figure><p>Even though Fargate containers don’t need EC2 instances, they still need to be registered to an ECS Cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*-kJ__TAR37PX0zYCWo6WFQ.png" /><figcaption>Set the cluster name to <strong>fargate</strong> and click next.</figcaption></figure><p>The next screen gives an overview of the components to be created: Task Definition, Service and Cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gul-Bv5xIUOVWMERJvVqQA.png" /><figcaption>Click Create to create the cluster.</figcaption></figure><p>The AWS resources are being created. It took 4 min for this run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSudiSIT0jaODO4EAQ_46A.png" /></figure><p>Under the hood, this is a <a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks">cloudformation stack</a> that is executed. The template can be exported and reused if you want to script this part, instead of going through the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kLKal1OnR965EywUVTXDow.png" /></figure><p>Once the Fargate cluster is created, let’s test that the container is up and can be reached through the ELB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c6NZtjv1X9q8B5ms2WpwOQ.png" /><figcaption>Click on View Service</figcaption></figure><p>The service lists the Target Group used for Load Balancing.</p><blockquote>Pro Tip: If you plan to run some containers through the cli (aws ecs run-task) you will need the subnets and securityGroups that are listed on this screen.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qkXX1KHVJINLDjSH6n3w5g.png" /><figcaption>Click on the target group to navigate to the Load Balancer</figcaption></figure><p>From the target group, we can navigate to the Load Balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zRcdEf1oDixaIBQ41DeTbQ.png" /><figcaption>Click on the Load Balancer name</figcaption></figure><p>The DNS name can be copied from this page. This is the public hostname for the Load Balancer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqHPVr9urbdaaUFaAJ5m1Q.png" /></figure><p>Now, let’s test by connecting to the copied URL. The load balancer redirects the HTTP request to the container, which returns the HTML page. Success!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FWK-654cckX0V2JXB7v0WQ.png" /></figure><h3>Scale the service by adding more containers</h3><p>Now, that we have one container, how can we add more container instance to our service?</p><p>First step, let’s go back to the <a href="https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/fargate/services/sample-app-service/details">ECS service</a>. The desired count and running count are set to 1. We want to bump this number to 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ui-bVayNGDq4YczDQg2PMQ.png" /><figcaption>Select Update to update Desired Count</figcaption></figure><p>Let’s set the number of tasks to 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bvQOebNi7dlQXRVFqKixgA.png" /><figcaption>Click on Next step</figcaption></figure><p>Click Next at each screen, until the Auto Scaling screen shows up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uN_UJ6IB9RKp_Uaft0LNUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uN_UJ6IB9RKp_Uaft0LNUw.png" /></figure><p>Select Auto Scaling and set the maximum number of tasks. For this example, we limited to 2 to prevent accidental auto-scaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQuvTJ9LTpHN_owMlKTmSw.png" /><figcaption>Click Next step</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bcL38B_njFALJmsW2Dp0bQ.png" /><figcaption>Confirm by clicking on Update Service</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDG38bap9YcK7SkViyAWeQ.png" /><figcaption>Once Service Auto Scaling is created click on View Service</figcaption></figure><p>Now the service has a Desired Count of 2, but the Running count is still 1. Let’s have a look at the Tasks tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N6TTQXqstwWYJlKu3E3mZA.png" /><figcaption>Select the Tasks tab</figcaption></figure><p>The newly created task goes from PROVISIONING to PENDING to RUNNING. It took 3 min during this test to reach the RUNNING state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZW6o8FbKEx8eJmZxk2SSCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qnc27DKG1kvqdqGVhx6-Zw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VIsd6rDRqKEk6zsRVq6g-Q.png" /></figure><p>If we go back to the <a href="https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#TargetGroups:">Target Group</a>, there are now 2 instances registered to the ELB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9b6FJgojkV8X9DiST5pEg.png" /></figure><p>So we have 2 registered targets on our Load Balancer but there is no instance of EC2 running. How is it possible?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMq9J9hyO5L57z8Y_gd6DQ.png" /></figure><p>If you check the <a href="https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#NIC:sort=networkInterfaceId">Network Interface</a>, you will find that for each container task, an entry created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nlBvoWpiT18g1Jm8b49zKA.png" /><figcaption>3 network interfaces were created: 1 for the ELB, 2 for the container tasks</figcaption></figure><h3>Review of AWS Fargate</h3><p>Does it live up to the promise?</p><blockquote>Run containers without managing servers or clusters</blockquote><p><strong>50% delivered</strong>: there is no EC2 instance, but there is still an ECS cluster, and the all the infrastructure that is needed to have an ECS cluster.</p><h4>Not managing server?</h4><p>Generally speaking, managing the EC2 instances can be a burden, there are so many instance types, and there is not simple rule to set up the auto-scaling efficiently, it’s usually a process of trial and error overtime.</p><p>With Fargate, you can’t set the EC2 instance type, this can be an issue for some use cases. Some applications would be optimized when running on a given type of instance. For example, P3 instances are suitable for Machine/Deep learning.</p><h4>Still managing the cluster?</h4><p>At the end of the day, Fargate is a pure ECS solution, with the exception of managing the EC2 instances. If you have never done ECS cluster management or come from Kubernetes, you might be confused with the notion of cluster, tasks, service, container definition.</p><h4>How can we make it better?</h4><p>ECS is complex. AWS is going in the right direction by trying to remove some layer and simplifying it.</p><p>But why not go all the way and make it plain simple?</p><p>Here is suggestion for this, extend the notion of container service and provide a <strong>one-screen setup</strong> with the following parameters:</p><p>Input:</p><ul><li><strong>Service Name</strong>: my-sample-app</li><li><strong>Docker Image</strong>: http:2.4</li><li><strong>Memory</strong>: 0.5GB</li><li><strong>Cpu</strong>: 0.25 CPU (256)</li><li><strong>VPC</strong>: Drop-down list. (if there is none, provide a one-screen UI for this)</li><li><strong>Subnets</strong>: List of subnets where the container would be hosted (public/private is derived from this choice)</li><li><strong>Desired count</strong>: number of container to start</li><li><strong>Load Balancer</strong>: yes/no (if yes, ask for port)</li></ul><p>Output:<br>- <strong>Load Balancer URL</strong><br>- <strong>List of container IPs</strong></p><p>The VPC creation is right now bundled into the Fargate wizard, this could be separated for clarity. Most likely, different set of users would be configuring the VPC and deploying containers.</p><p>With this solution, we could remove the notion of cluster entirely. Merge the concept of Task Definiton, Container into the Service. <strong>The user would only manages Container Services</strong>.</p><p>While making it simple, it would lean toward a solution that is agnostic from the container orchestration tool.</p><h4>Fargate still worth it?</h4><p>If you have never setup ECS, Amazon does a great job with providing a wizard that would get you started quickly. From then, you can save the cloudformation template and configure it properly. Most likely you will want to revisit the security (security groups and carefully select which components have public IPs).</p><p>During the initial tests, there were a few times, it took a few minutes for a container to be started. Behind the scene, AWS has to find EC2 instances for you, which can introduce some latency, if you manage your own servers (without Fargate), you could always pre-provision some instances for this. If you plan to use ECS in production, I would recommend benchmarking Fargate vs EC2/ECS.</p><p>Given that it’s very easy to set up, Fargate would be a very good tool for prototyping applications. If you were thinking of running your CI on docker container, Fargate would be a good answer as you won’t have to <a href="https://tech.ticketfly.com/our-journey-to-continuous-delivery-chapter-4-run-jenkins-infrastructure-on-aws-container-service-ef37e0304b95">manage autoscaling and cluster management</a>.</p><h3>Bottom Line</h3><p>AWS Fargate removes a layer of complexity from the current ECS offer, but still uses the concepts of Cluster, Service, Task Definition and Container Definition. The new wizard for Fargate hides a lot of the complexity. If you are using AWS and don’t have a container orchestration tool, it would be the easiest way to get you started.</p><p>Because Fargate doesn’t let you choose the EC2 instance type, you might miss out on some optimizations. For high performance applications, benchmarking Fargate vs EC2/ECS Cluster would be recommended.</p><p>If you are using Kubernetes, hold on, AWS Fargate support for Amazon EKS will be available in 2018. Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b52fca2148e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run Jenkins Infrastructure on AWS Container Service]]></title>
            <link>https://remy-dewolf.medium.com/our-journey-to-continuous-delivery-chapter-4-run-jenkins-infrastructure-on-aws-container-service-ef37e0304b95?source=rss-4a3788ab7bf2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef37e0304b95</guid>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Remy DeWolf]]></dc:creator>
            <pubDate>Thu, 31 Aug 2017 23:01:01 GMT</pubDate>
            <atom:updated>2017-12-07T16:32:46.654Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oH7hb4MAHx0Ng2_hhFk-mg.png" /></figure><h4>Our Journey to Continuous Delivery: Chapter 4 of 4</h4><p>One of the hardest challenges of delivering software frequently is the ability to build and test the high paced potential of code changes. In this chapter, we will present a scalable infrastructure to host Jenkins and its build farm.</p><p>Since we are already hosting our production in AWS, it was the natural place for us to host the build infrastructure. We wanted to simplify and standardize our solution. As such, we investigated existing services provided on AWS as our first consideration. Hence, we decided to use ECS for container management.</p><blockquote>Amazon EC2 Container Service (ECS) is a highly scalable, high performance <a href="https://aws.amazon.com/containers/">container</a> management service that supports <a href="https://aws.amazon.com/docker/">Docker</a> containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.</blockquote><p>First, we will first review the full AWS stack necessary to run a Jenkins build farm using ECS. Second, we will provide a fully working example to deploy the stack. When running this example, we will cover two use-cases:</p><ul><li><strong>Auto-Scaling(+/-)</strong>: When receiving a traffic spike, some EC2 instances are added to handle the load. When the system is idle, the extra instances are terminated until the minimum size is reached.</li><li><strong>Self-Healing</strong>: When Jenkins crashes, it is automatically restarted in a new container and attached to the load balancer.</li></ul><h3>Jenkins ECS Architecture Overview</h3><p>Being on the cloud, our architecture consists of <strong>virtual servers</strong> (EC2 instances, pictured as grey boxes), <strong>storage</strong> (EFS), and <strong>services</strong> (Load Balancer, Auto-Scaling Group, Alarms and Container Service).</p><p>To achieve <strong>auto-scaling</strong>, the alarms monitor the CPU usage and trigger the <em>Auto-Scaling Group</em> to add or terminate instances based on the number of Jenkins builds to run.</p><p>For <strong>self-healing</strong>, the ECS Cluster runs Jenkins as a service, ensuring that it is always up and healthy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*VNHlu_wTKEzWrApUclmq-w.png" /><figcaption>Scalable AWS Architecture based on ECS Cluster.</figcaption></figure><p>For each component, let’s highlight their role in the Jenkins Infrastructure.</p><ul><li><strong>ECS Cluster</strong>: manages Docker containers execution on the available pool of EC2 instances. Jenkins has an <a href="https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Container+Service+Plugin">ECS plugin</a> to execute tasks on the cluster.</li><li><strong>Auto-Scaling Group</strong>: contains a collection of EC2 instances. It can adjust the number of instances based on events. Our build system has spikes and low usage periods, we need to adjust our infrastructure accordingly. When properly configured, it prepares you for future growth as the infrastructure adjusts with your needs.</li><li><strong>Cloudwatch Alarms</strong>: send events to adjust the size of the <em>Auto-Scaling Group</em> based on current load. Setting alarms allows fine tuning how fast the infrastructure scales up and down. It is about finding the balance between preventing pipeline clogging and keeping AWS costs under control.</li><li><strong>EFS Storage</strong>: provides scalable file storage, can be shared across multiple instances. It prevents Jenkins from running out of disk space. EFS can be mounted from many instances, it works better for self-healing. This is unlike EBS which can be slow to detect that the volume has switched from “in-use” to “available”.</li><li><strong>Load Balancer</strong>: distributes incoming application traffic across multiple Amazon EC2 instances. It gives a consistent URL for the user while letting ECS decide where the application is hosted.</li><li><strong>VPC</strong>: The <em>Virtual Private Cloud</em> is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS cloud. Having this level of isolation for Jenkins increases the overall security in multiple ways, including that it: prevents someone with Jenkins access to reach production and allows defining VPC permissions for certain group of users. Billing can also be broken down by VPC.</li></ul><h3>AWS Hands-on: Scalable Jenkins Infrastructure using AWS</h3><blockquote>Cost Warning: This example is designed to run on micro instances (t2.micro), the maximum size of the Auto-Scaling Group has been set to 5 instances. Including storage and data transfer, it costs approximately <strong>$0.10/hour</strong> to run this stack.</blockquote><p>In the next section, we will deploy the Jenkins infrastructure to AWS. Once deployed on the cloud, we will be able to simulate <strong>Auto-Scaling </strong>and<strong> Self-Healing.</strong></p><h4>1. Launch the CloudFormation stack</h4><p>Here is the quick link that will take you directly to the <em>Create Stack </em>wizard with the expected parameters:</p><figure><a href="https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=jenkins-ecs-stack&amp;templateURL=https://s3.amazonaws.com/remydewolf-public/cloudformation/jenkins-ecs-stack.json"><img alt="" src="https://cdn-images-1.medium.com/max/144/1*Z2PO_ioDZEow7eb3SD6rsg.png" /></a></figure><p>Even if you have never used AWS, this stack will work as it contains everything it needs. If you have some existing infrastructure, it won’t mix up as it is isolated in its own VPC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*z3wLjrT8Y-R2nHb2V-yRFg.png" /><figcaption>The link will take you to the wizard page, the template is already selected, just click <strong>Next</strong>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Xd_bVZCrkk3XYifmAQLlqw.png" /><figcaption>You can optionally restrict the allowed IP range, once done click <strong>Next</strong> and <strong>Create</strong>.</figcaption></figure><p>It takes approximately 5 min to complete. Once done, copy the Jenkins URL from the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bvOZl2Ou0YAM-u4lpsVaGw.png" /><figcaption>CloudFormation completed successfully, Jenkins is up!</figcaption></figure><h4>2. Simulate <strong>Auto-scaling: Adding and terminating instances based on load</strong></h4><p>Go to the <strong>JenkinsELB</strong> URL. Jenkins is already setup for ECS and comes with pre-configured projects. To observe auto-scaling, we will simulate a traffic spike by building “<em>Run_All_Jobs</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*uzab9zrQNKQo70qPc23h0A.png" /><figcaption>Click on the dropdown for “Run_All_Jobs” and select “Build Now”.</figcaption></figure><p>Now in EC2 Container service, select the cluster called “<a href="https://console.aws.amazon.com/ecs/home?clusters/jenkins-cluster/services/jenkins-master/tasks&amp;#/clusters/jenkins-cluster/tasks">jenkins-cluster</a>”, a few tasks are listed:</p><ul><li>Jenkins Master: a service task.</li><li>Java or Javascript: these tasks are the first jobs in the queue making their way to ECS.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*R76kL_bBOKgO_XQgSkDuvQ.png" /><figcaption>ECS shows the tasks currently running, each task runs in a Docker container.</figcaption></figure><p>Next, Jenkins attempts to add more tasks, but the cluster is at full capacity. After a minute of high CPU reservation <em>the scale-up alarm goes off in </em><a href="https://console.aws.amazon.com/cloudwatch/home?#alarm:alarmFilter=ANY"><em>Cloudwatch</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*dXl6EORIVexTAU4FB5S8aw.png" /><figcaption>The Scale-up policy has changed to “Alarm” state</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*3R8fNSink4vIi0LV2W2RPg.png" /><figcaption>If the cluster reaches 75% CPU reservation, the alarm goes off, triggering new instances.</figcaption></figure><p>The alarm triggers the policy to create new EC2 instances in the <a href="https://console.aws.amazon.com/ec2/autoscaling/home?#AutoScalingGroups:view=details"><em>Auto-Scaling Group</em></a>. This process is repeated until it reaches the maximum allowed, set to <strong>five</strong> for this example.</p><blockquote>Pro Tip: Setting a max value is good practice to prevent unexpected costs.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kZOLhaIOasf0KXDtmcm2yw.png" /><figcaption>When notified by the alarm, the Auto-Scaling Group is launching new instances.</figcaption></figure><p>Back to <a href="https://console.aws.amazon.com/ecs/home?clusters/jenkins-cluster/services/jenkins-master/tasks&amp;#/clusters/jenkins-cluster/tasks">ECS Cluster</a>, the registered number of instances has increased from 2 to 4, more tasks are now running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*_tXlpnZBdMGF9BsXp2NC5A.png" /><figcaption>The cluster is placing the tasks in the container instances. One instance can run many tasks concurrently.</figcaption></figure><p>Back to Jenkins, the builds are running concurrently. Each job is executed in its own Docker container in an ECS task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*S9t3jW2Smxja7E7wQIbN2A.png" /><figcaption>As a result of the added capacity, Jenkins can now run more jobs in parallel.</figcaption></figure><p>Once the spike is over, the ECS cluster still has 5 instances. <strong>If there is another surge, the instances are available to handle it.</strong></p><p><strong>If the period of inactivity is longer, we would like to scale down</strong> so we don’t pay for unused instances. For this example, we have setup another <a href="https://console.aws.amazon.com/cloudwatch/home?#alarm:alarmFilter=ANY">CloudWatch alarm</a> that triggers after 10 minutes of inactivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kztl2LalBwIRkabR1x_sXw.png" /><figcaption>After 10 minute of inactivity, the scale-down policy is in “Alarm” state</figcaption></figure><p><strong>When receiving the event, the </strong><a href="https://console.aws.amazon.com/ec2/autoscaling/home?#AutoScalingGroups:view=details"><strong>Auto-Scaling Group</strong></a><strong> terminates some instances.</strong> This process is repeated until it reaches the minimum number of instances, set to 2:</p><ul><li>1 instance for Jenkins master.</li><li>1 for instance available for upcoming builds, to prevent waiting in queue.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bFXLecp9gp-qaMV2E9y4Hg.png" /><figcaption>When notified by the alarm, the Auto-Scaling Group terminates some instances.</figcaption></figure><h4>3. Simulate a Jenkins outage and observe self-healing</h4><p>Finally, what happens if Jenkins crashes? To simulate an outage, let’s shutdown Jenkins:</p><p>Go to the homepage and add <em>/exit</em> to the end of the URL to access this feature. Press the button “<em>Try POSTing”</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*UfVYhPeFWGKfZfX9oN68qw.png" /><figcaption>Jenkins is terminated through the UI.</figcaption></figure><p>Check the <a href="https://console.aws.amazon.com/ec2/v2/home?#LoadBalancers:">Load Balancer</a>, the instance is now listed as “<em>OutOfService”</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*_p6hV-5enw94SQfE8uqPHQ.png" /><figcaption>Since the Health Check fails, the ELB marks the instance as OutOfService.</figcaption></figure><p>The <em>Load Balancer</em> deregisters the instance from the ECS service, which results in the initialization of a new task in <a href="https://console.aws.amazon.com/ecs/home?clusters/jenkins-cluster/services/jenkins-master/tasks&amp;#/clusters/jenkins-cluster/tasks">ECS</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*2eiQi6zEb34k3CiPx8JnGw.png" /><figcaption>The ECS Service lists the succession of events. The downtime lasted 90 secs.</figcaption></figure><p>Once the service restarts the instance, Jenkins is restored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xNlwg4YYxPDifG-ByDyNag.png" /><figcaption>Jenkins is back up behind the ELB.</figcaption></figure><p>To recap, here is a video of the whole experiment to see if you missed something along the way.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7HcquHW3Uk4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7HcquHW3Uk4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7HcquHW3Uk4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/53422de89a6304e4f160fe6cc42d3ccb/href">https://medium.com/media/53422de89a6304e4f160fe6cc42d3ccb/href</a></iframe><h3>Lessons Learned about AWS Hosting</h3><p>We have accomplished our goal of scalable architecture for our build system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*BrWsUOavmirYaQ23mHsiGA.png" /><figcaption>Every build is executed in a container.</figcaption></figure><p>Let’s step back and reflect on some learnings of…</p><ul><li><strong>AWS Experience</strong>: Even if you are only interested by one AWS service, you will need more knowledge when troubleshooting.Most likely, you will stumble into network, security and permission issues.</li><li><strong>Permissions</strong>: Once we got our own AWS account with admin permissions, we started moving faster. It might be a big change for some companies. One suggestion is to work in different AWS Accounts or VPCs.</li><li><strong>Self-Healing</strong>: anticipate Jenkins outage and design your system to recover from it. It could be an application failure but also an AWS failure. Be ready for it. We used to get paged for Jenkins down but not anymore.</li><li><strong>EC2 Instance Types</strong>: Using containers gives you the flexibility of running multiple tasks on the same instance. It opens the door to multiple optimizations. Try out different scenarios and measure performance. We have some projects where we maintain a fixed group of cheap burst instances (t2) which are mostly idle but available when needed, and an <em>Auto-Scaling Group</em> of more expensive <em>Compute Optimized Instances</em> (c4) to handle spikes.</li></ul><blockquote>Pro Tip: In December 2017, AWS launched Fargate to not have to manage EC2 instances. Full review <a href="https://medium.com/@remy.dewolf/aws-fargate-first-hands-on-experience-and-review-1b52fca2148e">here</a>.</blockquote><ul><li><strong>Cost Optimization</strong>: be aware of the EC2 bill for your build farm and make adjustments when needed. It’s very easy to over-provision the infrastructure in the cloud. Use of auto-scaling should include some scaling-down policies as well. When this article was originally written, Amazon would charge per hour, but it has since switched to <a href="https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/">Per-Second Billing</a> (Sept. 2017).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CZxj0OgT83wWiBeHJyUm7w.png" /></figure><p>Now at the end of this series, let us review our goals of…</p><ul><li><strong>(Eliminating) Manual Work</strong>: Achieved by running Jenkins in Docker, configuring Jenkins through DSL and deploying infrastructure with CloudFormation.</li><li><strong>(Reducing) Build Time</strong>: Use Jenkins DSL to configure jobs to run in parallel. Scalable Infrastructure to prevent waiting in queue. EC2 optimized instance.</li><li><strong>Engineering Growth</strong>: Scalable architecture using <em>Auto-Scaling Group</em>(s).</li><li><strong>Engineering Satisfaction</strong>: Engineers can maintain their build config within their Github config. Jenkins downtime are minimal, self-healing allows us to recover when it crashes.</li></ul><p><strong>Thank you for following along.</strong> If you would like more information, please ask your questions in the comments section below. You can also review the <a href="https://github.com/Ticketfly/jenkins-docker-examples">source code</a> for the examples and check our <a href="https://www.youtube.com/channel/UCfK62eSGsJdheu3bukYAg_g">video tutorials</a>.</p><p>Previously in this series: <a href="https://tech.ticketfly.com/our-journey-to-continuous-delivery-chapter-1-it-begins-with-a-road-3ab175b80d42">Chapter 1</a>, <a href="https://tech.ticketfly.com/our-journey-to-continuous-delivery-chapter-2-run-jenkins-on-docker-49c32532cb7e">Chapter 2</a>, <a href="https://tech.ticketfly.com/our-journey-to-continuous-delivery-chapter-3-automate-your-configuration-with-jenkins-dsl-1ff14d7de4c4">Chapter 3</a>.</p><p>Related: <a href="https://medium.com/@remy.dewolf/aws-fargate-first-hands-on-experience-and-review-1b52fca2148e">AWS Fargate review</a>: ECS without EC2 instances.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef37e0304b95" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>