<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Laura Gonzalez on Medium]]></title>
        <description><![CDATA[Stories by Laura Gonzalez on Medium]]></description>
        <link>https://medium.com/@lauradgh?source=rss-67636d9766ce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Kx9RuhLQxTNeHOznM2BEOA.png</url>
            <title>Stories by Laura Gonzalez on Medium</title>
            <link>https://medium.com/@lauradgh?source=rss-67636d9766ce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:32:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lauradgh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The irony of software engineering]]></title>
            <link>https://medium.com/@lauradgh/the-irony-of-software-engineering-6325b4ba884b?source=rss-67636d9766ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/6325b4ba884b</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Laura Gonzalez]]></dc:creator>
            <pubDate>Fri, 06 Dec 2019 00:06:49 GMT</pubDate>
            <atom:updated>2019-12-06T00:08:50.226Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Being too clever makes things worse.</blockquote><blockquote>Dumb and simple is smarter.</blockquote><blockquote>Being productive is doing less.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6325b4ba884b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we moved to Kubernetes]]></title>
            <link>https://medium.com/@lauradgh/how-we-moved-to-kubernetes-f0fb961d5970?source=rss-67636d9766ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0fb961d5970</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[dcos]]></category>
            <dc:creator><![CDATA[Laura Gonzalez]]></dc:creator>
            <pubDate>Tue, 15 Oct 2019 16:53:21 GMT</pubDate>
            <atom:updated>2019-10-17T00:19:49.997Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>I work for a company called <a href="http://zeeto.io/">Zeeto</a>, we are company that focuses on online advertisement and data discovery. We are unique in our space by providing highly targeted advertising based on how users answer survey questions.</p><p>We started with a very ad-hoc and complex system that kept growing over the years to a point where we needed to re-design from scratch. The legacy system was built on PHP and Python running on AWS EC2 infrastructure. While there were some subsystems, it was more monolithic in nature.</p><p>Four years ago, we decided to take a step forward and build a new product, the best new product in the world using all the things we learned from our legacy system. We decided to go all in, redesigning a backend using micro services written in Java and front end applications written in React, for higher throughput. We also decided to leverage containers in order to support our planned growth and operating at web scale.</p><p><strong>Challenges</strong></p><p>But why containers? With a complex system with different languages and tasks, it made sense. It’s easy to package your application code including the entire ecosystem it needs to run on, and ship it anywhere.</p><p>Doing so created its own challenges such as:</p><ul><li>Need of an automated pipeline to ease the development of 20+ engineers</li><li>Need of multiple environments to stage and test our changes</li></ul><p>We reviewed and evaluated available frameworks for managing containers and compared the different feature sets we thought were critical. At the time Kubernetes was still new tech compared to the existing and large scale commercial Mesosphere/DCOS. We needed a tool to deploy, run and monitor our services.</p><p>It was summer of 2016, and at that time the final decision was DCOS. I remember talking to a DevOps friend that had experience with all these new shiny tools and telling him what we were up to, his immediate reaction was “Why DCOS? Just use Kubernetes, it’s simple, it’s well supported and it’s what everyone is moving to”.</p><p><strong>Initial Implementation</strong></p><p>After months and months of figuring out every piece of it, digging through their documentation for hours, creating a bunch of ansible scripts that set up every piece of the required infrastructure, networking and software to run it, we made it work:</p><ul><li>Multiple AWS availability zones.</li><li>Running about 40 services with 2–3 containers per application</li><li>Automated deployments through our pipeline</li><li>Running on ~15 big EC2 instances</li></ul><p>Jenkins is the foundation of our pipeline. It builds our applications into a docker image, then it publishes it to a docker registry we store in Artifactory, then sends a request to DCOS’ Marathon to run the fresh new version of our apps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xaqKrfreWlFToJ5H" /></figure><p><strong>Mesos DCOS is cool, BUT…</strong></p><p>We started running Mesosphere DCOS 1.6 on an AWS environment. We went with the community edition since the enterprise edition didn’t offer a significantly valuable set of features for the price. Our install lacked some features like secrets management, service discovery, rbac, and support. This definitely added some complexity and took us more time than it should have to make it work.</p><p><strong>Obscure Networking</strong></p><p>Networking was one of the most complex areas with Marathon, since our community edition didn’t support service discovery out of the box, we had to venture into figuring out how to make our services communicate.</p><p>Another big black box was marathon’s internal load balancer. We had some requests take a long time to move from one service to another, or sometimes they would get totally lost and it was impossible to monitor or log.</p><p><strong>Buggy UI</strong></p><p>Tasks would get stuck forever with no alerts, we’d have to ssh into one of the boxes and check the master logs to find out we couldn’t pull the docker image.</p><p>To click any action you had to “press harder”, or maybe just 5 times until it worked.</p><p>The most important one, if you’d modify the marathon JSON file through their UI editor that contains the spec of your application, it would overwrite some configurations and change critical settings for your application, making it unable to communicate with other services. And of course, we learned this the hard way in prod.</p><p><strong>Manual work to scale the cluster</strong></p><p>We had scripts to scale our cluster that we had to craft by hand, from creating the EC2 machine with all the right security groups, EBS volumes to install every piece of software to run a worker node.</p><p><strong>No easy maintenance</strong></p><p>After about three years of using DCOS we still did not understand the internals.</p><p>We had to manage and monitor the EC2 instances ourselves, test and roll any software upgrade by hand (we’d later migrated it into an ansible script). Telegraf and Cloudwatch came in handy to publish some basic metrics we could alert on when something happened.</p><p>Unfortunately no one really used Mesosphere/DCOS directly though it was successfully hosting our successful digital advertising network. Finding the right answer to debug any issue was hard and Kubernetes had clearly won the container orchestration wars. When you see companies like Google and AWS offering it as a service, you clearly know the popularity and demand of such tool. Everyone is now happily using Kubernetes.</p><p><strong>We need a change</strong></p><p>After a few months we decided to give Kubernetes a try. The main reason was DCOS was too complex and tedious to maintain for the engineering team. We wanted to explore autoscaling to scale our system out plus start running on EC2 Spot Instances for cost savings. Spot Instances are EC2 instances that you can bid on for way cheaper price than On-Demand but you can be out bid and lose your instance at any time.</p><p>We decided to make our changes incrementally. We wanted to just replace the orchestration tool that ran our applications in our infrastructure with a new one. Like moving to a new apartment with all of your belongings instead of fixing the drains of your current one.</p><p>About half a year ago, I went to a Mesosphere DCOS talk about how to run Kubernetes on it and asked the presenter: “Why would you install a container orchestration software ON another container orchestration software”… the answer was “Because people ask for it”. Which I, personally, still don’t understand.</p><p><strong>Hello Kubernetes</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xsGvhf4EGMGkYOutROi69A.png" /></figure><p>I took on the responsibility of migrating our entire live system to Kubernetes. There was a big learning curve understanding new terms and concepts like pods, stateful sets, services, etc. But it was good to know that I was not alone, I found a lot of very useful Kubernetes articles that helped me ramp up quickly and made the learning curve less steep.</p><p><strong>The plan</strong></p><p>Roll out changes incrementally, keep our pipeline as is except that instead of deploying our containers to DCOS, they need to go to Kubernetes!</p><p>Amazon had just released EKS, so I gave it a try. A few clicks and I had my kubernetes cluster running.</p><p>The first issue came in real quick, I tried to deploy one of our microservices and couldn’t talk to anything. At the time there was no option to run the cluster in our custom VPC/subnets, and that was a lot to take on. We have database services running on RDS, kafka, mongo and artifactory running on custom EC2 instances across different VPCs. All the security groups, VPC peering, route tables were already configured for it. Not to mention the<strong> AWS Data transfer fees</strong> we’d have to take on if we wanted to have our new cluster communicate to our data services in a separate VPC.</p><p>As a PoC, I was able to run one of our applications after a lot of manual networking setup to make it able to talk to a different VPC and configuring all the required security groups to be able to talk to all the data services.</p><h3>Out of the box solution: AgileStacks</h3><p><a href="https://www.agilestacks.com/">AgileStacks</a> is a DevOps platform that automates your infrastructure deployments. The automation hub provides support for infrastructure-as-code tools such as Terraform, CloudFormation, Deployment Manager, Helm, Ansible, Chef. Infrastructure stack templates are hosted in Git and can be deployed as a unit, helping to standardize and automate creation of Dev, Test, and Production environments on cloud providers such as AWS, GCP, and Azure. Agile Stacks also provides a Service Catalog of supported Kubernetes add-ons and stacks: such as Jenkins, GitLab, Kubeflow, Harbor, EFK. These stacks enable monitoring, certificate management, CI/CD pipelines, and other common infrastructure needs.</p><p>Benefits:</p><ul><li>They are the experts</li><li>Great support from their core developers</li><li>Great tools integrated on each environment</li><li>Infrastructure as a code</li><li>CLI options for automated scripts</li><li>Monitoring</li><li>Support for spot instances under our custom AWS accounts / existing VPN</li><li>Autoscaling</li></ul><p>We decided to give AgileStacks a try since it would reduce the overhead to create our own scripts to bring up our own cluster with the tools we needed. Plus, we wanted to start using Spot Instances to reduce our cloud costs. We run three different environments for our system, so any cost savings is multiplied by 3.</p><h3>How Agile Stacks approaches building Kubernetes clusters</h3><p>Agile Stacks provides an interface to configure all the components you need to bring up a customizable cluster with the tools you need.</p><h4>Cloud</h4><p>A cloud account is the entity used to sync your cloud provider to their system, in our case we created AWS Roles and generated a keys file and attached it to a cloud account per environment to give Agile Stacks scripts access to manage resources in our behalf.</p><h4>Environments</h4><p>There is a one-to-one relationship to a cloud account with some extra information, like other services keys, licenses, secrets, or extra parameters, that can be applied to your stack but are not necessarily tied to your cloud provider.</p><h4>Templates</h4><p>Templates describe the components each kubernetes cluster must have. In our case, we just have one template that defines all the components we need. Our template git repo contains a file called hub-component.yaml that we use to bring up the stacks.</p><p>A stack is an instantiation of a template. Each stack gets deployed to the environment using the same template. Each environment knows what aws key to use to deploy to the right AWS account. Each stack has its own parameters that are customizable so this way we can specify which VPC to deploy to, how many worker nodes we need, custom ACM certificates to use, etc.</p><p>So in our case, we have a stack per environment. One for dev, one for test and one for production.</p><h4>Components</h4><p>Components are a group of specific software tools that can be installed to a Stack. The Hub CLI reads the stack and template definition to bring up a new stack. Behind the scenes, each component has scripts that run scripts required to have a software toolchain fully running. For example, the traefik component deployment hooks into AWS to create a Load Balancer and ACM certificates then installs traefik through a helm chart connecting all the pieces together.</p><p>The Automation Hub</p><p>This is the automation tool Agile Stacks created to interact with the infrastructure. To bring up a new stack, all we have to do is to run this simple command in a shell:</p><p>$ hub deploy</p><p><strong>Our Implementation Strategy</strong></p><p>We decided to treat our container orchestration software as a black box and just replace the scripts responsible of bringing up DCOS with scripts that bring up Kubernetes. As simple as that.</p><p><a href="https://www.ansible.com/">Ansible</a> is an idempotent automation tool used to deploy and provision software. We use Ansible to create all of our AWS inventory and install required software on each provisioned machine, and we love it. Ansible uses playbooks to describe a set of tasks and integrates numerous plugins to manage and configure different types of infrastructure.</p><p>High level our simple steps in our main playbook look like this:</p><ul><li>Create AWS inventory</li><li>Set up networking (VPC, Subnets, Route Tables, Security Groups</li><li>Create AWS services (RDS, ElasticSearch, SQS, SageMaker)</li><li>Create EC2 machines for self hosted services and install software (Kafka, MongoDB, Grafana/Influx)</li><li>Set up EC2 Instances and install DCOS from scratch</li></ul><p>All we had to do was to run our k8s cloud provider scripts to create the stack and open up ports to their instances so we could have our microservices talk to our data services.</p><p>We utilized Kubernetes secrets and configmaps to configure our running applications with connection urls.</p><h3>The final picture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_nf1kKbPa8nIDkiQ" /></figure><p>Having the option to customize our AWS infrastructure via Agile Stacks templates and variables during runtime was huge to localize our services where we needed to without having to configure any VPC peering or add any extra overhead to our existing scripts. Again, we wanted to treat Kubernetes and DCOS as black boxes where we could just ship our containers to. Plus, defining the instance types to run Kubernetes out of the box was a huge win to us since we immediately saw cost improvements by switching to Spot Instances.</p><p>The final step was just to tell Jenkins to ship our containers to Kubernetes instead to DCOS. But how? One of my favorite tools was <a href="https://helm.sh/">helm</a>, it made it so easy to deploy our applications to the Kubernetes environment</p><p><strong>Helm</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*jhNO6G3B0s0WkmgKHJWXWQ.jpeg" /><figcaption>“I love helm” - Laura, 2019</figcaption></figure><p>Our Jenkins pipeline deploys the container to each environment in a single step. We had to handcraft the HTTP JSON request to marathon with all the custom configuration like: environment variables, container image version, networking settings, etc. So all the secrets management was being done in Jenkins.</p><p>All secrets were created in Kubernetes beforehand then we used a custom helm chart that assembled each kubernetes resource based on the application type.</p><p>We defined flags and based on these flags we dynamically injected certain environment variables the container needed to run the application. Plus each service could define the container resource requirements per environment.</p><p>One chart for our applications made the most sense for simplicity. This made our Jenkins pipeline simpler.</p><p>Each application has a custom <strong><em>values.yaml</em></strong> file that defines the properties then the helm chart will construct the right resources for it.</p><p>The main custom chart looks like this:</p><p>Default <strong><em>values.yaml</em></strong> file</p><pre>env: development</pre><pre>kafka: false<br>mongo: false<br>postgres: false<br>replicas:<br>  _default: 1</pre><p><strong><em>custom-chart/templates/deployment.yaml</em></strong></p><pre>apiVersion: apps/v1<br>{{- $replicaCount := pluck .Values.env .Values.replicas | first | default .Values.replicas._default }}<br>kind: Deployment<br>spec:<br>replicas: {{ $replicaCount }}</pre><pre>// …</pre><pre>template:<br>  spec:<br>    containers:<br>      name: {{ .Values.service }}<br>      image: &quot;custom-docker-registry/{{ .Values.service }}:{{ .Values.version }}&quot;<br>      env:<br>{{- if .Values.kafka }}<br>        — name: &quot;KAFKA_URI&quot;<br>          valueFrom:<br>            secretKeyRef:<br>              key: uri<br>              name: kafka-secrets<br>{{- end -}}<br>{{- if .Values.postgres }}<br>       — name: &quot;POSTGRES_URI&quot;<br>         valueFrom:<br>           secretKeyRef:<br>             key: uri<br>             name: postgres-secret<br>       — name: &quot;POSTGRES_USERNAME&quot;<br>         valueFrom:<br>           secretKeyRef:<br>             key: username<br>             name: postgres-secret<br>       — name: &quot;POSTGRES_PASSWORD&quot;<br>         valueFrom:<br>           secretKeyRef:<br>             key: password<br>             name: postgres-secret<br>{{- end -}}</pre><pre>// …</pre><p>Custom application <strong>values.yam</strong>l would determine what each application needs per environment</p><pre>service: app-name</pre><pre>kafka: true<br>postgres: true<br>replicas:<br>  prod: 5<br>  staging: 2</pre><p>Then in our Jenkins pipeline we just run</p><pre>$ helm upgrade --install app-name helm-charts/custom-chart -f values.yaml --set=env=prod --set=version=1.0.2</pre><p>And violá!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wVzM6C5Ym4ArqwoW" /></figure><p><strong>Cost Savings Round 1 — Spot Instances</strong></p><p>After moving to the new infrastructure we reduced our amazon bill for worker nodes by <strong>40% less</strong> than we used to spend by running our Mesosphere/DCOS cluster on On-Demand EC2 instances. This is per environment, summing a total of over $<strong>5K/month</strong> cost savings.</p><p>One of the main drawbacks is that you might get the spot instances revoked if you’re bidding less than the market price or retired from time to time. This risk was mitigated by ensuring we architected our system with high availability in mind, make sure you run more than one instance of each application across all of your nodes.</p><p><strong>Cost Savings Round 2 — Resizing applications</strong></p><p>Having the <strong>visibility</strong> of how much memory and CPU every application was using through Prometheus/Grafana made us realize how overprovisioned we were. So we decided to take a closer look to each microservice (over 40 of them) and resize accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3pQnruwI5BQgmxvl" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YrCZ4E8S6yRv3LZI" /></figure><p>We went from requiring <strong>~30</strong> instances to run our kubernetes cluster to needing about half the boxes. So our total cost savings went down by <strong>75%</strong> off our previous DCOS infrastructure.</p><p>Another cost savings tip is to make sure to shrink your cluster for dev, staging and load testing during non-working hours. We have cron jobs that reduce our Autoscaling groups down to 0 when we’re out of the office and back to the desired number of nodes when it’s office hours.</p><p><strong>Current Challenges</strong></p><p>AWS Spot instances are not as reliable as On-Demand. We recently had two big outages: in the first one we lost about 8 instances for no reason, or to be more exact “A health check failed” at the same time. The second outage was AZ related, we lost all of our instances in one of our AZ zones. Due to our autoscaling group, AWS immediately brought up new instances and our down time was about ~5 mins, we didn’t have to do anything manually and we were very happy with how things quickly recovered.</p><p>There are several strategies we will look into in the near future, like <a href="https://github.com/kube-aws/kube-spot-termination-notice-handler">Spot termination notice handler</a>, which notifies 2 minutes in advance you when AWS is going to terminate your instance, so that you can drain your node and migrate your applications to other available nodes. Another successful strategy is to diversify usage to several instance types in AZs using mixed autoscaling groups (ASGs). The reason is simple: by using multiple instance types we increase the chances of getting Spot capacity, instead of trying to get Spot capacity from a single instance type in a specific AZ and failing if there’s no spare capacity. Typically not all instance types will be interrupted at the same time, so it will be easily replenished by Auto Scaling groups thus avoiding any impact on availability.</p><h3>Conclusion</h3><h4>Benefits of Infrastructure Automation</h4><p>By migrating to Kubernetes and enhancing our infrastructure automation, we enabled Zeeto’s development team to be more agile and efficient. Kubernetes proved to be a great platform when it comes to automating a deployment pipeline. Zeeto’s deployments are not only more reliable, but also much faster, because we’re no longer dealing with VMs. Even if your company doesn’t have a dedicated DevOps team or if you lack experience doing so, implementing infrastructure as code is well worth the effort. With infrastructure as code you can save a lot of time and operational costs. Use a tool that is well adopted, flexible and a good fit for your use case. There are several services that can help to automate your deployments using infrastructure as code approach. Easy-to-use automation is the key to spinning up Kubernetes environments efficiently, and can also keep the cost of these environments low.</p><h4>Build your infrastructure with resilience and autoscaling in mind</h4><p>Losing a node should not require manual intervention. It could be as simple as configuring an autoscaling group that ensures you have X amount of running instances all the time. This will save you from waking up at night.</p><h4>Treat infrastructure as a first class citizen.</h4><p>Develop your infrastructure as how you would write your application code. Changes should be tracked in a Git repository, system health should be monitored, you should code around failures like how you would surround your code within a try/catch. And most important, do not reinvent the wheel and follow patterns, there are great tools out there you could use and learn from!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0fb961d5970" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>