<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Irving Popovetsky on Medium]]></title>
        <description><![CDATA[Stories by Irving Popovetsky on Medium]]></description>
        <link>https://medium.com/@irvingpop?source=rss-25ca8c726d9b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lTU9rVPIA0p5rOHX.jpeg</url>
            <title>Stories by Irving Popovetsky on Medium</title>
            <link>https://medium.com/@irvingpop?source=rss-25ca8c726d9b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:17:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@irvingpop/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Refactoring the Kubernetes infrastructure at OperationCode]]></title>
            <link>https://itnext.io/refactoring-the-kubernetes-infrastructure-at-operationcode-229e391e3ad9?source=rss-25ca8c726d9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/229e391e3ad9</guid>
            <category><![CDATA[aws-fargate]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Irving Popovetsky]]></dc:creator>
            <pubDate>Mon, 16 Mar 2020 02:12:02 GMT</pubDate>
            <atom:updated>2020-03-17T23:40:36.053Z</atom:updated>
            <content:encoded><![CDATA[<p>A story about getting our Kubernetes infra into a better state and slashing 2/3rd of our AWS bill in the process</p><h3>It all started when…</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ciOHIZ4xo-Oko1QOhoCkNA.png" /></figure><p><a href="https://medium.com/u/68b109307620">Nell Shamrell</a> noted in the #infrastructure-team channel that our monthly AWS bill was really high, and honestly it was way too high for an organization like OperationCode.</p><p>I’ve been helping out with ops stuff at OperationCode for about a year, and our Kubernetes (aka k8s) cluster definitely needed some work:</p><ol><li>It was still on Kubernetes 1.11 (21 months old! a hundred in k8s years) the steps to upgrade in-place were <a href="https://github.com/kubernetes/kops/blob/master/docs/releases/1.12-NOTES.md">fraught with peril</a>.</li><li>It had a cumbersome user onboarding process involving Google authentication and a secrets.json file that everyone (including the admins!) kept losing</li><li>It was costing us way too much, consisting of 6x t3.medium instances (3 masters, 3 worker nodes), 3 NAT gateways (these are surprisingly expensive!) and overly large EBS volumes (200 GB per host)</li></ol><p>As you can see, all of this really added up quickly for a small organization like ours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-LNgly900o3FrobSmLCag.png" /><figcaption>Our AWS costs for Feb 2020</figcaption></figure><h3>Part 0: Implement Monitoring and Observability</h3><p>Before proceeding with any changes, I took a good hard look at our monitoring situation. We have <a href="http://sentry.io/">Sentry.io</a> to notify us of application errors as well as Prometheus with <a href="https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/">AlertManager</a> to capture metrics and send alerts from inside the our Kubernetes cluster.</p><p>In addition I added 2 external (and free) points of visibility:</p><ul><li>Observability instrumentation from <a href="http://honeycomb.io/">Honeycomb.io</a> (my day-time employer), utilizing <a href="https://www.honeycomb.io/pricing/">the free Community plan</a> to give us 500MB of storage and 15/GB mo of telemetry data ingest. It was relatively easy to implement <a href="https://github.com/OperationCode/back-end/pull/277">the Python Beeline in our back-end application</a>, in fact most of the work involved fixing the build and updating dependencies and documentation.</li><li>External site monitoring with <a href="https://www.statuscake.com/">StatusCake</a>, also utilizing their <a href="https://www.statuscake.com/pricing">free plan</a> to give us 10 uptime test URLs and 1 page speed test.</li></ul><p>This way I had the confidence to proceed with a total infrastructure makeover, knowing we had sufficient test coverage for our production environment.</p><p>As <a href="https://medium.com/u/5587d135a397">Charity Majors</a> always says, you really do test in prod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*oi5RNqKtwUJSfcw79o-9PA.png" /><figcaption>Source: the charity.wtf blog</figcaption></figure><h3>Part 1: Clean up unused resources</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zaWaGHbT5H12_pVa03rzfw.png" /></figure><p>Our initial examination found nearly $200/mo of resources that were totally unused, left over from a previous version of the app that ran on ElasticBeanstalk and a few experiments. A 1-hour pairing session with two other OCers and we got those cleaned up quickly.</p><h3>Part 2: Reflecting on why Kubernetes matters to OperationCode</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*8HoB3yeVgtXj_TrmQbJjDw.jpeg" /></figure><p>One thing I love about OperationCode is how we use our own web application and infrastructure as a teaching resource, all the apps and infra <a href="https://github.com/OperationCode/START_HERE">are open source</a>. This allows the vets we help (as well as all volunteers) to gain experience in today’s popular development practices. I believe our choices to use and provide access to a “real, in production” Kubernetes cluster follow in the same spirit. It’s totally okay if it’s a little bit overkill for our needs, because it provides invaluable hands-on experience to anyone who wants it.</p><p>The feedback about our kops-based k8s cluster was that it was a little too hard, and that made it feel scary for people to experiment with. After all, this was our production infrastructure we’re talking about here and it needs to stay up and be easy to understand. Having had positive experiences with Amazon’s “managed” EKS offering in the past, this felt like the natural choice moving forward. It balances accessibility with reliability, while providing us ops volunteers a reasonable reduction in the complexity and effort of day-to-day maintenance.</p><p>Furthermore the idea of “serverless” Kubernetes with the <a href="https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-available/">new Fargate-based operating model</a> seemed an attractive way to even further reduce operations concerns. It didn’t quite work out, but more on that later.</p><h3>Part 3: Building out the new cluster</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*aG65bVTM3SGhhY103mD46A.jpeg" /><figcaption>An AWS Container, I don’t make this stuff up</figcaption></figure><p>Thanks to <a href="http://eksctl.io/">eksctl</a>, building out a new EKS cluster was incredibly easy. I appreciated that I could practice some Infrastructure as Code practices by utilizing <a href="https://github.com/OperationCode/operationcode_infra/blob/7396262afe38c2de4451c7f023d95e7c8649747c/kubernetes/eksctl/operationcode-backend.yaml">a config file</a>. eksctl also provides a really handy eksctl utils write-kubeconfig which made onboarding new users incredibly simple.</p><p>After the cluster was built (this took virtually no time at all!), the <a href="https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html">AWS docs</a> recommended installing the following components:</p><ul><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html">ALB ingress controller</a></li><li><a href="https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/external-dns/setup/">External DNS controller</a></li><li><a href="https://docs.aws.amazon.com/eks/latest/userguide/vertical-pod-autoscaler.html">Vertical Pod Autoscaler</a></li></ul><p>These were relatively easy to set up following the instructions, except for external-dns. The <a href="https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/v1.1.5/docs/examples/external-dns.yaml#L55">example configuration provided pointed to an outdated image</a> and unfortunately the <a href="https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md">external-dns docs</a> didn’t provide any guidance on how to make it work within Fargate. After some experimentation and help from the friendly folks in the <a href="https://slack.k8s.io/">Kubernetes slack</a> (#eks channel) I figured out that I needed to use the eksctl create iamserviceaccount command to create a role for external DNS, and then use the eks.amazonaws.com/role-arn annotation to map the kubernetes service account to the IAM role (<a href="https://github.com/OperationCode/operationcode_infra/blob/master/kubernetes/eksctl/external-dns/external-dns.yaml#L7">code here</a>).</p><p>Finally, we use <a href="https://argoproj.github.io/argo-cd/">Argo CD</a> for continuous delivery of our back-end services that run on k8s, so that was important to get deployed. One of the known limitations of EKS+Fargate is that it doesn’t support persistent volume claims, it can only support ephemeral pods. I ended up adding a single t3a.medium node to the cluster so that Argo wouldn’t lose all of its data every time a pod cycled :)</p><p>Rather than use Argo’s HA install template (that was overkill for us), I modified the quickstart template’s Redis server section to be a StatefulSet with a persistent volume claim (<a href="https://github.com/OperationCode/operationcode_infra/blob/master/kubernetes/argocd/install.yaml#L2294-L2304">code here</a>) which provides us a sufficient level of availability for Argo.</p><h3>Part 4: Updating our services k8s manifests</h3><p>In testing, we found a few changes that were going to be necessary to our application’s manifests in order to run on the new cluster.</p><p>Most importantly was modifying all the services from type: ClusterIP to type: NodePort as required by the ALB ingress controller and External DNS controller. After that it was simply implementing the ingress annotations and rules in the right fashion. The controllers watch for these annotations and then automatically create AWS ALB and Route53 DNS records, pretty cool!</p><pre>apiVersion: extensions/v1beta1<br>kind: Ingress<br>metadata:<br>  name: back-end<br>  annotations:<br>    kubernetes.io/ingress.class: alb<br>    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:633607774026:certificate/d59d030e-0239-4bfa-8553-e4bafb6481b4<br>    alb.ingress.kubernetes.io/listen-ports: &#39;[{&quot;HTTPS&quot;: 443}]&#39;<br>    alb.ingress.kubernetes.io/scheme: internet-facing<br>    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01<br>    external-dns.alpha.kubernetes.io/hostname: backend.k8s.operationcode.org<br>  labels:<br>    app: back-end<br>spec:<br>  rules:<br>  - host: backend.k8s.operationcode.org<br>    http:<br>      paths:<br>      - path: /*<br>        backend:<br>          serviceName: back-end-service<br>          servicePort: 80</pre><p>And here’s the complete pull request with all the changes:</p><p><a href="https://github.com/OperationCode/operationcode_infra/pull/143/">Rework Kubernetes deployment to use EKS+fargate for everything by irvingpop · Pull Request #143 · OperationCode/operationcode_infra</a></p><p>Finally cut-over was as simple as a DNS record change, which was performed (mostly) successfully on a Sunday evening. It helped that our PostgreSQL database is on RDS and didn’t need to be touched aside from a security group change.</p><h3>Part 5: Realizing our costs were still too high (ugh!)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSKS2OgbSk_5N_H07o_SZg.jpeg" /><figcaption>DevOps action set</figcaption></figure><p>After a week of operating the new setup, I cleaned up the kops cluster and then we began closely monitoring daily AWS spend. I implemented <a href="https://www.concurrencylabs.com/blog/introducing-miserbot-aws-cost-management/">MiserBot</a> to give us daily AWS spend updates. Unfortunately once it stabilized we were still running a daily cost of $14.51 day ($435/mo) and that still was way too high.</p><p>My analysis and recommendations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcrGLPoFMH82-TdzltHgDw.png" /><figcaption>Back in the #infrastructure-team chat in the OC slack</figcaption></figure><h3>Part 6: Transitioning to Spot Instance worker nodes</h3><p>Okay so Fargate was a cool idea, but it definitely cost more than I thought it would. The Fargate hours, NAT gateway and t3a.small were a combined spend of $6.16/day that I knew we could roll into a triplet of AWS t3.small instances that would cost a total of $0.45/day if spot priced. That was a worthwhile change.</p><p>I wanted to be careful about spot instances cycling (being terminated and re-created) impacting our operations, so I implemented additional monitoring with Cloudwatch alarms and <a href="https://github.com/widdix/cloudwatch-alarm-to-slack/">a Cloudwatch alarm to Slack sender</a>.</p><p>I was also a little bit nervous about transitioning the kube-system pods from Fargate to worker nodes, but that ended up going very smoothly as well as cutting over rest of the namespaces. I did discover after transitioning the staging environment that our RDS security groups were set to allow traffic from the NAT gateway (this also adds data transfer costs!), so I had to quickly implement VPC peering which was a better solution from both a security and cost perspective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4oieBzmuUFDMWB1bGkONRQ.png" /><figcaption>You can ninja-edit Infrastructure if nobody notices, right?</figcaption></figure><p>One additional thing I realized was how BAD our backend request latency was on Fargate, with many requests nearing 3 seconds! Post cutover the graph looks so much better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNIpw6waa_z6bYUHMevliw.png" /><figcaption>Honeycomb’s Slack unfurls are so good, right?</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wV6kXTxomp-53izM_2Wa4Q.png" /><figcaption>The difference in response time was shocking!</figcaption></figure><h3>Conclusion</h3><p>Running Kubernetes takes work, testing and vigilant monitoring. The project moves so fast that you can’t take your eye off it for a year and then expect a smooth upgrade. Let it go for 4–5 minor versions and you’ll find it much easier to lift-and-shift everything to a new cluster, as we did.</p><p>We’re now in a much better place in terms of cost and manageability for our Kubernetes cluster. The cluster itself now costs less than $5/day to operate (including the ingress ALBs) and our total AWS spend is $7.80/day which is a far more sustainable rate for an organization such as ours.</p><p>Fargate was the bit of a disappointment for this project: I wish AWS had been clearer upfront with EKS Fargate costs. Yes you can find it if you really dig around, but it is non-obvious. But the price/performance ratio was the biggest bummer.</p><p>Finally, organizations like OperationCode do good work and depend on the generosity of our service providers as well as your donations. <a href="https://operationcode.org/">Please donate!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=229e391e3ad9" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/refactoring-the-kubernetes-infrastructure-at-operationcode-229e391e3ad9">Refactoring the Kubernetes infrastructure at OperationCode</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey with Habitat on Amazon ECS, part 3]]></title>
            <link>https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-3-22d1256c25c?source=rss-25ca8c726d9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/22d1256c25c</guid>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <dc:creator><![CDATA[Irving Popovetsky]]></dc:creator>
            <pubDate>Sat, 16 Sep 2017 20:27:52 GMT</pubDate>
            <atom:updated>2017-09-16T20:27:52.485Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-1-39ae8af6542">first part of this series</a> I got the <a href="https://learn.chef.io/modules/chef-automate-pilot/scan-for-compliance#/">Chef Automate Pilot</a> container stack up and running on ECS. In <a href="https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-2-1115c0463f83">the second part</a>, I enabled it to survive the termination of any container or host. In this post my goal is to break this application out of the single-host box it’s been stuck in and enable the it to scale out.</p><h3>Solving the Initial Peer Problem</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/80c8b7dc04486fb97eb3a5cdd7504fac8e07aafc#diff-5defeeb087e0527b6a3ade7ca4a1af85">Move habitat initial peers to the hosts themselves, and have the host... · chef-customers/aws_native_chef_server@80c8b7d</a></p><p>As I mentioned in <a href="https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-1-39ae8af6542">part 1</a>, I was taking a bit of a shortcut by depending on the postgresql (and later rabbitmq) containers as the initial member of Habitat’s <a href="https://www.habitat.sh/docs/internals-supervisor/">gossip ring</a>. That was totally fine for operating a stack of containers that are bridged together on a single host, but now I need the gossip ring to extend to other hosts.</p><p>First, a digression about a wrong turn I took. I spent a few days with <a href="https://www.weave.works/">Weaveworks</a> and <a href="https://linkerd.io/">linkerd</a>, attempting get myself both overlay networking (routable container networks between my hosts) and service discovery (so that my habitat supervisors could find each other and form a gossip ring). I found the Weaveworks solution to be pretty well baked, even offering <a href="https://www.weave.works/docs/scope/latest/ami/">a drop-in replacement AMI for Amazon ECS</a> and I went so far as to build my own AMIs (with updated components) based on their <a href="https://github.com/weaveworks/integrations/tree/master/aws/ecs/packer">Packer template</a>.</p><p>Unfortunately, Habitat refused to play nicely with Weave’s networking scheme — it could only identify itself using the container’s bridge IP address (172.31.x.x), never seeming to grok the IP address that Weave supplied. There’s possibly something that could be fixed on the Habitat side to speak Weave, and is worth exploring in the future.</p><p>All was not lost, however, because if I hadn’t gone deep with Weave I wouldn’t have found <a href="https://github.com/chef-customers/aws_native_chef_server/commit/80c8b7dc04486fb97eb3a5cdd7504fac8e07aafc#diff-5defeeb087e0527b6a3ade7ca4a1af85">this awesome bit of code</a> that Weave uses to find peers by searching the AWS API for other members of the same Autoscale group. That gave me the idea to install a Habitat supervisor on each host and use the same discovery trick, also refactoring the host’s boot-time setup scripts while I was there.</p><p>Now I had a fleet of docker hosts, each running a habitat supervisor and forming a full gossip ring! And because I know that each of these hosts has a docker0 interface listening on 172.17.0.1 I can safely use that as the --peer argument for all of my containers</p><h3>Revisiting the ‘Host’ Networking Mode</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/1682d724653cf8b9c1a5733956292735f49a03f6#diff-5defeeb087e0527b6a3ade7ca4a1af85">Switch to the &#39;host&#39; networking mode, so that all containers have the... · chef-customers/aws_native_chef_server@1682d72</a></p><p>Now that all of my ECS hosts formed a gossip ring, I had a slight problem — the information stored in that ring becomes somewhat permanent, it isn’t thrown away and started fresh with each new ECS task deployment. And because the deployment can move from one host to another, I started seeing error messages like these:</p><pre>Could not send rumor to &quot;9fc4302dc0b14ea09dc1e4ba55e28d24&quot; @ &quot;172.31.31.122:9638&quot;; ZMQ said: Resource temporarily unavailable<br>Could not send rumor to &quot;9fc4302dc0b14ea09dc1e4ba55e28d24&quot; @ &quot;172.31.31.122:9638&quot;; ZMQ said: Resource temporarily unavailable</pre><p>That’s because each ECS host creates its own RFC1918 address space for the containers to talk to each other. But the containers on one host cannot reach the containers on another host (no overlay networking, remember!)</p><p><strong>Port conflicts begin to Rule Everything Around Me</strong></p><p><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#network_mode">Amazon’s documentation</a> makes the ‘host’ mode sound pretty awesome:</p><pre>The host network mode offers the highest networking performance for containers because they use the host network stack instead of the virtualized network stack provided by the bridge mode</pre><p>But also introduces a challenge:</p><pre>however, exposed container ports are mapped directly to the corresponding host port, so you cannot take advantage of dynamic host port mappings or run multiple instantiations of the same task on a single container instance if port mappings are used.</pre><p>As soon as I tried it out, I realized that this wasn’t going to work for Habitat, because each supervisor wants to listen on ports 9638 and 9631 — but since the host’s habitat has already claimed those ports, none of my containers would be able to start. It would be ideal if Habitat could automatically find a new port if the default ones are taken, but at this time it cannot.</p><p>After <a href="https://github.com/habitat-sh/habitat/issues/3025">some discussion with the Habitat team in an issue report I filed</a>, I implemented a simple workaround to specify in the ContainerDefinitions a unique --listen-gossip and --listen-http port for each container. That got my containers up and running in host mode, where each container now saw itself as the IP address of the host.</p><h3>Dealing with dead and alive members of a service group</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/70b3e303fc3a53fb2c5ec66f98104671f13580fb#diff-5defeeb087e0527b6a3ade7ca4a1af85">Implemented #isAlive helper in the habitat configs and did custom bui... · chef-customers/aws_native_chef_server@70b3e30</a></p><p>Hurray, my containers are back up! But why are all of the services broken?</p><p>The Habitat service discovery logic used by these applications was pretty simple, grabbing the first member found in the service group.</p><pre>{{#if bind.elasticsearch}}<br>  export ELASTICSEARCH_URL={{bind.elasticsearch.first.sys.ip}}<br>{{else}}<br>  export ELASTICSEARCH_URL={{cfg.elasticsearch_host}}<br>{{/if}}</pre><p>Looking at the census data ( curl <a href="http://localhost:9631/census">http://localhost:9631/census</a> ), I see the problem. There were multiple members of each service group — dead ones ( &quot;alive&quot;: false, &quot;departed&quot;: true) and alive ones. It was nice of Habitat to list them in chronological order, but the above logic always picked the oldest (and deadest!) one.</p><p>Chatting in the HabiSlack, I got some sage advice from Ian Henry and <a href="https://medium.com/u/9978525af468">Adam Jacob</a> — Use the #eachAlive helper. Awesome, now I can filter out the dead members! But I still had a problem — the way that ECS manages deployments it would start the new TaskDefinition before stopping the old one, which means that there would be two alive members — the new one and the one that’s about to go down. To deal with that, I needed to grab the last member of the list. The final result:</p><pre>{{#if bind.elasticsearch}}<br>  {{#eachAlive bind.elasticsearch.members as |member|}}<br>    {{#if @last}}<br>  export ELASTICSEARCH_URL={{member.sys.ip}}<br>    {{/if}}<br>  {{/eachAlive}}<br>{{else}}<br>  export ELASTICSEARCH_URL={{cfg.elasticsearch_host}}<br>{{/if}}</pre><p>This is where Habitat also does something VERY awesome. As soon as the topology changes, it automatically recomputes any impacted configuration files and reloads impacted services. For services like Nginx that reload very quickly and can handle multiple upstreams, I can skip the #if @last conditional entirely!</p><h3>Breaking up the Monolith</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/96f2640925c4100c08a3eb43dbfe86a8772add98#diff-5defeeb087e0527b6a3ade7ca4a1af85">refactor big combo task out into MICRO SERVICES · chef-customers/aws_native_chef_server@96f2640</a></p><p>Up until this point, I’d been treating my application as a monolith — all of the containers were defined in a single TaskDefinition, and in ECS parlance that means all of containers in the stack will be deployed together on to a single ECS host, and an update to one container results in a redeploy of the entire stack.</p><p>Now that I’d solved for networking and service discovery I could take the next step toward higher availability and scale-out — which meant breaking out each service into its own ECS Service andTaskDefinition. The functionality is identical to the previous stack, but these smaller task definitions are easier to manage and scale independently.</p><h3>First attempt at active-active</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/4fda6ce2be33552a948c45fc9acc1a23da037c09#diff-5defeeb087e0527b6a3ade7ca4a1af85">Hopefully fix an issue where the hosts ulimits kick in after about 24... · chef-customers/aws_native_chef_server@4fda6ce</a></p><p>Thinking that I was ready to see what would happen if I ran two of some containers, I flipped the containerDesiredCount setting on a number of TaskDefinition’s to equal the DesiredCapacity value of the ECS host ASG. Two hosts, two of each container — we’ve worked so hard to get here, we deserve this to work, right?</p><p>Unfortunately not quite. I experienced a couple of issues that caused services to fail on startup resulting in the container failing in a tight loop. Each one left behind a couple of zombie processes until the host hit its <a href="https://ss64.com/bash/ulimit.html">ulimit</a> on nproc (number of processes per user) so that I couldn’t even SSH into it!</p><p>It doesn’t make a ton of sense to apply per-user ulimits on an ECS host, particularly when all of the containers effectively map to the same user ID (psst, Amazon, if you’re listening please take note). To fix that I used the power of <a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html">cfn-init</a> to write out an /etc/security/limits.d/ecs.conf file to raise those limits to <a href="https://serverfault.com/questions/697936/pam-limits-operation-not-permitted-for-sshd">the highest allowable value without breaking PAM</a>.</p><p>And because I could safely terminate things without fear of data loss (thanks to the work done in part 2), I could simply nuke any busted ECS hosts, or all of the ECS hosts if I wanted to start with a clean and empty Habitat gossip ring. Autoscale would bring them all back and my data was safe in RDS, ES &amp; EFS.</p><h3>Better Monitoring and Starting to Understand ECS Deploys</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/32464811838da167f4ca76efd5a705e9b03419b1#diff-5defeeb087e0527b6a3ade7ca4a1af85">make a separate load balancer for the Workflow service, so we can mon... · chef-customers/aws_native_chef_server@3246481</a></p><p>But why was my Workflow container restarting in a tight loop, and what could I do about that? Cloudwatch logs would sometimes show that the Habitat supervisor can’t bind to the ports it needs, which is weird because I gave it unique ports.</p><p>That’s when I realized a flaw in my thinking — by setting the number of workflow containers equal to the number of ECS hosts, I introduced a problem. To understand why, you have to understand how Cloudformation works. When you update a Cloudformation template with any change to a <a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html">resource that requires replacement</a>, it does this:</p><ol><li>Launch a new version of that resource</li><li>Wait for it to be operational</li><li>Delete the old version (Cleanup phase)</li></ol><p>Since ECS Taskdefinitions require replacement, that means I needed to have <strong>double the number of containers running for a short period of time</strong>. But since I was constrained by named and bound TCP ports, I couldn’t run more containers than ECS hosts. To solve for this I backed off from setting container’s DesiredCount to host DesiredCapacity and just put them back to 1. More on that next.</p><p>While playing around with ECS, I also started to see the value of the monitoring it can provide when LoadBalancers was hooked up to a service. For example, the automate-nginx containers automatically join an <a href="http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html">ALB Target Group</a> called NginxTG that has HTTP-based monitoring enabled. If any of the containers started failing the TG’s health checks, ECS would automatically restart those containers for me! This can be quite a bit more useful than Habitat’s own monitoring, which can only report to the gossip ring that a particular service is unhealthy. In container-scheduled environments, I wonder if it would be better if the supervisor simply quit after a certain number of failed consecutive health checks.</p><h3>Eureka! Real HA</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/8afedcde1ee6771ef55eb270e44e33077ad0b1e0#diff-5defeeb087e0527b6a3ade7ca4a1af85">Elimiate port binding conflicts by defining ContainerPort for all lis... · chef-customers/aws_native_chef_server@8afedcd</a></p><p>There’s two things that ECS needs in order to prevent any containers playing “port bumper cars”: enough host headroom to launch another stack, and also an accurate representation of all the bound ports called out as PortMappings. I realized that my previous issues with the Habitat supervisor being unable to bind ports was because I hadn’t called them out in the PortMappings — so even if the container didn’t run a service that listened on any ports, I still need to help the scheduler out by telling it what ports Habitat listened on.</p><p>At this point I was able to successfully run two instances of each container, and Nginx was load balancing requests between them. And by doubling the size of my ECS cluster DesiredCapacity value to 4 I can safely deploy a new version of my stack.</p><h3>Scale Out</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mkwGQ9E8kafhtcWL98XLsg.png" /></figure><p>In the <a href="https://pages.chef.io/rs/255-VFB-268/images/ScalingChefAutomate_2017.pdf">Scaling Automate white paper</a> we identified the Logstash service as a key bottleneck that required scaling out because it is effectively a single-threaded process. In larger configurations we had 16 and 32 CPU servers running as many as 30 Logstash processes — not very cost effective or scalable!</p><p>Fortunately the Logstash service doesn’t need to listen for traffic, so I could revert that service to bridge mode and remove the PortMappings definition — thereby allowing me to run more Logstash processes than I have hosts. The only unfortunate downside of this configuration was that all of my habitat supervisors logs were spammed with messages like these:</p><pre>Could not send rumor to &quot;c3609ee274c74a7db4a735283c522499&quot; @ &quot;172.31.34.16:9652&quot;; ZMQ said: Resource temporarily unavailable</pre><pre>Could not send rumor to &quot;ad8abeab3b5a46be8a68370520a54812&quot; @ &quot;172.31.15.127:9652&quot;; ZMQ said: Resource temporarily unavailable</pre><pre>Could not send rumor to &quot;242135995cca438d9aafbb9b7eee5c11&quot; @ &quot;172.31.16.239:9652&quot;; ZMQ said: Resource temporarily unavailable</pre><p>That’s unfortunate because it means that a subset of my containers don’t have two-way communication, but it doesn’t seem to have any major impact on Habitat’s operation. As soon as a feature is developed for <a href="https://github.com/habitat-sh/habitat/issues/3025">Habitat issue 3025</a> that I mentioned previously, I could go back to host network mode for Logstash and eliminate those error messages.</p><h3>Conclusion</h3><p>Thanks for sticking with me throughout this series. I started with a laptop-centric application and built it out for resilience and scale. Along the way I learned a great deal about ECS and Habitat, documenting each step of the way with git commits which made writing this blog series possible. Thanks also to my awesome coworkers at Chef for help they provided along the way, I couldn’t have done it without them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22d1256c25c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey with Habitat on Amazon ECS, part 2]]></title>
            <link>https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-2-1115c0463f83?source=rss-25ca8c726d9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1115c0463f83</guid>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Irving Popovetsky]]></dc:creator>
            <pubDate>Mon, 11 Sep 2017 15:46:13 GMT</pubDate>
            <atom:updated>2017-09-13T19:12:31.341Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-1-39ae8af6542">first part of this series</a> we got the <a href="https://learn.chef.io/modules/chef-automate-pilot/scan-for-compliance#/">Chef Automate Pilot</a> container stack up and running on ECS. Now let’s make it survive termination of any container or EC2 instance without losing data by adding AWS RDS, EFS and Elasticsearch. A story told in 3 git commits:</p><h3>RDS and EFS</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/9e2d5e90e8a2f508d330be2cca2767bf7f0cec32#diff-5defeeb087e0527b6a3ade7ca4a1af85">working state: switch to external RDS postgres and start laying the g... · chef-customers/aws_native_chef_server@9e2d5e9</a></p><p>We know that Chef Automate stores almost all of its state in PostgreSQL, but is also a Git server and stores repositories on disk. We need to get all of that data out of the disposable container volumes and into highly available persistence stores.</p><p><strong>PostgreSQL</strong></p><p>For starters, I’ve borrowed a chunk of code from <a href="https://github.com/chef-customers/aws_native_chef_server/blob/master/backendless_chef.yaml#L609">another template I own</a> to add the AWS::RDS::DBInstance resource and friends — that was the easy part. Trickier is to extract the postgresql and postgresql-data containers from the stack.</p><p>Once you remove those two from the ContainerDefinitions, you have to take care of two things:</p><ol><li>All of the other containers previously considered postgresql to be the <em>Initial Peer</em>, so I’ll just re-point that to the rabbitmq container because we’ll be keeping that one, and it is now the most depended-on container.</li><li>Many of the containers also had binds to postgresql (--bind database:postgresql.default passed to the habitat supervisor) which have to be replaced with environment variables like so:</li></ol><pre>        Environment:<br>        - Name: HAB_WORKFLOW_SERVER<br>          Value: !Sub |<br>            sql_user = &quot;${DBUser}&quot;<br>            sql_password = &quot;${DBPassword}&quot;<br><br>            [postgresql]<br>            vip = &quot;${DBPostgres.Endpoint.Address}&quot;<br>            port = 5432</pre><p>This is where the magic of Cloudformation combines with the magic of Habitat — I can easily pass information about the RDS instance (hostname, username, password) in to Habitat’s runtime configuration and it does the right thing with that. As you can see, <a href="https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-references-simplified-substitution/">CloudFormation’s new-ish YAML format</a> makes variable interpolation delightful in multi-line strings, especially compared to the JSON format.</p><p>Now you may wonder: how are you supposed to know what configuration to pass in to a particular habitat package? The awesome Habitat <a href="https://bldr.habitat.sh/#/pkgs/frog-hornets-nest/workflow-server/1.6.81/20170808114045">depot site has the answer</a> (scroll down to the <em>Configuration</em> section to see all the variables Habitat’s <a href="https://github.com/toml-lang/toml">TOML</a> config format). The Habitat docs describe the <a href="https://www.habitat.sh/docs/run-packages-apply-config-updates/">methodology for passing in runtime configuration via environment variables</a> although I was never able to get the JSON format to work reliably (Habitat auto-detects the format of the variable).</p><p>Removing the <a href="https://www.habitat.sh/docs/run-packages-binding/">Habitat bindings</a> and switching to environment variables worked fine for all of the services in the stack except the new notifications service. I didn’t realize when making this first commit that notifications didn’t need to talk to Postgres at all, so I passed in the environment variables and then had to “fake it out” into skipping the startup-time bind wait by passing in a phony bind: --bind database:rabbitmq.default (friends, don’t try this at home, it’s a bad idea and won’t work).</p><p>Later on I realized that notifications configuration file was missing an important bit of code to make the binding conditional and used pkg_binds instead of pkg_binds_optional in the plan. That led to this PR back to automate:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7628ee94ee733db1249e4bc8ab283f0/href">https://medium.com/media/c7628ee94ee733db1249e4bc8ab283f0/href</a></iframe><p>While waiting for that to get accepted I build some of <a href="https://hub.docker.com/r/irvingpop/">my own docker containers</a> with those changes, which Habitat makes easy with the <a href="https://www.habitat.sh/docs/run-packages-export/">hab pkg export docker</a> command. That way I could quickly test those changes in my stack and get feedback quickly.</p><p><strong>Volumes (EFS)</strong></p><p>AWS EFS provides a way for persisting data across container restarts, and even providing concurrent access to files across containers (if you can do that safely) and that’s good old-fashioned NFS. Except it isn’t because EFS provides multi-AZ availability and NFS 4.1 isn’t exactly old fashioned — providing parallel access (pNFS), file locking and significant performance improvements. You still shouldn’t use it <a href="http://docs.aws.amazon.com/efs/latest/ug/performance.html#performance-overview">host your database files</a> but for low-intensity IO it is totally fine and buys us a ton of flexibility.</p><p>As <a href="https://aws.amazon.com/blogs/compute/using-amazon-efs-to-persist-data-from-amazon-ecs-containers/">various AWS articles</a> demonstrate, there are a few slightly awkward things about integrating EFS in to your ECS cluster:</p><ul><li>You have to add the commands to mount the EFS filesystem on each of your ECS instances, as I do in the UserData section of the AutoScaling LaunchConfiguration (AWS if you are listening, let’s make that automagical!)</li><li>You have to create an AWS::EFS::MountTarget for each subnet you operate in. That’s hard for shareable templates, where you may be operating in 2,3 or even <a href="https://twitter.com/irvingpop/status/881352166400606209">SIX Subnets</a> in your region! (psst, AWS, please just let users pass in an array of subnets like you do in most others services ;)</li></ul><p>Once that’s done, you can now tell ECS exactly where to put those volumes on the host (hint: on to the EFS mount) like so:</p><pre>      Volumes:<br>      - Name: maintenance<br>        Host:<br>          SourcePath: !Sub /mnt/efs/${AWS::StackName}/maintenance</pre><p>It took some digging around to realize exactly what containers data I should be mounting on EFS (I still don’t really know what that maintenance volume does), but in later commits I start putting the habitat data volume for key containers there. So let’s move on to that!</p><h3>AWS ES, A Signing Proxy, and HTTPS</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/29b6833f93f1c2bceb2816af806ef413ba1ede28">this battle station is now fully operational · chef-customers/aws_native_chef_server@29b6833</a></p><p>Okay so sometimes I get a bit punchy with my commit messages. Just like that Death Star, we’re hilariously not fully operational yet :D</p><p>What we’re doing here is replacing the elasticsearch container with AWS’s ES (Elasticsearch) service, which is where all that sweet visualization and reporting data is going to go. Now it would be super cool if ES had a simple access control scheme like RDS (VPC SecurityGroups) but no, they just had to be different!</p><p>AWS ES controls access by IAM roles, <a href="https://www.reddit.com/r/aws/comments/3qcvss/is_there_really_no_way_to_safely_limit_access_to/">and isn’t integrated with VPC at all</a> (your traffic goes to a public IP). Each request to ES must be signed <a href="http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html">the same way that AWS API requests are signed</a>. Fortunately my team and I had already appropriated a useful bit of code for that: the <a href="https://github.com/nsdavidson/aws-signing-proxy">aws-signing-proxy</a> (credit to Chris Lunsford for the original code).</p><p>All I needed to do was habitize that, which was super easy when using another go-based application as an example:</p><p><a href="https://github.com/nsdavidson/aws-signing-proxy/pull/5/files">Habitize v1 by irvingpop · Pull Request #5 · nsdavidson/aws-signing-proxy</a></p><p>One thing I realized was that you need to export http-port instead of port (in the plan.sh) just like the <a href="https://bldr.habitat.sh/#/pkgs/core/elasticsearch/5.5.0/20170726170852">habitat elasticsearch package</a> — that way binds that previously depended on elasticsearch could now depend on the aws-signing-proxy service as a drop-in replacement.</p><h3>EFS-Mounted Volumes, Revisited</h3><p><a href="https://github.com/chef-customers/aws_native_chef_server/commit/a1510bd6f9237b39937be26740cbb70a6ec4721b">Robustification · chef-customers/aws_native_chef_server@a1510bd</a></p><p>Watching the container logs in Cloudwatch Logs, I noticed that other services were taking a while to start up because they were re-creating various files at init time. For example, the automate-nginx container had a bit of code like this:</p><pre># Generate a private key if one does not exist.<br>cert_file=&quot;{{pkg.svc_data_path}}/cert&quot;<br>key_file=&quot;{{pkg.svc_data_path}}/key&quot;<br>if [[ ! -f &quot;$cert_file&quot; ]]; then<br>  openssl req \<br>    -newkey rsa:2040 -nodes -keyout &quot;$key_file&quot; \<br>    -x509 -days 3650 -out &quot;$cert_file&quot; \<br>    -subj &quot;/C=US/O=Chef Software/OU=Chef Delivery/CN=#{{cfg.server_name}}&quot;<br>  chmod 600 &quot;$cert_file&quot; &quot;$key_file&quot;<br>fi</pre><p>Thanks to some very forward-thinking developers, we can skip the expensive opensslkey generation step on subsequent starts by mounting the habitat data folder on EFS!</p><p>Habitat is already smart enough to instruct docker to mount the /hab/svc/servicename/data directory as a separate volume, as I found digging around via docker inspect. So instructing docker to mount that on EFS made sense for most of the containers.</p><p>One debatable service was rabbitmq. I chose not to mount this on EFS because I was concerned about the performance impact of doing so, particularly in high scale scenarios (which is our ultimate goal, after all!) Also, in our experience RabbitMQ tends to be greatly impacted by slow disk when handling large durable queues, so let’s give it as fast disk as we can.</p><h3>Conclusion</h3><p>Now we have a Chef Automate container stack that can survive a wide variety of faults up to and including instance termination — and can recover with all of its data in just a couple of minutes.</p><p>In the next post, I’ll start working on multi-host operations for even better availability as well as options to scale-out.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1115c0463f83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey with Habitat on Amazon ECS, part 1]]></title>
            <link>https://medium.com/@irvingpop/a-journey-with-habitat-on-amazon-ecs-part-1-39ae8af6542?source=rss-25ca8c726d9b------2</link>
            <guid isPermaLink="false">https://medium.com/p/39ae8af6542</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <dc:creator><![CDATA[Irving Popovetsky]]></dc:creator>
            <pubDate>Sat, 09 Sep 2017 20:07:42 GMT</pubDate>
            <atom:updated>2017-09-10T03:25:38.908Z</atom:updated>
            <content:encoded><![CDATA[<p>Amazon’s ECS (EC2 Container Service) provides a compelling offering to developers and operators who are already very comfortable with AWS and its tooling. In this series of posts I explore deployment strategies for Containerized applications built using <a href="http://habitat.sh/">Habitat</a> on ECS.</p><p>The mission: Taking an application stack from “works on my laptop” to “works in production”.</p><h3>Step 1, Picking an app</h3><p>In my day job I work with a fairly complex microservices-based application, <a href="https://docs.chef.io/chef_automate.html">Chef Automate</a>. Specifically the containerized version you see published as the <a href="https://learn.chef.io/modules/chef-automate-pilot#/">Automate Pilot</a> — and meant for non-production demo and trial purposes only. But once I took that demo apart and understood what made it tick, I realized that it could be scaled up and out into a robust Production-ready app.</p><p>A stretch goal for me personally is to continue the underlying technical work we did for this whitepaper: <a href="https://pages.chef.io/rs/255-VFB-268/images/ScalingChefAutomate_2017.pdf">Scaling Chef Automate Beyond 50,000 nodes</a> but to see if we can reimagine the scaling solutions in a containerized version of the app — while adding high availability and possibly even saving on cost.</p><h3>Step 2, Picking a Container Scheduler</h3><p>ECS is a container scheduling system built on top of <a href="http://docker.com/">Docker</a> and AWS EC2. As somebody who uses AWS daily, I found it pretty easy to understand how it worked and to integrate it into my workflow.</p><ul><li>You define what you want to do with Task Definitions (which define which containers you want to run) combined with ECS Services.</li><li>These can be submitted directly via the AWS API, or more commonly via CloudFormation.</li><li>You can even take a regular docker-compose.yml and import it straight into ECS! That lowers barrier slightly when transitioning from local workstation development to ECS.</li></ul><p>The reason that I chose ECS for this project is that it provides an elegant solution to running data persistence services (Postgres, Elasticsearch, etc) in the AWS RDS and Elasticsearch DBaaS, rather than trying to run those services in containers. The magic that ties it all together is Cloudformation, which allows you to provision persistent data services (RDS, etc) and your ECS cluster in a single command, from a single infrastructure specification.</p><h3>Why Habitat on ECS?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rA7gEHieHlXoUDotxl6ehg.png" /></figure><p>Unlike some <a href="https://kubernetes.io/">competing</a> <a href="https://mesosphere.com/product/">solutions</a>, ECS doesn’t provide you niceties like overlay networking and service discovery out of the box — which are really important if you want to run your service on more than one docker host! For service discovery there are many proposed solutions to solve for this that involve setting up <a href="https://aws.amazon.com/blogs/compute/service-discovery-an-amazon-ecs-reference-architecture/">Lambdas</a>, <a href="https://aws.amazon.com/blogs/compute/service-discovery-via-consul-with-amazon-ecs/">Consul</a>, <a href="https://github.com/Netflix/eureka">Netflix Eureka</a>, or <a href="https://aws.amazon.com/blogs/compute/service-discovery-via-consul-with-amazon-ecs/">installing agents to report to DNS</a>. There are also 3rd offerings that provide both overlay networking and service discovery from <a href="https://www.weave.works/">Weave</a> and <a href="https://linkerd.io/">Linkerd</a> — of which I found the Weave offering compelling, and could honestly eliminate the need for Habitat if it had config management capabilities.</p><p>The compelling thing about Habitat is that it’s <em>the automation that travels with the app</em> so I didn’t have to build external services and add dependencies on those systems — which should simplify my deployment and reduce operational complexity. It also helped that much of Chef’s internal development is shifting from <a href="https://blog.chef.io/2012/06/29/omnibus-chef-packaging/">Omnibus</a>-style apps to Habitized apps, so a lot of the hard work had already been done on Automate by awesome people like <a href="https://twitter.com/libsysguy">Elliott</a> (hi!).</p><h3>What’s hard about Habitat on ECS?</h3><p>ECS’s lack of service discovery makes it harder for Habitat supervisors to initially find each other (the service discovery system needs service discovery to bootstrap itself!) and the lack of overlay networking makes it harder for supervisors to all have 2-way communication with each other.</p><p>That’s why the <a href="https://www.habitat.sh/docs/container-orchestration-ecs/#ec2-compute-service">Habitat docs section on ECS</a> simply tells you to use docker links (a deprecated feature) like so:</p><pre>version: &#39;2&#39;<br>services:<br>  mongo:<br>    image: aws_account_id.dkr.ecr.ap-southeast-2.amazonaws.com/billmeyer/mongodb:latest<br>    hostname: &quot;mongodb&quot;<br>  national-parks:<br>    image: aws_account_id.dkr.ecr.ap-southeast-2.amazonaws.com/mattray/national-parks:latest<br>    ports:<br>      - &quot;8080:8080&quot;<br>    links:<br>      - mongo<br>    command: --peer mongodb --bind database:mongodb.default</pre><p>What this does is leverage Docker (and ECS’s) <em>Bridge</em> mode networking, which is the default networking mode and required in order for linking to work. In this case the initial peer is the mongodbcontainer which the national-parks container can find because linking takes care of service discovery (the hostname mongodb is resolvable from any other linked containers) as well as 2-way communications (they are on the same bridged network, thus having unencumbered communications with each other).</p><p>ECS has another networking mode (called <a href="http://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RegisterTaskDefinition.html#ECS-RegisterTaskDefinition-request-networkMode">Host mode</a>) where each container doesn’t have its own IP address, and instead identifies as the IP address of the host. Any port that the container tries to listen on becomes a listening port on the host. More on that in a future post.</p><h3>A “simple” deployment</h3><p>Let’s port Automate Pilot’s <a href="https://chef-automate-artifacts.s3-us-west-2.amazonaws.com/stable/latest/pilot/docker-compose.yml">docker-compose.yml</a> file to ECS. I started with a fairly stock <a href="http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/quickref-ecs.html">ECS cloudformation snippet</a> and replaced the example service with the container definitions from the docker-compose config. Here’s what that looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7a38767d999ec26dcbb0487952f3405/href">https://medium.com/media/c7a38767d999ec26dcbb0487952f3405/href</a></iframe><p>What you see to the left is a complete and working Cloudformation template that is made up of 3 important parts:</p><ol><li>Parameter definitions — asking you important questions like what VPC to deploy into and server sizing (lines 2–35)</li><li>The ECS Cluster — really the underlying EC2 instances that live in an AutoScaling group, plus associated networking and security rules (lines 359–493)</li><li>The ECS Service and Task definition — The part that once was our plucky docker-compose.yml file, but now presented in a fashion that ECS can schedule, run and even monitor (lines 52–284)</li></ol><p>Let’s dive deeper into that ECS TaskDefition, because that deserves some unpacking. There are 9 container instances, providing a range from data persistence services (postgresql, elasticsearch), data processing services (logstash, rabbitmq) and API services (workflow-server, compliance, notifications andautomate-nginx). There’s also a trick-container called postgresql-data which lays down the initial Postgres data volume and then exits — we have to set that to Essential: false so that ECS is okay with it quitting.</p><p>Each container definition has some important components:</p><ul><li>The image name (which is pulled from <a href="https://hub.docker.com/u/chefdemo/">dockerhub</a>)</li><li>Tuning options (Cpu, Memory and Ulimits)</li><li>Links (as mentioned above for service discovery and networking between containers)</li><li>Command Line arguments that are passed to Habitat: A peer name (to form a <a href="https://www.habitat.sh/docs/internals-supervisor/">gossip ring</a>) and bindings (dependencies on a <a href="https://www.habitat.sh/docs/run-packages-service-groups/">service group</a>)</li><li>Environment variables passed to Habitat — TOML formatted parameters which inject <a href="https://www.habitat.sh/docs/run-packages-apply-config-updates/">configuration parameters </a>to be used at runtime by this container (and also consumable by dependent services on other containers)</li><li>Logging configuration, which is handy because it makes all container output available via CloudWatch Logs</li></ul><p>Another important aspect of this configuration is the use of Volumes — right now in a simple way to share the postgresql-data and maintenance volumes. More on this in a future post, there’s a ton of interesting things we can do with it</p><p><strong>Know your Habiterms: The Initial Peer</strong></p><p>In this stack the initial peer is defined as the postgresql container — all of the other containers must be linked to it in order to join the gossip ring and enable and perform service discovery.</p><p><strong>Know your Habiterms: Service bindings</strong></p><p>Service bindings are a super important part of gluing services together in Habitat. It says “Wait until the postgresql.default service group is available before starting” — which makes a number of order-of-operations and other orchestration problems magically disappear. It also allows you to explicitly depend on other services, and know what ports to bind to or export to.</p><h3>Neat things about the underlying ECS cluster</h3><p>One thing I really like about ECS is how it gives you full control over the underlying ECS instances. They’re essentially EC2 instances with a simple expectation: they run the ECS service and the cluster name is put in the /etc/ecs/ecs.config file. That’s it! It means you could replace the stock ECS-Optimized Amazon Linux images with CoreOS or Weave and it still works just as well (but with cool features).</p><p>In an upcoming post I’ll show exactly what I do with that, but for now I’ll show one very real problem I could solve easily: Elasticsearch 5 will refuse to start unless you set a certain sysctl value like so: sysctl -w vm.max_map_count=262144</p><p>That’s not a setting that can be set <em>inside</em> the container, it has to be set on the host — and guaranteed consistent on every docker host in your cluster. Cloudformation makes that easy, because I can set it in the UserData (boot time commands) for each instance in the cluster.</p><h3>Up Next</h3><p>In the next post, I’ll start to make this more robust by leveraging AWS RDS, ES and EFS services so that our application can survive termination of the containers or even the ECS instance!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39ae8af6542" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>