<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Christopher Lane on Medium]]></title>
        <description><![CDATA[Stories by Christopher Lane on Medium]]></description>
        <link>https://medium.com/@clanecfa?source=rss-14500ec7ee58------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*kHdl8qbc_r3F7_nw</url>
            <title>Stories by Christopher Lane on Medium</title>
            <link>https://medium.com/@clanecfa?source=rss-14500ec7ee58------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:45:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@clanecfa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing Kubernetes Manifests with Kustomize Feeder Repository]]></title>
            <link>https://medium.com/chick-fil-atech/writing-kubernetes-manifests-with-kustomize-feeder-repository-b67b0be7ad2e?source=rss-14500ec7ee58------2</link>
            <guid isPermaLink="false">https://medium.com/p/b67b0be7ad2e</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Christopher Lane]]></dc:creator>
            <pubDate>Tue, 28 Mar 2023 17:00:54 GMT</pubDate>
            <atom:updated>2024-03-15T18:21:13.566Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In our <a href="https://medium.com/chick-fil-atech/hundreds-of-weekly-deployments-managed-with-gitops-51f6cc60c6b3">last post on Chick-fil-A’s application platform</a>, we shared an overview of our GitOps strategy, but only briefly mentioned a critical component: the kustomize feeder repositories that are the foundation of the process. This post will focus on the design and implementation of these repositories.</p><p>Today, a significant portion of our sales are flowing through our digital channels, including <a href="https://www.chick-fil-a.com/one">The Chick-fil-A App</a> and <a href="http://chick-fil-a.com/">chick-fil-a.com</a>. We have seen significant growth in these channels since March 2020, when the impacts of COVID-19 started in North America.</p><p>Our digital properties are powered by our Digital Experience Engine (DXE). DXE is a cloud-based microservices architecture composed of ~150 services, running in our Amazon Elastic Kubernetes Service (EKS)-based application platform. We have hundreds of developers working on DXE and, collectively, they push thousands of commits and open hundreds of pull requests every week.</p><p>To start, let’s review our traffic pattern and GitOps-based deployments.</p><h3>Traffic Pattern</h3><p>We’ve shared a few examples of our daily traffic pattern previously, but let’s update those figures, as the volume of requests through DXE continues to grow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*80y8mWbxa0GpGkWur7CU7g.png" /><figcaption>DXE Daily Traffic</figcaption></figure><p>All times are US/Eastern. This figure probably isn’t terribly surprising: requests ramp up during breakfast, spike during the lunch rush, and hit another, smaller peak during dinner.</p><p>Overall, we exceed ~430K rpm at lunch, average about ~190K rpm, and handle more than 150M requests during a given day. If we’re running a promotion, these numbers can easily double.</p><h3>D<strong>eployment Process</strong></h3><p>The deployment process looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*eHYEXxWPgx1e3DoK8DeDCA.png" /><figcaption>DXE Deployment Process</figcaption></figure><p>The steps:</p><ol><li>The process is started by a developer pushing a commit to the mainline branch of the application repository. This triggers the build workflow in <a href="https://docs.github.com/en/actions">GitHub Actions</a>.</li><li>The application container is built, all tests are run, and the image is pushed to our <a href="https://jfrog.com/artifactory/">Artifactory</a> registry.</li><li>Next up is the focus on this post: the base manifests for the given application type are pulled from our feeder repository. We’ll discuss this in detail below.</li><li>The complete application manifests are committed and pushed to the atlas.</li><li><a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a> watches for changes to the atlas repository, and once the new manifests are pushed, it applies the changes to the cluster.</li><li>The new image is pulled from Artifactory.</li><li>The new version of application is deployed to cluster.</li></ol><h3>Feeder Repository</h3><p>When we started migrating DXE services from Elastic Beanstalk to our new EKS-based platform in 2018, there were only a limited number of folks in the organization that had experience developing and maintaining Kubernetes manifests. (How we addressed this knowledge gap will be, yet another, blog post!) It was clear from the beginning that we didn’t want each team writing the entirety of their own manifests. This would have immediately created a barrier of entry to the new platform, in addition to a confusing mess of requirements, tooling, and team silos.</p><p>We decided to build a set of base manifests for each of our common application types. Teams could pull in these manifests and “patch-in” any changes for their particular application using kustomize. This made the learning curve we were asking teams to climb less daunting: folks didn’t need to write a (potentially gigantic) pile of yaml to deploy their applications to Kubernetes. Rather, they just needed to learn how patch the base manifests.</p><p>So, what does this feeder repository look like? We’ve published a version of our application feeder repository on our public Github organization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxpUnFvenAhnfyETGzEcfA.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application">https://github.com/chick-fil-a/kustomize-application</a></figcaption></figure><p>Let’s explore this repository. At the root, we have the base directory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0NQwlaeXlo4Yg-uumMVog.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/tree/main/base">https://github.com/chick-fil-a/kustomize-application/tree/main/base</a></figcaption></figure><p>This is the starting point: the base layer of deployment and service manifests, as well as tangential resources like the horizontal pod autoscaler (HPA), pod disruption budget (PDB), and service monitor.</p><p>On top of base, we define our backend and frontend tiers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nzlUkFWQ1J8l9rqSFSfkdw.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/blob/main/backend/kustomization.yaml">https://github.com/chick-fil-a/kustomize-application/blob/main/backend/kustomization.yaml</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SAvQngt2ve8ovTwnRlSwDg.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/blob/main/frontend/kustomization.yaml">https://github.com/chick-fil-a/kustomize-application/blob/main/frontend/kustomization.yaml</a></figcaption></figure><p>As you can see, these are just thin layers that add common labels and annotations for their respective tiers.</p><p>Finally, we have the application type layer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*49j8u46D4dxjKDR_UjplxQ.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/tree/main/go-api">https://github.com/chick-fil-a/kustomize-application/tree/main/go-api</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BDC_FAGfz_DF51yybbEdJQ.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/tree/main/java-api">https://github.com/chick-fil-a/kustomize-application/tree/main/java-api</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DHDcZgkDUJ_GXLCy9kStSA.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/tree/main/python-api">https://github.com/chick-fil-a/kustomize-application/tree/main/python-api</a></figcaption></figure><p>These are the layers that teams reference in their own application repository.</p><p>Let’s write a complete set of application manifests by pointing to this layer with kustomize. We’ll assume we’re writing a Python API and we’re ready to deploy v0.0.1 to our development cluster. (We’ll use busybox:latest as a stand-in for our application in the next examples.)</p><p>First, let’s add a k8s/kustomization.yaml file to the root of our application repository:</p><pre>mkdir -p k8s<br>&lt;&lt;eof &gt;k8s/kustomization.yaml<br>resources:<br>- https://github.com/chick-fil-a/kustomize-application/python-api?ref=main<br><br>commonLabels:<br>  app: my-k8s-api<br><br>commonAnnotations:<br>  cm.chick-fil-a.com/system-tag: &quot;CAP_BLOG_POST&quot;<br><br>namePrefix: my-k8s-api-<br>namespace: my-k8s-api<br><br>images:<br>- name: application-image-placeholder<br>  newName: busybox<br>  newTag: latest<br><br>patches:<br>- target:<br>    kind: Deployment<br>  path: patches/deploy.yaml<br>eof</pre><p>The key is the resources list: the first item (and only item in our example) references the base manifests for a Python API from kustomize-application.</p><p>Our application isn’t doing much, and so, doesn’t need the default memory or CPU resources. Let’s patch our resource requests and limits to reflect this:</p><pre>mkdir -p k8s/patches<br>&lt;&lt;eof &gt;k8s/patches/deploy.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: deployment<br>spec:<br>  template:<br>    spec:<br>      containers:<br>      - name: application-container<br>        resources:<br>          limits:<br>            cpu: 200m<br>            memory: 256Mi<br>          requests:<br>            cpu: 100m<br>            memory: 128Mi<br>eof</pre><p>We can now build the complete manifest using kustomize build:</p><pre>mkdir -p k8s/artifacts<br>cd k8s<br>kustomize build . &gt;artifacts/kube.yaml</pre><p>This will output the application manifests to k8s/artifacts/kube.yaml:</p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  annotations:<br>    cfa-base-name: python-api<br>    cfa-base-version: v0.1.0<br>    cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>  labels:<br>    app: my-k8s-api<br>    owner: cfa<br>    tier: backend<br>  name: my-k8s-api-sa<br>  namespace: my-k8s-api<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  annotations:<br>    cfa-base-name: python-api<br>    cfa-base-version: v0.1.0<br>    cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>  labels:<br>    app: my-k8s-api<br>    owner: cfa<br>    tier: backend<br>  name: my-k8s-api-service<br>  namespace: my-k8s-api<br>spec:<br>  ports:<br>  - name: app-port<br>    port: 80<br>    protocol: TCP<br>    targetPort: app-port<br>  selector:<br>    app: my-k8s-api<br>    owner: cfa<br>    tier: backend<br>  type: NodePort<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  annotations:<br>    cfa-base-name: python-api<br>    cfa-base-version: v0.1.0<br>    cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>  labels:<br>    app: my-k8s-api<br>    owner: cfa<br>    tier: backend<br>  name: my-k8s-api-deployment<br>  namespace: my-k8s-api<br>spec:<br>  revisionHistoryLimit: 2<br>  selector:<br>    matchLabels:<br>      app: my-k8s-api<br>      owner: cfa<br>      tier: backend<br>  strategy:<br>    rollingUpdate:<br>      maxUnavailable: 0<br>    type: RollingUpdate<br>  template:<br>    metadata:<br>      annotations:<br>        cfa-base-name: python-api<br>        cfa-base-version: v0.1.0<br>        cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>        prometheus.io/scrape: &quot;true&quot;<br>      labels:<br>        app: my-k8s-api<br>        owner: cfa<br>        tier: backend<br>    spec:<br>      containers:<br>      - env:<br>        - name: AWS_REGION<br>          valueFrom:<br>            configMapKeyRef:<br>              key: AWS_REGION<br>              name: env-region-cm<br>        - name: ENV_NAME<br>          valueFrom:<br>            configMapKeyRef:<br>              key: ENV_NAME<br>              name: env-region-cm<br>        image: busybox:latest<br>        imagePullPolicy: Always<br>        livenessProbe:<br>          httpGet:<br>            path: /health<br>            port: 8080<br>            scheme: HTTP<br>          initialDelaySeconds: 30<br>          timeoutSeconds: 5<br>        name: my-k8s-api-application-container<br>        ports:<br>        - containerPort: 8080<br>          name: app-port<br>        readinessProbe:<br>          httpGet:<br>            path: /health<br>            port: 8080<br>            scheme: HTTP<br>          initialDelaySeconds: 30<br>          timeoutSeconds: 5<br>        resources:<br>          limits:<br>            cpu: 200m<br>            memory: 256Mi<br>          requests:<br>            cpu: 100m<br>            memory: 128Mi<br>      imagePullSecrets:<br>      - name: artifactory-docker<br>      serviceAccountName: my-k8s-api-sa<br>      terminationGracePeriodSeconds: 30<br>---<br>apiVersion: policy/v1beta1<br>kind: PodDisruptionBudget<br>metadata:<br>  annotations:<br>    cfa-base-name: python-api<br>    cfa-base-version: v0.1.0<br>    cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>  labels:<br>    app: my-k8s-api<br>    owner: cfa<br>    tier: backend<br>  name: my-k8s-api-pdb<br>  namespace: my-k8s-api<br>spec:<br>  maxUnavailable: 1<br>  selector:<br>    matchLabels:<br>      app: my-k8s-api<br>      owner: cfa<br>      tier: backend<br>---<br>apiVersion: autoscaling/v2beta1<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  annotations:<br>    cfa-base-name: python-api<br>    cfa-base-version: v0.1.0<br>    cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>  labels:<br>    app: my-k8s-api<br>    owner: cfa<br>    tier: backend<br>  name: my-k8s-api-hpa<br>  namespace: my-k8s-api<br>spec:<br>  maxReplicas: 10<br>  metrics:<br>  - resource:<br>      name: cpu<br>      targetAverageUtilization: 80<br>    type: Resource<br>  minReplicas: 2<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: my-k8s-api-deployment<br>---<br>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  annotations:<br>    cfa-base-name: python-api<br>    cfa-base-version: v0.1.0<br>    cm.chick-fil-a.com/system-tag: CAP_BLOG_POST<br>  labels:<br>    app: my-k8s-api<br>    owner: cfa<br>    prometheus-enabled: &quot;true&quot;<br>    release: prometheus-operator<br>    tier: backend<br>  name: my-k8s-api-servicemonitor<br>  namespace: my-k8s-api<br>spec:<br>  endpoints:<br>  - interval: 30s<br>    path: /prometheus<br>    port: app-port<br>    scheme: http<br>  namespaceSelector:<br>    matchNames:<br>    - my-k8s-api<br>  selector:<br>    matchLabels:<br>      app: my-k8s-api<br>      owner: cfa<br>      tier: backend</pre><p>And we’re done! Committing these files to our application repository will kick off the <a href="#f2f3">deployment process</a> and my-k8s-api application will be deployed to our development cluster.</p><p>To put everything together, let’s review the path we took through our kustomize overlays:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7kqJpXo9Y0ss8Fqz_Jg8pw.png" /></figure><p>You can see a few complete examples here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_nb72anCgm0E4QRlaI5CDQ.png" /><figcaption><a href="https://github.com/chick-fil-a/kustomize-application/tree/main/examples">https://github.com/chick-fil-a/kustomize-application/tree/main/examples</a></figcaption></figure><h3>Wrap Up</h3><p>We’ve used this setup to generate manifests for our applications for a few years. It’s proven flexible enough to handle requirements from many different teams running in many different environments and regions. We continue to add new application types and improve the base layers as we onboard new teams and discover new requirements (or run into issues).</p><p>One word of caution: keep the number of layers to a minimum! We don’t enforce a maximum number of overlays a team can add, but more than 5 seems to start to cause issues.</p><p>We are actively working on and investing in our Kubernetes and GitOps platform at Chick-fil-A. If this is an area of interest for you, we welcome feedback and community partnership on this. Feel free to leave us a comment or send me a message on LinkedIn!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b67b0be7ad2e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/chick-fil-atech/writing-kubernetes-manifests-with-kustomize-feeder-repository-b67b0be7ad2e">Writing Kubernetes Manifests with Kustomize Feeder Repository</a> was originally published in <a href="https://medium.com/chick-fil-atech">chick-fil-atech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>