<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zack Smith on Medium]]></title>
        <description><![CDATA[Stories by Zack Smith on Medium]]></description>
        <link>https://medium.com/@zack_smith?source=rss-cf76d6623721------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Nb_EfV1CdCPkRCKTRq7glA.jpeg</url>
            <title>Stories by Zack Smith on Medium</title>
            <link>https://medium.com/@zack_smith?source=rss-cf76d6623721------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:32:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zack_smith/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How our Edge Kubernetes Platform has Evolved]]></title>
            <link>https://medium.com/chick-fil-atech/how-our-edge-kubernetes-platform-has-evolved-12609006bc92?source=rss-cf76d6623721------2</link>
            <guid isPermaLink="false">https://medium.com/p/12609006bc92</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[edge-computing]]></category>
            <dc:creator><![CDATA[Zack Smith]]></dc:creator>
            <pubDate>Tue, 09 Sep 2025 20:33:06 GMT</pubDate>
            <atom:updated>2025-09-09T20:33:06.686Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>By </strong><a href="https://www.linkedin.com/in/zack-smith12"><strong>Zack Smith</strong></a><strong>, </strong><a href="https://www.linkedin.com/in/jake-wasdin-9b047620"><strong>Jake Wasdin</strong></a><strong> and </strong><a href="https://www.linkedin.com/in/kevin-koleck-936b824a"><strong>Kevin Koleck</strong></a></p><p>Since we <a href="https://medium.com/chick-fil-atech/edge-computing-at-chick-fil-a-2621f4b5a969">first shared the work Chick-fil-A is doing at the Edge in 2018</a>, we have come a long way. A couple of years ago, we wrote about some of our updates, including <a href="https://medium.com/chick-fil-atech/enterprise-restaurant-compute-f5e2fd63d20f">the state of the platform</a>, <a href="https://medium.com/chick-fil-atech/what-we-learned-from-launching-edge-compute-from-enterprise-architecture-1dc34e49482f">how we built support for it internally</a>, and <a href="https://medium.com/chick-fil-atech/stumbling-into-gitops-at-the-edge-92924dfb2ca1">how we do GitOps</a>. Not only is the platform now present in all our stores, but it is increasingly essential for more areas of store operations, yielding more use cases and increasing the importance of the platform’s reliability. We have done work to both increase the reliability of our platform and add support for more types of hardware for emerging use cases.</p><h3>The challenges of redundancy</h3><p>When designing our edge platform, we designed a solution with three edge nodes per restaurant running the k3s distribution of Kubernetes. This architecture allowed us to have capacity to schedule workloads reliably across several nodes and recover from failures on individual hardware, software, or other nodes.</p><p>As shared in the past, our edge platform was built on bare metal using minimal tooling at the host level, with most of our tooling focused on the k3s environment, including managing containers and deployments. We did have some lightweight tooling to help identify node issues, but the focus was always on the cluster.</p><p>Our design worked, perhaps too well. While things looked healthy on the surface in that we had running clusters, there were opaque node failures occurring. We discovered our host-level processes were not catching all our offline nodes and that we had quite a few to remedy across the chain. With more demand for our edge platform to support critical components of our restaurant, including emerging computer vision use cases and elements of our point-of-sale system, we needed to close the gap on node observability, fix the offline nodes, and re-think some of our node management processes.</p><p>We decided to invest in even more proactive health monitoring, increase automation around node recovery, and shift toward declarative management of node state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*V1vyY_vZcA5uDkIx91wfwg.png" /><figcaption>Inspired by the Go Gopher created by Renée French, licensed under CC BY 4.0</figcaption></figure><h3>Enter the host agent</h3><p>To help alleviate support stresses and improve reliability, we decided to switch the way we initialize, update, and recover nodes. Previously our methods were reactionary. If we had a misbehaving node or an unhealthy cluster, an alert would generate and our auto-remediation process would remotely wipe the node, or our support team would step in if the issue persisted. While this worked well at solving misbehaving nodes, it had downtime associated with it and could not recover nodes that were not remotely accessible to us. We wanted something running at the edge, but outside of the cluster to manage the node for us. Enter the host agent.</p><p>The host agent is comprised of plugins, which are small components responsible for a single task. Like most of our team’s code, they’re written in Go. This allows us to compose a collection of these plugins together to manage the node’s state. The host agent on each node will reach out to our host service in the cloud to get its desired state. We have settings related to each plugin that can be set on a per node, location or chainwide level. This helps us to quickly or slowly role out features depending on risk associated with the change. The agent will run all enabled plugins at a set interval based on priority and dependency tree, preventing us from running and subsequently throwing errors in plugins that rely on other plugins to be completed first. This composability allows us to have robust error handling as well as easily and quickly implement changes without affecting other components.</p><p>An example of a host agent plugin is our k3s plugin which is responsible for installation, upgrading, and ensuring the k3s server or agent is running. It also ensures the node is in the location’s cluster. We only need to specify the k3s version we want to run on our nodes and the k3s plugin handles the upgrade or install process. If any errors are encountered during the upgrade process it will revert to the old version and alert the team it failed. This simple plugin made what used to be a cumbersome and complicated process of updating k3s into something as simple as setting a value and letting the plugins do the upgrade (and if necessary, recover gracefully).</p><p>We also wanted more robust observability data from our nodes. We incorporated an event messaging system to achieve this, both in the host agent and in the plugins themselves. The host agent sends an event every time a plugin starts and every time it encounters an error. The plugins use the same messaging system to report actions the plugin is performing and what triggered the actions. This helps with root cause analysis and ensures our plugins are behaving as intended. Our old approach would often lose error messages and logs. This event system allows us to uncover reoccurring problems we can either address or take the time to add remediation steps to a plugin that is designed to check and quickly fix issues while we work on correcting the root cause. Overall, the addition of the host agent has provided a significant increase in reliability of the number of nodes in our clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*mZIBjeN7qnn_NvI9NpCajw.jpeg" /><figcaption>An example of an ERC cluster with GPU capabilities</figcaption></figure><h3>Adding GPU support</h3><p>We began testing Computer Vision projects in some of our stores, like an improved <a href="https://medium.com/chick-fil-atech/unseen-connections-the-role-of-embeddings-in-quality-assurance-at-chick-fil-a-78e1720cce8c">Automated Holding Assistant</a>. These add the requirement of having a discrete GPU, which our other edge nodes did not have.</p><p>We needed to weigh compatibility with existing work while balancing scalability, sourcing, and cost. While we weighed different GPU and architecture types, we ended up settling on using edge nodes very similar in specifications to what we had, but only with discrete GPU installed. This meant simple, x86 mini-PCs, but not quite as mini because of the physical size needed for the GPU.</p><p>While we investigated supporting GPU workloads on ARM-based nodes, we needed to build out cross architecture support in our platform, which delayed our time to deliver and deploy the feature. We found we are much less dependent on having a homogenous cluster. Originally, we used one vendor for a single model, which still runs at many of our stores. But now we also have edge nodes from various manufacturers, including at the same store. We can use what makes the most sense at the time and have multiple vendors available.</p><p>Another change that made a positive impact to our reliability evolution was moving from a cluster with <em>exactly</em> 3 nodes to <em>at least</em> 3 nodes. We can now add additional GPU nodes to scale compute needs dynamically based on the individual location’s needs and reliability expectations of a given project. Scaling is not as easy as in the cloud as it requires installing additional hardware, but it is seamless from a software perspective.</p><p>Currently, we utilize a GPU plugin in the host agent to look at the hardware resources available. If it detects a GPU, then it will install the applicable <a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/">GPU operator</a> into that location’s cluster so the GPU resource becomes available for scheduling in Kubernetes. By running these workloads at the edge, we are not bandwidth constrained. We can keep our bandwidth open for processing orders while still gaining valuable insights to help our team members serve guests.</p><h3>Final thoughts</h3><p>Today our Enterprise Restaurant Compute platform offers a significantly higher level of reliability at a node level view by pulling much of our tooling outside of the cluster and into the host agent. We are now better equipped to serve the growing needs of the business with more resources available at the edge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12609006bc92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/chick-fil-atech/how-our-edge-kubernetes-platform-has-evolved-12609006bc92">How our Edge Kubernetes Platform has Evolved</a> was originally published in <a href="https://medium.com/chick-fil-atech">chick-fil-atech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>