<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sameer Kapoor on Medium]]></title>
        <description><![CDATA[Stories by Sameer Kapoor on Medium]]></description>
        <link>https://medium.com/@sameerkapoor?source=rss-99f633e55234------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ge0B9e8c2ZiTWSom.</url>
            <title>Stories by Sameer Kapoor on Medium</title>
            <link>https://medium.com/@sameerkapoor?source=rss-99f633e55234------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:45:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sameerkapoor/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implementing a Fully Automated Sharding Strategy on Kubernetes for Multi-tenanted Machine Learning…]]></title>
            <link>https://medium.com/workday-engineering/implementing-a-fully-automated-sharding-strategy-on-kubernetes-for-multi-tenanted-machine-learning-4371c48122ae?source=rss-99f633e55234------2</link>
            <guid isPermaLink="false">https://medium.com/p/4371c48122ae</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Sameer Kapoor]]></dc:creator>
            <pubDate>Fri, 28 Jan 2022 00:12:06 GMT</pubDate>
            <atom:updated>2023-02-14T18:06:57.839Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing a Fully Automated Sharding Strategy on Kubernetes for Multi-tenanted Machine Learning Applications</h3><p>In our <a href="https://medium.com/workday-engineering/scaling-multi-tenanted-machine-learning-applications-on-kubernetes-3f744ae543e2">previous article</a>, we discussed a bin packing sharding strategy to allocate machine learning models to multi-tenanted machine learning inference applications and how these shards look in Kubernetes. I highly recommend reading that article to get a good understanding of what we’ll be discussing here.</p><p>In this article we’ll explore how to implement this sharding strategy along with some of the challenges in doing so for enterprise machine learning production systems and how we address them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_Q8QemhfLTyYhvfr" /><figcaption>Figure 1: System Design of a sharding strategy for ML inference applications</figcaption></figure><h3>Sharding Service</h3><p>The <strong>sharding service</strong> is the brains behind the operation. It receives notifications from the Model Repository whenever a model is created, updated or deleted. The sharding service also receives notifications from the Kubernetes API server whenever a user deploys, updates, or deletes their ML application.</p><p>Once a user has deployed their ML application, everything is automated — model training, uploading to the Model Repository, and then the subsequent allocation of these models to shards. The sharding service is able to perform the following tasks:</p><p><strong>Shards:</strong></p><ol><li>Create a new shard</li><li>Delete existing shard</li><li>Trigger rolling updates of shards</li></ol><p><strong>Models:</strong></p><ol><li>Add new models to an existing shard</li><li>Remove models from a shard</li><li>Update the version of an existing model in an existing shard</li><li>Move updated models from one shard to another</li></ol><p>The sharding service is able to determine how much memory a model will consume and how much free memory exists in any given shard. Based on these calculations it’s able to bin-pack models by performing the tasks listed above.</p><h3>Challenges</h3><ol><li><strong>Zero downtime</strong>: The ML inference application must always be able to serve traffic; while the application is being updated, models are upgraded even when the models are being moved from one shard to another.</li><li><strong>Stability</strong>: The sharding strategy and the allocation of models to shards should never lead to instability in the underlying inference application.</li></ol><h3>Zero Downtime</h3><p>Zero downtime is really a requirement in any production system. However, this poses a unique set of challenges when dealing with a multi-tenanted ML inference application. There are a few scenarios that can result in downtime serving models, so let’s take a look at each use case.</p><h4>Application Updates</h4><p>When the ML inference application is updated, the updated docker container hosting the application should propagate to each shard without causing any downtime. Luckily, Kubernetes takes care of this out of the box; since each shard of the ML application contains its own deployment, we can specify a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment">rolling update strategy</a> in the deployments to allow for this.</p><pre>spec:<br>  replicas: 10<br>  selector:<br>    matchLabels:<br>      app: inference-application-shard-1<br>  strategy:<br>    rollingUpdate:<br>      maxSurge: 50%<br>      maxUnavailable: 50%<br>    type: RollingUpdate</pre><p>In this example we’ve specified <em>maxUnavailable </em>to be 50%, this means that only half of the pods in the shard’s deployment (i.e. 5 in our example) can be taken down to update at any given moment; the rest of the replicas will serve traffic.</p><p>The sharding service issues a rolling update to all shards with the updated docker container when an updated deployment is detected.</p><h4>Model updates, additions and deletions</h4><p>Frequently tenanted models are added, deleted, or updated within shards. Each time this happens we need to still serve traffic to the rest of the models. To enable this, there are a few requirements that the ML inference applications need to abide by:</p><ol><li>The application needs to implement and expose a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes">readiness probe</a> or a <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes">startup probe</a> to indicate when all of the models assigned to the shard have successfully been loaded to memory and the application is ready to serve traffic. The application needs to load these models into memory on startup. <br><strong>This requirement allows the sharding service to add or delete models from the shard, and then issue a rolling update to allow the application to reload the models without dropping requests.</strong></li><li>The application should poll the configmap containing the list of models to detect upgrades of model versions. The application should be able to load an upgraded model into memory and switch over to it without dropping requests.<br><strong>This requirement allows updating of existing models within the shard without issuing a rolling update.</strong></li></ol><h4>Moving models from shard to shard</h4><p>From time to time, models get updated and grow larger. This may lead to situations where the model would need to be moved to another shard where there is enough available memory. When this occurs, there’s a special sequence of procedures that need to be executed to ensure that requests are not dropped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yt94KdkzVzSPmHOL" /><figcaption>Figure 2: Moving models from shard to shard</figcaption></figure><p>Figure 2 shows Model B1 upgraded to Model B2, however, the memory consumption of Model B2 has increased and no longer fits in Shard 1. We first find a suitable shard to fit Model B2 and then add the model to Shard 2. Once Model B2 has successfully been loaded into Shard 2 and is ready to serve traffic, we modify the virtual service associated with Shard 1 and Shard 2 to reroute the traffic for the tenant associated with the model. <strong>This sequence of procedures ensures that the tenanted traffic for Model B never drops</strong>; there may be a short period of time during which traffic is served by an older version of the model, but this is acceptable.</p><h3>Stability</h3><p>In any production system, the more moving parts there are, the greater the instability. Sometimes this results in a few trade-offs that need to be made, but there are times where we can design for this. Let’s look at a couple of ways we can design to increase the overall stability of the sharding strategy.</p><h4>Bin Packing</h4><p>In order to fit models optimally into shards we apply a <a href="https://en.wikipedia.org/wiki/Best-fit_bin_packing">best fit bin packing algorithm</a> while allocating models to shards. What’s important to note is that <strong>once models have been allocated to shards, we do not attempt to optimize the memory utilization by reapplying the bin packing algorithm,</strong> which would result in shuffling the models into their optimal shards. Though this may result in slightly suboptimal allocation of the models, the trade off is that model movement from one shard to another is minimized, thus resulting in fewer rolling updates and greater stability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IJWl-TT4cM6xqQU4" /><figcaption>Figure 3: Best fit bin packing sharding strategy</figcaption></figure><h4>Grouping Model Operations</h4><p>In order to add, delete, or update models and their respective routing rules, we use 3 commands which are idempotent;</p><pre><strong>set_models(shard,[models…])</strong> — Add, delete or update models within a given shard</pre><pre><strong>set_route(shard, [tenants…])</strong> — Set routing rules for a given shard</pre><pre><strong>update_shard(shard)</strong> — Issue a rolling update of a given shard</pre><p>Let’s take a look at a few examples of some basic model operations and how we use the above 3 commands to instrument them.</p><h4>Example 1: Add Model to Existing Shard</h4><p>Assuming there’s a shard named <strong>shard_1</strong> and it already contains a model <strong>model_A_v1</strong> let’s add another model <strong>model_B_v1</strong> which belongs to tenants <strong>tenant_A</strong> and <strong>tenant_B</strong> respectively.</p><pre>set_models(shard_1, [model_A_v1, model_B_v1])<br>update_shard(shard_1)<br>set_route(shard_1, [tenant_A, tenant_B])</pre><h4>Example 2: Combining operations</h4><p>Building on the previous example, say we want to now upgrade <strong>model_A_v1</strong> to <strong>model_A_v2</strong>, and add <strong>model_C_v1</strong> to <strong>shard_1</strong> for <strong>tenant_C</strong>.</p><pre>set_models(shard_1, [model_A_v2, model_B_v1, model_C_v1])<br>update_shard(shard_1)<br>set_route(shard_1, [tenant_A, tenant_B, tenant_C])</pre><p>Here we see that when we have multiple operations to perform on a single shard, we can group them together.</p><p>In production we accumulate such operations over a period of time, such as 5 minutes or 1 hour, depending on the SLAs of the machine learning application. This gives us the ability to accumulate several of these operations and merge them into a smaller set. <strong>This avoids multiple rolling updates and changes to routing rules, thus increasing stability.</strong></p><h3>Final Thoughts</h3><p>In this series of articles we’ve gone from conceptualizing what a shard looks like for a multi-tenanted machine learning application on Kubernetes to looking at some of the challenging aspects of implementing an automated sharding strategy. Hopefully these articles spark some ideas and interest in the topic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4371c48122ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/workday-engineering/implementing-a-fully-automated-sharding-strategy-on-kubernetes-for-multi-tenanted-machine-learning-4371c48122ae">Implementing a Fully Automated Sharding Strategy on Kubernetes for Multi-tenanted Machine Learning…</a> was originally published in <a href="https://medium.com/workday-engineering">Workday Technology Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Multi-tenanted Machine Learning Applications on Kubernetes]]></title>
            <link>https://medium.com/workday-engineering/scaling-multi-tenanted-machine-learning-applications-on-kubernetes-3f744ae543e2?source=rss-99f633e55234------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f744ae543e2</guid>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sameer Kapoor]]></dc:creator>
            <pubDate>Thu, 27 Jan 2022 23:39:53 GMT</pubDate>
            <atom:updated>2022-01-27T23:39:53.337Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Note: Some of the examples of Kubernetes resource definitions have been overly simplified to make it easier to read and may not work without some editing</em></p><p>At Workday most of our ML inference services utilize tenanted models which are either fully or partially loaded in memory. These models can grow in size over time with the addition of feature sets and even tenant data. Moreover, with the growth of Workday and the adoption of ML features, the number of tenants constantly grows. With ML inference services having to load thousands of these tenanted models in memory it poses a problem for memory capacity; a single instance of a service can only hold so many models in memory.</p><p>It’s apparent that we need some sort of sharding strategy to be able to distribute models and serve them efficiently. In this article we’ll go through a few potential sharding strategies and then share how we ended up solving this problem.</p><h3>Potential Sharding Strategies</h3><h4>Per-tenant shards</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VgiqMMm3HLIIOptbZG-LDQ.jpeg" /><figcaption>Figure 1: Per-tenanted shards where Tenant A, Tenant B and Tenant C are the tenanted models stored in memory</figcaption></figure><p>Here, as shown in Figure 1 each shard would contain multiple instances, each serving one tenanted model. This would be relatively easy to implement and deploy; we could have a single Kubernetes deployment per model and so each deployment would serve only 1 tenant. The biggest drawback of this solution is the resource utilization overhead; each instance would need to run the ML inference application, the operating system and whatever other sidecars associated with the deployment. In other words, there isn’t any sharing of common resources in this solution.</p><h4>Tenants as features</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Xraq1qKD4Rj8MddOzly_w.jpeg" /><figcaption>Figure 2: Shards where a single model contains multiple tenants as features</figcaption></figure><p>Figure 2 shows instances of shards that each host a single model with multiple tenants. Each tenant is a feature when training a model. This solution solves the problem of resource utilization overhead; we now share common resources such as the ML application memory and OS memory.</p><p>There are a couple of problems with this strategy:</p><ol><li>You have to know the topology of the tenants in the model while training them. This would entail knowing the size of the resulting model to determine which tenants can fit into each model beforehand.</li><li>Another problem with this solution is that, since a single model hosts several tenants, whenever a single tenant’s features are updated, the entire model must be updated. This could lead to the ML inference application constantly refreshing its model which wouldn’t be ideal.</li></ol><h4>Bin packed shards</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hRtvnK95WVwOnoSC270ZRg.jpeg" /><figcaption>Figure 3: Shards where models are bin packed</figcaption></figure><p>Figure 3 looks very similar to Figure 2, the difference here is that each instance in a shard contains multiple models; one for each tenant.</p><p>Here, we determine how much memory is left in a shard and can pack models into the shard depending on the model’s memory consumption. This even gives us the flexibility to pack models into shards based on other cost factors such as CPU utilization, traffic patterns, etc. Now when a tenanted model is updated, only that particular model needs to be updated.</p><p>We now have the flexibility of training tenanted models independently and we’ve managed to solve the resource utilization problem by sharing common resources. We ended up going with this approach, which provides us the best of both worlds.</p><h3>Shards on Kubernetes</h3><p>Now that we’ve decided on an approach to our sharding strategy, let’s define what a shard looks like in terms of Kubernetes (and Istio) constructs. It’s well known that Kubernetes is great at managing deployments, handling zero downtime and scheduling instances to underlying nodes, another useful feature is to programmatically define and manage deployments; this makes automation fairly simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KrK2ejraAqWvRIqNnab7-g.jpeg" /><figcaption>Figure 4: Shard as defined in Kubernetes</figcaption></figure><p>As seen in Figure 4, we define a shard as a set of 4 main components</p><ul><li><strong>Deployment: </strong>This is what tells Kubernetes to create or modify instances of the pods that run the inference applications. The deployment defines how to roll out updates to the inference application and how to maintain zero downtime while doing so. Each shard has a unique deployment which instantiates its own set of pods. This gives us the ability to scale up or down the number of replicas of pods depending on the load.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ecc53de927fba32a33946612ade2ced/href">https://medium.com/media/4ecc53de927fba32a33946612ade2ced/href</a></iframe><ul><li><strong>Service:</strong> This is the logical abstraction of the deployment, it provides a DNS entry for the set of pods and load balances requests to them. Each shard has a unique service defined and thus a unique entrypoint.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f21040f35617d1182eb54eb6aac98075/href">https://medium.com/media/f21040f35617d1182eb54eb6aac98075/href</a></iframe><ul><li><strong>ConfigMap:</strong> The configmap is a dictionary consisting of key-value pairs. It’s a file that is mounted on to each pod’s filesystem which the inference application can read. The configmap contains the list of models that are assigned to the shard so that the application can download them and load them into memory. Each shard contains a unique conigmap and this is how we distribute unique sets of models into individual shards.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38eaf71a52c3d84a4f8fa8a2b59fcfc1/href">https://medium.com/media/38eaf71a52c3d84a4f8fa8a2b59fcfc1/href</a></iframe><ul><li><strong>Virtual Service:</strong> This is an Istio construct, it defines the set of traffic routing rules to the shard’s <em>service</em>. We can programmatically define match rules that match tenant headers specified in the request and route these requests to the corresponding shard on which the tenanted model is being hosted. Here’s an example of a route that matches on the <em>Tenant</em> header and looks for a value <em>Tenant-A</em> to route to shard 1.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca31bcfa55d73bbc678247ad989886f5/href">https://medium.com/media/ca31bcfa55d73bbc678247ad989886f5/href</a></iframe><h3>Final Thoughts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fy12HNrI-GmIyev9ssc8_g.jpeg" /><figcaption>Figure 5: Multiple shards running on Kubernetes</figcaption></figure><p>We’ve gone through various sharding strategies and arrived at the conclusion that bin packing models to shards was the best strategy for us. We finally showed how this bin packing strategy works in Kubernetes (see image above).</p><p>This not only gives us the ability to dynamically scale the inference applications based on the number and size of tenanted models, but also gives us the ability to horizontally scale each shard based on load.</p><p>Stay tuned for our next post on how we actually implement this strategy in production.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f744ae543e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/workday-engineering/scaling-multi-tenanted-machine-learning-applications-on-kubernetes-3f744ae543e2">Scaling Multi-tenanted Machine Learning Applications on Kubernetes</a> was originally published in <a href="https://medium.com/workday-engineering">Workday Technology Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>