I was suddenly struck by the thought that everything I knew was powered by a ball of gas billions of years old, flying over mountains that were millions of years old, visiting a city that was thousands of years old, on a contraption devised a hundred years ago.

I wondered if the ancient Romans dreamt of flying vehicles.

Rome was a revelation. The Colosseum, crumbling and pock marked, is a grand spectacle, a shocking sight even in the age of skyscrapers and megastructures— I can only imagine how it looked to people two millennia ago in all its glory. I learned that 400,000 people died here over the course of the 400 years it was used for gladiatorial combat. 400 years! We have archaeologists now working on sites from the 1700s and 1800s. How would it have felt to know that the building you’re working in or visiting, seating tens of thousands, has been used actively for hundreds of years for the same macabre purpose?

The Pantheon takes it to another level. It’s one thing to see a ruin of the Roman Empire’s might, but entirely another to see an intact building from 2000 years in the past. I spent hours at the Pantheon, admiring it from the outside and from within. The level of detail in the interior and the coffered dome, still the largest unreinforced concrete dome in the world, is truly a sight to see. I was left confounded as I tried to understand how the intricate artwork at such massive scale could’ve been achieved without modern precision tools.

The biggest question I had, though, as I explored Rome and saw stunning, ancient sites around seemingly every corner was: Why? Why go to this level of effort? What motivated them to do this and push the boundaries of what was possible at the time? What drove them?

In Florence, the City of Art, I found one of my favourite views in the world at the very top of the bell tower of the Santa Maria del Fiore Cathedral. It threw off my itinerary for the day because I spent two hours up there just enjoying the view of a stunningly beautiful city and that of the cathedral itself.

It was also in Florence that I saw the Santa Croce Basilica, home to the tombs of Machiavelli, Galileo, and Michelangelo. They were laid to rest here with intricate, beautiful artwork celebrating their lives. But as with all tombs, I wondered, who was this meant for? A promise kept to the departed, or was it more a way for future generations to remember them? Or maybe a mix of the two; a message to everyone that sees the tombs: “Become this great, and you too could be celebrated centuries after your time.”

In Napoli, founded as Neapolis by the ever enterprising Greeks more than 2500 years ago, I fell in love with the city’s chaos, Pizza™️, and the Neapolitan sense of style. But what I enjoyed the most was the city’s hidden underground. The Greeks painstakingly built an extensive underground aqueduct system that supplied the entire city with water till as recently as the 1800s. The tunnels are deep underground and stretch out over tens of kilometres through solid rock. The engineering brilliance, ingenuity, and persistence of those who came before us always leave me speechless.

In Pompeii, I saw the symbolic plaster casts of people that died when Mount Vesuvius erupted 2000 years ago. Contrary to what I’d vaguely thought before, there was nothing organic there. It was just plaster in a roughly humanoid shape; nothing impressive.

Except… I felt more connected to these unfortunate souls than I had to all of the “famous” people who’d been “immortalised” in bronze, marble, and paint. The people of Pompeii were going about their daily lives, unaware that their city was built on and prospered due to the volcanic soil from even earlier eruptions of Vesuvius. And as they must’ve uncomprehendingly witnessed the wrath of the Gods obliterating a mountain, they sought comfort in their last moments on the planet, curling up alone or huddling with others. Their names will never be found, and their identities, thoughts, memories, and desires are irrecoverably lost to time. But they were us, and we are them.

As I saw these ancient places and more surviving neglect, looting, bombing, natural calamities, or simply the ravages of time, and as I saw the descendants of those ancients still living their lives there today, I felt a certainty take root in me. This is probably a textbook example of hubris from Greek tragedies, but I’m sure that humanity will never perish — some form of it will always survive. Maybe in the stars after an unified, expansionist push by all countries — something that seems increasingly unlikely with every passing day. Or maybe in isolated communities deep underground after aeons of devolution and an apocalyptic nuclear war.

I thought about all the people who’d spent their lifetimes obsessed with legacy: Alexander the Great, Julius Caesar, Napoleon Bonaparte, Genghis Khan, and countless others who’ve achieved varying levels of success in having their names immortalised. But these are just names to us, most likely not even spelled or pronounced the right way. Their thoughts and actions have been filtered and modified through generations of propagandists and people wearing rose or red tinted glasses. A million people speak Alexander’s name today and he’s not alive to hear it. Tens of millions of people are said to be descendants of Genghis Khan and he’s not here to witness it. So why go through all of that effort just to have yourself turned into a figurehead or a caricature?

People living today struggle to accurately represent themselves to the ones that love them the most. Most people simply hide their deepest thoughts, fears, and worries. So why is it so important to have a legacy if at best, even when you’re a world-altering conqueror, only a pale imitation of you will be remembered?

I have a strong desire to leave the planet a slightly better place than when I came into it, but that’s just something that gives me direction. Why would I let my short lifespan be consumed with thoughts of what future generations might think of me? Why do I need to be remembered?

In front of the Castel Sant’Angelo in Rome, I listened to a violinist play the theme from the movie Interstellar. It was one of the most moving experiences of my life, but it was just another evening of busking for him. Whatever money I put down and my quiet, mispronounced “grazie” before his next song began was a paltry effort to convey how I felt. But I will carry that experience with me forever — even if I don’t know his name and to be quite honest, have forgotten what moved me so much in the first place. Although… maybe that’s the secret?

Life isn’t about the details. It isn’t about past or the future.

It’s a series of “Right Now”s.

I remember my mom had a cactus when I was younger. And as we eagerly waited, on the same day every year, a beautiful flower would bloom overnight and it would be gone the next day. And once or twice to our sadness, it was there and gone even when we weren’t around to witness its beauty.

A thing isn’t beautiful because it lasts.

Maybe Legacy is a tool like Religion. A shared belief across time and cultures. A way to channel humanity’s best and worst instincts. To make us work tirelessly for the perceived good of future generations and the pre-determined answer to the question when we inevitably ask: “But what am I getting out of it?”

I don’t have the answers, nor I believe will anyone, ever. Because I think there is no question to be had.

Life doesn’t need a purpose. And Humans don’t need a legacy to live their lives to the fullest.

Be passionate about everything you do and this should make you happy in all of the “right now”s that lead to the day you die. If things work out, then maybe people will barely remember you two thousand years from now as they gaze in awe at your name on the side of an imposing marble monument. If they don’t, well… that’s no longer your problem, is it?

Please note: I work at a startup and maintain our automated ML training systems myself, so a lot of my decision making is informed by this. Between complex systems that have low buy-in but require a lot of upkeep, and complex systems that require a lot of engineering effort to set-up but require little upkeep, I always choose the latter. I doubt that approach is valid when larger teams and more engineering resources are involved, so many of the points we discuss here might not be relevant or an issue for you.

TL;DR

• I think Ray on AWS/GCP VMs is the way to go today if you want critical workloads to be run on a dependable system that can be maintained with minimal resources.
• KubeRay is the way to go in the long-term, especially when paired with a powerful Kubernetes (K8s) cluster autoscaler like Karpenter. So explore KubeRay on the side and have an integration ready to go, because I think it’s a question of when to use it in production, not if.

Let’s now look at different categories for KubeRay and for Ray clusters on cloud VMs. I’ve only used Ray clusters on EC2 instances, but as I understand it, GCP is also officially supported and the functionality is very similar. For brevity, whenever I mention VMs, I mean an EC2 instance or a GCE Virtual Machine.

Cluster Configuration

The cluster configuration is a YAML file that lets you configure the composition of your cluster and its autoscaling characteristics.

KubeRay

• You have the ability to specify the pod spec for the head and each of the worker pods. This gives you near-complete control of the pods, except for the command that specifies the primary Ray process that runs in the main container.
• However, your Ray cluster goes only as far as your K8s cluster does. If you have a K8s cluster that doesn’t allow pod-level VM configuration, you’re going to have to resort to node groups along with node-selectors and tolerations in order to control the machines on which your Ray pods end up.
   — Considering the best practice is to schedule a single Ray pod per machine, you’re looking at quite a few node groups if you have a complex Ray cluster configuration with many differing types of worker groups.
   — This equation completely changes if you have a powerful K8s cluster autoscaler like Karpenter because it lets you specify the instance type and more VM configuration options right from the pod specification.

Ray on VMs

• You can specify the entire cloud-specific VM configuration for the head and worker groups.
• You can control every aspect of the infrastructure through this — security groups, subnets, instance market type, etc.

Conclusion

You need Karpenter or an equally powerful K8s cluster autoscaler to truly unlock the power of KubeRay and make it flexible.

But even with that, you can never match the sheer simplicity and flexibility of using Ray on cloud VMs. Not to mention, as a rule, even if you optimize by baking Docker images into your AMIs, cold starts for KubeRay will always be slower than for Ray on VMs.

The Core Ray Process

KubeRay

• [You] –kubectl exec→ [Pod [Ray Python Process]].
• The Ray initialization command (ray start …) is used to start the pod, so you have no direct access to the command even if you exec into the pod’s container, because interrupting the process will kill the container.
• You have access to rayStartParams in the YAML manifest of the RayCluster Custom Resource (CR) through which you can configure ray start's parameters.
• According to this document, you can also specify a command and args in the YAML manifest for each one of the head or worker group specifications. This is then prepended to the ray start command before the pod is started.
   — Example: command args && ulimit -n 65536; ray start …

Ray on VMs

• [You] –SSH→ [Machine [Ray Python Process]]
• Each of the following fields takes a list of strings (commands) and are executed in the order specified.
• initialization_commands are the first to be run on each machine when it’s started.
• setup_commands, head_setup_commands, and worker_setup_commands come next.
   — The head_setup_commands are merged into setup_commands and run on the head node.
   — The worker_setup_commands are merged into setup_commands and run on the worker nodes.
• head_start_ray_commands and worker_start_ray_commands are the final step and are responsible for initializing Ray on the head node and worker nodes respectively.

Conclusion

With KubeRay, there’s no interfering with the Ray process once it has been started, either the cluster works or it doesn’t, you can’t fix things in real-time if any issue develops in a cluster.

With Ray on VMs, you have unlimited access to the machine. In case something goes wrong with the Ray process, you can manually kill and start it yourself with the commands cleanly listed out in the cluster manifest. Not that you’d do it often — or if you’re lucky, at all — but the fact remains that you can “heal” a cluster or very easily debug issues since you have direct access to the ray code on the machine. Anecdotally, I’ve used this a few times to implement hot-fixes in obscure parts of the Ray code — and creating a new machine image with the change baked in — with the actual fixes sometimes months away in an official release.

Autoscaling

The Ray autoscaler is responsible for monitoring resource demands from the Global Control Service (GCS) and making the appropriate changes to the underlying node provider — KubeRay, AWS, GCP, etc.

While KubeRay is a GitHub project separate from the core Ray library, the autoscaler used for KubeRay resides in the core library, alongside the one used for AWS and GCP.

KubeRay

• Autoscaling has to be explicitly enabled with enableInTreeAutoscaling: true in the RayCluster CR. The config for the autoscaler container is then built like so and injected into the head node’s pod like so.
• The command that’s used for the autoscaler pod runs this function.
• The autoscaler here doesn’t make direct autoscaling changes. Instead, it updates the RayCluster Custom Resource (CR), and those changes are picked up by KubeRay’s operator which then constructs the pods, deletes them, etc.
• Any updates made to the RayCluster CR while the cluster is running are picked up in real-time by KubeRay’s operator. This lets you endlessly tweak and manage your cluster
• As with the Ray process when working with KubeRay, there’s no option of interrupting the autoscaling process if needed.

Ray on VMs

• Autoscaling is integrated into the head node’s Ray initialization. If you take a look at the head_ray_start_commands, you’ll notice a --autoscaling-config section that triggers the launch of the autoscaler.
• Unlike with KubeRay, the autoscaler here does make direct changes to the underlying node provider. For example, with AWS, it launches and terminates EC2 instances and other infrastructure components as needed.
• Each Ray cluster’s head node contains a ~/ray_bootstrap_config.yaml file that describes the cluster. Any change made to this YAML file is immediately picked up the autoscaler which makes the necessary changes to the cluster. Like with KubeRay, this lets you maintain fine-grained control over your cluster throughout its lifetime.
• Since the autoscaler is launched as part of the Ray initialization, stopping ray also stops the autoscaler. This lets you directly modify the underlying code to debug issues if any pop up. You can also choose to start Ray without the autoscaler integrated and launch the autoscaler as a separate process.

Conclusion

Autoscaling for Ray on VMs is straight-forward, has complete configurability on all levels, has a minimal number of moving parts, and can be thoroughly trusted in production with many years of a proven track record.

KubeRay is a long way from that today.

Potential for Catastrophes and Complications

KubeRay

• Multiple Ray clusters reside within a single Kubernetes (K8s) cluster.
• Depending on your needs, you need a team dedicated to maintaining KubeRay, a powerful cluster autoscaler, RBAC policies and enforcement, network policies and enforcement, and much more to ensure that you get proper isolation for your Ray clusters. Both from the usage perspective and the underlying infrastructure perspective.
• Even with all of that effort, a K8s cluster-wide issue will instantly knock out all your Ray clusters.

Ray on VMs

• Each Ray cluster is completely isolated. Once the head node has been launched, there are no more centralized components to worry about.
• The only thing that can take out multiple Ray clusters would be a region-wide or cloud account-wide issue, at which point you’ll have bigger issues than Ray to contend with.
• Blast Radius for a cluster going down is exactly zero, since it’s trivial to enforce networking constraints at the VM level with security groups.

Conclusion

You can get to a very high level of stability and isolation with KubeRay. Kubernetes is the farthest thing from an obscure technology and it will only become more mature and resilient as the years pass. But, today with KubeRay, the amount of effort it will take you to get to that safe, stable state of your system is astronomical in comparison to what you almost instantly get with Ray on VMs.

Project Stability

KubeRay

• The KubeRay autoscaler MVP was introduced in March 2022.
• KubeRay entered beta in August 2022.
• Autoscaler support is still listed as in beta as of August 2023.
• KubeRay is at 0.6.0 as of August 2023.
• A lot of development focus seems to be on KubeRay at the moment. And while to my knowledge it hasn’t been explicitly stated anywhere, looking at the activity around it, I get the strong sense that it will eventually become the officially recommended way to deploy Ray in any production scenario.

Ray on VMs

• Has been stable and in use for at least the last 3–5 years.
• Very strong anecdotal evidence of its stability.

Final Conclusion

If you’re undecided, you’ll first need to answer these questions carefully. Keep a rough note of the number of “Yes”es.

• Do you need to define the characteristics of the underlying virtual machine at the Ray cluster’s head/worker node level?
• Do you need infrastructure level isolation for your Ray Clusters? This means that each Ray cluster is in an isolated infrastructure pool with one cluster being unable to interfere with another’s operation.
• Do you have strict security requirements?
• Do you have strict RBAC requirements?
• Do you have strict uptime requirements?
• Do you lack access to cloud K8s clusters that you can easily experiment with?
• Do you lack engineering resources that are highly skilled in K8s administration?
   — More importantly, if you do have those resources, do they lack bandwidth to dive deep into any issues that crop up in the many K8s components that will be involved in productionizing KubeRay?
• Do you have a short timeline to get to production?
• Are your workloads critical?

More “Yes”es than “No”s

• It will take a significant investment of resources to get production-ready on KubeRay. It will also have a high ongoing maintenance cost.
• With Ray on VMs on the other hand:
   — It should be much easier to productionize it.
   — It’s easier and cheaper to find engineering resources for it because experienced AWS/GCP devs are more readily available than experienced K8s ones.
   — You’ll have a much lower maintenance cost since there aren’t as many moving parts.
   — And trust me on this, there will be no catastrophes thanks to the near-perfect isolation of each Ray cluster. Only rare “oopsies” that you can easily debug and fix.

More “No”s than “Yes”es

• It should be very easy to get up and running with KubeRay. Definitely worth investing a bit of time to see if it can work for your needs.

Though I’ve chosen Ray on EC2 for the near future, I’m operating with the understanding that a shift to KubeRay is inevitable. It’s just a matter of when, not if.

Even before I could formulate it in words, I’ve had this vague, sinking feeling every time I interacted with an LLM over the last few months. This isn’t right, my brain told me, you’re not talking to a human. You don’t have to say please and thank you and be polite, it’s a program. But I couldn’t do it, it’s more engaging, patient, smarter, and kinder than many humans I’ve interacted with in forums online. And that’s when I realized something. Until some point in time last year, if we saw text anywhere, we implicitly knew a human must’ve written them, there was just no alternative. But from now and until the day the human race ends, there will always be the doubt: who’s doing the talking? Are these your thoughts? Or am I speaking to an AI?

It’s not yet a societal issue, since not all people currently see its potential or face its effects. If they do use it, they’re critical of the flaws in the system, proud that they can spot fakes. But I have to say, before November ’22 when ChatGPT was released, AI was just an amusement to most, the bumbling antics of Google Assistant, Siri, or Alexa compounding this. They were like the shiny robot puppies of the 2000s, wonderfully silly but nothing more. But now, everything has changed, AI is everywhere, offering to do the writing for you, clamouring to reduce your brain’s labours.

Even as I type this on Notion, I’m a single forward slash (/) away from accessing an AI that can write this entire article for me and make it more intelligent than I could ever hope to make it. It understands linguistics and esoteric philosophy from 2000 years ago, how can I compete?

And yes, I use the word “understands” consciously even though I know that’s not what it’s doing. Why do we suddenly care about the circumstances of the writer? We’ve been communicating through blocks of text with disembodied agents since the dawn of the internet and even that of writing itself, and we’ve certainly never cared about the humanity of the writer. The words were what have always indicated the intelligence, empathy, and a thousand other human characteristics of the writer and they continue to do so now. The fact that the words come from an AI is irrelevant if it can comfort someone going through a tough time or give financial advice to someone on the brink of losing their money. For the first time in history, we have an intelligent, all-knowledgeable agent that has no angle — no love, passion, hatred, greed, kindness, empathy, or any emotion influencing it. It’s just intelligent for the sake of being intelligent.

So… what are my words worth? I can feed an outline to an AI and get it to write this entire article to specifically cater to the reading ability of my target demographic. It could correct mistakes, fix my awkward phrasing at times, and so much more. And I can’t even make the argument that it would rob my writing of its tone. All I have to do is feed it my old works and it would precisely mimic my tone to a level beyond human perfection, with none being the wiser.

So I feel as if my words are no longer a strong indicator of my humanity or intelligence when I could fake it in seconds with AI.

The ideas I have, thankfully, are still valuable. But only for now. I already see that LLMs have the uncanny ability to put 2 and 2 together to come up with a million. They will, at some point, surpass a human’s ability to create planned chaos from order, and isn’t that what creativity is? The only thing we can do better is to test our hypotheses and draw on past experience to validate ideas. But when you give these models some mechanism to remember past experience and the agency to act in real-world systems… who can compete?

So… what are my words — these words — worth? What do they mean? What do they indicate? I have no clue. None.

But I do feel the need to write more, now more than ever, and ensure we don’t give up our voices and stay quiet now that there’s a smarter kid on the block.

An entry in a series of blogs written during the Vector Search Hackathon organized by the MLOps Community, Redis, and Saturn Cloud.

The Top2Vec paper explains the concepts behind the Top2Vec library in a more accessible way than I ever could. So instead of diving into the concepts, we’ll look at the practicalities of using Top2Vec and the data generated from it.

We have three major types of objects in Top2Vec: words, documents, and topics. All the unique, cleaned-up words in all the documents fed into the model make up the vocabulary. Topics refer to centroid vectors representing dense clusters of documents, the clusters having been determined using UMAP and HDBSCAN.

The following is my simplified understanding of the code in the top2vec library. This is what happens once Top2Vec(documents) gets called and the training begins.

Looking at this code, we find that these objects have the most relevance:

• vocab: list of words
• word_vectors: array of word vectors
• word_indexes: dictionary of words to word indexes. Useful to get the word index given the word without iterating over the list of words.
• documents: numpy array of all documents
• document_vectors: array of document vectors
• topic_vectors: array of topic vectors
• topic_words: 2-D array of 50 words to describe every topic
• topic_word_scores: 2-D array of 50 scores to show the dist of the 50 words for every topic in topic_words
• doc_top: topic for each document
• doc_dist: scores per topic for each document
• topic_sizes: frequency of documents per topic

We then come up with a data model to load this data into Redis:

class Paper

Dealt with by RedisOM.

class Paper(HashModel):
    paper_id: str = Field(index=True)
    title: str = Field(index=True, full_text_search=True)
    year: int = Field(index=True)
    authors: str
    categories: str
    abstract: str = Field(index=True, full_text_search=True)
    input: str

In addition, each paper will have a topic index and a topic score assigned to it to indicate the topic the document is closest to.

The input field is a cleaned version of title + abstract.

paper_vector:

• paper_pk
• paper_id
• doc_idx
• categories
• year
• vector

word_vector:

• word_id
• word
• vector

topic_vector:

• topic_id
• size
• vector
• word_0 … word_15
• word_score_0 … word_15

A total of 33 fields per topic_vector.

Indexes

We create three different HNSW indexes for the vector key with Redisearch. The separation between the indexes is assured by the prefix we provide it; it will only look for hashes that begin with the prefix.

papers

prefix: paper_vector:

words

prefix: word_vector:

topics

prefix: topic_vector:

Conclusion

We’ve barely scratched the surface here. There is a ton of information contained within the Redis DB now, both in terms of metadata and vectors. Correlating them and exploring relationships should yield very interesting results. And the promise of this not being in a single computer is tantalizing. We’re no longer dealing with a Top2Vec model we have to port everywhere.

An entry in a series of blogs written during the Vector Search Hackathon organized by the MLOps Community, Redis, and Saturn Cloud.

I’ve used Redis extensively before, but like most, I’ve only used it as a cache. So I was quite excited coming into the Vector Search Hackathon; it’s always fun when something you’re familiar with still holds secrets.

In this blog, we’ll look over different aspects of Redis and how we can use it as a vector database that’s capable of vector similarity search.

Managed Redis

As I noted here, I’m a fan of self-hosted software and that includes deploying Redis wherever we’d needed it previously. But working with RedisCloud now, it’s hard to understate how much simpler using high-availability Redis has become. I don’t have to think about sharding, availability, uptime, throughput, etc. This is perfect for teams that are unable to devote resources to maintaining a large, self-hosted Redis deployment.

Redis Modules

Redis is a key-value store. But there are certain tasks you can do on top of that store, and instead of having the user implement those externally, Redis Modules bring a lot of common use-cases like search, time-series data, and graphs into the Redis DB directly. For the hackathon, we primarily used Redisearch, which offers basic and full-text search and vector-similarity search. The API is very mature and the Python SDKs are great to work with.

Data Loading

I shouldn’t be surprised, but I was blown away by how quick Redis is at loading a ton of vectors into the Database. It’s very reasonable to do this near real-time. So if you have a use-case where an application needs vectors.

Redis Object Mapping Python

This is probably one of my favourite tools that I got from this already great experience. Using Pydantic models, we’re able to create detailed data models that can be directly stored/queried in/from Redis without ever having to look at a single backend command like hget.

class Paper(HashModel):
    paper_id: str = Field(index=True)
    title: str = Field(index=True, full_text_search=True)
    year: int = Field(index=True)
    authors: str
    categories: str
    abstract: str = Field(index=True, full_text_search=True)
    input: str

Since we’re dealing with Pydantic, there are no holds barred on validation — perfect!

Here, you’ll notice that the paper_id, title, year, and abstract variables are of type Field. The title and abstract alone though have full_text_search set to True. Now whenever equality or range checks are done on paper_id or year, it’ll use an index to speed things up. And when you search for specific words in the title or abstract, Redisearch will use full-text search to deliver the papers containing those words to you.

paper_instance = Paper(
    paper_id="0704.3780",
    title="Stochastic Optimization Algorithms",
    year="2006",
    categories="cs.NE",
    abstract="---",
    input="---",
)

This will now create an instance of the Paper class on your machine, but it still hasn’t been sent over to Redis. When you now do the following, it will create a hash in Redis and save this data.

paper_instance.save()

Querying

When querying, you don’t have to worry about the hashes, you can just reuse the same model you used to create the hash: Paper.

A few examples:

# Return all papers after 2004
Paper.find(Paper.year > 2004).all()

# Return the first paper that contains the word "generative"
# in their abstract.
Paper.find(Paper.abstract % "generative").first()

This makes code so much more readable. This is a great fit for situations where you want the rigidity of a strongly typed DB with advanced validation features, but at the same time don’t want the overhead that comes with using a proper RDB.

Vectors

Vectors in the DB have to be bytes. So if you’re dealing with Numpy Arrays, you’ll have to convert them to bytes form first before storing them.

An interesting problem is that the datatype choice is up to you, so if you choose np.float32 to store a vector to the DB and retrieve it using np.fromBuffer with the data type as np.float64, you’re going to end up with a vector that’s half the size being returned. So use the same data type on both ends, when the vector enters and exits the DB.

np.array(vec, dtype=np.float32).tobytes()

Vector Indexes

There are hashes and there are keys beneath those hashes. We store vectors as values under a specific key. So when indexing the vectors, we need a way to group vectors of a kind together. A practical example is what we did with Top2Vec. We have three different kinds of vectors: word, document, and topic. And needed three indexes, one for each kind of vector. So we kept the key names for all three types the same, it was still “vector”. But when creating the index, we can pass a prefix option that will let us filter out a specific set of hashes whose “vector” fields will be indexed. So in our case, we had three functional indexes: “papers”, “words”, and “topics” and the hashes for each type began thus: paper_vector:, word_vector:, topic_vector.

redis_conn.ft(index_name).create_index(
    fields=fields,
    definition=IndexDefinition(
        prefix=[prefix],
        index_type=IndexType.HASH
    ),
)

For a detailed explanation of vector similarity in redis, visit this document.

Vector Search

The vector search queries are rather straightforward:

You define:

• The number of results you want to be returned.
• The field which contains the vector.
• The bytes representation of the vector for which you want to find the k-nearest neighbours.

With the Python client for Redis, this is made even simpler, especially when using code from Tyler Hutcherson’s library.

query = search_index.vector_query(number_of_results=50)

redis_conn.ft("papers").search(
    query,
    query_params={
        "vec_param": np.array(vec, dtype=np.float32).tobytes()
    },
)

Something to note is that the redis-py library doesn’t seem to be directly communicating with the API. Rather, it’s just constructing CLI commands through a Pythonic interface.

I see a lot of doors opening up for my team with vector capabilities from Redis. Just having it as a live vector store is more than valuable enough. We’re really looking forward to working further with Redisearch and try out more efficient ways of storing vectors and retrieving them.

An entry in a series of blogs written during the Vector Search Hackathon organized by the MLOps Community, Redis, and Saturn Cloud.

I’m a die-hard (note the italics) fan of deploying and self-hosting open-source software for production. But that said, I’ve found myself enjoying the freedom that Saturn Cloud provided. Starting our project only three days ago, when my teammate, Chirag Bajaj, and I had to double down and focus on the ML and Redis exploration, I was ecstatic to not have to think about infrastructure. Things just worked out of the box; a unique experience for someone like me, since I usually have to assemble the parts before getting started with anything. I can see why teams comprising solely of ML engineers would gravitate here. Zero thought gets put into infrastructure, but their work gets done on time.

Our approach in this hackathon was two-pronged, one on the MLOps side and one on the ML side.

MLOps

Machine Learning or Data Science tend to plateau after a point without solid tools backing them. There’s only so far you can go with manually naming your models, storing your scripts on disk, and using log files to understand how each training run progressed. So we made the conscious decision to devote our attention primarily to building a rudimentary MLOps platform. Both in the hackathon and in the real world, this causes ML work to crawl along at a snail’s pace when the platform is being set up; the uncertainty is excruciating and it feels like nothing gets done. But once everything is available and running, the pace of research accelerates to levels that would never be possible without the platform.

ML

Our goal here was to extract relationships between papers in the arXiv dataset and explore the natural clusters that formed in the data using topic modelling. To this end, we used Top2Vec, an excellent Python library which implements the ideas explored in this paper by Dimo Angelov.

Let’s take a quick look at the mechanisms that Saturn Cloud offers to enable what we’re trying to do.

Saturn Cloud

There are three concepts relevant to us:

Jupyter Servers

Used as easy development environments backed by powerful machines. Accessible through the Jupyter UI on your browser or through SSH. The SSH connection also lets you connect to this server from the familiar comfort of your IDE, which is always a wonderful experience.

Deployments

These are like servers but use images, startup scripts and commands to automate the process of deploying an application. So once the data scientist finishes developing something with a Jupyter Server, it should be trivial for them to clone their environment and launch it as a deployment that will be permanently accessible.

Jobs

Similar to deployments, but these are meant to be ephemeral, dying once their task is complete. These are perfect for orchestrating jobs that have to be triggered on a periodic schedule or on a specific signal.

Notes

• Saturn Cloud, the community edition, exposes only one port: 8000. This is the case on your deployments, and even on your Jupyter servers should you wish to use it. This port is automatically mapped to a website that looks like this:

https://<subdomain you choose>.community.saturnenterprise.io

• To protect the platform from the execution of malicious code, external images are disabled on the community version of Saturn Cloud Hosted. But since all sensible OSS have a simplified installation process, this doesn’t really affect us as you’ll see below. We can just install whatever we need during runtime or build a custom image with their image builder tool.

We go into some detail on the components of our platform below.

Model Training

We needed a fair bit of compute to train our Top2Vec model. It had to chew through a couple of million papers, and the compute requirements were made worse by the fact that we weren’t going to be generating embeddings from a pre-trained model, but rather training a Doc2Vec model from scratch. And to add to this, Top2Vec doesn’t support GPU acceleration.

So we used the 4XLarge machine, with its 16 CPU cores and 128 GB of memory. Thankfully the library was able to fire at a 100% on all 16 cores and made it through the entire dataset in ~7 hours.

The auto-shutoff feature helps you save costs if you leave a server running without having it open. But if you do have a lengthy training running on it, make sure to connect to it via SSH from a stable server so the SSH connection prevents the auto-shutoff from being triggered.

As noted earlier, the interesting part is that even though we developed an ad-hoc script and trained with a notebook on a Jupyter server, Saturn Cloud allows you to clone your environment and recreate that as a job or a deployment. Combine this with the feature that lets you automatically pull code from a GitHub repository and a simple Docker image builder to pre-package your dependencies, you now have a platform in Saturn Cloud that can take any bit of code that a data scientist tests and plays around with and translates that into a periodic job with little effort. This is an inspired workflow.

Object Storage

Since we have a system that’s spread out across multiple components, having our files on disk doesn’t quite cut it for us. We deployed MinIO to act as an S3-compatible object store. I don’t think I’ll ever get over how well the S3 API has unified the landscape for object storage. It was a breeze to use MinIO with boto3.

Installation and startup

wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20221029062133.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb

mkdir -p ~/minio

# Startup command for the deployment.
minio server ~/minio --address :8000

Interfacing with MinIO from boto3

The only difference from using this with S3 is that we specify an endpoint URL as follows.

s3_resource = boto3.resource(
    "s3",
    endpoint_url="https://xyz-minio.community.saturnenterprise.io",
    config=boto3.session.Config(signature_version="s3v4"),
)

Model Tracking

Behind every great model is a model registry rolling its eyes.

Mistakes are a core part of the data science journey, but mistakes only generate value if they are tracked. A model registry lets you do this and more.

We deployed MLFlow on Saturn Cloud to track our training runs. Though Top2Vec doesn’t quite generate metrics during training that we can track, we plan to create a metric from Topic Information Gain as described in equation 5 of the Top2Vec paper. It’s also useful to get an idea of the number of documents, words, and topics after each model is trained; we log these as params to MLFlow.

Installation and startup

# Startup command for the deployment.
mlflow ui --host 0.0.0.0 --port 8000 --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./artifacts

Interfacing with MLFlow from Python

It’s very straightforward.

• Set the Tracking URI.

mlflow.set_tracking_uri(os.environ[“MLFLOW_TRACKING_URI”])

• When your training/task begins, start your run.

mlflow.start_run(run_name=run_name)

• To log parameters that feed into the task or describe the task at a high level, use log_param or log_params. In the screenshot above, num_topics, num_words, etc are parameters.

mlflow.log_param(key=”num_workers”, value=workers)

• When your training/task ends, end your run.

mlflow.end_run()

Logging

Distributed anything is a dream for scalability, but it’s also a nightmare for logs. Ideally, you’d want a centralized logging interface so that you can make sense of everything. Log files sound good, and do the job, but are practically hard to work with when you want to get a sense of your system as a whole.

I swear by Loki for anything to do with logging, because its ability to process large volumes of logs and extract useful information and metrics from them is nothing short of magical.

Installation and startup

curl -O -L "https://github.com/grafana/loki/releases/download/v2.6.1/loki-linux-amd64.zip"
sudo apt update; sudo apt install unzip -y
unzip "loki-linux-amd64.zip"
chmod a+x "loki-linux-amd64"

wget https://xyz-minio.community.saturnenterprise.io/topvecsim/platform/loki/loki_config.yaml

# Startup Command for the deployment.
./loki-linux-amd64 --config.file loki_config.yaml

In the script above, you’ll note a link to xyz-minio. We store our Loki configuration file in our MinIO object store and download it whenever the Loki deployment starts up. Loki listens by default on port 3100, so we use the config file to redirect it to port 8000; specifically, we set http_listen_port: 8000.

Interfacing with Loki from Python

Loki has a simple API that’s easy to use, but we can use a library called python-logging-loki which makes the process easier by providing a logging handler that we can consume for our logger.

Here’s a sample:

loki_handler = logging_loki.LokiHandler(
    url=https://xyz-loki.community.saturnenterprise.io/loki/api/v1/push,
    version="1",
)

logger = logging.getLogger("topvecsim")
logger.addHandler(loki_handler)
logger.setLevel(logging.INFO)

logger.info("Starting model training.")

The library also provides a Queue Handler which is well-suited for high log-volume situations.

Monitoring

Grafana needs no introduction as an observability interface. It pairs perfectly with Loki and lets you manipulate logs into any number of intuitive visualizations.

Installation and startup

sudo apt-get update
sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_9.2.3_amd64.deb
sudo dpkg -i grafana_9.2.3_amd64.deb

# Change port 3000 to 8000.
sudo sed -i 's/3000/8000/g' /etc/grafana/grafana.ini
sudo sed -i 's/3000/8000/g' /usr/share/grafana/conf/defaults.ini

# Startup Command for the deployment.
sudo /usr/sbin/grafana-server  --homepath='/usr/share/grafana' --config='/usr/share/grafana/conf/defaults.ini'

What was missed out/in-progress for the Hackathon?

Jaeger

Distributed tracing to help you track the performance of your APIs and even your Python code that’s deployed in different places.

Loki Metrics on Grafana Dashboards

Sending logs to Loki as a JSON encoded string lets it deconstruct it and extract values from it. Doing this will let us plot any metrics from our application on Grafana through LogQL queries.

There are certain low-effort tasks you can take up that will make your life infinitely better. This is especially true when it comes to something as inherently chaotic and disorganized as ML/DS research and exploration. Invest in simple tools like the above for exponential workflow improvement.

Container Images have become the de facto mechanism for packaging dependencies and code. A few years ago, when the task of building these Images was under the purview of operations specialists, there existed the luxury of optimization — with beautiful multi-stage builds, clean-up after every step, etc. But today, any developer working on a modern stack gets to build their own images and pass them along to a container orchestrator like Kubernetes or Docker Swarm to deploy their application.

This decoupling between operations and application development has been a boon. A tiny team of Ops engineers — whatever you call them: Ops, DevOps, Platform, Infra — can serve a wide range of products at virtually any scale. At no point does an operations specialist need to get involved in this process of Image building once a simple build pipeline has been set up.

On the flip side, this decoupling now means that the Images built by app developers are barely optimized, contain a lot of unused packages and dead weight, and little thought is given to breaking an application into multiple pieces. Which is… completely fair. What we lose with the efficiency of these Images, we gain exponentially with the increase in developer velocity. There’s nothing worse than having Ops be a bottleneck to development. So, I’ll gladly take a few extra Gigabytes any day.

But, that said, here’s one of the biggest problems I face as a platform engineer: Large Startup Times. It’s pretty much the villain in my book, the silent killer of efficiency, and has implications up and down the stack.

In the case of Kubernetes with AWS EKS, here’s what happens when a new pod is launched but no node currently exists to serve it.

1. A new EC2 instance is launched.
2. The instance joins the cluster and becomes a K8s ‘node’.
3. The Docker image necessary for the pod to run is pulled to the node.
4. The container is launched and the application is started on it.

Each step takes time, but there’s nuance to this:

• Steps 1 and 2 are inevitable, and only EKS and K8s devs can contribute to any improvements here.
• Step 4 is dependent on the app developer. If they use a potato language like Python (I say this with love) with a lot of module imports, the app launch is going to be quite drawn out.

So, steps 1, 2, and 4 for a particular application take a fixed amount of time that’s out of the hands of an Ops engineer. But step 3, the Image pull time, is interesting, because it introduces the biggest opportunity for guaranteed improvement in this process.

Let’s say a particular task takes four minutes to run, and the 4 GB Image needed for the task takes two minutes to be pulled. This means that 33% of the computing time is taken up by the Image pull step, which is significant. And this only gets worse when the Image gets even larger and autoscaling gets aggressive. You could end up spending thousands of minutes of computing time a day on nothing but Image pulls.

How does this affect the larger picture? Say the app team wants a task to start in 20 seconds flat from a cold start, this is pretty much impossible unless you always have a pool of warmed up nodes that have the Container Images pulled and waiting for new tasks to be scheduled. This “headroom” is incredibly wasteful, and since many workloads are rather hard to predict, you’re paying round the clock for compute capacity that is guaranteed never to be used. (Disclaimer: I refuse to take responsibility if your finance team comes across this.)

The Solution

Every major cloud has the concept of a machine image from which a virtual machine can be launched. If the Container Image is ‘baked’ into a Machine Image, then when the machine is booted, the Container Image is already present and ready to go, thus completely eliminating the Container Image pull time.

The following steps assume the use of AWS EKS since it’s the most popular way to deploy K8s today, but the underlying concepts are incredibly straightforward and can be used for any cloud/platform.

Overview

Steps

1. Trigger Container Image build.
2. Store Image in a remote Container Image Registry.
3. Trigger Machine Image build with Packer.
4. Packer starts an EC2 instance from an EKS AMI.
5. It then runs a script on the instance that pulls the newly built Container Image from ECR.
6. Once done, it stops the EC2 instance and triggers the creation of an AMI from the stopped instance.
7. Once the AMI is created, Packer cleans up everything else.
8. We now make use of this AMI in our Kubernetes cluster manager — Karpenter, Cluster Autoscaler, Spotinst Ocean, etc.

Packer

Packer is an automation tool that lets you build, among a lot of other things, EC2 AMIs, which are AWS’ version of Machine Images. We’ll be using it to bake our Container Images into EKS AMIs so that we can use these new AMIs to launch the EC2 instances that will back the nodes of our K8s cluster.

Packer uses its own Domain Specific Language, but it’s well thought out and thus easy to grasp. Glance through the following script, and we’ll break it down in the section that follows it. I do have to note at this point that this is a bare-bones introduction to Packer to serve our current needs. For a comprehensive overview, I suggest following the official tutorials, they’re structured in an easy-to-consume manner.

This is executed using the following command:

packer build \
--var IMAGE=xyz.dkr.ecr.us-east-1.amazonaws.com/app:potato-0.2 \
--var AMI_NAME=baked-potato-ami-0.2 \
--var SOURCE_AMI=amazon-eks-gpu-node-1.23-* \
prepull.pkr.hcl

Breakdown

Our goal is to:

1. Launch an instance from a base EKS AMI. This is the default AMI that is used when new nodes are launched.
2. Execute the nerdctl-script.sh file on the launched instance.
3. Create a new AMI.

Let’s forget about the nerdctl-script.sh file for now, because its contents have no bearing on the packer build process, we only care that it modifies the instance in some beneficial way.

In the packer build command above, you’ll notice that we specify three var arguments. These are variables that configure the specific build. Each var is a key=value pair, and the corresponding keys will be defined as variable “key” in the pkr.hcl script.

We’ve used this mechanism to configure:

• IMAGE: This is the Container Image that will be baked into the AMI.
• AMI_NAME: The name of the AMI that will be created.
• SOURCE_AMI: The base AMI on top of which the customization will take place.

The source “amazon-ebs” “eks-ami” section captures information about the EC2 instance that will be launched, including the instance type, the volume size, IAM role attached to it, and the source AMI that’s used. You’ll spot the use of SOURCE_AMI through ${var.SOURCE_AMI} in the source_ami_filter section here. More details here and here.

The build section makes use of the source section and adds the shell provisioner on top of it. As the name suggests, the shell provisioner does nothing but execute a shell command or script on the source. In this case, the source is an EC2 instance and nerdctl-script.sh is the script that gets executed on it.

The Script

Kubernetes is an incredible container orchestrator, but for all its magic, it still requires a container runtime to run the Images it’s given. Up to K8s version 1.22, at least on EKS, Docker was the default runtime, but starting with version 1.23, containerd is used instead. For our purposes, this just means that for versions 1.22 and lower, we use Docker to pull the images from the remote Image registry. For version 1.23+, we use nerdctl, which is a Docker compatible CLI for containerd.

The “Docker compatible” part is pretty important for us because it lets us trivially authenticate nerdctl with AWS ECR to retrieve our Images in the same way we’d do it with Docker.

The following script, meant for K8s nodes with containerd instead of Docker, installs and uses nerdctl to pull Images to the machine. The comments in the script provide an explanation for each step.

The next script is meant for K8s nodes using Docker. Quite straightforward as opposed to the above.

As discussed earlier, the shell provisioner in the packer script just mentions a script that should be executed on the machine. So your folder structure is now the following, and you’d run the packer build command above from the packer-build folder.

packer-build/
  prepull.pkr.hcl
  nerdctl-script.sh

Karpenter

Once you have your new AMI from the previous steps, you just need to update your cluster manager. In this case, it’s Karpenter, which is an incredible K8s native cluster manager that currently only supports EKS. If you haven’t come across it, I urge you to read more about it here; it’s a true game-changer in my opinion.

Karpenter has the concept of a provisioner which is used to configure the instances it launches. There can be multiple provisioners, each with different characteristics for different purposes. For example, I have a provisioner each for CPU, GPU, and High Availability workloads. We can also link an AWSNodeTemplate to each provisioner to precisely configure the EC2 instances that are launched. Here’s a detailed example of a provisioner and the AWSNodeTemplate it uses, just for reference.

TL;DR: There is an aws-ids field in the AWSNodeTemplate that can be used to select the AMI. There are also other ways of specifying the AMI.

With Karpenter + AMI baking, I’ve seen cold starts of 40–50 seconds even with 4–5 GB Docker Images. (This feels like a poor man’s version of discussing quarter mile times in drag races.)

Conclusion

Optimizing large Container Images is a long-term goal that should be addressed through thorough training sessions, code reviews, and more. In the meantime, Ops teams can’t ignore the nightmare of torturous Image pull times and the effect it has on their platforms. So as much as I hate the bodging that this article promotes, we really do need it until a solution comes along to fix this issue for good (maybe it’s image streaming?). I can only hope it’s sooner rather than later.

So this article goes over some necessary tools for your Terminal to make life easier when working with Kubernetes.

Kube PS1

kube-ps1

Displays your current Kubernetes cluster and namespace information in the terminal prompt. It’s a script that can be installed with the Brew package manager if you already have it:

$ brew update
$ brew install kube-ps1

If you don’t have brew, don’t bother with it, just install kube-ps1 from source, it’s nothing but a script that needs to be sourced.

When you use managed clusters, your cluster names tend to be unwieldy, something like arn:aws:eks:us-east-1:<aws-account-id:cluster/<cluster-name>. So you can shorten it with a function before sourcing the kube-ps1 script. For example, here’s what my .bashrc contains:

# kube-ps1
function get_cluster_short() {
  echo "$1" | cut -d : -f6 | cut -d / -f2
}

KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
. $HOME/.kube-ps1/kube-ps1.sh

PS1='$(kube_ps1) ${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\[\033[00m\]\[\033[01;34m\]\W\[\033[00m\]\$ '

This gives me a concise cluster name, namespace, and local folder name. For brevity, my PS1 displays only the current directory’s name.

(⎈|<cluster-name>:kube-system) directory$

Note

PS1 is the primary prompt variable on Linux. It’s what defines the contents of the prompt you see whenever you open the terminal.

Aliases for kubectl

kubectl-aliases

This project contains aliases for more than 1200 useful kubectl commands in a consistent manner. The consistency guarantee is key because it makes it easy to guess commands without referring to the aliases or memorizing them. For example, kubectl get pods is kgpo

and kubectl get ingress -o yaml is kgingoyaml

Thus, kubectl get pods -o yaml is kgpooyaml

I was expecting that it’d take me a month to get used to it, but I became quite comfortable with it in just a day.

Installation

A straightforward process since it just needs the ~/.kubectl_aliases file to be sourced during shell start.

Kubectl Tree

kubectl-tree

There are a lot of interrelated resources in Kubernetes and kubectl doesn’t do the relationships justice when you look at your resources category by category. Using ownerReferences, kubectl-tree shows you the relevant resources in one view. Here’s an example, when we take a look at the ‘karpenter’ deployment with: kubectl tree deploy karpenter -n karpenter

A minor convenience, but a welcome one.

Installation

Uses the krew kubectl plugin manager. Once you have that, installation is just: kubectl krew install tree

Kubectx and Kubens

Link

Context management is a pain. Namespace management is a pain. These tools take care of that in a visually appealing and efficient manner.

Instead of going through kubectl config get-contexts and then use kubectl config use-context <context>. We just do kubectx <context>.

Similarly, just using kubens <namespace> will also work.

But the main value-add here is not the alias, but rather the autocompletion and fuzzy finder for both kubectx and kubens. This makes dealing with multiple clusters and endless namespaces so much easier.

Installation

Both can be installed with the ‘krew’ kubectl plugin manager.

kubectl krew install ctx
kubectl krew install ns

If you want the interactive mode shown in the gif above, you only need to install fzf in addition to the two plugins above.

K9s Terminal UI

K9s

I was previously a heavy user of the desktop UI Lens, but more recently, I’ve found that I much prefer K9s for its snappiness and keyboard shortcuts. You do miss the IDE feel that Lens gives you, but overall, I think K9s is the best Kubernetes graphical interface out there. And the minimal resource usage is the cherry on top.

It excels at giving you a bird’s eye view of your cluster from the command line. It also lets you edit resources, view logs, attach to a pod, etc. I’m surprised by how intuitive it was; where I expected a learning curve, I found none.

Managing multiple Kubectl Versions

I’ve covered this in this article. I highly recommend this if you routinely deal with multiple Kubernetes clusters running different K8s versions.

Even a task as simple as providing a dedicated set of nodes for a group of users requires creating new node groups with custom labels and taints. It greatly limited the potential of Kubernetes and caused nightmares for replication because we’d have to capture application requirements partially in K8s YAML files and partly in the node group configurations. Not to mention, having too many node groups is an anti-pattern, so we couldn’t even commit to that completely.

I’d reached a state of acceptance that this was as far as we’d ever get with cluster management and that I’d just have to learn to live with it. But out of seemingly nowhere…

Enter Karpenter. The casual overachiever.

Can I request infrastructure resources directly from the pod spec? Uh-huh.

Can I use Kubernetes YAML files to define my infrastructure? Files that I could then deploy and manage with ArgoCD? Sure why not?

*Thinking of something wildly unrealistic* What about isolated, dynamic node groups that need nothing more than a node selector or node affinity specified in the workload definition? Pfft, trivial.

Honestly, I have whiplash from how Karpenter just upended everything I thought I knew about cluster autoscaling; there’s just no going back from such a K8s native infrastructure interface now.

The rest of the article goes over some high-level features of Karpenter followed by a short discussion of the concepts underlying Karpenter and some questions.

Features

Rapid node launch times

I’ve consistently seen unschedulable pods start execution in as little as 40 seconds. This speed is attributed to Karpenter determining instance types and provisioning them from the EC2 fleet API within a couple of seconds of the pod entering the unschedulable state.

Because of this, I’m already rethinking how I view autoscaling. What previously used to be a task of warming up node groups to handle workload spikes is now just a passing thought, “Eh, I’ve got Karpenter, it’ll be up in half a minute.”

Complete Control of Instance Provisioning

Using the provisioner spec or the pod spec, you can control everything about your EC2 instances from your K8s cluster. No more fiddling with infrastructure interfaces like the AWS console/CLI or Spotinst Ocean to provision node groups.

Cost Management

You can use Spot instances with the guarantee of fallback to On-Demand. Karpenter uses the EC2 Fleet API to ensure long-lived spot instances.

Kubernetes Native

Karpenter does a great job of being K8s native, and the more I think about it, the more I see that this was a really good design choice. Because they’ve taken three core components of pod specifications: resource requests, node-selectors, and node-affinity. And they’ve transformed it into an interface for your infrastructure by requiring you to do nothing but match specific labels. For example, specifying karpenter.k8s.aws/instance-family and c5 lets you limit your workloads to EC2 instances from the c5 instance family. Something that would've previously taken a node group and a cluster administrator to achieve.

Zero Third-Party Cost

Directly managing node groups on AWS is of course free. But as soon as you start using external tools like Spotinst Ocean, you run into charges. For example, Ocean reportedly charges you ~20% of the costs you save by using Spot instances [1, 2, 3]. This is a not-so-insignificant amount when you’re talking about running hundreds of Spot instances; and what exactly are you paying for? Not the instances themselves, you still have to pay AWS for that. Are you paying all that money just for them provisioning your Spot instances on your AWS account? Is it really worth that?

With Karpenter, you only use AWS native services like the EC2 Fleet API, so you pay for nada but your EC2 usage.

Deprovisioning and Consolidation

Karpenter can save costs by terminating empty and underutilized nodes.

Concepts

Karpenter Controller

This component looks for unschedulable pods and launches (provisions) EC2 instances (nodes) to schedule these pods.

When a node is unused for a pre-configured period of time, the controller also deletes (deprovisions) the corresponding nodes.

The controller also takes care of consolidation; when nodes are underutilized, it will move workloads around and deprovision nodes to more efficiently bin-pack your cluster.

Layered Constraints

AWS has hundreds of instance types and dozens of instance families. Instead of forcing either the developer or the cluster administrator to take a call on the instance types and other attributes of the instances before the workload is even deployed, Karpenter allows both parties to specify constraints in different places.

An example:

The cluster administrator wants to keep the costs low and prevent instances with more than 50 GB of RAM from ever being provisioned for the cluster. So they specify this constraint in the ‘provisioner’.

The application developer finds this constraint too permissive for their application. Too many instance types have less than 50 GB of RAM, and the dev wants their pods to run only on a node that has between 15 and 25 GB of RAM, is network-optimized, and has a directly mounted SSD volume. Previously, they would have to put out a request to the cluster administrator, who would then provision a node group with a particular node label and taint, following which the app developer would be able to use it. Any follow-ups or modifications would have to be through the cluster admin.

With Karpenter, however, they can define this constraint directly in the pod specification of their workload using resource requests and node-selectors/node-affinity. Because the cluster admin has already enforced their constraints on the provisioner, they’re not involved in any way with this process. I’ve said this before, but this is game-changing! The infrastructure configuration resides inside your application YAML definition. And most importantly, you don’t have to bother your overworked infrastructure team for every little change, which means your development enters hyperspeed!

Constraints

Constraints are the key/value pairs specified in the provisioner or the pod spec that can control:

• Instance type — memory, CPU, architecture, and accelerators.
• Availability zone in which the instance is launched. (Topology)
• Where pods are deployed in relation to other pods — pod affinity/anti-affinity.

Provisioner (K8s Custom Resource)

Used at the cluster level, usually by the cluster administrator, to impose high-level constraints. The provisioner has a requirements field that allows the same constraints that can be specified in the pod spec, so instance types, topology, etc can be managed here. When pods then go into the unschedulable state, Karpenter tries to find a matching provisioner that can launch instances which will satisfy the constraints of the pod.

We can also specify the behaviour of the cloud provider, AWS for example. So things like the IAM InstanceProfile, AMI, User data, EC2 tags, etc can be configured here.

Provider

Even though Karpenter only supports AWS at present, it’s built to be cloud provider agnostic. So anything specific to a cloud is relegated to the provider definition. For AWS, this is the ‘AWSNodeTemplate’ Custom Resource, which can be referred to from the provisioner.

How does Karpenter schedule pods?

When there’s an unschedulable pod, Karpenter looks at the pod spec and determines the instance to be launched based on the resource requests and node-selectors/node-affinity.

It looks for a provisioner that is able to satisfy those constraints. When there are multiple provisioners eligible, Karpenter chooses one at random, so it’s advisable to make your provisioners mutually exclusive.

Twenty instance types are selected following this process, and a request is made to the EC2 fleet API to provision the necessary number of instances.

How are cost savings possible?

Karpenter allows you to use Spot instances with the option to fall back to On- Demand. So you get both low costs and guaranteed scheduling. If your provisioner spec or pod spec doesn’t limit the pool of possible instance types/zone by too much, you will get a spot instance 100% of the time. There are hundreds of EC2 instance types and multiple zones in every AWS region, which makes it practically impossible for the Spot market to ever completely “run out”.

How does Karpenter interact with the Spot Market?

It doesn’t! It interacts with the EC2 fleet API specifying the capacity-optimized-prioritized allocation strategy. This tells EC2 to provide spot instances that have the least likelihood of being interrupted so that the Spot instances on your cluster are not constantly being reclaimed by AWS. By letting the Fleet API do what it’s good at, Karpenter is saving itself from an impossible task — one of monitoring and keeping up with the Spot market in real-time.

How stable is Karpenter?

It’s more than stable enough for me to use it without worry in a volatile environment with regular and rapid autoscaling. But setting that aside, since I haven’t used it for very long, you can take the following points to the bank:

1. Community engagement is high.
2. It has well-recognized adopters using it in production.
3. Development is highly active and any necessary features are either already available or in design, development or testing.
4. AWS seems to be all in on it! In all honesty, this is pretty much the only assurance I needed. This project is here to stay.

Conclusion

Considering the issues that plague Kubernetes Cluster Scaling today, I see Karpenter’s revolutionary features gaining it immense traction over the course of the next year. As more people begin to use it, we will also start seeing its adoption by other cloud providers. If that doesn’t happen for some reason, they will at the very least adopt a similar model of bringing in infrastructure configuration into the cluster which can only be great news for anyone that uses managed Kubernetes (which seems to be the majority).

So a few years down the line, we’ll be looking at cluster scaling that will be baked into and native to every managed Kubernetes cluster.

Or so goes my dream.

Right now, for example, I have to use a 1.18 cluster and a 1.22 cluster simultaneously, which is painful because the version of kubectl you use to access a cluster can only be a maximum of one minor version away: skew ≤ 1. But in my case, the skew = 4, which means that I need a different kubectl binary for each cluster.

Managing these binaries directly is out of the question, it’s just not a clean solution. Instead, we use the ‘asdf’ tool version manager.

• Install asdf.
• Install the asdf kubectl plugin.
• Before accessing a specific cluster, we request asdf for the correct kubectl version to be enabled.
• That’s it!

Steps

Core Installation

Taken from here.

# Clone the asdf Github Repository to ~/.asdf
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.10.2

# Add the following lines to ~/.bashrc if on Linux.
# Or to ~/.bash_profile if on MacOS.

. $HOME/.asdf/asdf.sh
. $HOME/.asdf/completions/asdf.bash

Kubectl Plugin Installation

Taken from here.

# Install the kubectl plugin.
asdf plugin-add kubectl

# Install specific versions of kubectl as needed.
asdf install kubectl 1.18.20
asdf install kubectl 1.22.10

Use asdf

# To enable 1.18 access, run:
asdf global kubectl 1.18.20

# To enable 1.22 access, run:
asdf global kubectl 1.22.10

In cases where a single machine is shared by multiple people, use local instead of global to use multiple versions of kubectl simultaneously.

asdf local kubectl 1.18.20
asdf local kubectl 1.22.10