<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Duncan Campbell on Medium]]></title>
        <description><![CDATA[Stories by Duncan Campbell on Medium]]></description>
        <link>https://medium.com/@alchemicduncan?source=rss-2c470c7c96a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LKjFg8XDojVDTgL9</url>
            <title>Stories by Duncan Campbell on Medium</title>
            <link>https://medium.com/@alchemicduncan?source=rss-2c470c7c96a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:54:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alchemicduncan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Practical Guide to Reading Research Papers]]></title>
            <link>https://alchemicduncan.medium.com/a-practical-guide-to-reading-research-papers-50cfca2a7a3a?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/50cfca2a7a3a</guid>
            <category><![CDATA[reading]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[analysis]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 18:26:59 GMT</pubDate>
            <atom:updated>2026-03-28T18:26:59.803Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometimes it can be intimidating to figure out where to begin when reading a paper. Especially if you might be new to the process. There can be a lot of terminology that you might not be familiar with right away. Math that is referenced in the paper might appear complex, maybe you don’t have the right background right away to understand it. The learning process to understand what you’re even reading might be complex and seem to be filled with hurdles.</p><p>Personally I faced the same thing when I started off in industry working as a robotics engineer in research and development for various robots. I was intimidated by the sheer volume of papers that were being released. Technically I hadn’t even completed my bachelors at that point, so I felt like quite the fish out of water reading these things. Over time though, I’ve picked up some tips on how to look through things, determine if the paper might be worth reading, how to digest it bit by bit, and how to leverage current tools to make the process much easier!</p><h3>How To Find Papers to Read</h3><p>Beyond traditional methods, modern AI tools can drastically accelerate your discovery process:</p><ul><li><strong>Google’s AI Mode in Search (AI Overviews):</strong> For general subject area exploration, the new AI Mode tab in Google Search allows you to pose complex, multi-part questions in natural language. It uses powerful models to intelligently organize and synthesize information from multiple sources, which can help surface key papers or trends you were not explicitly searching for. AI Mode is designed to handle specific, complex queries that combine multiple searches and satisfy “endless curiosity”.</li><li><strong>Following Key Researchers/Labs:</strong> Platforms like <a href="https://scholar.google.com/">Google Scholar</a> or <a href="https://arxiv.org/">arXiv</a> remain excellent starting points. You can also trace the foundational work in a field by diligently reviewing the references of any paper you already found useful.</li><li><strong>Social Media Platforms (X, LinkedIn, etc.):</strong> Many researchers and labs share their latest findings directly on social media. Following industry leaders on <a href="https://twitter.com/">X (formerly Twitter)</a> and <a href="https://www.linkedin.com/">LinkedIn</a> can provide real-time updates on trending papers and discussions within your field.</li></ul><h3>Leveraging AI Tools for Synthesis</h3><p>Once you have found a paper, AI can help you quickly assess its relevance and condense its core ideas, saving you hours of initial reading time:</p><ul><li><a href="https://gemini.google.com/"><strong>Gemini</strong></a><strong>’s Deep Research and Summarization:</strong> Gemini’s Deep Research feature is designed to tackle complex research tasks by breaking them down, exploring the web for answers, and synthesizing findings into comprehensive, cited reports. Some models, like Gemini 3.1 Pro, offer a 1M token context window, allowing you to process multiple large documents for comprehensive analysis. Deep Research is built to summarize extensive texts with ease.</li><li><strong>Audio Overviews (</strong><a href="https://notebooklm.google/"><strong>NotebookLM</strong></a><strong>/Illuminate):</strong> If you prefer auditory learning, tools like NotebookLM and Illuminate can transform your uploaded documents (or publicly available URLs) into engaging audio discussions or summaries. These “Audio Overviews” use two AI hosts to summarize the material, make connections between topics, and stay grounded in the source content.</li><li><strong>Comparative AI Tools (e.g., </strong><a href="https://www.anthropic.com/product/claude"><strong>Claude</strong></a><strong>):</strong> Other sophisticated models like Claude (e.g., Claude 4.6 Opus) also offer advanced reasoning and long-context windows (up to 1M tokens), making them viable alternatives for tackling complex, detailed research papers.</li></ul><h3>The Three-Pass Approach: Deep Reading Strategy</h3><p>A strategy for deep technical reading is the “Three-Pass Approach.” Instead of reading from start to finish, you process the paper in increasing levels of depth. This method is crucial for maximizing comprehension and retaining information.</p><h3>1. The First Pass (The Bird’s Eye View)</h3><p><strong>Goal:</strong> To quickly determine the paper’s relevance and core idea (5–10 minutes).</p><ul><li>Read the title, abstract, and introduction carefully.</li><li>Glance at the section headings to understand the structure.</li><li>Read the conclusion and look over the references to gauge the paper’s scope and foundation.</li></ul><p><strong>Outcome:</strong> You should be able to identify the main contribution, the context (where the work fits into the field), and decide if the paper is worth a second pass.</p><h3>2. The Second Pass (Grasping the Content)</h3><p><strong>Goal:</strong> To understand the paper’s main argument and methods without getting bogged down in detail.</p><ul><li>Read the paper with greater care, focusing on figures, diagrams, and tables. Try to internalize the methodology without reading all the complex math or proofs.</li><li>Identify the key components of the paper: The main results, the evidence supporting them, and the limitations acknowledged by the authors.</li></ul><p><strong>Action:</strong> Take notes, highlight key definitions, and write brief summaries of each major section in your own words.</p><p><strong>Outcome:</strong> You should be able to summarize the main thrust of the paper, including the experiments and results, for a colleague.</p><h3>3. The Third Pass (Deep Understanding)</h3><p><strong>Goal:</strong> To critically evaluate the paper and gain a complete mastery of the material.</p><ul><li><strong>Detail-Oriented Review:</strong> Challenge every assumption. Follow the math in detail and replicate the logic of the proofs.</li><li><strong>Critical Analysis:</strong> Identify potential failings, overlooked constraints, and future work. Think about how you would implement the system described, or how you would modify the experiment.</li></ul><p><strong>Action:</strong> Write down questions for the authors, compare the work to the cited papers, and mentally “re-implement” the paper’s core ideas.</p><p><strong>Outcome:</strong> You should be able to describe the work’s strengths and weaknesses, explain the ideas to someone with little background, and potentially propose follow-up research.</p><h3>Conclusion</h3><p>Reading research papers is a skill that improves with practice. By finding the right sources and using a structured approach like the Three-Pass method, you can turn an intimidating stack of jargon into a manageable and exciting way to stay at the cutting edge of your field. To further enhance your learning journey, consider the following steps:</p><ul><li><strong>Join a Journal Club:</strong> Discussing papers with peers can offer new perspectives and clarify difficult concepts.</li><li><strong>Start a Research Log:</strong> Use tools like <a href="https://www.notion.so/">Notion</a> or <a href="https://obsidian.md/">Obsidian</a> to track the papers you’ve read and how they connect.</li><li><strong>Practice with </strong><a href="https://notebooklm.google/"><strong>NotebookLM</strong></a><strong>:</strong> Upload your next paper and use the AI to quiz yourself on the core methodology.</li></ul><p><em>Ready to dive in? Pick your first paper today and try the First Pass. Happy reading!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50cfca2a7a3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Misadventures in Kubernetes: Autoscaling Workers]]></title>
            <link>https://medium.com/google-cloud/misadventures-in-kubernetes-autoscaling-workers-40af6e0485f7?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/40af6e0485f7</guid>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Thu, 12 Mar 2026 01:30:09 GMT</pubDate>
            <atom:updated>2026-03-16T02:58:59.053Z</atom:updated>
            <content:encoded><![CDATA[<p>So we’ve already done a few things for setting up our own custom cluster. We’ve manually configured a Kubernetes Control Plane and joined worker nodes by hand. While that was a great way for us to learn the components, let’s be honest: setting up every server by hand is just not scalable for a real production environment. Or for creating a more resilient cluster as well! Really the main issue boils down to how we set up the initial pool of worker nodes.</p><p>If what I’m talking about us doing isn’t familiar you should first read our <a href="https://medium.com/@alchemicduncan/kubernetes-the-kinda-hard-way-creating-workers-45c493715547">prior post</a>!</p><h3>The Problem with Manual Nodes</h3><p>Right now, our cluster is static, and honestly, it’s a bit of a liability. If worker-1 decides to take an unscheduled vacation and crashes, it’s just gone. Our capacity takes a hit, and we are stuck in the dark until someone manually notices and provisions a replacement. In a modern setup, having to SSH into every new VM just to run a join command isn’t just tedious, it’s a bottleneck that keeps us from scaling when it actually matters. Not only that, but it also makes it hard to meet the expectations of what we thought Kubernetes would be able to help as well, no!?</p><p>There are a few things that we need our cluster to be able to do:</p><ul><li>Automatic Joining: New nodes should just join the cluster the moment they boot up, no human needed.</li><li>Self-Healing: If a node dies, the system should recognize the loss and automatically spin up a healthy replacement.</li><li>Smart Scaling: The cluster needs to get bigger when the load increases and shrink back down when things quieten down to save money.</li></ul><h3>Startup Scripts</h3><p>The key to our automation is ensuring a new VM joins the cluster automatically when it boots. We can’t be there to SSH in and run kubeadm join every time.</p><p>To achieve this, we will use a GCP Startup Script to run the join command for us.</p><h4>Step 1: Generate a Permanent Token</h4><p>Standard kubeadm tokens expire after 24 hours. For our autoscaling group that might last months, we need a permanent token.</p><p>On your Control Plane, run:</p><pre>kubeadm token create --print-join-command --ttl 0</pre><p>Copy the output command, as you’ll need it for the next step.</p><h4>Step 2: Create an Instance Template</h4><p>An Instance Template tells GCP how to build a VM by specifying the image, machine type, and scripts to run. Note that the <em>k8s-node-family</em> referenced here is a custom image we built in the previous part of this series.</p><p>We use the — metadata startup-script flag to inject our join command.</p><pre>gcloud compute instance-templates create k8s-worker-template \<br>--image-family=k8s-node-family \<br>--machine-type=e2-standard-2 \<br>--tags=k8s-worker \<br>--metadata startup-script=&#39;#! /bin/bash &amp;lt;PASTE_YOUR_JOIN_COMMAND_HERE&amp;gt;&#39;</pre><h4>Step 3: Create the Managed Instance Group (MIG)</h4><p>We will create a Regional MIG. This means GCP will spread our nodes across multiple zones, such as us-central1-a, b, and c, for high availability.</p><pre>gcloud compute instance-groups managed create k8s-worker-mig \<br>--template=k8s-worker-template \<br>--size=1 \<br>--region=us-central1</pre><p>GCP will immediately spin up 1 node. It will boot, run the startup script, and join your cluster automatically.</p><h4>Step 4: Enable Autoscaling</h4><p>Now we tell GCP to watch the CPU usage of these nodes. If the average CPU usage exceeds 60%, it will add more nodes for us (up to 5).</p><pre>gcloud compute instance-groups managed set-autoscaling k8s-worker-mig \<br>--max-num-replicas=5 \<br>--min-num-replicas=1 \<br>--target-cpu-utilization=0.60 \<br>--region=us-central1</pre><h3>Stress Testing the Autoscaler</h3><p>Let’s prove it works. We’ll use a local kubectl connection to create artificial load.</p><ol><li>Create a Load Generator We’ll deploy a simple busybox container that does nothing but an infinite loop to burn CPU.</li></ol><pre>kubectl create deployment load-generator --image=busybox -- /bin/sh -c &quot;while true; do :; done&quot;</pre><p>2.<strong> Request CPU</strong> This step is absolutely critical. We must tell Kubernetes. exactly how much CPU this pod requires; without these resource requests, the cluster autoscaler will not recognize that the node is at capacity and will fail to trigger the scale-up event.</p><pre>kubectl set resources deployment load-generator --requests=cpu=200m</pre><p>3. Scale the Load Scale it to 20 replicas.</p><pre>kubectl scale deployment load-generator --replicas=20</pre><p>4. Watch the Magic Open two terminal windows. In one, watch your nodes:</p><pre>kubectl get nodes -w</pre><p>In the other, watch GCP instances:</p><pre>gcloud compute instance-groups managed list-instances k8s-worker-mig --region=us-central1</pre><p>You will see the single node fill up, pods go into Pending state, and the GCP Autoscaler provision new VMs to handle the load.</p><h3>The Easy Way (GKE)</h3><p>It’s always worth pausing to appreciate just how much effort we put into this. If you were using Google Kubernetes Engine (GKE), this entire process could have been replaced by a single “easy button” command:</p><pre>gcloud container clusters create k8s-easy-cluster \<br>--zone us-central1-a \<br>--num-nodes 3 \<br>--machine-type e2-medium</pre><p>While GKE is powerful, understanding the “hard way” makes us better operators because we know exactly which components to investigate when things go wrong.</p><p>Ready for the next challenge? Follow along to the next part of the series: upgrading your control plane without downtime!</p><h3>Are We Done? (Or Just Getting Started?)</h3><p>At this stage, we have a functional Kubernetes cluster on Compute Engine that includes self-healing and auto-scaling capabilities. This setup provides control over the operating system, kernel, and networking without GKE management fees, demonstrating that automation can be integrated into a manual build. However, this configuration is merely the foundation for a much broader architectural exploration. What are some things that you might want this cluster to do that it doesn’t already?</p><h3>Go Deeper</h3><ul><li>The Original Blueprint: This entire series is inspired by Kelsey Hightower’s foundational guide, <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way">Kubernetes The Hard Way</a>.</li><li>Official Documentation: Get familiar with the source of truth for all components: <a href="https://kubernetes.io/docs/">Kubernetes.io Documentation</a>.</li><li>Official GKE Documentation: Access the complete guide for <a href="https://cloud.google.com/kubernetes-engine/docs?utm_campaign=CDR_0x586bd5d5_default_b479225639&amp;utm_medium=external&amp;utm_source=blog">Google Kubernetes Engine (GKE) Documentation</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40af6e0485f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/misadventures-in-kubernetes-autoscaling-workers-40af6e0485f7">Misadventures in Kubernetes: Autoscaling Workers</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Misadventures in Kubernetes: Creating Workers]]></title>
            <link>https://medium.com/google-cloud/kubernetes-the-kinda-hard-way-creating-workers-45c493715547?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/45c493715547</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Sat, 28 Feb 2026 00:56:10 GMT</pubDate>
            <atom:updated>2026-03-14T01:14:35.759Z</atom:updated>
            <content:encoded><![CDATA[<p>So we have built a control plan from scratch. That’s kinda useful and was also kinda hard to make? But, really what we want to make is a proper, functioning, multi-node cluster that can actually run our containerized applications. A control plane by itself is just the brain; we need the muscle (the worker nodes) to do the actual work.</p><p>When we set up the control node, setting up the Control Plane required us to manually configure every single detail of the seed VM, line-by-line. While that was an important exercise in understanding the components, trying to repeat those steps for 10, 50, or 100 worker nodes would quickly become a nightmare. This part of the series is about making the process scalable by creating a “Golden Image” from our pre-configured machine, allowing us to rapidly provision identical workers and efficiently join them to our existing cluster.</p><h3>Creating a Golden Image</h3><p>The core principle here is to stop doing repetitive manual configuration. We already spent the time when we set up the control node getting the operating system tuned exactly right. Remember all those steps? We loaded the overlay and br_netfilter kernel modules, tweaked the network bridge settings for iptables, and, critically, disabled swap entirely because Kubelet won’t run otherwise. We also installed containerd and configured it to use systemd as the cgroup driver.</p><p>That entire stack of prerequisites, the OS configuration, and the necessary Kubernetes binaries (kubeadm, kubelet, kubectl) are now perfectly baked into our k8s-seed VM. A <strong>Golden Image</strong> is simply a frozen, ready-to-use snapshot of that perfect setup. Instead of repeating those steps on every new node, we use this image as the template to rapidly spin up as many identical worker nodes as we need. This moves us from a manual, configuration-based process to an automated, image-based provisioning process, which is the foundational step toward true cloud-native infrastructure.</p><h3>Step 1: Generalize the Seed Instance</h3><p>Before we can use our k8s-seed VM as a template, we need to “seal” it. Every running Linux machine has unique identifiers — a machine ID and specific network logs. If we duplicate these across multiple nodes, the Kubernetes Control Plane will be confused, seeing multiple machines with the same identity. We need to clear these unique details so that when a new VM boots from the image, it generates its own fresh identity and network configuration.</p><ol><li><strong>SSH into the seed machine:</strong></li></ol><pre>gcloud compute ssh k8s-seed --zone=us-central1-a</pre><p>2. <strong>Clean up unique identifiers:</strong> Run the following command to remove the machine ID and reset cloud-init logs. This ensures that when new VMs boot from this image, they generate their own fresh identity.</p><pre>sudo cloud-init clean --seed --logs --machine-id<br>exit</pre><p>3. <strong>Stop the instance:</strong> Back on your local machine, stop the VM so we can snapshot its disk.</p><pre>gcloud compute instances stop k8s-seed --zone=us-central1-a</pre><h3>Step 2: Create the Custom Image</h3><p>With the seed machine prepped and stopped, we now create a Google Compute Engine image from that disk. This image, k8s-node-image-v1, will serve as the foundation for every node in our cluster, whether they eventually become a control plane member or a worker.</p><p>Run the following command to create the image:</p><pre>gcloud compute images create k8s-node-image-v1 \<br>  --source-disk=k8s-seed \<br>  --source-disk-zone=us-central1-a \<br>  --family=k8s-node-family</pre><p><strong>Note:</strong> We added the — family flag so we can easily reference the “latest” version of this image later without needing to know the exact version name. This is a great cloud-native practice for image management!</p><h3>Step 3: Provision Worker Nodes</h3><p>Now for the easy part. Because the heavy lifting of configuration is done, spinning up new workers is instantaneous. We can create two worker nodes identically, pre-loaded with all the necessary Kubernetes prerequisites. This process takes minutes, a stark contrast to the hours spent manually configuring the seed VM in when we setup the control node.</p><p>Run these commands to create worker-1 and worker-2 using the image family we just created:</p><pre>gcloud compute instances create worker-1 \<br>  --zone=us-central1-a \<br>  --image-family=k8s-node-family \<br>  --tags=k8s-worker<br><br>gcloud compute instances create worker-2 \<br>  --zone=us-central1-a \<br>  --image-family=k8s-node-family \<br>  --tags=k8s-worker</pre><h3>Step 4: Join the Cluster</h3><p>Your worker VMs are running, but they are just generic machines with Kubernetes components installed; they don’t know they belong to a cluster yet. This is where the magic of the kubeadm join command comes in. The worker’s kubelet agent needs to securely authenticate and register with the Control Plane’s API server. The join command provides the token and the hash that validates the worker’s identity, effectively making it a true node.</p><ol><li><strong>Locate your join command:</strong> Find the kubeadm join command you saved from the output of kubeadm init when we set up the control node. It looks something like this:</li><li><strong>Join the workers:</strong> SSH into each worker and run that command with sudo. For Worker 1:</li></ol><pre>kubeadm join 10.128.0.x:6443 --token &lt;token&gt; --discovery-token-ca-cert-hash sha256:&lt;hash&gt;</pre><p>3. <strong>For Worker 2:</strong> Repeat the same steps for worker-2.</p><pre>gcloud compute ssh worker-1 --zone=us-central1-a<br>sudo kubeadm join ... # Paste your command here<br>exit</pre><h3>Step 5: Verify the Cluster</h3><p>Head back to your <strong>Control Plane</strong> to verify that everyone has checked in. This is the satisfying moment where all that hard work pays off and you see your multi-node cluster come alive.</p><pre>kubectl get nodes</pre><p>You should see output similar to this, with all nodes showing a Ready status:</p><pre>NAME            STATUS   ROLES           AGE   VERSION<br>control-plane   Ready    control-plane   10m   v1.35.0<br>worker-1        Ready    &lt;none&gt;          2m    v1.35.0<br>worker-2        Ready    &lt;none&gt;          1m    v1.35.0</pre><h3>The Easy Way (GKE)</h3><p>It’s always worth pausing to appreciate just how much effort we put into this. Throughout this series, we’ve manually provisioned VMs, installed binaries, configured systemd services, and joined nodes — all to get a basic Kubernetes cluster running.</p><p>If you were using <strong>Google Kubernetes Engine (GKE)</strong>, this entire process (for both the control plane and worker nodes) could have been replaced by a single command:</p><pre>gcloud container clusters create k8s-easy-cluster \<br>  --zone us-central1-a \<br>  --num-nodes 3 \<br>  --machine-type e2-medium</pre><h3>What’s Next?</h3><p>You now have a multi-node cluster, which is a huge win. But there’s a big problem: your cluster is still static. If worker-1 crashes, it’s gone forever and your capacity is reduced until you manually replace it. If your application load increases, you have to manually provision worker-3, SSH in, and run the kubeadm join command. Future-proofing your infrastructure means moving beyond these manual interventions.</p><p>To achieve a truly “cloud-native” and resilient cluster, we need to automate the lifecycle of our worker nodes, ensuring they can be recreated and scaled without manual effort. This is something to think about, maybe this is something that we might touch on in the future possibly?</p><h3>Go Deeper</h3><p>The journey through “Kubernetes the Kinda Hard Way” is fundamentally about building foundational knowledge. Ready to solidify your understanding and tackle advanced concepts? Check out these resources:</p><ul><li><strong>The Original Blueprint:</strong> This entire series is inspired by Kelsey Hightower’s foundational guide, <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way"><strong>Kubernetes The Hard Way</strong></a>.</li><li><strong>Official Documentation:</strong> Get familiar with the source of truth for all components: <a href="https://kubernetes.io/docs/"><strong>Kubernetes.io Documentation</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45c493715547" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-the-kinda-hard-way-creating-workers-45c493715547">Misadventures in Kubernetes: Creating Workers</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Misadventures in Kubernetes: Provisioning the Control Plane]]></title>
            <link>https://medium.com/google-cloud/kubernetes-the-kinda-hard-way-provisioning-the-control-plane-f60ebc078a8b?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f60ebc078a8b</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Sat, 28 Feb 2026 00:26:55 GMT</pubDate>
            <atom:updated>2026-03-13T05:33:26.499Z</atom:updated>
            <content:encoded><![CDATA[<p>When using managed platforms like Google Kubernetes Engine (GKE), there are many things that you don’t have to worry about that it takes care of for you! But have you ever paused to consider what machinery operates beneath that simple surface? Perhaps wondered how you might be able to do it on your own, or how to at least go about trying to do it?</p><p>My personal motivation for this project was simple: I wanted to give this a try myself to truly understand Kubernetes at its core. Moving past the abstraction of managed services, I set out to peel back the layers and inspect the core components. This approach involves setting up standard Google Compute Engine (GCE) Virtual Machines, manually installing and configuring every component, and wiring the networking together ourselves. This deliberate, hands-on process helps build the foundational knowledge necessary to effectively troubleshoot, optimize, and better understand our own clusters.</p><h3>Prerequisites</h3><p>Before we dive in, ensure you have the following:</p><ul><li>A Google Cloud Platform (GCP) account.</li><li>The <a href="https://docs.cloud.google.com/sdk/docs/install-sdk">gcloud CLI</a> installed and authenticated on your local machine.</li></ul><h3>Step 1: Provision the Base Instance</h3><p>We start by creating a “seed” or a base instance. This VM serves as our foundation where we install all necessary software, providing a clear template for how the Kubernetes nodes/VMs are constructed. The E2 series is often chosen because it offers the most cost-optimized VMs on GCP, providing a performance-to-cost balance suitable for a Kubernetes control plane. We are using the e2-standard-2 machine type because Kubernetes requires at least 2 vCPUs and 2GB of RAM to run comfortably.</p><p>Run the following command to create your VM:</p><pre>gcloud compute instances create k8s-seed \<br>    --zone=us-central1-a \<br>    --machine-type=e2-standard-2 \<br>    --image-project=ubuntu-os-cloud \<br>    --image-family=ubuntu-2204-lts \<br>    --boot-disk-size=50GB</pre><p>Once created, SSH into the machine:</p><pre>gcloud compute ssh k8s-seed --zone=us-central1-a</pre><h3>Step 2: Configure the OS</h3><p>Kubernetes has specific requirements for the underlying Linux OS. We need to load specific kernel modules and tweak network settings to allow Kubernetes to manipulate traffic correctly.</p><h4>1. Load Kernel Modules</h4><p>These modules allow Kubernetes to manipulate network traffic for Pods and Services.</p><pre>cat &lt;&lt;EOF | sudo tee /etc/modules-load.d/k8s.conf<br>overlay<br>br_netfilter<br>EOF<br><br>sudo modprobe overlay<br>sudo modprobe br_netfilter</pre><h4>2. Network Bridge Settings</h4><p>Ensure bridged traffic is passed to iptables for correct filtering.</p><pre>cat &lt;&lt;EOF | sudo tee /etc/sysctl.d/k8s.conf<br>net.bridge.bridge-nf-call-iptables  = 1<br>net.bridge.bridge-nf-call-ip6tables = 1<br>net.ipv4.ip_forward                 = 1<br>EOF<br><br>sudo sysctl --system</pre><h4>3. Disable Swap</h4><p>This is critical. The Kubelet will fail to start if swap is enabled.</p><pre>sudo swapoff -a<br># Edit fstab to prevent swap from turning on after reboot<br>sudo sed -i &#39;/ swap / s/^\(.*\)$/#\1/g&#39; /etc/fstab</pre><h3>Step 3: Install the Runtime (Containerd)</h3><p>Kubernetes needs a container runtime to launch pods. We will use containerd.</p><p>We choose containerd because it is the industry-standard core container runtime and is lightweight. Kubernetes now defaults to containerd, and runtimes like Docker are being deprecated in modern Kubernetes versions.</p><h4>1. Add Docker’s Apt Repository</h4><pre>sudo apt-get update<br>sudo apt-get install -y ca-certificates curl gnupg<br>sudo install -m 0755 -d /etc/apt/keyrings<br>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg<br>sudo chmod a+r /etc/apt/keyrings/docker.gpg<br><br>echo \<br>  &quot;deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \<br>  $(. /etc/os-release &amp;&amp; echo &quot;$VERSION_CODENAME&quot;) stable&quot; | \<br>  sudo tee /etc/apt/sources.list.d/docker.list &gt; /dev/null</pre><h4>2. Install Containerd</h4><pre>sudo apt-get update<br>sudo apt-get install -y containerd.io</pre><h4>3. Configure Systemd Cgroups</h4><p>Kubernetes recommends using systemd as the cgroup driver.</p><pre>sudo mkdir -p /etc/containerd<br>sudo containerd config default | sudo tee /etc/containerd/config.toml<br>sudo sed -i &#39;s/SystemdCgroup = false/SystemdCgroup = true/&#39; /etc/containerd/config.toml<br>sudo systemctl restart containerd</pre><h3>Step 4: Install Kubernetes Tools</h3><p>Now we install the “big three”:</p><ul><li>kubeadm: The bootstrapper.</li><li>kubelet: The node agent.</li><li>kubectl: The CLI.</li></ul><pre># Add Kubernetes repo<br>sudo apt-get update<br>sudo apt-get install -y apt-transport-https ca-certificates curl gpg<br><br>curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg<br><br>echo &#39;deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /&#39; | sudo tee /etc/apt/sources.list.d/kubernetes.list<br><br># Install<br>sudo apt-get update<br>sudo apt-get install -y kubelet kubeadm kubectl<br>sudo apt-mark hold kubelet kubeadm kubectl</pre><p><strong>Note:</strong> We use apt-mark hold to prevent automatic updates from breaking the cluster.</p><h3>Step 5: Initialize the Cluster</h3><p>At this point, our “seed” machine is fully prepped. We will use it now to initialize the Control Plane.</p><h4>1. Run Init</h4><p>We specify a pod network CIDR that is compatible with our chosen networking plugin (Calico).</p><pre>sudo kubeadm init --pod-network-cidr=192.168.0.0/16</pre><h4>2. Configure Kubectl</h4><p>To run commands against your new cluster:</p><pre>mkdir -p $HOME/.kube<br>sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config<br>sudo chown $(id -u):$(id -g) $HOME/.kube/config</pre><h3>Step 6: Install Networking (Calico)</h3><p>Nodes cannot communicate until a Container Network Interface (CNI) is installed. We’ll use Calico.</p><p>Calico offers high-performance networking by deploying without encapsulation or overlays, and provides a distributed firewall for flexible network policy enforcement.</p><pre>kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml</pre><p>Run kubectl get nodes and within a minute, you should see your node transition to Ready.</p><h3>What if I’d Just Used GKE?</h3><p>To appreciate the work we just did, it’s worth noting that you could have achieved a fully managed, production-hardened equivalent with a single Google Kubernetes Engine (GKE) command:</p><pre>gcloud container clusters create k8s-easy \<br>    --zone us-central1-a \<br>    --machine-type e2-standard-2 \<br>    --num-nodes 3</pre><p>This single command provisions the control plane (managed by Google), creates worker nodes, configures networking, and sets up authentication. But by building part of it manually, you now understand <em>how</em> those components work together.</p><h3>What’s Next?</h3><p>Congratulations! You have successfully built a functional Kubernetes control plane from scratch.</p><p>By building the cluster “the kinda hard way,” you’ve moved beyond being a user of managed services to understanding more about the components that orchestrate modern containerized applications. This foundation is key to troubleshooting, optimizing, and scaling production-grade Kubernetes environments. But what if you want to use more than control nodes?? Maybe think about how you might be able to create worker nodes to work with your existing cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f60ebc078a8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/kubernetes-the-kinda-hard-way-provisioning-the-control-plane-f60ebc078a8b">Misadventures in Kubernetes: Provisioning the Control Plane</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Nemotron 3 on A3 Spot VM]]></title>
            <link>https://alchemicduncan.medium.com/running-nemotron-3-on-a3-spot-vm-505a37a98d33?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/505a37a98d33</guid>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Fri, 19 Dec 2025 01:30:10 GMT</pubDate>
            <atom:updated>2025-12-19T01:30:10.722Z</atom:updated>
            <content:encoded><![CDATA[<p>Nvidia just released a new model this week, <a href="https://huggingface.co/blog/nvidia/nemotron-3-nano-efficient-open-intelligent-models"><strong>Nemotron 3</strong></a>. A3 instances on Google Cloud, powered by NVIDIA H100 GPUs, are perfect for running this model but can traditionally be a bit pricey. By utilizing <strong>Spot VMs</strong>, we can access this hardware at a fraction of the cost to run high-performance inference server that we can play around with!</p><p>This guide will walk you through provisioning an A3 Spot VM, setting up the environment, and serving the nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 model using vLLM.</p><h3>Setting up the Instance</h3><h4><strong>Creation</strong></h4><p>We will use the Google Cloud CLI (gcloud) to provision an A3 Spot VM. This instance comes with 8 NVIDIA H100 GPUs.</p><p>Ensure you have your project ID ready and the necessary APIs enabled. Run the following command to create the instance:</p><pre>export INSTANCE_NAME=&quot;nemotron-inference&quot;<br>export PROJECT_ID=&quot;your-project-id&quot; # Replace with your project ID<br>export ZONE=&quot;us-east5-a&quot; # Verify availability in your preferred zone<br><br>gcloud compute instances create &quot;${INSTANCE_NAME}&quot; \<br>    --project=&quot;${PROJECT_ID}&quot;\<br>    --zone=&quot;${ZONE}&quot; \<br>    --machine-type=a3-highgpu-8g \<br>    --provisioning-model=SPOT \<br>    --instance-termination-action=STOP \<br>    --create-disk=boot=yes,device-name=nemotron-disk,size=2000,type=hyperdisk-balanced,image=projects/ml-images/global/images/c0-deeplearning-common-cu113-v20241118-debian-11</pre><p><strong>Key Flags:</strong></p><ul><li>— machine-type=a3-highgpu-8g: Specifies the A3 instance with 8 H100 GPUs.</li><li>— provisioning-model=SPOT: Requests the discounted Spot pricing.</li></ul><h4>Initial Setup</h4><p>Once the instance is running, SSH into it:</p><pre>gcloud compute ssh &quot;${INSTANCE_NAME}&quot; --project=&quot;${PROJECT_ID}&quot; --zone=&quot;${ZONE}&quot;</pre><h4>Environment Configuration</h4><p>Luckily there’s really only one thing to do here, download the specific reasoning parser required for Nemotron 3.</p><pre>wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/resolve/main/nano_v3_reasoning_parser.py</pre><h3>Running the Inference Server via Docker</h3><p>For this tutorial we’re going to leverage the base vLLM Docker image. You can run the following command (ensure you have your Hugging Face token set as $HF_TOKEN):</p><pre>sudo docker run --runtime nvidia --gpus all \<br>    -v ~/.cache/huggingface:/root/.cache/huggingface \<br>    -v $(pwd):/workspace \<br>    --env &quot;HF_TOKEN=$HF_TOKEN&quot; \<br>    -p 8000:8000 \<br>    --ipc=host \<br>    vllm/vllm-openai:latest \<br>    --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \<br>    --max-num-seqs 8 \<br>    --tensor-parallel-size 8 \<br>    --max-model-len 262144 \<br>    --trust-remote-code \<br>    --tool-call-parser qwen3_coder \<br>    --reasoning-parser-plugin /workspace/nano_v3_reasoning_parser.py \<br>    --reasoning-parser nano_v3</pre><h3>Testing the Model</h3><p>Once the server is up and running (listening on port 8000), you can test it using curl from a separate terminal window (or your local machine if you’ve set up port forwarding/firewall rules).</p><pre>curl http://localhost:8000/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer token&quot; \<br>  -d  &#39;{ <br>    &quot;model&quot;: &quot;nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;system&quot;,<br>        &quot;content&quot;: &quot;You are a helpful assistant capable of complex reasoning.&quot;<br>      },<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Assume I am in a game show with 3 doors. Behind one is a car, behind the others, goats. I pick Door 1. The host opens Door 3, which has a goat. Should I switch to Door 2? Explain your reasoning clearly.&quot;<br>      }<br>    ],<br>    &quot;temperature&quot;: 0.6,<br>    &quot;max_tokens&quot;: 1024<br>  }&#39;</pre><h3>Cleanup</h3><p>Remember to stop or delete your Spot VM when you are finished to avoid incurring unnecessary costs.</p><pre>gcloud compute instances delete &quot;${INSTANCE_NAME}&quot; --project=&quot;${PROJECT_ID}&quot; --zone=&quot;${ZONE}&quot;</pre><h3>Conclusion</h3><p>This guide demonstrated how to efficiently provision an A3 Spot VM on Google Cloud, configure the environment, and run the Nemotron 3 model using vLLM and Docker. By leveraging Spot VMs, you can significantly reduce the cost of running high-performance inference servers with powerful NVIDIA H100 GPUs. <strong>Remember to clean up your resources after use to prevent unexpected charges.</strong> Ready to try running inference with large language models and vLLM? Perhaps consider using vLLM with spot instances for small scale experiments to gauge the performance of new open models when they get released!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=505a37a98d33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training Nanochat on a A3 Spot VM]]></title>
            <link>https://medium.com/google-cloud/training-nanochat-on-a-a3-spot-vm-9eae1710d9a0?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9eae1710d9a0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nanochat]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Fri, 24 Oct 2025 23:21:31 GMT</pubDate>
            <atom:updated>2025-10-27T20:44:33.947Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*WKiaF7N1FIoRmdWSCR0hRw.png" /></figure><p>Training your own model from scratch might seem like a luxury reserved for tech giants with massive budgets. That used to be the case. However, this post will demystify the process, demonstrating how to provision a temporary, supercomputer-class machine on Google Cloud at a fraction of the typical cost. This allows you to train nanochat, a minimal, hackable, and entirely open-source implementation of a ChatGPT-style LLM.</p><p><a href="https://github.com/karpathy/nanochat">Nanochat</a> is an open source project created by Andrej Karpathy<strong>. </strong>It’s designed to be the simplest, most approachable implementation possible, boiling down the complexities of LLM training into a single, understandable Python script. In this post, we’ll guide you step-by-step through provisioning the right hardware, configuring the environment, and running the training script. By the end, you’ll have trained your own model from scratch and be able to chat with it.</p><h3>A Cheaper Compute Option — Spot VMs</h3><p>Training an LLM typically involves millions, sometimes billions, of matrix multiplications, a task that GPUs, and TPUs, are exceptionally good at. For this, we turn to Google Cloud’s A3 machine series, specifically instances equipped with NVIDIA’s H100 GPUs.</p><p>This instance type is a power machine; an instance with 8 of these GPUs is effectively a personal supercomputer. While such power can be expensive on-demand, for instance an A3 can be <a href="https://cloud.google.com/products/calculator?hl=en&amp;dl=CjhDaVJsWm1VeU5EaGtPUzFtWWpaaExUUmpNRGN0WVdWbE5TMWhaVEprTXpVeE16aG1NalFRQVE9PRAIGiQ0NkRBNzE4Ri1CMkEyLTQ4MzgtOTQyMy1BMTMyNkQzNkY4Rjc&amp;utm_campaign=CDR_0x586bd5d5_default_b454357970&amp;utm_medium=external&amp;utm_source=blog">upwards of $80</a> on demand! So how do we make this more affordable? We can use <a href="https://cloud.google.com/solutions/spot-vms?hl=en&amp;utm_campaign=CDR_0x586bd5d5_default_b454357970&amp;utm_medium=external&amp;utm_source=blog">Spot VMs</a>.</p><p>Spot VMs leverage spare Google Cloud capacity, offered at steep discounts (often 60–90% off). The primary caveat is that Google can reclaim this capacity at any time through a “preemption.” However, for lengthy training runs, the nanochat scripts mitigate this risk by saving checkpoints frequently. If preemption occurs, we can easily spin up a new instance and resume training from the last checkpoint, ensuring we only pay for the compute time used. This approach can give us a VM that can <a href="https://cloud.google.com/products/calculator?hl=en&amp;dl=CjhDaVF4TUdJd1lXUmtZeTAwWmpVMUxUUmlORFV0WW1VeFlTMDRaV1EwWldVME5XVm1aRFVRQVE9PRAIGiQ0NkRBNzE4Ri1CMkEyLTQ4MzgtOTQyMy1BMTMyNkQzNkY4Rjc&amp;utm_campaign=CDR_0x586bd5d5_default_b454357970&amp;utm_medium=external&amp;utm_source=blog">cost under $20</a> an hour versus our initial cost that was upwards of $80 an hour!</p><h3>Setting up the Instance</h3><h4>Creation</h4><p>Before running through these steps make sure that you have a <a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects?utm_campaign=CDR_0x586bd5d5_default_b454357970&amp;utm_medium=external&amp;utm_source=blog">project setup</a> and have <a href="https://cloud.google.com/billing/docs/how-to/modify-project?utm_campaign=CDR_0x586bd5d5_default_b454357970&amp;utm_medium=external&amp;utm_source=blog">billing enabled</a>. Next make sure the APIs that we’ll need are enabled for the project using something like the following command below where PROJECT_ID is the name of the project you created:</p><pre>gcloud services enable compute.googleapis.com \<br>    --project=&quot;${PROJECT_ID}&quot;</pre><p>To provision this powerful machine, we’ll use the gcloud command-line tool. The command to create an A3 Spot VM is comprehensive, so let’s break down its essential parts. Remember to replace INSTANCE_NAME and PROJECT_ID with your specific values.</p><pre>export INSTANCE_NAME=&quot;nanochat-machine&quot; # Or your own custom name <br>export PROJECT_ID=&quot;&quot; # Replace with your project name <br>export ZONE=&quot;us-east5-a&quot; # You can use this region or choose a different one <br><br>gcloud compute instances create &quot;${INSTANCE_NAME}&quot; \<br>    --project=&quot;${PROJECT_ID}&quot;\<br>    --zone=&quot;${ZONE}&quot; \<br>    --machine-type=a3-highgpu-8g \<br>    --provisioning-model=SPOT \<br>    --instance-termination-action=STOP \<br>    --create-disk=boot=yes,device-name=nanochat-disk,size=2000,type=hyperdisk-balanced,image=projects/ml-images/global/images/c0-deeplearning-common-cu113-v20241118-debian-11</pre><p>Let’s break down the key flags:</p><ul><li><strong>— machine-type:</strong> Selects one of the most powerful publicly available machine types for AI training.</li><li><strong>— accelerator-type:</strong> Explicitly requests 8 NVIDIA H100 80GB GPUs.</li><li><strong>— provisioning-model:</strong> This is the crucial flag that makes this affordable.</li><li><strong>— image-family:</strong> We’re using a Debian 11-based Deep Learning image. This is a huge time-saver, as it comes pre-installed with the correct NVIDIA drivers, CUDA toolkit, and common ML frameworks.</li></ul><p>Once you run this, Google Cloud will find available capacity and provision your instance. This might take a few minutes or there is a chance that it might not be able to find capacity and return an error indicating that. In that case, you can either wait a bit and try again, or look for another zone with A3 capacity to try this in, although note pricing can vary from zone to zone. Once it’s ready, you can SSH into it and start setting up.</p><h4>Initial Setup</h4><p>For the initial setup we first need to SSH into the instance using the command below:</p><pre>gcloud compute ssh &quot;${INSTANCE_NAME}&quot; --project=&quot;${PROJECT_ID}&quot; --zone=&quot;${ZONE}&quot;</pre><p>Next make sure that the Python environment manager uv is installed:</p><pre>pip install uv</pre><p>And finally, we just need to clone the nanochat repository onto the spot instance:</p><pre>git clone https://github.com/karpathy/nanochat.git</pre><h3>Training nanochat</h3><p>The nanochat repository comes with a fantastic speedrun.sh script that automates the entire training pipeline. We’ll follow its lead to understand the journey from raw text to a conversational model. Let’s get it started with training, to do this navigate into the nanochat repository, then, run the <a href="http://speedrun.sh/">speedrun.sh</a> script. The script will first handle the environment setup, including installing the Rust compiler (for the tokenizer) and Python dependencies using uv. The commands should look like this:</p><pre>cd nanochat<br>bash speedrun.sh</pre><p>Now what is that script doing? Well it can be broken down into three different parts of it’s execution, each one executes after the other.</p><ol><li>Tokenizer Training — text is converted to numbers for the model to understand, using a custom tokenizer for efficiency.</li><li>Base Model Pre-training — this phase involves downloading a large dataset and pre-training the model to learn language patterns, grammar, and facts by predicting the next word.</li><li>Supervised Finetuning (SFT) — SFT teaches the pre-trained model conversational skills using a dataset of high-quality conversations, transforming it into a conversational agent.</li></ol><p>So what do we do once this script has finished running after a couple(about 4ish) of hours?</p><h3>Working with Nanochat</h3><p>After the speedrun.sh script completes (which will take about 4 hours), you will have your very own trained nanochat model! Luckily the Nanochat repo provides a command that we can run to standup up chat server we can use:</p><pre>uv run python -m scripts.chat_web</pre><p>You can access it by navigating to your VM’s public IP address at port 8000 (e.g., <a href="http://YOUR_VM_IP:8000).">http://YOUR_VM_IP:8000).</a> You’ll be greeted by a clean, familiar chat interface. Ask it questions, have it write a poem, or test its knowledge. See what the limits and capabilities of this model are!</p><p>Note though that you’re still running on the A3 spot VM so you will probably want to copy the model to your own local machine for further testing later. Once done here don’t forget to delete your spot instance and <a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects?utm_campaign=CDR_0x586bd5d5_default_b454357970&amp;utm_medium=external&amp;utm_source=blog">cleanup/delete your project</a>!</p><h3>So What’s Next?</h3><p>You’ve just completed creating your own model from scratch for just under $100 on Google Cloud. You’ve provisioned the hardware on a budget and run a full-scale training workload. You now possess a model that has been trained entirely from scratch.</p><p>What are some thighs you could try with this model! Start digging into the code of Nanochat and seeing how it’s composed/put together. Read through the <a href="https://github.com/karpathy/nanochat/blob/master/README.md#bigger-models">README</a> and think about some of the suggestions on how to scale the model to achieve different benchmarks! Maybe think about how might this be able to train on things other than maybe even GPUs perhaps! There’s quite a bit of learning and exploration one can do to learn about large language models after training Nanochat once!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9eae1710d9a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/training-nanochat-on-a-a3-spot-vm-9eae1710d9a0">Training Nanochat on a A3 Spot VM</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own AI Hypercomputer: Training Qwen2 on a Slurm Cluster]]></title>
            <link>https://medium.com/google-cloud/build-your-own-ai-hypercomputer-training-qwen2-on-a-slurm-cluster-6c8361196b30?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c8361196b30</guid>
            <category><![CDATA[slurm]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[qwen-2]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 19:02:15 GMT</pubDate>
            <atom:updated>2025-10-14T02:25:26.538Z</atom:updated>
            <content:encoded><![CDATA[<p>So, you’ve been hearing all about the power of large language models (LLMs) and you’re ready to get your hands dirty. You want to move beyond just using pre-trained models and start training one yourself. You’ve heard that for the big stuff, you need a high-performance computing (HPC) cluster, but setting one up sounds incredibly complex and, frankly, expensive. You’ve looked at a few guides, but they seem to assume you’re already a systems administrator.</p><p>Does this sound familiar? It’s a common hurdle for many of us who are diving deeper into the world of AI. How can you get access to the kind of multi-node, multi-GPU power you need to train a model like Qwen2 without getting lost in a sea of configuration files and command-line arguments?</p><p>That’s exactly what we’re going to tackle today. We’ll walk through how to set up your very own production-grade, high-performance Slurm cluster on Google Cloud and use it to train a 1.5 billion parameter Qwen2 model. We’re using Qwen2 as our initial base to zero out since it’s an open source model and has a version small enough for us to leverage. And we’ll do it using tools that make the process surprisingly manageable. This blog post is based off of a <a href="https://cloud.google.com/ai-hypercomputer/docs/tutorials/train-qwen2-a4-slurm-cluster">tutorial</a> that we wrote earlier!</p><h3>The Tools of the Trade: Cluster Toolkit and Slurm</h3><p>Before we dive in, let’s get acquainted with the two main tools that will make this all possible: the <a href="https://cloud.google.com/cluster-toolkit/docs/overview">Google Cloud Cluster Toolkit</a> and <a href="https://slurm.schedmd.com/documentation.html">Slurm</a>.</p><h4>Cluster Toolkit: The Architect of Your Cluster</h4><p>How do you take a collection of powerful virtual machines, each with multiple GPUs, and get them to work together as a single, cohesive unit? That’s where the Cluster Toolkit comes in. Think of it as the architect and construction crew for your HPC cluster. You provide it with a blueprint — a configuration file that describes exactly what you want to build — and it handles the rest. It provisions the virtual machines, configures the high-speed networking they need to communicate effectively, sets up shared storage, and gets everything ready for you to start your work. Without it, you’d be stuck manually configuring every piece of the puzzle. The Cluster Toolkit automates the heavy lifting of infrastructure setup.</p><h4>Slurm: The Conductor of Your Orchestra</h4><p>Once you have your cluster built, you need a way to manage the work it’s doing. If you have a massive training job, how do you distribute it across all of those GPUs? How do you queue up multiple jobs and make sure they run efficiently? That’s the job of the Slurm Workload Manager. Slurm acts as the conductor for your cluster. You give it a job (in the form of a script), and Slurm figures out the best way to allocate resources, schedule the work, and run your script across the nodes of your cluster. It’s the brains of the operation, ensuring that your powerful hardware is being used effectively.</p><h3>Before You Begin: Laying the Foundation</h3><p>Like any good construction project, we need to do a bit of prep work before we start building. This involves setting up your Google Cloud project and getting the necessary permissions in place.</p><p>First, you’ll need to have the gcloud command-line interface (CLI) <a href="https://cloud.google.com/sdk/docs/install">installed and configured</a>. If you’re working in your local shell, make sure you’ve installed it, logged in, and initialized it for your project.</p><p>You’ll need to create a new Google Cloud project or select an existing one. If you’re just trying this out, it’s often best to create a new project so you can easily delete all the resources when you’re done. Once you have your project, make sure billing is enabled.</p><h4>Enable the APIs</h4><p>Next, we’ll enable the necessary APIs. These give your project access to the services we’ll be using, like Compute Engine (for the virtual machines), Filestore (for shared storage), and Logging.</p><pre>gcloud services enable compute.googleapis.com file.googleapis.com logging.googleapis.com cloudresourcemmanager.googleapis.com servicenetworking.googleapis.com --project=your-project-id</pre><h4>Grant User Account Permissions</h4><p>These commands grant your user account the necessary permissions to create and manage the resources for your Slurm cluster.</p><pre>gcloud projects add-iam-policy-binding PROJECT_ID --member=&quot;user:USER_IDENTIFIER&quot; --role=roles/compute.admin<br>gcloud projects add-iam-policy-binding PROJECT_ID --member=&quot;user:USER_IDENTIFIER&quot; --role=roles/iam.serviceAccountUser<br>gcloud projects add-iam-policy-binding PROJECT_ID --member=&quot;user:USER_IDENTIFIER&quot; --role=roles/file.editor<br>gcloud projects add-iam-policy-binding PROJECT_ID --member=&quot;user:USER_IDENTIFIER&quot; --role=roles/storage.admin<br>gcloud projects add-iam-policy-binding PROJECT_ID --member=&quot;user:USER_IDENTIFIER&quot; --role=roles/serviceusage.serviceUsageAdmin</pre><p>Replace the following:</p><ul><li>PROJECT_ID: Your project ID.</li><li>USER_IDENTIFIER: The identifier for your user account. For example, myemail@example.com.</li><li>ROLE: The IAM role that you grant to your user account.</li></ul><h4>Enable and Configure the Default Service Account</h4><p>Next, you’ll enable the default service account for your Google Cloud project, which the resources we create will use to interact with other Google Cloud services.</p><pre>gcloud iam service-accounts enable PROJECT_NUMBER-compute@developer.gserviceaccount.com \<br>    --project=PROJECT_ID</pre><p>Replace PROJECT_NUMBER with your project number. To review your project number, see <a href="https://www.google.com/search?q=https://cloud.google.com/resource-manager/docs/creating-managing-projects%23identifying_projects">Get an existing project</a>.</p><p>Grant the Editor role (roles/editor) to the default service account:</p><pre>gcloud projects add-iam-policy-binding PROJECT_ID \<br>    --member=&quot;serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com&quot; \<br>    --role=roles/editor</pre><p><strong>Note:</strong> If you use a local shell and an external identity provider (IdP), and you encounter an authentication error after running the preceding command, then sign in to the gcloud CLI with your federated identity.</p><h4>Enable OS Login</h4><pre>gcloud compute project-info add-metadata --metadata=enable-oslogin=TRUE</pre><h3>Gaining Access to Qwen2</h3><p>To work with the Qwen2 model, you’ll need a <a href="https://huggingface.co/login">Hugging Face</a> account. Once you’re signed in, you’ll need to sign the consent agreement to use the <a href="https://huggingface.co/Qwen/Qwen2.5-1.5B">Qwen2 1.5B model</a>. Then, create a “read” access token and then copy it down somewhere so you don’t forget it. This token will allow your training script to download the model from Hugging Face.</p><h3>Preparing Your Environment</h3><p>Now, let’s get your local environment ready for building the cluster. First, you’ll need to clone the Cluster Toolkit GitHub repository, which contains the scripts and blueprints we’ll be using.</p><pre>git clone https://github.com/GoogleCloudPlatform/cluster-toolkit.git</pre><p>You’ll also need a Cloud Storage bucket. This bucket will be used by the Cluster Toolkit to store its configuration and state.</p><pre>gcloud storage buckets create gs://your-unique-bucket-name --project=your-project-id</pre><h3>Building Your Slurm Cluster</h3><p>With the prep work done, it’s time for the exciting part: building the cluster. We’ll be using a pre-made blueprint from the Cluster Toolkit examples that’s specifically designed for creating a Slurm cluster with A4 VMs, which are optimized for AI workloads.</p><p>Navigate into the cluster-toolkit directory you cloned earlier. If this is your first time using the toolkit, you’ll need to build the gcluster binary. Make sure that you have make and Go installed on your machine before building this binary as those are required dependencies.</p><pre>cd cluster-toolkit<br>make</pre><p>Now, go to the directory containing the A4 Slurm example:</p><pre>cd examples/machine-learning/a4-highgpu-8g/</pre><p>Inside this directory, you’ll find a file named a4high-slurm-deployment.yaml. This is the blueprint for our cluster. Open it in a text editor and you’ll see a section for vars. This is where you’ll customize the blueprint for your project. You’ll need to provide your bucket name, project ID, the region and zone where you want to build your cluster, and the details of your A4 VM reservation.</p><p>Once you’ve edited the blueprint, you’re ready to deploy the cluster.</p><pre>./gcluster deploy -d examples/machine-learning/a4-highgpu-8g/a4high-slurm-deployment.yaml examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml --auto-approve</pre><p>This command kicks off a two-phase process. First, it builds a custom virtual machine image with all the necessary software pre-installed. This can take a little while, so be patient. Once the image is built, the second phase begins, which uses that image to deploy the cluster. This part is much faster.</p><h3>Preparing the Training Workload</h3><p>While the cluster is deploying, let’s prepare the scripts we’ll need for our training job. We’ll need a few different files to set up the environment, preprocess the data, and run the training. You can copy over the samples from the <a href="https://cloud.google.com/ai-hypercomputer/docs/tutorials/train-qwen2-a4-slurm-cluster#create-scripts">tutorial</a> to get this working, for now we’ll just briefly describe each file you create at what it does.</p><p>First, create an install_environment.sh script. This script will set up a Python virtual environment on the cluster and install all the necessary libraries, including PyTorch and the Hugging Face libraries.</p><p>Next, create an accelerate_config.yaml file. This file configures the Hugging Face Accelerate library, which makes it easy to run training jobs across multiple GPUs and multiple nodes.</p><p>The heart of our job submission is the submit.slurm script. This is the script we’ll give to Slurm to run our training job. It’s a shell script with some special #SBATCH directives at the top that tell Slurm what resources we need (in this case, 2 nodes, with 8 GPUs each). The script has three main stages:</p><ol><li><strong>Environment Setup and Data Pre-processing:</strong> This stage runs once on each node. It copies the Python environment to the node’s local SSD for faster access and then runs a Python script to download and preprocess the training data.</li><li><strong>Training:</strong> This is the main event. It uses srun to launch the training script on all 16 GPUs across the two nodes.</li><li><strong>Copying the Final Model:</strong> Once training is complete, this stage copies the trained model from the local SSD of one of the nodes back to the shared home directory, so it’s safely stored.</li></ol><p>You’ll also need a requirements.txt file that lists all the Python libraries our job depends on.</p><p>The preprocess_data.py script is responsible for downloading the dataset from Hugging Face, tokenizing it, and saving it in a format that’s ready for training.</p><p>Finally, the train.py script contains the code for the actual training process. It initializes a new Qwen2 model from a configuration, loads the preprocessed data, sets up the training arguments, and then uses the Hugging Face Trainer to run the training loop.</p><p>Once you’ve created all these files, you’ll need to upload them to the login node of your Slurm cluster. You can find the name of your login node by listing the A4 VMs in your project. Then, use gcloud compute scp to copy the files to the login node.</p><h3>Running and Monitoring Your Training Job</h3><p>Now that your cluster is up and your scripts are in place, it’s time to kick off the training. First, connect to the login node of your Slurm cluster using SSH.</p><p>Once you’re on the login node, set up an environment variable for your Hugging Face access token. Then, run the install_environment.sh script to set up the Python environment.</p><p>With the environment ready, you can submit your training job to Slurm:</p><pre>sbatch submit.slurm</pre><p>You can monitor the progress of your job by tailing the output files that are created in the logs directory on the login node.</p><p>You can also monitor the GPU utilization in the Google Cloud console to make sure your job is running efficiently. For a healthy training job, you should see the utilization of all 16 GPUs rise and stay high throughout the training process. The entire job should take about an hour to complete.</p><h3>Your Hard-Earned Model</h3><p>After about an hour, your job will complete, and you’ll have a fully trained model waiting for you. It’s saved in the ~/qwen2-from-scratch-on-smollm-fineweb/ directory on the login node. Because this is a persistent shared directory mounted across all nodes, your model checkpoints are safe and sound, even after the job completes and the compute nodes are deallocated.</p><p>You can download the model from the login node to your local machine. From your local machine, run the following command:</p><pre># From your local machine<br>LOGIN_NODE_NAME=&quot;your-login-node-name&quot; # e.g., a4-high-login-001<br>PROJECT_ID=&quot;your-project-id&quot;<br>ZONE=&quot;your-cluster-zone&quot; # e.g., us-west4-a<br><br>gcloud compute scp --project=&quot;$PROJECT_ID&quot; --zone=&quot;$ZONE&quot; --tunnel-through-iap \<br>  &quot;${LOGIN_NODE_NAME}&quot;:~/qwen2-from-scratch-on-smollm-fineweb/ ./qwen2-trained-model/ --recurse</pre><p>With your model downloaded, a world of possibilities opens up. You can:</p><ul><li><strong>Load it for inference:</strong> Use the Hugging Face Transformers library to load your newly trained model and see what it can do.</li><li><strong>Continue fine-tuning:</strong> Use the saved checkpoint as a starting point for more specialized training on a different dataset.</li><li><strong>Share it with the world:</strong> Push your model to the Hugging Face Hub to share your work with the community.</li></ul><h3>Cleaning Up Your Workshop</h3><p>This is an important step! The resources we’ve used — powerful VMs with multiple GPUs — can be expensive. To avoid any unexpected charges to your Google Cloud account, it’s crucial to clean up when you’re done.</p><p>If you created a new project just for this tutorial, the easiest way to clean up is to delete the entire project.</p><pre>gcloud projects delete YOUR_PROJECT_ID</pre><p>If you used an existing project, you’ll want to delete the Slurm cluster itself. To do this, navigate back to the cluster-toolkit directory on your local machine and run the gcluster destroy command.</p><pre>cd cluster-toolkit<br>./gcluster destroy a4-high --auto-approve</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c8361196b30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/build-your-own-ai-hypercomputer-training-qwen2-on-a-slurm-cluster-6c8361196b30">Build Your Own AI Hypercomputer: Training Qwen2 on a Slurm Cluster</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your AI Hypercomputer: Slurm on ARM]]></title>
            <link>https://medium.com/google-cloud/build-your-ai-hypercomputer-slurm-on-arm-1d2a728aaed1?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d2a728aaed1</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Sat, 31 May 2025 00:11:34 GMT</pubDate>
            <atom:updated>2025-06-03T00:03:55.941Z</atom:updated>
            <content:encoded><![CDATA[<p>You’ve done some research on the AI Hypercomputer. You’ve realized that it’s something that you might be able to make on your own. You’ve learned that you can do training on an HPC cluster, but you don’t really have the know-how on how to set one up and use one. You’ve tried to go through a tutorial on setting one up but ran into issues because it seemed like it would be way too expensive to try and set one up yourself.</p><p>Does this situation sound familiar to you at all? It’s one thing that I ran into when first trying to learn how to create my own AI Hypercomputer. Especially when it came to trying to set up my own High Performance Compute(HPC) cluster and wanting to keep it affordable. So, I wondered how I might be able to explore setting up a HPC cluster, while keeping it affordable? That’s where ARM compute instances enter the picture, but before delving into creating my own I had to develop some background knowledge first.</p><h3>Creating an ARM Cluster: The Tools</h3><p>For creating a HPC cluster that runs with ARM compute instances there are a couple of tools that we need to get familiar with. One tool, for managing the cluster and scheduling jobs to be run, called Slurm. And another tool for setting up the cluster and maintaining the infrastructure behind it, known as Cluster Toolkit. With the combination of these two tools we should be able to set up our own ARM based cluster to try things on.</p><h4>Cluster Toolkit: The Groundwork</h4><p>So, how do we create and manage the infrastructure that the cluster is run on? That’s where Cluster Toolkit comes into play. It’s a toolkit you can use for your HPC cluster to manage and create the resources that it consumes. You know you will ARM compute instances that can talk to each other, share resources, and to act like one cohesive unit for performing tasks. You also might need some storage as well. Cluster Toolkit helps you create all of that. It’s the tool that takes your individual ARM instances and helps you configure them, connect them, and essentially get them all lined up and ready to work as a team. Without this, you’d be manually trying to piece together a complex puzzle, and who has time for that!? Cluster Toolkit aims to simplify that initial, often daunting, setup process, letting you focus more on the ‘what’ and less on the ‘how’ of your cluster’s physical (or virtual) assembly.</p><h4>Slurm: The Scheduler</h4><p>Now that you’ve got your cluster assembled, thanks to something like Cluster Toolkit, how do you actually tell it what task that you want it to do? You have several tasks that you want to run, and a bunch of instances ready to go, but how do you know where to run the tasks? That’s the problem Slurm solves. Slurm, or Slurm Workload Manager, acts as the conductor or scheduler for your cluster. You tell Slurm, “Hey, I have these jobs I need to run,” and Slurm figures out the best way to distribute those jobs across your available compute instances. It queues them up, allocates resources, and makes sure everything runs as smoothly as possible. It’s the brain that ensures your ARM cluster isn’t just sitting there, but is actually crunching numbers and or reading files, making sure things actually get done. Getting an understanding of how Slurm works and how to work with it ensures that you are able to effectively work with the ARM cluster that you have created.</p><h3><strong>The Blueprint: A Guide for the Cluster</strong></h3><p>Think of the blueprint file (the yaml file, like the one we have here) as the detailed architectural plan for our HPC cluster. It tells Cluster Toolkit exactly what to build, how to configure it, and what components it needs. It might look a bit technical at first glance, but let’s break it down section by section. Our example blueprint here is named “arm-cluster”, a straightforward name for our ARM-based setup.</p><h4>Vars: Configuration Definitions</h4><pre>blueprint_name: arm-cluster<br><br>vars:<br>  project_id:  your-project-id<br>  deployment_name: arm-inference-cluster<br>  region: us-west1<br>  zone:  us-west1-a<br>  zones:<br>  - us-west1-a<br>  new_image:<br>    family: ml-slurm<br>    project: $(vars.project_id)<br>  disk_size_gb: 200<br>  metadata:<br>    VmDnsSetting: GlobalOnly</pre><p>Above, we have the vars section, which allows for us to define things ahead of time. These are things that we might use more than once throughout the project such as our project name. Instead of typing out your-project-id (your Google Cloud Project ID for this cluster) every single time we need it, we just define project_id once and then can refer to it as $(vars.project_id). We do the same for our deployment_name (what we’ll call our cluster once it’s built), the region and zone where it’ll live in the cloud, the details for a new_image we’re going to create, disk_size_gb for our virtual machines, and some specific metadata. This keeps things tidy and makes it much easier to change common values later if we need to!</p><h4>Primary Group: Shared Resources</h4><pre>deployment_groups:<br>- group: primary<br>  modules:<br>  - id: network<br>    source: modules/network/vpc<br><br>  - id: private_service_access<br>    source: community/modules/network/private-service-access<br>    use: [network]<br><br>  - id: homefs<br>    source: modules/file-system/filestore<br>    use: [network, private_service_access]<br>    settings:<br>      local_mount: /home<br>      size_gb: 2560<br>      filestore_tier: BASIC_SSD<br><br>  - id: script<br>    source: modules/scripts/startup-script<br>    settings:<br>      runners:<br>      - type: shell<br>        destination: install-ml-libraries.sh<br>        content: |<br>          #!/bin/bash<br>          # this script is designed to execute on Slurm images published by SchedMD that:<br>          # - are based on Debian distribution of Linux<br>          # - have NVIDIA drivers pre-installed<br><br>          set -e -o pipefail</pre><p>The primary group is all about setting up the essential share resources:</p><ol><li><strong>network</strong>: This module, sourced from modules/network/vpc, sets up our Virtual Private Cloud (VPC). The private, isolated network in the cloud where our cluster components can securely talk to each other.</li><li><strong>private_service_access</strong>: This module helps our private network securely connect to other Google Cloud services, like Filestore, without needing public IP addresses. It uses the network we just defined, building upon an already existing resource.</li><li><strong>homefs</strong>: This sets up a shared file system using Google Cloud Filestore. We’re calling it homefs because it will be mounted as /home on all our cluster nodes. This means you can have your files and scripts in one place, and every machine in the cluster can access them. We’re allocating 2560 GB (or 2.5 TB) of BASIC_SSD storage for this. It uses both the network and private_service_access modules.</li><li><strong>script</strong>: This module defines a startup script. The content here is a basic shell script named install-ml-libraries.sh. While the current script content is just a placeholder comment block, this is where you’d typically put commands to install common libraries or tools (like PyTorch, TensorFlow, etc.) that you want on your compute nodes or custom image. We’ll see this script being used later.</li></ol><h4>Packer Node: Creating our Base Image</h4><pre>- group: packer<br>  modules:<br>  - id: custom-image<br>    source: modules/packer/custom-image<br>    kind: packer<br>    use:<br>    - network<br>    - script<br>    settings:<br>      omit_external_ip: false<br>      source_image_family: &quot;slurm-gcp-6-3-ubuntu-2204-lts-arm64&quot;  <br>      source_image_project_id: [&quot;schedmd-slurm-public&quot;]<br>      disk_size: $(vars.disk_size_gb)<br>      disk_type: hyperdisk-balanced<br>      image_family: $(vars.new_image.family)<br>      machine_type: c4a-highmem-1<br>      state_timeout: 20m</pre><p>The packer group focuses on creating a custom virtual machine image. Why do we do this? Well, instead of configuring every single compute node from scratch each time it starts, we can create a “golden image” that already has much of our needed software and configuration built in. This speeds up cluster creation and ensures consistency.</p><p>The <strong>custom-image</strong> module uses Packer (a tool for creating machine images) to do its job.</p><ul><li>It uses the network (so Packer can build the image within our VPC) and our script (the install-ml-libraries.sh we defined earlier would be run during the image creation process if it had installation commands).</li><li>omit_external_ip: false gives the temporary VM used for building the image a public IP, which can be useful if it needs to download software from the internet.</li><li>The source_image_family points to an existing ARM64 Ubuntu image optimized for Slurm provided by SchedMD (slurm-gcp-6–3-ubuntu-2204-lts-arm64 from the schedmd-slurm-public project). This is our base.</li><li>We then customize it, specifying disk_size, disk_type, and the image_family name for our new custom image (which we defined in vars as ml-slurm).</li><li>machine_type: c4a-highmem-1 specifies an ARM-based machine type for building the image. It’s a cost-effective choice as image building itself isn’t usually a super intensive task.</li><li>state_timeout: 20m is just how long Packer will wait for certain operations.</li></ul><p>So, this just takes a base ARM image, potentially runs our setup script on it, and saves the result as a new custom image named ml-slurm within our project.The packer group creates a custom virtual machine image to avoid configuring each compute node individually at startup. This “golden image” contains pre-installed software and configurations, speeding up cluster creation and ensuring consistency.</p><h4>Cluster Group: Creating What we Wanted</h4><pre>- group: cluster<br>  modules:<br>  - id: examples<br>    source: modules/scripts/startup-script<br>    settings:<br>      runners:<br>      - type: data<br>        destination: /var/tmp/torch_test.sh<br>        content: |<br>          #!/bin/bash<br>          source /etc/profile.d/conda.sh<br>          conda activate pytorch<br>          python3 torch_test.py<br>      - type: data<br>        destination: /var/tmp/torch_test.py<br>        content: |<br>          import torch<br>          # ... (rest of the python script) ...<br><br>  - id: c4a_nodeset<br>    source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset<br>    use: [network]<br>    settings:<br>      node_count_dynamic_max: 20<br>      bandwidth_tier: gvnic_enabled<br>      machine_type: c4a-highmem-1<br>      disk_size_gb: $(vars.disk_size_gb)<br>      disk_type: hyperdisk-balanced<br>      instance_image: $(vars.new_image)<br>      instance_image_custom: true<br><br>  - id: c4a_partition<br>    source: community/modules/compute/schedmd-slurm-gcp-v6-partition<br>    use: [c4a_nodeset]<br>    settings:<br>      partition_name: a2<br>      is_default: true<br><br>  - id: slurm_controller<br>    source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller<br>    use:<br>    - network<br>    - c4a_partition<br>    - homefs<br>    settings:<br>      enable_controller_public_ips: true<br>      instance_image: $(vars.new_image)<br>      instance_image_custom: true<br>      machine_type: &quot;c4a-highmem-1&quot;<br>      login_startup_script: $(examples.startup_script)<br>      disk_size_gb: $(vars.disk_size_gb)<br>      disk_type: hyperdisk-balanced<br>      controller_state_disk: null</pre><p>This is the final group where we actually define the Slurm cluster itself.</p><p><strong>examples</strong>: This startup-script module is a bit different from the one in the primary group. Here, it’s not about building an image but about placing some example files on our Slurm controller’s login node.</p><ul><li>It creates two files in /var/tmp/: torch_test.sh and torch_test.py</li><li>Notice the type: data. This means the content is written directly into the destination file.</li></ul><p><strong>c4a_nodeset</strong>: This module defines a “nodeset” for Slurm. A nodeset is a group of identical compute nodes.</p><ul><li>It uses network which we defined above as our VPC.</li><li>node_count_dynamic_max: 20 means we can scale up to 20 of these nodes. Slurm can dynamically add or remove nodes based on workload, which is very cool for saving costs!</li><li>machine_type: c4a-highmem-1 specifies the ARM-based machine type for our compute nodes. These are general-purpose ARM instances.</li><li>instance_image: $(vars.new_image) and instance_image_custom: true tell Slurm to use the custom ml-slurm image we built in the packer group for these compute nodes.</li><li>Other settings define disk size, type, and enable Google Virtual NIC (gvnic_enabled) for better network performance.</li></ul><p><strong>c4a_partition</strong>: In Slurm, a partition is a logical grouping of nodes. You submit jobs to a specific partition.</p><ul><li>This module uses the c4a_nodeset we just defined, meaning this partition will consist of those c4a-highmem-1 ARM nodes.</li><li>We’re naming this partition a2 and making it the is_default: true partition, so if you don’t specify a partition when submitting a job, it’ll go here.</li></ul><p><strong>slurm_controller</strong>: This is what controls our Slurm cluster and where we login to as well.</p><ul><li>It uses the network, our c4a_partition (so it knows about the compute nodes it can manage), and homefs (so the controller also has access to the shared /home directory).</li><li>enable_controller_public_ips: true gives the controller a public IP address so we can SSH into it from outside our private network. This is essential for managing the cluster and submitting jobs.</li><li>It also uses our $(vars.new_image) custom image for the controller itself.</li><li>machine_type: “c4a-highmem-1” is also used for the controller. For many workloads, the controller doesn’t need to be massively powerful, but it should be reliable.</li><li>login_startup_script: $(examples.startup_script) means that the torch_test.sh and torch_test.py files we defined in the examples module will be placed on the controller when it starts up. This is super convenient for having test scripts ready to go!</li></ul><p>And that’s it! When Cluster Toolkit processes this blueprint, it will create all these resources in the specified order, linking them together as defined. You’ll end up with a fully functional Slurm cluster running on ARM instances, with a shared home directory and a handy PyTorch test script waiting for you on the controller.</p><h3>What’s Next?</h3><p>We’ve now covered the essentials of defining an Arm-based Slurm cluster on Google Cloud using the Cluster Toolkit and dissected its key components. In the next part of this series, we’ll dive into interacting with our newly minted cluster. Covering how to submit your first jobs, and exploring how to set up the cluster in your project as well!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d2a728aaed1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/build-your-ai-hypercomputer-slurm-on-arm-1d2a728aaed1">Build Your AI Hypercomputer: Slurm on ARM</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your AI Hypercomputer: Bringing Your ARM Blueprint to Life]]></title>
            <link>https://medium.com/google-cloud/ai-hypercomputer-bringing-your-arm-blueprint-to-life-2e7e944c99f5?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e7e944c99f5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Sat, 31 May 2025 00:11:14 GMT</pubDate>
            <atom:updated>2025-06-03T00:03:43.278Z</atom:updated>
            <content:encoded><![CDATA[<p>So, we’ve meticulously gone through our arm-cluster blueprint, section by section, to understand what each part creates. We now have a detailed blueprint in our hands describing the ARM cluster that we want to create. Now, how does this plan turn into an actual AI Hypercomputer? That’s precisely where the Cluster Toolkit steps in to help us out!</p><p>Think of Cluster Toolkit as a tool assisting you to create, deploy, and delete the infrastructure that you defined in your blueprint. You provide the detailed YAML blueprint, and Cluster Toolkit begins to work. It will read the blueprint section by section standing up the resources and infrastructure that you specified for the cluster.</p><p>Cluster Toolkit orchestrates all the underlying cloud resources. It makes the necessary API calls, provisions the virtual machines, configures the storage, establishes the networking rules, and ensures all the pieces defined in your blueprint are not just created, but are also connected and configured to interact correctly.</p><h3>Using the Toolkit: Using the Command-Line Interface</h3><p>This blueprint, comprehensive as it is, requires a mechanism to initiate the building process. This is accomplished through the Cluster Toolkit’s command-line interface (CLI). On Google Cloud, this is typically the ghpc (Google Cloud HPC Toolkit) command. Generally to install this, you need to clone the Github repository, build it locally, and add it to your system path. Then, once your blueprint file (e.g., arm-cluster.yaml) is prepared, interacting with the CLI generally follows a clear sequence:</p><h4>1. Blueprint Processing and Deployment Preparation</h4><p>The first step is to instruct Cluster Toolkit to process your blueprint and prepare for deployment. This is often done with a command like:</p><pre>ghpc create arm-cluster.yaml</pre><p>This command doesn’t immediately build the entire cluster. Instead, it validates your blueprint and generates a deployment directory. This directory contains the necessary configurations and scripts, often utilizing Terraform, which Cluster Toolkit uses to materialize your infrastructure.</p><h4>2. Deploying the Infrastructure</h4><p>With the deployment directory created, you navigate into the directory that was created. From there, you issue the command to begin the actual construction of the cluster:</p><pre>cd arm-cluster-deployment # The name of your deployment directory<br>ghpc deploy</pre><p>At this stage, Cluster Toolkit provisions all the resources defined in the blueprint — virtual machines, networking, storage, and the Slurm components. The terminal will display the progress of these operations. This process can take some time, especially for new or complex cluster configurations.</p><h4>3. Deleting the Cluster</h4><p>When the cluster is no longer needed, it’s important to release the resources to avoid ongoing charges. From within the same deployment directory, Cluster Toolkit provides a command to dismantle the cluster:</p><pre>ghpc delete</pre><p>This command systematically removes all the resources that were provisioned, ensuring a clean and complete teardown.</p><p>In essence, Cluster Toolkit, guided by your blueprint and operated via its CLI, transforms your blueprint into a functional Slurm cluster on ARM. It automates the intricate processes of infrastructure deployment and management, allowing you to concentrate on utilizing the cluster for your workloads.</p><h3>Slurm: Running Your First Arm Job</h3><p>You now understand the blueprint and have used Cluster Toolkit to construct your ARM-based Slurm cluster. The infrastructure is in place, the nodes are ready, and the scheduler, Slurm, is waiting for tasks to perform. Now, it’s time to put it to work and run your first job on this new ARM-powered high-performance computing environment.</p><h4>Connecting to Your Cluster</h4><p>The first step is to log into the Slurm controller node. In our blueprint, we configured the slurm_controller to have a public IP address, making it accessible. You’ll use SSH to connect. You’ll need the public IP address of your controller node (which you can find in the Google Cloud Console or through ghpc output if it provides it) and the SSH key you configured for access.</p><pre> ssh -i /path/to/your/ssh-private-key your-username@CONTROLLER_PUBLIC_IP_ADDRESS</pre><p>Replace /path/to/your/ssh-private-key with the path to your private SSH key and CONTROLLER_PUBLIC_IP_ADDRESS with the actual IP address. The your-username would typically be the local username from which the SSH key was generated or a common username on the image. Alternatively you can find the controller node listed under the instances from the Google Cloud Console and SSH from there directly.</p><p>Once connected, you’ll be at the command prompt of your Slurm controller. This is where you’ll interact with Slurm to submit and manage your jobs.</p><h4>A Quick Primer on Slurm Concepts</h4><p>Before submitting a job, let’s briefly touch upon a few key Slurm terms:</p><ul><li><strong>Job:</strong> A unit of work submitted to the cluster, typically a script that executes your program.</li><li><strong>Partition:</strong> A logical grouping of compute nodes. Jobs are submitted to a partition. In our blueprint, we created a default partition named a2 consisting of our ARM-based c4a-highmem-1 nodes.</li><li><strong>Nodes:</strong> The individual compute instances (your ARM virtual machines) where your jobs actually run.</li><li><strong>Job Script:</strong> A shell script that, at a minimum, contains the commands you want to execute on a compute node. It can also include Slurm directives (lines starting with #SBATCH) to request resources like number of nodes, tasks, time limits, etc.</li></ul><h4>Your First Slurm Job: A Simple “Hello ARM”</h4><p>Let’s start with a very basic job to confirm everything is working.</p><p>On the controller node, create a file named hello_arm.sh using a text editor like nano or vim.</p><p>Add the following lines to the script:</p><pre>#!/bin/bash<br>#SBATCH --job-name=hello_arm_job<br>#SBATCH --output=hello_arm_output_%j.txt<br>#SBATCH --nodes=1<br>#SBATCH --ntasks=1<br><br>echo &quot;Hello from an ARM node!&quot;<br>echo &quot;My hostname is: $(hostname)&quot;<br>echo &quot;I am running on architecture: $(uname -m)&quot;</pre><p><strong>Submit the Job:</strong> Use the sbatch command to submit your script to Slurm:</p><pre>sbatch hello_arm.sh</pre><p>Slurm will respond with a message like Submitted batch job JOB_ID, where JOB_ID is a unique number for your job.</p><p><strong>Check Job Status:</strong> You can see the status of your submitted jobs using squeue:</p><pre>squeue -u your-username</pre><p>(Replace your-username with your actual username). You might see your job briefly in PD (pending) state before it moves to R (running). If it’s a very short job, it might complete before you even see it running.</p><p><strong>View the Output:</strong> Once the job is complete, a file named hello_arm_output_JOB_ID.txt will appear in your current directory on the controller (because we launched the job from there and the output path was relative). View its contents:</p><pre>cat hello_arm_output_JOB_ID.txt</pre><p>You should see something like:</p><pre>Hello from an ARM node!<br>My hostname is: c4a-nodeset-1-1  # Or similar<br>I am running on architecture: aarch64</pre><p>The aarch64 confirms your job ran on an ARM compute node!</p><h4>Some Essential Slurm Commands</h4><p>Here are a few fundamental Slurm commands you’ll find useful:</p><ul><li>sbatch &lt;job_script&gt;: Submits a job script to the queue.</li><li>squeue: Displays the status of jobs in the queue.</li><li>squeue -u &lt;username&gt;: Shows jobs for a specific user.</li><li>scancel &lt;job_id&gt;: Cancels a running or pending job.</li><li>sinfo: Displays information about Slurm nodes and partitions.</li><li>scontrol show job &lt;job_id&gt;: Shows detailed information about a specific job.</li><li>sacct -j &lt;job_id&gt;: Displays accounting information for a completed job.</li></ul><h4><strong>A Note on the ARM Architecture</strong></h4><p>Remember, your compute nodes are ARM-based. This means any software or libraries you intend to use must be compiled for the ARM64 (aarch64) architecture. Many common data science and AI libraries like PyTorch, TensorFlow, and NumPy now offer ARM64-compatible versions. When setting up your custom image or environment, always ensure you’re getting the ARM versions of your dependencies.</p><h3>What’s Next?</h3><p>We’ve now covered the essentials of creating an Arm-based Slurm cluster on Google Cloud using the Cluster Toolkit and running a simple job on it. Next, we’ll dive into seeing what running a fine-tuning job looks like on our newly created cluster. Is this something even worth trying? Something that we’ll have to test out!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e7e944c99f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/ai-hypercomputer-bringing-your-arm-blueprint-to-life-2e7e944c99f5">Build Your AI Hypercomputer: Bringing Your ARM Blueprint to Life</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PyTorch Models on Cloud TPUs: An Introduction]]></title>
            <link>https://medium.com/google-cloud/pytorch-models-on-cloud-tpus-an-introduction-bc83f211933c?source=rss-2c470c7c96a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc83f211933c</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[tpu]]></category>
            <dc:creator><![CDATA[Duncan Campbell]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 00:01:40 GMT</pubDate>
            <atom:updated>2024-07-19T07:05:25.831Z</atom:updated>
            <content:encoded><![CDATA[<p>So, maybe you’re a data scientist that has been prototyping a model using PyTorch on your local machine looking for flexible ways to accelerate training. Or you are an engineer who has been handed a PyTorch model by your data science team, looking for a way to run the model that they just created other than on just GPUs. In both these cases, or many others, you are developing or deploying a model using PyTorch and are looking for a flexible way to allow your model to run on different forms of machine learning accelerators.</p><p>One of these machine learning accelerators that you may have heard of are Cloud TPUs, the latest version of which <a href="https://cloud.google.com/blog/products/compute/introducing-trillium-6th-gen-tpus">you heard was announced</a>. You also may have learned through some research that in order for your PyTorch models to run on Cloud TPUs you need to use the PyTorch/XLA library. But what exactly is that?! How does it help me out or even work?</p><h3>What are TPUs?</h3><p>TPUs (Tensor Processing Units) are specialized hardware accelerators developed by Google, designed to significantly boost the speed and efficiency of machine learning workloads. They are custom-built chips optimized for the kind of matrix and vector operations that form the backbone of neural networks and other AI models. They are specifically optimized for linear algebra based operations, such as matrix multiplications, and others in the machine learning domain. One thing to note though is that if you are trying to do any custom operations, then there is a chance that you may not be speeding up your workload by running on TPUs in some cases such as:</p><ul><li>Models with custom PyTorch operations in the main training loop.</li><li>Workloads that access memory in a sparse manner.</li><li>Workloads that require require high-precision arithmetic.</li></ul><p>We will not be focusing on that aspect though as a part of our series as that can be considered a more advanced use case and will assume users are using the standard PyTorch libraries without anything custom.</p><h3>OpenXLA Demystified</h3><h4>What is OpenXLA?</h4><p>OpenXLA, which is separate from XLA, is an open-source machine learning compiler ecosystem developed by companies like Google, AMD, Arm, Intel, and more. It aims to simplify ML development by providing a unified compiler API and pluggable device-specific backends, making it easier to compile and optimize ML models for various hardware platforms. It provides added flexibility and simplification to aid developers when it comes to training and running their models.</p><h4>How Does it Work?</h4><p>The process for getting a model to run on various types of platforms is a tricky one. OpenXLA does this by introducing several tools for us: StableHLO, XLA, and PJRT. You can look at the OpenXLA reference at the end to learn more about them in detail but at a high level this is how they each work:</p><ul><li>Stable High-Level Operations (HLO) provides a common high-level instruction set that each of the machine learning frameworks can write their models out into, acting as the portability layer between the frameworks and the compilers.</li><li>Accelerated Linear Algebra (XLA) is the one example of an open source machine learning compiler that takes the StableHLO that is output by an ML framework and then optimizes for execution across the various different hardware platforms. For TPUs it outputs Low-Level Optimizations (LLO) which can then be executed.</li><li>Pretty Much Just Another Runtime (PJRT) acts as a hardware and framework independent interface or a generic set of application programming interfaces. It supports operations that involve the targeted device, such as compilation, execution, or transferring the model/data.</li></ul><h4>Connecting the Dots</h4><p>PyTorch/XLA introduces a new concept called an XLA Tensor, analogous to CPU or CUDA tensors but tailored for XLA-compatible devices. Unlike its counterparts, XLA Tensors are “lazy.” They record operations, building a computation graph as they progress, delaying execution until necessary. This approach allows for potential grouping of multiple operations into a single computation, optimizing performance.</p><p>Eventually, the computation graph is constructed, and the operations are converted into StableHLO. In the case for Cloud TPUs, it’s sent to the XLA compiler, which translates it into LLO, an executable format for TPUs. In order for this to happen the XLA compiler leverages the PJRT interface. PyTorch/XLA takes care of graph construction, device communication, and data synchronization, abstracting away many of the implementation details for developers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*hQ8uiQJLetnbhTjK" /></figure><p>For a concrete example, let’s consider multiplying two matrices together, and how it would go about executing this on an XLA device. First the device needs to get acquired and then the two tensors we are multiplying are transferred to the memory of the device. Next, the graph is being constructed but not executed, so the matrix multiplication gets recorded. Once we reach a point for execution, the graph is then optimized, followed by execution on the device. Finally the results are transferred back from the XLA device to the original machine.</p><h3>Where to Next?</h3><p>In our next post, we will show you how to migrate a simple existing PyTorch model to run on TPUs using PyTorch/XLA. Meanwhile, check out the following resources:</p><ul><li><a href="https://github.com/pytorch/xla">Enabling PyTorch on XLA Devices (e.g. Google TPU)</a></li><li><a href="https://cloud.google.com/tpu/docs/setup-gcp-account">Set up the Cloud TPU environment</a></li><li><a href="https://cloud.google.com/tpu/docs/run-calculation-pytorch">Run a calculation on a Cloud TPU VM using PyTorch</a></li><li><a href="https://openxla.org/">OpenXLA Project</a></li><li><a href="https://pytorch.org/blog/understanding-lazytensor-system-performance-with-pytorch-xla-on-cloud-tpu/">Understanding LazyTensor System Performance with PyTorch/XLA on Cloud TPU</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc83f211933c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/pytorch-models-on-cloud-tpus-an-introduction-bc83f211933c">PyTorch Models on Cloud TPUs: An Introduction</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>