“Inside the Silicon Stampede: How GPUs Turned Supercomputing Into a Sprint”

GPUs and HPC: Inside the Machines That Run Modern Science

There’s a reason “GPU” and “supercomputer” have become almost synonymous over the last decade. Climate models, protein-folding pipelines, particle physics simulations, and of course large language models all lean on the same underlying idea: break a huge problem into millions of small, identical pieces of arithmetic and run them all at once. That’s exactly what a GPU is built to do, and it’s why high-performance computing (HPC) centers have reorganized themselves around GPU-dense hardware.

This post walks through how a GPU differs from a CPU, how an HPC cluster is put together, and the actual commands you’d type to get work running on one — from checking a GPU’s health to submitting a multi-node job through a scheduler.

1. Why GPUs, Not Just Faster CPUs

A CPU has a handful of powerful cores optimized for doing one thing at a time, very fast, with lots of branching logic. A GPU has thousands of small, simple cores optimized for doing the same thing to many pieces of data simultaneously — a pattern called SIMD (Single Instruction, Multiple Data).

CPU                              GPU
 ┌───────────────────┐          ┌───────────────────────────┐
 │  Core   Core       │          │ SM  SM  SM  SM  SM  SM   │
 │ ┌────┐ ┌────┐      │          │ ▓▓  ▓▓  ▓▓  ▓▓  ▓▓  ▓▓   │
 │ │ALU │ │ALU │      │          │ SM  SM  SM  SM  SM  SM   │
 │ │Ctrl│ │Ctrl│      │          │ ▓▓  ▓▓  ▓▓  ▓▓  ▓▓  ▓▓   │
 │ └────┘ └────┘      │          │ SM  SM  SM  SM  SM  SM   │
 │   Big cache, few    │          │ ▓▓  ▓▓  ▓▓  ▓▓  ▓▓  ▓▓   │
 │   cores, low latency│          │ Thousands of small cores  │
 └───────────────────┘          │ organized into SMs         │
                                  │ (Streaming Multiprocessors)│
                                  └───────────────────────────┘

A CPU is a small team of generalists. A GPU is a stadium full of specialists doing the same repetitive task in lockstep. Matrix multiplication, convolution, and n-body simulation all map naturally onto that model — which is why GPUs dominate both AI training and traditional scientific computing.

2. Anatomy of an HPC Cluster

A supercomputer isn’t one giant machine — it’s hundreds or thousands of individual servers (“nodes”) wired together by a very fast network, coordinated by a job scheduler.

┌─────────────────┐
                         │   Login Node     │  ← you SSH here
                         │ (submit jobs)    │
                         └────────┬─────────┘
                                  │
                         ┌────────▼─────────┐
                         │   Scheduler       │  (Slurm / PBS)
                         │  decides who runs │
                         │  where and when   │
                         └────────┬─────────┘
              ┌───────────────────┼───────────────────┐
              │                   │                   │
     ┌────────▼───────┐ ┌────────▼───────┐ ┌────────▼───────┐
     │  Compute Node 1  │ │  Compute Node 2  │ │  Compute Node N  │
     │ ┌────┐ ┌────┐    │ │ ┌────┐ ┌────┐    │ │ ┌────┐ ┌────┐    │
     │ │GPU0│ │GPU1│... │ │ │GPU0│ │GPU1│... │ │ │GPU0│ │GPU1│... │
     │ └────┘ └────┘    │ │ └────┘ └────┘    │ │ └────┘ └────┘    │
     └────────┬───────┘ └────────┬───────┘ └────────┬───────┘
              └───────────── High-speed interconnect ─────────────┘
                       (InfiniBand / NVLink / NVSwitch)
                                  │
                         ┌────────▼─────────┐
                         │  Parallel Storage │  (Lustre / GPFS)
                         └───────────────────┘

Key pieces:

• Login node — where you edit code and submit jobs. You never run heavy computation here.
• Scheduler (Slurm is the most common today) — queues jobs and allocates nodes/GPUs fairly.
• Compute nodes — the actual muscle, usually with 4–8 GPUs each.
• Interconnect — InfiniBand between nodes, NVLink/NVSwitch between GPUs within a node, since GPU-to-GPU bandwidth is often the real bottleneck.
• Parallel filesystem — shared, high-throughput storage every node can read/write concurrently.

3. Checking What You’ve Got: nvidia-smi

Before running anything, it’s standard practice to check GPU status:

nvidia-smi

This prints a table of every GPU on the node — utilization, memory used, temperature, and which processes currently hold onto them:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.90       Driver Version: 550.90    CUDA Version: 12.4        |
|-------------------------------+----------------------+----------------------|
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf   Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  H100-SXM5        On      | 00000000:1A:00.0 Off |                    0 |
| N/A   38C    P0    72W / 700W |   1024MiB / 81920MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+

Useful variants:

# live-updating view, refreshed every second
nvidia-smi -l 1

# just the numbers you care about, in CSV
nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,temperature.gpu --format=csv
# topology: how GPUs are connected to each other
nvidia-smi topo -m

That last command matters more than people expect — it tells you whether GPU-to-GPU traffic goes over fast NVLink or has to cross the slower PCIe bus, which can make or break performance for multi-GPU jobs.

4. Submitting Work: Slurm

Almost every HPC center uses Slurm (Simple Linux Utility for Resource Management) to schedule jobs. A few commands you’ll use constantly:

# see the state of the cluster: node counts, partitions, availability
sinfo

# see your jobs in the queue
squeue -u $USER
# cancel a job
scancel 123456
# ask for an interactive session with 1 GPU for 30 minutes
srun --gres=gpu:1 --time=00:30:00 --pty bash

For anything longer than a quick test, you write a batch script and submit it:

#!/bin/bash
#SBATCH --job-name=train_model
#SBATCH --partition=gpu
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=8
#SBATCH --time=04:00:00
#SBATCH --output=logs/%x_%j.log

module load cuda/12.4 openmpi/4.1
srun python train.py --batch-size 256 --epochs 50

Submit it with:

sbatch train_job.sh

Slurm queues this, finds two free nodes with 4 GPUs each, and launches it — automatically setting up the environment variables your MPI/NCCL code needs to know which rank is running where.

5. Writing and Compiling GPU Code

At the lowest level, GPU code is written in CUDA (for NVIDIA hardware) or portable alternatives like HIP or SYCL. A minimal CUDA program compiles with nvcc, the NVIDIA CUDA compiler:

nvcc -O3 -arch=sm_90 vector_add.cu -o vector_add
./vector_add

-arch=sm_90 tells the compiler to target Hopper-generation GPUs (like the H100); you'd use sm_80 for Ampere (A100), and so on — GPU code is often compiled for a specific architecture to get the best instruction scheduling.

Get ashish chettri’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Most scientists don’t write raw CUDA, though — they use libraries that already speak GPU fluently:

# PyTorch will use CUDA automatically if it's available
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

For multi-node, multi-GPU training or simulation, MPI (Message Passing Interface) handles communication between nodes, often paired with NVIDIA’s NCCL library for fast GPU-to-GPU collectives (all-reduce, broadcast, etc.):

mpirun -np 8 --map-by ppr:4:node python distributed_train.py

6. Watching a Job While It Runs

Once a job is live, a few tools help you see what’s actually happening on the silicon:

# GPU utilization and memory, live
watch -n 1 nvidia-smi

# detailed performance profiling / timeline
nsys profile -o profile_report python train.py
# kernel-level compute profiling
ncu ./vector_add
# cluster-wide GPU health and telemetry (used by admins)
dcgmi diag -r 3

nsys (Nsight Systems) is particularly useful for spotting the classic HPC problem: a GPU sitting idle waiting on data from disk or from another node, which shows up as gaps in the timeline rather than solid blocks of compute.

7. Putting It Together

A typical workflow looks like this:

1. SSH into the login node.
2. Write and test a small version of your code with srun --pty bash on a single GPU.
3. Package the full run into an sbatch script requesting the nodes/GPUs you need.
4. Submit with sbatch, monitor with squeue, check logs as they stream in.
5. Once running, spot-check GPU utilization with nvidia-smi to confirm you're not leaving expensive hardware idle.
6. Profile with nsys if performance looks off, especially checking whether time is going to compute, memory transfer, or network communication.

The pattern is the same whether the job is training a neural network, running a fluid dynamics simulation, or folding proteins — the difference is almost entirely in the code that runs on the GPU, not in how you get access to the GPU in the first place. Once you’re comfortable with nvidia-smi, Slurm, and a profiler, you have the operational vocabulary to work on essentially any modern HPC system.