Total Segmentator
Image segmentation is one of the most fundamental ideas in computer vision. At its core, it’s just about dividing an image into meaningful regions so we can understand what exactly is inside. In medical applications, segmentation forms the backbone of accurate analysis, diagnosis, and decision-making.
Segmentation gives structure to raw CT scans. It pulls out organs, bones, vessels, and soft tissues that doctors need to measure, analyze, or monitor. Without this, medical imaging is just grayscale data. With segmentation, it becomes a map of human anatomy.
And out of all the open-source models I’ve worked with, TotalSegmentator is easily one of the best. It’s clean, reliable, detailed, and covers almost the entire human anatomy; all with a single model.
Let’s start with:
dataset, annotations, design choices, model architecture, and training pipeline.
Dataset
The TotalSegmentator model was trained on a large and diverse set of real-world CT examinations.
- 1,204 CT examinations
- Sourced from routine clinical workflows
- Includes wide variation in scanner types, patient conditions, contrast phases, and body coverage
- Covers head to knee regions across many demographics
This diversity is one of the reasons the model generalizes so well.
Annotations
This is where the method truly stands out.
The authors manually annotated 104 anatomical structures, including:
- 27 organs
- 59 bones
- 10 muscles
- 8 vessels
Very few open-source datasets offer this level of granularity in 3D medical segmentation. This richness directly increases the model’s usefulness for radiology, surgery planning, organ measurement, and research.
Training Strategy
TotalSegmentator builds on the philosophy of nnU-Net, which automatically configures the entire training pipeline based on the dataset.
This includes:
- patch size
- batch size
- learning rate
- normalization strategy
- data augmentation
- loss functions
- network depth and architecture configurations
The point is:
the model configures itself.
You don’t need manual hyperparameter tuning.
how can we achieve that?
Instead of relying on hand-tuned hyperparameters, the framework first analyzes the dataset and derives optimal settings directly from it.
Concretely, the process looks like this:
Dataset fingerprinting
The model scans the training data to understand:
- Image dimensions and spacing (voxel resolution)
- Intensity distributions
- Number of samples and class imbalance
- Memory constraints imposed by the data
Rule-based configuration
Based on this fingerprint, nnU-Net applies a set of empirically validated rules to decide:
- Patch size that maximizes context while fitting GPU memory
- Batch size that balances stability and efficiency
- Network depth and number of feature maps
- Appropriate normalization strategy
- Loss functions suited for multi-class medical segmentation
- Data augmentation tailored to the dataset’s spatial properties
Automatic architecture selection
Depending on image resolution and anisotropy, the framework chooses between 2D, 3D full-resolution, or 3D cascade variants; without user intervention.
Get Nandini Lokesh Reddy’s stories in your inbox
Join Medium for free to get updates from this writer.
The result is that TotalSegmentator does not require manual hyperparameter tuning.
The model adapts itself to the dataset in a structured, reproducible way, allowing researchers and clinicians to focus on application rather than optimization.
Resolution Settings
The authors trained two models:
- 1.5 mm isotropic resolution
- Higher precision
- More computationally expensive
2. 3 mm isotropic resolution
- Lighter, faster
- Works on lower-end hardware
Both are released open-source, making the model accessible to beginners and practitioners without heavy GPUs.
Architecture (U-Net + nnU-Net principles)
TotalSegmentator is built on a 3D U-Net–style encoder–decoder architecture, a design that has become the standard for volumetric medical image segmentation. However, the strength of the system does not come from a novel architecture, but from how that architecture is automatically adapted to the data.
The network follows a classic encoder–decoder structure:
- Encoder (downsampling path)
Progressively captures higher-level, semantic features while reducing spatial resolution. This allows the model to understand large anatomical context. - Decoder (upsampling path)
Gradually restores spatial resolution to produce dense, voxel-wise segmentation maps. - Skip connections
Feature maps from the encoder are passed directly to the decoder at corresponding scales, preserving fine-grained spatial details that are critical for accurate anatomical boundaries. - 3D convolutions throughout
All operations are performed in 3D, enabling the model to fully leverage volumetric context in CT scans rather than treating slices independently.
What makes this architecture powerful in TotalSegmentator is nnU-Net’s automated design logic. Instead of fixing architectural choices upfront, nnU-Net derives them from the dataset itself:
- Network depth is adjusted based on image resolution and field-of-view
- Kernel sizes and strides are adapted to voxel spacing and anisotropy
- Feature map widths scale with available GPU memory
- Input patch size is chosen to balance global context and computational feasibility
As a result, there is no hand-crafted architecture tuning.
The same U-Net blueprint is dynamically reshaped to match the data’s geometry, spacing, and statistical properties.
This data-driven architectural adaptation is what allows TotalSegmentator to remain robust across diverse CT datasets while maintaining high anatomical precision at scale.
Implementation:
This is the implementation of TotalSegmentator on CPU
pip install Totalsegmentator
import nibabel as nib
from totalsegmentator.python_api import totalsegmentator
input_path = "Case1.nii.gz"
output_path = "Case1"
totalsegmentator(input_path, output_path)import nibabel as nib
import numpy as np
import matplotlib.pyplot as plt
import imageio
from io import BytesIO
from IPython.display import display, Image
file_path = 'Case1/liver.nii.gz'
img = nib.load(file_path)
data = img.get_fdata()
images = []
segmented_slices = []
for i in range(data.shape[2]):
if np.any(data[:, :, i] != 0):
segmented_slices.append(i)
plt.figure(figsize=(5, 5))
plt.imshow(data[:, :, i], cmap='gray')
plt.axis('off')
buf = BytesIO()
plt.savefig(buf, format='png')
buf.seek(0)
images.append(imageio.imread(buf))
buf.close()
plt.close()
if images:
gif_buffer = BytesIO()
imageio.mimsave(gif_buffer, images, format='GIF', duration=0.5)
gif_buffer.seek(0)
display(Image(data=gif_buffer.read(), format='png'))
print("GIF displayed with segmented slices.")
else:
print("No segmented slices found in the data.")
for non-commercial license model, there are more in-depth structures like coronary arteries, heart chambers highres, appendicular bones, tissue types, brain, vertebrae body, face, thigh shoulder muscles and brain aneurysm
Performance
On its validation set, TotalSegmentator achieves a strong:
- Mean Dice score ≈ 0.94
This is extremely high considering the diversity and number of structures.
Some structures naturally perform higher (large organs), while thin or small structures are more challenging — but overall, the model shows very consistent performance.
Limitations & Failure Modes
- Very small structures can be harder to segment reliably.
- Severe pathologies or post-operative anatomy may deviate from training distribution.
- CT scans with very unusual fields-of-view can lead to incomplete predictions.
- Bone-related predictions are affected by metal artifacts.
- Performance may vary across institutions with different acquisition protocols.
But for an open-source tool, the robustness is genuinely impressive.
