Hao·Aug 25From FlatParameter to DTensor: PyTorch’s Native Parallelism Stack in the FSDP2 EraThe 2024 AWS blog post Accelerating Llama3 Training with HSDP and Model Parallelism [14] described an FSDP1-era approach that combined…
Romeo Kienzler·Nov 19, 2025FSDP vs DeepSpeedRomeo Kienzler in collaboration with Mark Sosman— IBM Research & ETH Zürich
Sanket Thakur·Nov 11, 2025Scaling Deep Learning with Distributed Training — Part II(a): How GPUs talk ?This post is a small novella in our Distributed Training series — think of it as the “Dunk & Egg” to our “House of the Dragon.”
Pratish Dewangan·Sep 27, 2025ZeRO (Zero Redundancy Optimizer): The Secret Behind Scaling LLM TrainingIntroduction
InAI Simplified in Plain EnglishbyFrank Morales Aguilera·Aug 17, 2025Fine-Tuning Llama 3 on Colab TPUs: A Deep Dive into Efficient Language Model AdaptationFrank Morales Aguilera, BEng, MEng, SMIEEE
