Shail·17h agoPart 1.5: Scaling Laws and Compute-Optimal TrainingThe previous posts focused on feasibility:
Shail·Jul 8Part 1.4: Training Memory: Weights, Gradients, Optimizer States, and ActivationsParameter count tells us how large the model is. It does not tell us whether the model can be trained on a given GPU.
Shail·Jul 5Part 1.3: FLOPs per Token and Total Training ComputeOnce we can count parameters, the next question is the compute.
Shail·Jul 5Part 1.2: Counting Parameters in Decoder-Only TransformersBefore we estimate FLOPs, memory, or distributed training requirements, we need a reliable parameter count.
Shail·Jul 4Part 1.1: From Zero to Distributed LLM Training Decisions: Before Training an LLM, Define the…When people discuss LLM training, they often start with the model size: 1B, 7B, 70B, or some larger number. That is useful shorthand, but…
Shail·Jul 4Part 0: From Zero to Distributed LLM Training DecisionsBuilding LLMs From First Principles: Series Roadmap
Shail·Feb 15From Vanilla Transformers to Modern LLMs: What Changed After the Original Transformer (Part 3)Modern LLM Architecture (Part 3): MoE, Low-level Hardware KV Cache Optimizations, and MuonClip Optimizer
Shail·Feb 7From Vanilla Transformers to Modern LLMs: What Changed After the Original Transformer (Part 2)Modern LLM Architecture (Part 2): Efficient Attention and Long Context
Shail·Feb 4From Vanilla Transformers to Modern LLMs: What Changed After the Original Transformer (Part 1)Modern LLM Architecture (Part 1): From Vanilla Transformers to Stable Training
Shail·Oct 1, 2024Mastering Tokenization: Part 3 — Comprehensive Guide To SentencePiece for Training Multilingual…When dealing with language models, tokenization is a crucial step, and SentencePiece offers a highly flexible approach. Unlike traditional…A response icon1A response icon1