himanshuprajapati·Sep 7Batching is supposed to make one GPU serve many people almost for free.TL;DR: My last post showed that my laptop reads about a gigabyte of model weights for every single word it writes, and that read costs the…
Nisha·Aug 12Speculative Decoding in LLM Inference: Why Wasting Compute on Wrong Guesses Makes AI FasterYou are paying your GPU to generate words it fully expects to throw away — and that is exactly why your inference bill is about to get…
Sriram Penumatcha·Jul 3The Hidden Engineering Behind ChatGPT: FlashAttention, PagedAttention, and Continuous Batching…An introductory guide to how large language models interact with modern GPU hardware, and the software innovations that make AI systems…
Naren Suri·Jun 9Serving LLMs Without Burning MoneyFast LLM Inference, KV Cache, Quantization, PagedAttention, vLLM and Benchmarking
The_Turingetic_Guy·May 31Large-Scale Distributed LLM Inference — Part 2 : Modern Attention Mechanisms That Make Large-Scale…In Part 1, we explored why LLM inference is fundamentally different from traditional deep learning inference.
Nikhil Rasineni·May 26LLM Inference Optimisation: Part2, MQA & GQAIn the part 1 of the LLM inference optimisation series we talked about how KV cache speeds up inference but creates two new problems w.r.t…
InTowards AIbyJiten Bhalavat·Mar 9Understanding LLM Quantization: Why FP32, FP16, BF16 and INT8 Matter for Modern AI SystemsLarge Language Models are extremely large and computationally expensive, so techniques like reduced-precision data types (FP16, BF16) and…
