Member-only story
Compressing Large Language Models (LLMs)
Make LLMs 10X smaller without sacrificing performance
This article is part of a larger series on using large language models (LLMs) in practice. While the immense scale of LLMs is responsible for their impressive performance across a wide range of use cases, this presents challenges in their application to real-world problems. In this article, I discuss how we can overcome these challenges by compressing LLMs. I start with a high-level overview of key concepts and then walk through a concrete example with Python code.
The AI mantra of 2023 was "Bigger is Better," where the equation for improving language models was pretty simple: more data + more parameters + more compute = better performance [1].
While this is likely still the case (GPT-5 coming soon?), there are obvious challenges with working with 100B+ parameter models. For example, a 100B parameter model using FP16 requires 200GB just for storage!
Needless to say, most consumer devices (e.g. phones, tablets, laptops) can’t handle models this big…
