Member-only story
Google Searched 25,000 Transformer Programs. Two Small Changes Made Language Models Much Cheaper to Train.
Inside Primer EZ: squared ReLU, convolutional attention projections, an independent PyTorch implementation, and a surprising failure when the sequence pattern changes
An engineering tutorial for machine learning researchers and developers. Based on the NeurIPS 2021 paper “Primer: Searching for Efficient Transformers for Language Modeling” by David R. So and colleagues.
Click here to access this for free: https://inamdaraditya.medium.com/google-searched-25-000-transformer-programs-53ce2c808617?sk=d66bbcd788be6f53a44b210e3675e5ed
There is something strange about the way we optimize neural networks.
We will spend weeks designing a distributed training stack, choosing a new optimizer, implementing kernel fusion, profiling memory bandwidth, and deciding whether to double the GPU count. Yet the actual function executed millions of times inside a Transformer can remain almost untouched.
What if the most valuable optimization is neither a faster GPU kernel nor a bigger model?
