Reimplementing CodecSlime
How Dor Tirosh and I rebuilt and trained CodecSlime from scratch, fixed broken pseudocode, learned the hard truth about reproducing AI research that without released source code, made our code public and AI generated a Medium post about it.
tl;dr — It works and it can be found here.
Audio compression has come a long way from the rigid, math-based signal processing of the MP3 era. Today, neural audio codecs can maintain crystal-clear, natural speech at ultra-low bitrates by learning directly from data. Models like EnCodec and SoundStream have revolutionized how we transmit voice.
But despite these massive advancements, modern codecs share a fundamental, almost philosophical limitation: they process all sound at a constant rate.
Human speech is highly uneven. A rapid burst of consonants carries a heavy acoustic load, while a drawn-out vowel or a moment of silence carries almost none. The theoretical information rate needed to perfectly understand human speech is estimated at just 50 to 100 bits per second. Yet, because today’s neural codecs rely on a Fixed-Frame-Rate (FFR), they are forced to spend the exact same number of tokens on a complex consonant cluster as they do on absolute silence.
They waste their token budget simply repeating themselves.
To solve this, we looked to a recent, fascinating paper: CodecSlime. CodecSlime introduces a Dynamic Frame Rate (DFR) plugin that mathematically decouples what a sound is from how long it lasts. We set out to reproduce it. Here is how we did it, the roadblocks we hit, and what we learned.
Rebuilding
Because CodecSlime didn’t release public source code, and its backbone model (BigCodec) didn’t have pre-trained checkpoints available, we had to build and train the entire pipeline from the ground up.
Our architecture relied on two major innovations of the writers:
1. ScheDFR
If you want to compress audio dynamically, you can’t just drop frames at random. The paper introduces ScheDFR, a dynamic programming algorithm that finds the optimal way to merge adjacent, redundant frames into a single, high-density token with minimal information loss.
To make this mathematically heavy algorithm viable for training, we had to optimize it heavily using Python’s joblib for parallel batch processing.
2. The Melt-and-Cool Manager
Even if ScheDFR finds the perfect way to downsample frames, a standard neural network will panic when fed these unfamiliar, merged tokens. To fix this, the network undergoes a “Melt-and-Cool” training regimen.
- The Melt: We slowly introduce the model to an increasing amount of randomly downsampled frames, forcing it to stretch its understanding of time.
- The Cool: We freeze the encoder and fine-tune the model specifically on the optimized ScheDFR frames.
Fixing some Pseudocode
Reproducing a paper without source code is an exercise in forensic science.
Get Noamshakedcohen’s stories in your inbox
Join Medium for free to get updates from this writer.
While the core ScheDFR algorithm was mathematically detailed in the paper, the Melt Manager was only provided as pseudocode on a landing page. As we implemented it, we noticed a critical logical flaw.
According to the text of the paper, the Melt Manager should slowly increase the amount of randomly compressed frames from zero to the maximum. But the provided pseudocode did the exact opposite — it initialized aggressively and slowly backed off. We rewrote the manager, patched the bug, and tested both versions. Our corrected version performed significantly better, proving that you should always trust the math over the pseudocode.
Furthermore, we upgraded the model’s quantization method. We moved away from traditional Vector Quantization (VQ) — which is notoriously unstable and prone to “codebook collapse” — and integrated Finite Scalar Quantization (FSQ). FSQ skips complex lookup tables entirely and uses simple mathematical rounding, making the network far more stable.
The Results: Does it Actually Work?
The short answer: Yes, beautifully.
Our reproduction successfully validated the core claims of CodecSlime. When we took an untrained Fixed-Frame-Rate backbone and forced it to rigidly compress audio by a factor of two, intelligibility plummeted. The Word Error Rate (WER) spiked to a disastrous 65%.
However, when we applied the intelligent ScheDFR algorithm to that exact same backbone, the WER dropped to 37%. When we applied the full Melt-and-Cool training pipeline, the WER improved even further to roughly 26% (with excellent STOI and PESQ acoustic scores).
The “Reproducibility Gap”
While our relative trends perfectly mirrored the original paper, our absolute scores did not. The original authors reported a highly optimized WER of ~4%, while our best model hit ~26%. Why the massive gap?
- The ASR Mismatch: We evaluated our audio using OpenAI’s Whisper, while the original authors used NeMo ASR. Word Error Rate is notoriously sensitive to the specific text-transcription model used.
- The Foundation: Because we had to train the massive 159-million-parameter BigCodec backbone entirely from scratch, our baseline was lower. A plugin can only be as good as the foundation it sits on.
- The Secret Sauce: Finding the bug in the pseudocode confirmed our suspicion that the authors’ private codebase likely contains undocumented heuristics, optimizations, or parameter tweaks.
We repeated the ablation study from the paper, too:
Looking Forward: The Future of Audio Codecs
CodecSlime proves that dynamic frame rates are the future of audio compression, but there is still massive room for innovation.
Currently, the encoder remains frozen during the “Cooling” phase. What if we co-trained the encoder alongside the dynamic scheduler, allowing it to proactively generate representations that it knows will be merged?
While real-time, low-latency streaming remains a challenge (dynamic programming requires looking ahead into the future of the audio stream), the broader impact is undeniable.
By proving that temporal redundancy can be dynamically squeezed out of neural networks, we are one step closer to codecs that finally listen to speech the way humans do: paying attention when it matters, and resting when it’s quiet.
