How to Prepare Speech Datasets

If you’ve ever opened a folder of “ready to use” audio data and felt your soul leave your body, this one’s for you.

Speech datasets have a dirty secret: the model is rarely the hard part. The hard part is the hundreds of hours of audio sitting in inconsistent formats, transcribed by three different people who couldn’t agree on whether to write “okay” or “ok,” with silence at the start of some clips and clipped speech at the end of others.

I’ve put together the workflow I now use every time, mistakes included. Steal it.

Start by deciding what you actually need

Before you touch a single file, lock down your basics: sample rate (16kHz for most ASR work, 22–48kHz if you’re doing TTS), mono audio, and which languages or accents you actually care about.

This sounds obvious. It is not obvious at 2am when you discover half your dataset is 8kHz phone audio and the other half is 48kHz studio audio, and now everything has to be resampled and you’re questioning every life choice that led you here.

Decide early. Future you will send a thank-you note.

Collecting audio: track everything, trust nothing

Grab your raw recordings, and for the love of all that is holy, keep them separate from anything you process later. Never overwrite the originals. You will need to go back to them. You always do.

For every file, log the boring stuff: speaker ID, device, environment, language, accent, timestamp. It feels like busywork until six months from now when you’re trying to figure out why one speaker’s data is tanking your model’s accuracy, and that one column saves you a full day of detective work.

Transcription is where datasets quietly die

This is the part nobody warns you about. Transcription isn’t just “writing down what was said” — it’s where inconsistency creeps in and quietly poisons everything downstream.

Get Alex Morgan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A few rules that have saved me real pain:

• Pick one orthographic standard and enforce it. Punctuation, casing, how you write numbers — pick once, never deviate.
• Tag non-speech events clearly: [noise], [laughter], [overlap]. Don't just... leave them out.
• For multi-speaker audio, diarize. Speaker tags and timestamps aren’t optional extras, they’re the difference between usable data and a mess.
• Get a second pass. Have someone else check a sample, measure how often annotators disagree. If your annotators don’t agree with each other, your model won’t have a chance either.

Cutting the audio into usable pieces

Long recordings need to become short utterances, typically 1–15 seconds, cut at natural pauses. Never mid-word. If you want this done properly, lean on voice activity detection or a forced aligner like the Montreal Forced Aligner rather than eyeballing waveforms at 1am (I’ve done this; don’t).

The cleanup pass nobody enjoys but everyone needs

This is the unglamorous middle stretch:

• Hunt down clipping, DC offset, and background noise that’s louder than it should be.
• Normalize loudness so one clip isn’t a whisper and the next isn’t a shout.
• Trim silence from the edges — but gently. Over-trimming kills natural speech rhythm, and your model will learn the wrong thing about how people actually talk.
• Pick one format and stick to it. PCM 16-bit WAV is the boring, reliable choice. Boring is good here.

Make sure the text and audio actually match

This step catches more errors than you’d expect. Run sanity checks: does the clip length roughly match the word count? Is there actual audio energy in files that are supposed to contain speech? A two-second clip with a forty-word transcript isn’t a hidden gem, it’s a bug.

Splitting your data without sabotaging yourself

Check your diversity: speakers, genders, accents, recording conditions. One speaker shouldn’t be hogging half your dataset.

And here’s the mistake that gets almost everyone at least once: split your train/val/test sets by speaker, not by random utterance. Otherwise the same voice shows up in training and testing, your evaluation numbers look fantastic, and then the model falls apart the moment it meets a real, unfamiliar voice. It’s a quiet kind of self-deception, and it’s everywhere.

Write it all down

Make a dataset card. Where the data came from, how it was licensed, who’s represented in it, what biases might be lurking, what preprocessing you did. This isn’t bureaucracy for its own sake — voice data is personal, often legally sensitive, and “I don’t remember where this came from” is not a sentence you want to say out loud later.

A folder structure that’s saved me more than once:

dataset/
├── wavs/
│   └── speaker01_0001.wav
├── transcripts/
│   └── speaker01_0001.txt
├── metadata.csv
└── README.md

The mistakes I see (and make) over and over

• Mixing sample rates and formats without a consistent resampling pass
• Silent or corrupted files slipping through because nobody checked
• Transcription style drifting as different annotators take over
• Speaker leakage between train and test sets, quietly inflating your results

None of this is glamorous. None of it is the part you’ll show off in a demo. But it’s the difference between a model that genuinely works and one that just looks like it works until real users show up.