I Fine-Tuned Three LLMs. The Data Pipeline Was the Problem Every Time.

TL;DR: Fine-tuning is usually presented as a modeling problem. Pick a base model, adjust the learning rate, choose a LoRA rank, and start training. But in my experience, the problems that caused the most trouble happened before the GPU job even started. Across three fine-tuning projects, I ran into repeated training examples, validation leakage across conversations, and a data conversion pipeline that silently dropped thousands of examples. Here’s what broke, how I found it, and the data engineering checks I’d put in place before launching another training run.

I Fine-Tuned Three LLMs. The Data Pipeline Was the Problem Every Time.

Fine-tuning an LLM is usually presented as a modeling exercise. Choose a base model, prepare a dataset, configure the training parameters, launch a GPU job, and evaluate the results.

That description skips a lot of engineering work.

I’ve worked on three LLM fine-tuning projects over the last two years. The first was a customer-support model for a fintech use case. The second was a code-generation assistant for internal tooling. The third was a medical summarization model where incorrect or incomplete information required human review.

The models had different objectives, different datasets, and different training requirements. But the problems I spent the most time fixing weren’t learning…