You’ve run 47 hyperparameter sweeps. Each one takes 6 hours on 4 GPUs. Your validation metric has improved by 0.4% in two weeks, and it may have been a lucky run.

Your experiment tracker has a graveyard of rejected ideas:

• Cosine annealing: no improvement.
• Weight decay: hurts performance.
• Muon: unstable.

But you tested cosine annealing with a learning rate tuned for a constant schedule. You tested weight decay on SGD but never tried it with Adam. You tested Muon with Adam’s learning rate range. None of those ideas were bad. The method was, and most of them probably would have worked given a fair shot.

This article shows you how to give them one. You’ll learn how to separate the parameters you’re studying from the ones you need to control, how to explore search spaces efficiently with quasi-random sampling, and how to sequence your experiments so each round builds on the last instead of invalidating it.

This methodology draws on my years of experience — directing research teams at 42 AI’s research lab, and teaching at Paris 1 Sorbonne and Amsterdam Tech University — as well as sources I’ll link throughout, particularly the Google Deep Learning Tuning Playbook [1] and Goodfellow et al.’s debugging strategies [1b].

One piece of context before we dive in: this work was part of migrating our production image classifiers from TensorFlow to PyTorch. Our TensorFlow baseline was a formidable target: it encodes a decade of accumulated knowledge from researchers and engineers, some of whom are now at Mistral and DeepMind. So “beating the baseline” meant out-tuning years of refinement with a PyTorch stack built from the ground up. Across our 13 production tasks, this methodology did exactly that, with an average gain of 4.6%.

In This Article

1. Heuritech:
   The company behind this work
2. The Problem With Ad-Hoc Tuning:
   Why sequential one-at-a-time tuning produces biased conclusions
3. Scientific vs. Nuisance Parameters:
   The key insight that makes fair comparisons possible
4. Beyond Grid Search: Quasi-Random Sampling:
   Sobol sequences, and why not Optuna (yet)
5. Foundations Before Tuning:
   Metric, batch size, and compute budget
6. The Ordering Rationale:
   Why architecture → optimizer → scheduler → regularization
7. The Iterative Loop:
   Four phases of compounding gains on 13 production tasks
8. The 5-Phase Experiment Workflow:
   From research question to reproducible results
9. Results: From Matching TensorFlow to Beating It:
   +4.6% average, up to +14.5% on individual tasks
10. Making It Your Own:
    The distilled framework

Heuritech

A word about the system this work lives in:

Heuritech’s computer vision pipeline processes over 1.5 billion images sourced from 16 markets (US, Europe, Japan, and beyond) to help fashion & sportswear brands forecast demand more accurately and reduce overproduction.

The AI stack runs deep: object detection models identify garments in the wild, a dozen image classifiers label their attributes (shoe type, fabric, silhouette…), and from these visual signals we extract over 2,000 individual trends that get combined combinatorially (red × skirt × midi length) and mapped across geographies to produce more than 200,000 time series that we forecast into the future using advanced ensembling methods. We serve inference through Triton for production-grade throughput, use VLMs for specialized detections where traditional classifiers fall short, and run active learning pipelines to target our annotation budget (500K to 1M newly labeled images per month) where it matters most.

This article focuses specifically on one piece of that system: retraining our image classifiers, the dozen models that turn raw detections into the attribute labels everything downstream depends on.

For context on the rest of the pipeline, the team has written extensively across both sides of the stack:

On the vision side — directly upstream of this article — Paul Morel’s A social media image is worth a thousand fashion labels… walks through how we extract fashion attributes from social media images using deep learning, using metal-studs jackets as a worked example.

On the forecasting side — what happens downstream once these labels feed the time series — Etienne David’s three-part series:

1. Is fashion predictable? — the foundation: translating fashion dynamics into measurable time series.
2. Is Fashion Predictable? From Basic to Accurate Forecasting Models — applying statistical and deep learning forecasting models, with code.
3. Is Fashion Predictable? The Rise of Foundation Models — evaluating time series foundation models from Amazon, Microsoft, and IBM on fashion forecasting.

The Problem With Ad-Hoc Tuning

Most hyperparameter tuning in practice looks like this:

1. Pick a parameter (usually learning rate)
2. Try a few values
3. Pick the best one
4. Move to the next parameter
5. Repeat until deadline

This approach has four fatal flaws:

Confounded comparisons: Hyperparameters interact.
When you compare lr=1e-3 vs lr=1e-4 with weight decay (wd) fixed at wd=1e-5, you’re not comparing learning rates, you’re comparing:

• “lr=1e-3 with appropriate weight decay”
• “lr=1e-4 with the wrong weight decay.”

Each learning rate has its own optimal weight decay. A fair comparison requires tuning the parameters separately for each.

Incomplete exploration. Finding a good run isn’t enough, you need to find where performance degrades.
If your best result sits at the edge of your search range, you haven’t found the optimum; you’ve found the boundary of where you looked. Keep expanding until the best value is surrounded by worse ones on all sides.

Biased conclusions. If augmentation A gets 30 experiments and augmentation B gets 10, A may win simply because it was searched more thoroughly. Worse: if A’s sub-parameters are well-tuned but B’s aren’t, you’ll conclude A is better, when B might win with equal tuning budget. Fair comparison demands equal search effort across the options you’re evaluating.

Wasted compute. Grid search tests every combination, including those in unproductive regions. Random search is better but still inefficient.

Scientific vs. Nuisance Parameters: The Key Insight

The single most important concept in systematic tuning is the distinction between scientific and nuisance parameters.

Scientific parameters are what you’re trying to measure. They answer your research question directly.
Example: “What optimizer works best?” → The scientific parameter is optimizer and could take values sgd, adam, adamw and muon.

Nuisance parameters must be tuned per scientific parameter value to ensure a fair comparison. When comparing optimizer types, the learning rate is one of the nuisance parameters: different optimizers need different optimal learning rates. If you hold it constant, you might not be fairly evaluating the optimizer types.

Critically, nuisance parameters don’t have to be the same across scientific parameter values. When comparing SGD vs Adam, SGD’s nuisance parameter is learning rate (range 1e-3 to 1e-1), while Adam’s nuisance parameters are learning_rate (range 1e-5 to 1e-3) and beta1 (range 0.8 to 0.99). Each scientific value gets its own nuisance search space, tailored to what makes sense for that option. This is what makes the comparison fair.

The same applies to scheduler comparison: a constant learning rate has no nuisance parameters beyond LR itself, while cosine annealing has warmup epochs, restart multiplier, and decay length — all nuisance parameters that must be tuned to give cosine annealing a fair shot.

Fixed parameters are everything else, held constant to reduce the search space.

The Common Mistake

An easy mistake is to confuse nuisance parameters with scientific ones.
Here’s a real example. Suppose your research question is: “Which optimizer achieves the best validation metric: Muon or Adam?”

That question already tells you everything about the parameter classification, but in practice an easy mistake is to reach for the parameter suspected to have the most impact and place it as scientific:

In this setup, you’re treating a sub-parameter as the thing under study, and burying the actual decision — which optimizer? — as something to hold constant. The right setup flips this: the optimizer type is what you’re measuring, and each option gets its own nuisance search space.

Notice that the nuisance spaces are not just different values : they’re different parameters entirely. Muon doesn’t have a beta1, Adam doesn't have ns_steps.

Each optimizer gets tuned on its own terms:

• Muon:
  learning_rate ∈ [1e-3, 1e-1] ← search space is not shared
  ns_steps ∈ [3, 10]
  momentum ∈ [0.8, 0.95]
• Adam:
  learning_rate ∈ [1e-5, 1e-3] ← search space is not shared
  beta1 ∈ [0.8, 0.99]
  epsilon ∈ [1e-9, 1e-7]

Notice that even for nuisance parameters sharing the same name, their search spaces do not have to be shared across scientific parameters.

Now, the instinct to study learning rate isn’t wrong: it’s premature. Learning rate is one of the highest-impact continuous hyperparameters, and wanting to understand it deeply is exactly the right reflex. But the sequential logic matters: first, settle the optimizer choice (where LR is a nuisance parameter that must be tuned per optimizer to ensure a fair comparison). Then, once the optimizer is locked, that same instinct to study learning rate becomes the next scientific question : you’re now comparing LR schedules, warmup strategies, and decay shapes. The parameter is promoted from nuisance to scientific, in the right order.

This distinction seems pedantic until you realize it prevents the most common source of bias in ML experiments: unfairly comparing a well-tuned option against a poorly-tuned one.

Beyond Grid Search: Quasi-Random Sampling

Once you’ve defined your search space, how do you explore it efficiently?

To build intuition, consider a simple two-dimensional example: tuning two hyperparameters with a budget of just 16 experiments.

• One parameter has a high impact on our metric of interest (F1-score in this example): a narrow optimal region that’s easy to miss.
• The other has low impact : it matters, but not nearly as much.

Grid search places experiments on a rigid lattice. With 16 trials, you get a 4×4 grid, only 4 unique values per dimension. If the optimal region falls between two grid lines, every single experiment misses it. The heatmap tells the story: the grid’s best configuration scores 0.13, completely blind to the peak.

Random search breaks free from the lattice. Bergstra and Bengio (2012) showed that random search is more efficient than grid search precisely because it samples each dimension independently [2]. With 16 random trials, you get 16 unique values per dimension instead of 4. But randomness comes with a cost: clusters and gaps. Some regions get over-sampled while others are ignored entirely. Here, a lucky draw lands near the peak — best = 0.48, but coverage is uneven, and another random seed might miss it entirely.

Quasi-random sampling (Halton/Sobol sequences) combines the best of both worlds: the structural coverage of a grid with the dimensional independence of random search [3a]. Sobol sequences are designed to fill the space as uniformly as possible, no clusters, no gaps. Every new point is placed to maximally reduce the largest remaining hole in the coverage. The result: best = 0.51, with points distributed evenly across the entire space.

The marginal F1 plots on each figure make the mechanism visible. Look at the dots projected onto the high-impact parameter curve (top):

• Grid search gives you only 4 probes, none near the peak.
• Random search gives you 16, but they cluster on the left.
• Sobol gives you 16 evenly spaced probes, guaranteeing that at least one lands close to any narrow optimum.

Where It Really Matters: Higher Dimensions

The two-dimensional case is illustrative. The real payoff comes in higher dimensions.

In a 5-dimensional search with a grid of 5 values per dimension, you need 3,125 trials, but each dimension still only sees 5 unique values. That’s 3,125 experiments to test the same 5 learning rates. 80% of your compute is spent re-confirming what the low-impact dimensions do.

Random search with 30 trials gives you 30 unique values per dimension : 6× better coverage than the 3,125-point grid, at less than 1% of the cost.

Halton and Sobol sequences with 30 trials give you 30 unique values with guaranteed uniformity, no clusters, no gaps, reducing lucky-seed dependence. As dimensionality grows, the advantage of quasi-random over both grid and random search compounds dramatically.

Both SciPy’s Sobol and Halton samplers work the same way: generate uniform points in [0, 1], then scale to your parameter ranges. Here’s a minimal Sobol example with log-scaled learning rate and a categorical optimizer:

from scipy.stats.qmc import Sobol
import numpy as np

sampler = Sobol(d=2, scramble=True)
points = sampler.random(n=32)  # Sobol works best with powers of 2

lr = 10 ** (np.log10(1e-5) + points[:, 0] * (np.log10(1e-2) - np.log10(1e-5)))
optimizer = np.array(["sgd", "adam", "adamw"])[
    (points[:, 1] * 3).astype(int).clip(max=2)
]
# lr.shape = (32,)       → [2.1e-5, 7.4e-4, 3.1e-3, ...]
# optimizer.shape = (32,) → ["adam", "sgd", "adamw", ...]

And the Halton equivalent, same interface, but accepts any sample size and is naturally extensible (add trials by continuing the index):

from scipy.stats.qmc import Halton

sampler = Halton(d=2, scramble=True)
points = sampler.random(n=30)  # Halton: any n, not just powers of 2

lr = 10 ** (np.log10(1e-5) + points[:, 0] * (np.log10(1e-2) - np.log10(1e-5)))
optimizer = np.array(["sgd", "adam", "adamw"])[
    (points[:, 1] * 3).astype(int).clip(max=2)
]
# lr.shape = (30,)       → [1.2e-3, 4.8e-5, 6.0e-4, ...]
# optimizer.shape = (30,) → ["sgd", "adam", "adamw", ...]

Sobol vs. Halton :

Sobol sequences (used above) have slightly better theoretical uniformity in high dimensions (measured by discrepancy [3d], how evenly points fill a space), but require power-of-2 sample sizes for optimal coverage.

Visual example of Sobol sampling, from Wikipedia:

Halton sequences accept any sample size and are naturally extensible (you can add trials by continuing the index sequence, and new points automatically fill gaps in the existing coverage; however, regions added by widening the search space afterwards remain less explored).

Visual example of Halton sampling, from Wikipedia:

Both are vastly superior to grid or random search. If you anticipate extending a running experiment (widening a search range, adding more trials), Halton is the more practical choice.

Visual example of Random sampling, from Wikipedia:

How many trials?
As a rule of thumb: 5–30 trials per scientific parameter value, depending on the number of nuisance dimensions and the type of hyperparameter.

• With 1–2 nuisance parameters, 5–10 trials suffice.
• With 4–5, you need 20–30.
• Avoid tuning more than 5 hyperparameters simultaneously, above that, the space is too large for a reasonable trial budget to cover meaningfully (unless you are GPU rich!).

The type of parameter greatly impacts the number of trials needed. Continuous numeric parameters require the fewest trials (e.g. learning rate), then discrete numeric parameters (e.g. batch size, number of layers), and finally categorical parameters are the most costly — they will exceed the recommendations above as soon as they can take more than a couple of values.

In practice, we keep most experiments to 2–4 nuisance dimensions, and we try to exclude categorical parameters (or to treat them as scientific ones in later rounds of optimization).

Why Not Optuna or Hyperopt?

A natural question: if we’re trying to find the best configuration efficiently, why not use Bayesian optimization tools like Optuna[3b] or Hyperopt[3c]? They’re designed for exactly this : they model the objective function (our error metric) and focus trials on the most promising regions of each of our hyperparameters.

That’s precisely the problem. We’re not just trying to find the best configuration (yet) we’re trying to understand the system to direct our compute resources.

Bayesian optimizers tend to bias exploration toward the first good results they encounter. After a few trials, they build a surrogate model and start exploiting : concentrating new trials near the current best. This is efficient for finding a single optimum, but it leaves large regions of the search space unexplored. You can get a great configuration, but you learn almost nothing about why it’s great, or what else might work.

Quasi-random search doesn’t bias. Every region gets equal coverage, which means:

You can run all trials in parallel. Sobol generates every configuration upfront, so you can launch them all simultaneously across available GPUs. Bayesian optimization needs completed results to update its surrogate before sampling the next trial; you’re either waiting between runs or batching blindly, losing the adaptive advantage that justified using it in the first place.

You can distinguish high-impact from low-impact parameters. When all configurations near lr=1e-3 perform well regardless of weight decay, you know weight decay doesn't matter, and stop wasting time on it. Biased sampling makes this kind of sensitivity analysis unreliable.

You can extend a search range without starting over. If your best learning rate sits at the boundary, you need to explore further. With Sobol, you extend the range and add trials; all previous results remain valid data points. A Bayesian surrogate model built for the old range doesn’t transfer cleanly.

You can promote nuisance parameters to scientific ones. Sometimes you discover mid-experiment that a nuisance parameter actually matters. Because quasi-random search explored it uniformly, your existing data is already an unbiased sample of its effect. Bayesian optimization concentrated its exploration wherever the objective was high; you can’t draw fair conclusions about independent effects from biased coverage.

You build transferable knowledge. “Adam’s optimal LR for DINOv3[4a] backbones is broadly 1e-4 to 1e-3” is reusable across tasks and projects. “Optuna found lr=3.7e-4” is not.

Bayesian optimization has its place: once you understand your system, it’s powerful for squeezing out the last few points. But starting with it means optimizing a system you don’t understand, and that’s how you end up with fragile configurations that break when anything changes.

One notable exception, is when using a quasi-random sampler (like Sobol or Halton) specifically — for example, Optuna’s QMCSampler [4b]

Search Space Boundaries: A Critical Diagnostic

One diagnostic that’s easily missed in practice: always make your search space large enough to include clearly bad configurations at the edges. You want to see performance degrading at both boundaries: that’s how you know the optimum is enclosed, not just clipped by where you stopped looking.

If your best configuration is at the boundary of your search space, you haven’t found the best configuration — you’ve found the edge of where you looked.

This sounds obvious, but it’s an easy mistake to make : setting a learning rate range of [1e-4, 1e-3] and finding the best result at 1e-3. Is 1e-3 optimal, or would 2e-3 be better? You can’t know without expanding the range. Design search spaces to include the bad runs on purpose: they’re not wasted compute — they’re confirmation that you’ve explored far enough.

Sources:
[2] Bergstra & Bengio, 2012 (JMLR) : https://www.jmlr.org/papers/v13/bergstra12a.html
[3a] SciPy QMC Documentation : https://docs.scipy.org/doc/scipy/reference/stats.qmc.html
[3b] Optuna : https://github.com/optuna/optuna
[3c] HyperOpt : https://github.com/hyperopt/hyperopt
[3d] Discrepancy : https://en.wikipedia.org/wiki/Low-discrepancy_sequence
[4a] DINOv3 : https://arxiv.org/abs/2508.10104
[4b] Optuna QMCS https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.QMCSampler.html

Foundations Before Tuning

Before running a single experiment, three decisions shape everything that follows.

Know Your Compute Budget

Your compute budget shapes your experimental design. With 4 GPUs and 6-hour runs, you get ~20 experiments per week, enough for 2–3 nuisance parameters per round, not 5. This constraint is precisely why sequential tuning works: you can’t search a 10-parameter space on a realistic budget, but you can search three 3-parameter spaces in sequence.

Choose a Single-Number Metric

Whenever it’s possible to optimize for a single metric, do it.
A single target simplifies every decision in the tuning process: you always know whether a configuration is better or worse.

When you have multiple optimization targets, experiments become much harder to exploit. You’ll frequently find that improving one metric degrades another, making comparisons blurry and conclusions uncertain. Many experiments end up with ambiguous results that are hard to act on. A single number cuts through that ambiguity.

Your metric will depend on your task, but the principle is the same: collapse your evaluation to one number whenever the problem allows it. It’s not always possible, but when it is, it transforms the tuning process.

Set Batch Size First

Set batch size before anything else. Batch size is primarily a hardware efficiency decision, not a model quality decision.

Choose the largest batch size that maximizes GPU utilization.
The effect of batch size on optimization dynamics (larger batches → smoother gradients → potentially different optima) can almost always be compensated by adjusting the learning rate downstream. This means batch size doesn’t need to participate in the hyperparameter search — it’s a fixed precondition.

The notable exception: tasks where batch size has semantic meaning, such as contrastive learning, where the batch defines the ratio of negative samples. In those cases, batch size is a scientific parameter. For standard classification, it’s not.

The Ordering Rationale

The sequence in which you tune dimensions is not arbitrary : it’s chosen to minimize cascading retuning. The principle: tune the most globally impactful decisions first, and the most dependent decisions last.

Why does this work? Hutter et al. (ICML 2014) showed through functional ANOVA decomposition that even in high-dimensional HP spaces, most performance variance is explained by just a few parameters [4]. Bergstra and Bengio (2012) confirmed this with their effective dimensionality finding: for most tasks, only a small subset of hyperparameters truly matter, but which ones matter varies by task [2]. Wang et al. (2025) measured this directly on CNN image classification: architectural depth alone explained 38.5% of performance variance, learning rate 22.8%, dropout 13.1%, batch size just 1.5% [5]. Their Grouped Sequential Optimization Strategy, tuning groups in decreasing importance order, achieved a 32% reduction in total optimization time versus simultaneous search, with equivalent final accuracy.

The practical consequence is what we call cascading invalidation: changing a high-impact parameter shifts the optimal values of everything downstream, but not vice versa. Tuning in the right order means later decisions rarely invalidate earlier ones.

Architecture first. Architecture determines the entire function class, computational budget, and memory footprint. Changing the backbone invalidates optimizer settings, scheduler parameters, and regularization, everything downstream must be re-tuned. This is why Wang et al. found it explains the most variance: it’s the most global decision. Our approach: try the largest model that fits on your GPU for each architecture family you’re considering, then scale down if needed. This gives each family its best shot.

Optimizer second. The optimal learning rate for SGD (typically 1e-2 to 1e-1) differs from Adam’s (typically 1e-4 to 1e-3) by several orders of magnitude [1]. As the Google Tuning Playbook states: switching optimizers after tuning the learning rate wastes all prior LR work, the learning rate is effectively a different parameter for each optimizer. Fix the optimizer early, and everything you build on top stays valid.

Scheduler third. Scheduler parameters (warmup length, decay rate, restart multiplier) are tightly coupled to the optimizer and learning rate; they’re essentially a more granular tuning of the learning dynamics. Karpathy’s advice is emphatic: disable learning rate decay entirely until you’ve established the base learning rate, because the best schedule is sensitive to nearly every other optimization choice [6]. The Google Playbook recommends starting with a constant LR, then adding cosine or linear decay only once the base rate is locked [1].

Regularization last. Three reasons:

• First, regularization addresses the gap between training and validation metrics — you can only meaningfully measure that gap once architecture, optimizer, and scheduler are stable. Tuning regularization earlier means optimizing against a moving target.
• Second, large learning rates themselves act as implicit regularizers through optimization noise (Smith, 2018) [7]. The optimal dropout and weight decay values therefore depend on your learning rate: tune regularization before LR is set, and your results are misleading.
• Third, Hernandez-Garcia & König (2018) showed that data augmentation is robust to other HP changes, while weight decay and dropout are brittle, they can dramatically hurt performance if not carefully tuned in the context of the final configuration [8]. Regularization is the most dependent dimension, which is exactly why it comes last.

One interaction deserves special mention: learning rate × weight decay. Loshchilov & Hutter (2019) proved that L2 regularization and weight decay are not equivalent for Adam, which motivated AdamW [9]. More subtly, with BatchNorm layers, weight decay’s primary effect is modulating the effective learning rate rather than providing traditional regularization [10]. This means weight decay tuned before the learning rate is set will produce doubly misleading results, it’s partially an LR modifier masquerading as a regularizer.

Sources:
[1] Google Research Deep Learning Tuning Playbook ·
[2] Bergstra & Bengio, 2012 (JMLR) ·
[4] Hutter, Hoos & Leyton-Brown, 2014 (ICML) ·
[5] Wang et al., 2025, Grouped Sequential Optimization Strategy (arXiv:2503.05106) ·
[6] Karpathy, A Recipe for Training Neural Networks ·
[7] Smith, 2018, A Disciplined Approach to Neural Network Hyper-Parameters (arXiv:1803.09820) ·
[8] Hernandez-Garcia & König, 2018 (arXiv:1806.03852) ·
[9] Loshchilov & Hutter, 2019, Decoupled Weight Decay Regularization (ICLR) ·
[10] van Laarhoven, 2017, L2 Regularization versus Batch and Weight Normalization (arXiv:1706.05350)

The Iterative Loop: Building Systematically

You can’t tune everything at once, the search space is too large. A 5-dimensional search with continuous ranges requires orders of magnitude more trials than a 2-dimensional one. The solution is sequential: fix one dimension, tune the next, build on previous results.

The rest of this article applies that principle to a concrete case: Heuritech’s image classification pipeline. Our computer vision stack detects garments in social media images, then classifies their attributes through a set of hierarchical classifiers : one model identifies the type of garment (jacket, dress, top…), others label its texture, silhouette, pattern, color, and finer attributes like neckline shape or sleeve length. These predictions combine to define the thousands of fashion trends we track and forecast.

In total, we retrained 13 production classifiers spanning diverse visual tasks: outerwear type, dress type, shoe type, bag model, gender, brand logo, fabric, and more. Each task has different class distributions, visual complexity, and sensitivity to hyperparameters, which is precisely what makes them a good stress test for a general-purpose methodology.

One crucial design choice: We ran this entire progression on a single dataset, outerwear classification. The goal was never to find the best configuration for outerwear specifically. It was to find a general-purpose configuration that would work out of the box across all 13 production tasks, without any per-task tuning (our compute budget couldn’t afford tuning all 13 datasets directly). This is a much harder constraint, and it influenced decisions throughout. The other 12 datasets were held out entirely: the methodology was blind to them until final evaluation.

All results below are reported in p@r60 (precision at 60% recall): the precision our classifier achieves when recall is held at 60%. We need a recall floor for coverage, and we optimize precision above that floor : a single number, following the principle described earlier in this article.

As a reference point: DINOv3-ViTS16 with default PyTorch training parameters scored 0.650 p@r60 on our outerwear test set. Every phase below builds from there.

We followed a 4-phase progression (after the batch size and metric foundations described above), each phase building on the previous one’s best configuration:

Phase 1: Backbone Selection → p@r60 = 0.676

Scientific parameter: backbone architecture.
Nuisance: learning rate.
Fixed: constant scheduler, SGD optimizer, no regularization.

Result: DINOv3-ViTS16 selected as the foundation. Starting metric on our outerwear classification task: 0.676 p@r60, by far the best option among all the tested architectures!

Phase 2: Optimizer Choice → p@r60 = 0.692 (+1.6%)

Scientific parameter: optimizer (SGD, Adam, AdamW, Muon).
Nuisance: learning rate, betas, momentum, with different ranges per optimizer.
Fixed: DINOv3, constant scheduler, no regularization.

Result: Four optimizers, four different stories:

A quick note on experiment counts: the number of trials per optimizer is proportional to its nuisance dimensions, following the guidelines from the quasi-random sampling section. SGD has one nuisance parameter (learning rate), so it got fewer trials. Adam has three (learning rate, beta1, epsilon), so it required more trials to cover its larger search space. This isn’t unequal effort — it’s appropriately scaled effort. Each optimizer gets the budget its search space demands.

For SGD, we ran 4 experiments because we only had one nuisance parameter. SGD achieved the highest single-configuration score, but it had a very narrow optimal region, the peak was sharp and easy to miss:

Adam had more nuisance parameters to tune, so 15 experiments were needed. It scored lower but had a broad, flat optimal region of good learning rates:

Since we needed a single configuration that works across 13 classification tasks (shoe type, bag model, dress type, gender, etc.), Adam’s robustness mattered more than SGD’s peak performance. A configuration that’s 2% worse but works across all tasks beats one that’s 2% better on one task but fails on others.

Adam sensitivity analysis:

Recommended: Adam with lr=5e-4, beta1=0.86, epsilon=1e-3, beta2=0.999.

Phase 3: Scheduler Choice → p@r60 = 0.722 (+3.0%)

Scientific parameter: scheduler type (constant, linear decay, cosine annealing).
Nuisance: scheduler-specific parameters — constant LR has just the learning rate, while cosine annealing has warmup epochs, restart multiplier, decay length, and more.
With so many nuisance parameters, we increased the number of experiments to 30.
Fixed: DINOv3, Adam.

Result: Cosine annealing was the clear winner at 0.722 vs. constant LR at 0.699. The gain was not free, it required more epochs (training went from optimal at epoch 2–3 to optimal at epoch 10–20). But 3% relative improvement justified the additional compute (internally we mostly care about inference-time compute).

The first round of cosine experiments revealed an important diagnostic: the best configurations clustered at the boundary of our search space (especially for learning_rate), meaning we hadn’t explored far enough:

After extending the search space and doing our 30 experiments, we had a clearer picture, though the full scatter is dense. Filtering to the top 8 runs (>25% of our 30 experiments) makes the sweet-spot regions visible :

Recommended scheduler: Cosine Annealing with:

Phase 4: Regularization → p@r60 = 0.727 (+0.5%)

Scientific parameter: regularization strategy.
Nuisance: dropout, weight decay, image size.
Fixed: DINOv3, Adam, cosine annealing.

Result: Data augmentation did not improve results for our tasks — the sweet spot was simply dropout=0.10, image_size=224, weight_decay=0. The key takeaway: not every addition of complexity helps. Regularization should be validated, not assumed.

Regularization comes last for a reason: it addresses the gap between training and validation metrics. You can only meaningfully measure and reduce that gap once everything else — architecture, optimizer, scheduler — is stable. Tuning regularization earlier would mean optimizing against a moving target.

The Compound Effect

Each phase’s improvement was modest individually: +1.5%, +3%, +0.5%. But they compounded:

And that was just outerwear. On other tasks, the gains were larger: up to +14.5% on dress type classification.

The 5-Phase Experiment Workflow

The methodology above tells you what to tune. But in practice, even well-designed experiments fail: wrong YAML configs, unlogged parameters, GPU crashes wasting hours of compute. We developed a 5-phase workflow to make experiments reliable:

Phase 1: Collaborative Design

Start with a research question: “Does label smoothing improve DINOv3 accuracy?” Define the scientific parameter (label_smoothing, range 0.0–0.2), nuisance parameters (dropout, data augmentation), and fixed parameters (everything else). Research the literature for informed range choices.

Phase 2: Configuration Generation

A generation script creates all experiment configurations using quasi-random sampling. Critically, it also generates a TEST configuration: a single config with max_epochs=3, limit_train_batches=10, designed to validate the setup in minutes rather than hours.

Phase 3: Pre-Launch Verification

Before burning any real compute:

1. Verify hyperparameter logging: Run a verification script against the TEST config to confirm all scientific and nuisance parameters will appear in your experiment tracker. We use TensorBoard’s hparams tab: it logs every hyperparameter alongside training metrics, making post-experiment analysis possible. If a parameter isn’t logged, you can’t analyze it.

2. Verify code state: git status and git log to ensure you’re running committed code. Nothing is worse than discovering a great result came from uncommitted changes you can’t reproduce.

3. Run the TEST config: Launch the TEST yaml on a single GPU. This catches YAML syntax errors, missing dataset paths, shape mismatches, and configuration bugs. Catching these before launching 30 full experiments saves days.

Phase 4: Documentation

Write the experiment hypothesis before seeing results. This prevents post-hoc rationalization, the tendency to explain whatever result you got as if you expected it.

Our experiment documentation template includes:

• Objective (the research question)
• Hypothesis (what we expect and why)
• Configuration (dataset, model, metric, trial count)
• Parameter tables (scientific, nuisance, fixed — with ranges and rationale)

Phase 5: Launch and Monitor

After TEST validation passes, launch the full experiment. Distribute across available GPUs, monitor training curves for early divergence, and document results while they’re fresh.

The discipline of this workflow is what makes systematic tuning practical. Without it, the friction of running rigorous experiments pushes teams back toward ad-hoc tuning.

The Results: From Matching TensorFlow to Beating It

After applying this methodology across our full production workload, here are the results. Each task is a different image classification problem evaluated on precision-at-recall-60.

The TF Baseline column is not default TensorFlow, it’s the best TensorFlow configuration our team had refined over years of production use, based on an Xception backbone. This is the bar we needed to clear:

A few things to note:

Not every task improved : Brand logo and shoe type showed regressions of -2.9% and -1.8%. This is expected: the entire methodology was tuned on a single dataset and applied blindly to 12 others.
The fact that 11 out of 13 tasks improved (with an average gain of +4.6%) validates the general-purpose approach. But the two regressions point to the natural next step: now that we have well-characterized search spaces and a solid baseline configuration, we can run task-specific optimization on the underperforming tasks.
Data augmentation — which didn’t help on outerwear — may matter for brand logos, which have fundamentally different visual patterns. And this is precisely where Bayesian optimization earns its place: the exploration phase is done, the search spaces are mapped, and tools like Optuna can now squeeze out per-task gains within those spaces without risk of optimizing blindly.

Remember the starting point. On gender classification, our TensorFlow baseline (Xception backbone, refined over years) scored 82.5%. A naive PyTorch port with the same Xception architecture and default parameters scored 70.5% — a 12pp gap that made the migration look like a regression.

The final result — 91.0% — came from two compounding improvements: switching to a DINOv3 backbone (the architecture phase), and systematically tuning the training configuration (the methodology phases). Neither alone would have gotten there. The architecture gave us a better starting point; the methodology ensured we actually found its optimal operating point.

This is the broader lesson: an architecture upgrade without proper tuning leaves performance on the table, and tuning without the right architecture has a lower ceiling. The methodology’s value is making sure you reach whatever ceiling your architecture allows.

A telling sign of how unexpected these results were: when first presented, the team suspected a measurement error. The improvements, particularly the 10pp jump on gender classification, seemed too large to be real. This is exactly why the phased verification methodology matters: every result was traceable to a documented experiment with logged hyperparameters, committed code, and reproducible configurations. The skepticism was healthy; the evidence was airtight.

Making It Your Own

This methodology is framework-agnostic. The same principles apply whether you’re tuning PyTorch, TensorFlow, JAX, or anything else, and whether you’re recovering from a migration or starting from scratch.

Here’s the distilled framework:

The tools we built (Halton sequence generators, YAML configuration scripts, hyperparameter verification scripts, experiment documentation templates) are not complex, and the heavy lifting still runs on standard PyTorch. The value isn’t in the tools, it’s in the discipline of using them consistently.

Sources:
[1] Google Research Deep Learning Tuning Playbook :https://github.com/google-research/tuning_playbook
[1b] Goodfellow, Bengio & Courville, Deep Learning, Ch. 11.5: Debugging Strategies : https://www.deeplearningbook.org/

The Bottom Line

An experienced engineer’s intuition about learning rate might be right 80% of the time. But across 6 hyperparameters, 4 architectural choices, and 13 classification tasks, that 80% accuracy compounds to less than a coin flip. Systematic methodology beats intuition not because intuition is wrong, but because the investment in systematic methodology delivers compounding interest after each new experiment.

This approach gave us an average +4.6% over configurations our team had refined for years, not from a single insight, but from the compound effect of many small, rigorous, well-validated improvements. The two tasks that regressed (−2.9% on brand logos, −1.8% on shoe type) point to the natural next step: now that search spaces are mapped, Bayesian optimization can squeeze out per-task gains where the general-purpose config falls short.

Now it’s your turn to share: what’s your tuning methodology?

I’m particularly curious whether you’ve found techniques that transfer reliably across diverse tasks, the interaction between optimizer choice and task characteristics is something we’re still investigating.

Louis Develle is an ML Research Engineer at Heuritech, where he led the TensorFlow-to-PyTorch migration and designed the systematic tuning methodology described in this series.

Stop Guessing: A Systematic Methodology for Tuning Deep Learning Models was originally published in Heuritech, a Luxurynsight Company on Medium, where people are continuing the conversation by highlighting and responding to this story.