Synthetic Data for Virtual Try-On
It’s been quite a year for the Virtual Try-ON (VTON) team here at Glance. We’ve presented a paper at SynlRgy (https://synirgy-workshop.github.io), put the data on our Hugging Face account, shared details of a qualitative study on user attitudes to VTON at Fashion — Textiles — Wearables in the UK, amongst a host of other VTON related items.
We wanted to shine a light on the path that got us to this point across a series of blog posts, and today we will start with the driving factors behind creating our VTON datasets.
What is VTON?
The world of online fashion retail began to take off in the late 1990s and 2000s. However, the inability of customers to physically try on garments before making a purchase has remained a persistent obstacle. This limitation continues to drive high return rates and customer drop-off in the purchase funnel. Virtual Try-On (VTON) technology offers a significant improvement in the current experience by enabling customers to generate a virtual image of themselves wearing a garment selected from an online catalogue. VTON image generation involves realistically warping the garment image, preserving the garment’s pattern, texture and drape while shaping it to the customer’s body pose and proportions.
The data used to train these models is typically in the form of multiple sets of image pairs, comprising of a photo of the garment on its own and a person wearing the garment.
When we started our work, it became clear that while there are many wonderful datasets powering the state-of-the-art, the vast majority of these come with restrictive licensing/usage. They often have no path to license the data due to the nature of how the datasets were constructed by the community and third-party providers. Models trained with these datasets by extension are unsuitable for commercial use.
We also found that they typically don’t provide a good representation of people in the real world. For example, some datasets show a bias towards white, lean, elegant, model style physiques and often focused on women.
At Glance we strive to build for everyone, all body shapes and sizes, all skin tones and ethnicities. Everyone should be able to enjoy Glance images that truly capture the likeness of the individual.
Our goals for VTON went beyond the data that was available, so it became clear that we would need to build our own datasets to realise our ambitions.
Dataset Creation
VTON models are typically trained on paired images: a standalone product image of a garment and an image of a person wearing that same item. Creating these pairs manually at scale is a significant undertaking. Each garment must be sourced, catalogued and photographed before being matched with a model, styled correctly and captured under controlled conditions. Producing a diverse dataset also requires variation across body types, poses, backgrounds and garment categories, multiplying the number of shoots, and the associated costs of samples, models, photographers, studios and post-production. Even with substantial investment, it can be difficult to achieve both the scale and breadth needed for a robust model. Synthetic data offers a compelling alternative: rather than photographing every garment–person combination, it may be possible to generate training pairs programmatically, reducing the cost of data creation while enabling much greater control over the dataset’s size and diversity.
However, generating useful synthetic pairs introduces a different challenge. Modern image-generation models are highly effective at producing plausible garments that look similar to a reference, but visual similarity is not enough for VTON training. The garment shown on the person must correspond precisely to the standalone product image, preserving details such as shape, colour, texture, pattern, logos and construction. Even subtle changes, an altered neckline, misplaced graphic or simplified fabric pattern, can break the integrity of the pair and introduce unreliable supervision during training. The central challenge, therefore, is not simply to generate a convincing image of someone wearing a similar garment, but to generate one in which the identity of the original garment is preserved exactly.
The Synthetic Advantages
If the challenge of garment fidelity can be addressed, synthetic data offers an important advantage: control. Real-world datasets reflect the products, models and photography styles that happen to be available, often producing uneven coverage across garment colours and categories. A synthetic pipeline makes it possible to deliberately rebalance these distributions, for example, by increasing the representation of colours that appear less frequently in the original data. This could help prevent a VTON model from performing well only on the most common products while struggling with rarer shades, patterns or garment types.
The same principle applies to the people represented in the dataset. Commercial imagery often concentrates around a relatively narrow range of body shapes and sizes, leaving models with less evidence from which to learn how garments should drape and fit across a broader population. Synthetic generation provides a way to target these gaps intentionally, increasing representation of body types, sizes, skin tones and other characteristics that may be underrepresented in the source data. Crucially, this coverage can be planned and measured rather than left to chance.
Synthetic data can also introduce greater variation in pose, viewpoint and presentation. A garment might be shown on people standing, sitting, turning or holding their arms in positions that are uncommon in conventional product photography but challenging for VTON systems. By generating examples around known weaknesses or rare scenarios, the dataset can evolve alongside the model. Synthetic generation is therefore not only a route to more data; it offers a way to create a more balanced, diverse and purposeful training set.
The Synthetic Disadvantages
Synthetic data also introduces limitations. Its realism is bounded by the capabilities of the underlying image-generation model, which may produce subtle anatomical errors, unnatural fabric behaviour or inconsistencies in lighting and texture. Even when individual images appear convincing at first glance, a dataset composed primarily of generated examples can retain a recognisably artificial quality. If these artefacts occur repeatedly, a VTON model may learn patterns that do not transfer reliably to real-world product photography.
Pair fidelity remains an especially important concern. A generated image may depict a garment that resembles the reference while changing details that should remain constant, such as the cut, colour, sleeve length, fastenings or material. These mismatches weaken the relationship between the two images and can introduce noisy or misleading supervision into the training process. Scaling synthetic generation is therefore only valuable if the quality and consistency of the resulting pairs can also be maintained.
The problem becomes more pronounced for garments with complex visual details. Intricate patterns, fine textures, typography and brand logos can be distorted, misplaced or replaced entirely by generative models. Repeated motifs may lose their structure, and designs can change as they wrap around the body or become partially occluded. Synthetic data should therefore not be treated as a straightforward replacement for real imagery: its usefulness depends on careful generation, validation and filtering, as well as a clear understanding of where the underlying models remain unreliable.
Our Approach
Having worked out why synthetic data was worth pursuing, and understanding its risks, we set out to actually build it. That meant creating a pipeline that could generate large numbers of garment-and-person image pairs, checking automatically that each pair was good enough to use, and producing all of the extra information a VTON model needs to learn from an image, such as where a person’s joints are, or which parts of the image belong to which piece of clothing.
Getting from a promising idea to something we were confident releasing publicly took a lot of trial and error. We worked on the problem iteratively: generate a batch of data, train or test a model on it, look closely at where it fell short using both metrics and human assessment, and then use what we learned to inform the next round of generation.
Generating the Pairs
The first question we had to answer was how to generate a well-matched pair of images. We tried asked a VLM to generate the garment on its own, then separately to generate a photo of a person wearing the same garment. We quickly found this two-prompt approach didn’t work well: generating the two images independently meant colours shifted, patterns got simplified, and small details disappeared between the two images, so the garment image no longer matched the one shown on the person.
Instead, we asked the model to generate the garment and the person wearing it alongside within a single image, so the VLM would have a better chance of keeping each view consistent. Our first attempts, using Google’s Imagen-3–002 and Imagen-3-fast-001 models, only produced usable pairs around half of the time. We had more success with Flux Dev, an image generation model with a licence that allows commercial use. Flux Dev lets us describe an image with two separate prompts at once:
- A short CLIP prompt (77 tokens) that sets the scene, lighting and pose.
- A much longer T5 prompt (512 tokens) that describes the garment itself in fine detail, right down to zips, stitching and graphics.
Splitting the description this way lifted the proportion of good, consistent pairs to somewhere between 70 and 75 percent.
Get Paul Duff’s stories in your inbox
Join Medium for free to get updates from this writer.
We also made some deliberate trade-offs along the way. We experimented with a more classical computer-vision approach to spotting pattern mismatches using fiducial markers, but ran into difficulty reliably locating the same region of a garment across two different images once lighting, warping and pose got involved. A general-purpose multimodal model turned out to generalise better with less engineering effort. On the generation side, we also tried switching to lower FP16 precision to speed things up. This worked but visibly reduced output quality, so we kept full FP32 precision and instead accelerated image generations using ParaAttention, which caches some of the model’s internal calculations for later reuse. That cut generation time by around 38%, from 37 seconds down to 23 seconds per image on an A100 GPU.
Checking Every Pair
Generating a plausible-looking pair is only half the job. At the volumes we were working with — well over ten thousand pairs — there was no realistic way to check each one by hand. So, we built an automated checking step powered by Gemini-2.0-flash-exp, a multimodal AI model that can look at images and reason about them. Rather than asking it for a single yes or no answer, we walk it through the same kind of reasoning a human reviewer might use:
- Examine each image and describe what it sees, independently, across a long list of specific features (zips, pockets, stitching, colours, textures, patterns, collars, cuffs, graphics and more).
- Rate how similar those descriptions are to one another, feature by feature.
- Write a short overall assessment of the differences it observed between the two images.
- Critique and reflect on its own findings before giving a final answer.
We deliberately tuned this process to be cautious, preferring to throw away a good pair rather than risk keeping a bad one, and it took several rounds of tuning to get that balance right. Tested against a golden set of 2,000 human-checked pairs it reached 88.6% accuracy, 89.9% precision and 93.4% recall. This automated checking turned out to be one of the most important pieces of the whole pipeline: without it, generating pairs at this kind of scale wouldn’t have been practical.
Pairs that pass the check are then enriched with the extra information a VTON model needs to learn from, including:
- Pose estimates (using Detectron2), marking out where a person’s joints and limbs are.
- A DensePose map, essentially a body-shape and orientation guide.
- A human parsing segmentation mask, showing which pixels belong to which item of clothing versus skin, hair or background.
- A short text caption describing the garment itself.
Building in Diversity
None of this on its own guarantees a diverse dataset. Generation and checking only tell you whether an individual pair looks convincing, not whether the dataset as a whole reflects the range of bodies, poses and colours we need it to have. So, alongside generation, we deliberately asked for variation across several axes at once:
- Pose (leaning, hand on hip, clasped hands, and other positions uncommon in typical catalogue photography).
- Body type, including plus-sized, curvy, athletic and muscular builds.
- Skin tone, hairstyle, age and ethnicity.
We started by replicating results from the IDM-VTON and VITON-HD papers as a real-world baseline, then began replacing an increasing share of that training data with our own synthetic pairs. A first blend, using around 10% synthetic data alongside real VITON-HD samples, integrated cleanly and matched the real-data baseline almost exactly. Moving to 100% synthetic data, however, produced VTON outputs with a subtle but consistent distortion around the shoulders. Tracing the cause led us back to a lack of variety in the body shapes used to generate the training pairs; the original VITON-HD dataset, it turned out, had contained enough natural variation to avoid the problem. Once we injected pose and body-type variation into the generation stage instead, the distortion disappeared, and two standard image-quality metrics improved alongside it: FID moved from 8.630 to 7.943, and LPIPS from 0.155 to 0.134 (lower is better for both). It was a clear demonstration that dataset diversity is paramount to how well the resulting model actually performs.
Another exercise with colour told a similar story. By looking at which hues were common across our dataset, we found we were under-representing greens and purples relative to real-world catalogues, which was causing clothes in those colours to render with unnaturally strong saturation. Generating more examples specifically in those hues, this time using a different model called CatVTON-Flux, fixed the problem.
How Well Did It Work?
To check how well our approach worked we trained two different VTON architectures on our datasets. IDM-VTON was our primary model throughout the project, giving us a consistent basis for comparison as the dataset evolved. We also trained CatVTON-Flux, a newer, simpler architecture that skips the need for pose and text inputs altogether by combining the garment and person images directly and found it noticeably better at reproducing fine details like buttons, logos and patterns, details that matter significantly when a customer is trying to judge whether a garment is really “them”.
A few of the numbers behind these experiments are worth sharing directly:
- A modest blend of just 10% synthetic data alongside real VITON-HD samples already matched, and in some respects slightly improved on, a real-data-only baseline (FID 5.86 versus 6.29; LPIPS 0.098 versus 0.10), at comparable SSIM and CLIP scores.
- Moving to 100% synthetic training initially regressed quality overall (FID 8.63), for the shoulder-distortion reasons described above. Injecting diversity closed most, though not quite all, of that gap (FID 7.94), highlighting a limitation of taking a fully synthetic approach at this stage.
- Once extended across all four garment categories, per-category scores were strong and reasonably consistent: FID ranged from 3.09 (men’s lower wear) to 4.94 (women’s lower wear), SSIM between 0.82 and 0.95, and CLIP between 0.97 and 0.99.
Metrics like FID, SSIM, LPIPS and CLIP were essential for this kind of benchmarking, and for tracking whether a change had helped or hurt overall. However, we found they were not sensitive enough on their own to catch some of the finer mistakes, like a shirt losing a button or a logo becoming slightly misaligned. So human judgement remained essential throughout: we had multiple people independently review batches of a few hundred outputs at a time, record their reasons for accepting or rejecting each one, and then combined their notes with automated summaries and group discussion to work out what needed further attention. Although more labour-intensive this consistently helped us spot systemic issues faster than the metrics alone would have. We think building better automated ways to catch this kind of fine-grained error remains an open problem worth solving.
Extending Coverage
Having built a reliable pipeline, we used it to help close some gaps in public VTON data: menswear, and lower-body garments for both genders, categories that are consistently under-represented in datasets like VITON-HD. For each new category, we identified around 19,000 existing products from third-party catalogue metadata, generated textual descriptions for them, and produced roughly 18,500 additional training pairs using the same generation and moderation pipeline described above. This is a fairly direct illustration of the advantage synthetic data offers: once the pipeline exists, extending it to a whole new category is mostly a matter of pointing it at a new set of products, rather than organising a new photoshoot from scratch.
Releasing the Data
Taken together, this work convinced us that synthetic data, generated carefully and refined iteratively rather than produced in one pass, can be a good substitute for the expensive, restrictive datasets that have limited virtual try-on research and development until now. Having benefited from existing research and datasets we wanted to give back to the VTON community, so we have shared our synthetic datasets publicly and for free on Hugging Face (https://huggingface.co/datasets/glanceai/VTON-Synthetic-Pairs). We’ll shortly be publishing a second post going into more detail on exactly what we released, why we structured it the way we did, and how you can start using it.
