In a recent blogpost, Étienne David introduced how it is possible to forecast a fashion trend based on the time series representing its visibility on social media. In case you missed it, you can read it here.

I’m Paul Morel, Data Science Lead at Heuritech. In this article, I will present how we extract meaningful insights from social media images using Deep Learning. Together, we will illustrate the detection of a fashion trend using the example of metal studs jackets for men.

Computer Vision at Heuritech

Each fashion image on social media contain many layers of valuable insights. At Heuritech, our taxonomy enumerates more than 2,000 clothing attributes (including colors, shapes, types and other attributes) detected on more than 750 million images. By crossing these attributes, we designed a catalog of more than 85,000 fashion trends.

For our example trend, we need 3 distinct Deep Learning models:

• One detecting the fashion items, including the human and the jacket.
• A second detecting the clothing attributes of the jacket, including the metal studs.
• A third inferring the gender of the human.

To be able to do that, several considerations are key to understanding how we handle fashion attributes detection, which makes Heuritech unique.

Exploring fashion with AI and social media images

Defining and Predicting Specific Classes

Let’s focus on how we define and learn to predict a specific class for our clothing attributes detector, such as metal studs. This model performs image classification, identifying one or more classes among those it’s been trained to recognize. Most of Heuritech models are doing classification.

Acquire the right data, for the right classes

The first step is defining classes that make sense for a fashion expert. Keep in mind that these classes will be combined with other ones to define our fashion trends. These classes must be:

• Granular enough to define and distinguish trends properly, avoiding confusion.
• Exhaustive to cover all types and attributes of a given clothing while ensuring classes are mutually exclusive, i.e. there is no overlap in the classes definition.
• Visible enough to get decent volumes of high-quality samples for training.

One key challenge is avoiding false detections, such as confusing metal studs with rhinestones or pearls. Below are examples of attributes that must not be confused with metal studs.

One key challenge is avoiding false detections, such as confusing metal studs with rhinestones or pearls. Below are examples of attributes that must not be confused with metal studs.

We tackle this issue in two ways:

1. Learning these classes if they’re useful for detecting other trends.
2. Training the model with negative samples of metal studs. However, defining all possible classes a model might “see” is often impractical (too expensive in tagging as there would be too many classes), leading us to rely on open set learning. This field allows models to predict confidently even when encountering unforeseen classes.

Data quality is king and defining accurately the classes as well as finding the suitable training samples is a long, meticulous job performed by our Fashion experts. Shoutout to them!

Building a hierarchy of classes

Our classification models don’t simply only pick a single class from the list of trained classes. Many of our models involve both:

• Classifiers parts detecting single features (e.g., type of top).
• Multilabel identifying multiple attributes (e.g., sleeve length, neckline shape).

At the same time. Classes may also have hierarchical relationships. For instance, our clothing attributes model identifies specific denim types (e.g., bleach, tie-dye) while consolidating them under a broader “denim” class. If confidence is low, the model predicts only the broader class.

For metal studs, no complex hierarchy is required. However, if we wanted a broader trend like “jacket embellishments for men”, we could consolidate predictions of metal studs, pearls, rhinestones, etc., into a parent class.

Adjust the training process to fit our need

Now that we have the right data for the right model architecture, we train and evaluate the model. A classification model usually predicts a score for every single class between 0 and 1, representing its confidence in detecting the given class. During training, scores above 0.5 are considered detections, and the model is corrected if it makes errors. This threshold is later fine-tuned using a validation set to maximize performance metrics.

The threshold value impacts 2 important metrics, precision and recall:

• A higher threshold increases precision but lowers recall.
• A lower threshold improves recall but risks more false positives, harming precision.

Chosing this threshold value is a tradeoff that needs to be made for each class. The predictions of our models are aggregated to build trend lines, and we want them to be insightful while having enough volume so they can be statistically representative of the given trend.

At Heuritech, we value precision twice as much as recall. To balance this, we use the F-β score, an adjusted harmonic mean of precision and recall where β reflects the weight of recall. For instance, using β = 0.5 prioritizes precision twice as much as recall.

This score is computed for a wide range of possible threshold values, so we can use it to select the threshold that maximizes it.

Below curve illustrates how precision, recall, F-β and detected volume of metallic studs on clothes evolve with threshold changes, across our valdation set. The optimal threshold maximizes the F-β score while maintaining adequate detected volume.

Vertical plain and dashed lines represent 2 distinct values of threshold, the one from the model during a previous training (dashed line) and the latest one (plain line). By looking at the intersection with the colored lines, we can find the associated values of the various metrics measured (precision, recall, etc.). Doing so, we see that it is useful to slightly increase the former threshold from ~0.7 to 0.75 to maximize the F-β score.

We can notice here that the new threshold is the one where the F-β metric is the highest, although we could have better precision by increasing the threshold value. Yet, this would harm too much the predicted volume of detected metallic studs so it is preferrable to keep a non extreme threshold value.

Provided that we are able to ensure the model has good performance (which is no simple thing, as usual with Deep Learning), we should be good to go to start detection of metal studs.

Building a pipeline of model to track fashion trends

While we’ve discussed training a single class within a classification model, fashion insights require multiple models working together. Unlike trendy large foundation models, we use a modular pipeline:

• Image-wise models: Detect fashion objects (e.g., humans, tops, dresses) and classify context (e.g., outdoor vs. indoor, gym).
• Item-specific models: Analyze detected items for features like attributes, colors, and patterns. For instance, a top classifier detects sleeve length, neckline shape, and type. We also perform regression to spot the color of the clothes.
• Aggregation: Combine predictions to define trends, such as red V-neck shirts for women.

By aggregating the images and predictions for a specific geography, we can track a given fashion trends worldwide. Our modular design allows independent updates to specific scopes, such as adding new patterns without retraining unrelated models. Below image showcases the behavior of our deep learning pipeline.

Conclusion

Building deep learning models for fashion involves multiple steps and expertise. This article touched on:

• Training classification models.
• Avoiding common pitfalls like false detections.
• Using modular pipelines for modularity.

For the sake of simplicity, we didn’t cover many problems that can show up in the detection of trends, even the most obvious ones: how do we ensure the stability and improvement of a given model from one training to an other? How to control the impact of a model on the related trends? Etc.

If you are technical or simply curious about it, please let me know below if you are interested about a detailed article related to this below, by giving it a 👍 or letting me know in the comments below!

Also, we didn’t tackle the topic of analyzing hundreds of millions of images in reasonable amounts of time and costs, which is a major topic for our solution to work properly. Indeed, modularity is one thing, but our approach needs also to be scalable. This will be the topic of an upcoming article about Computer Vision Model Serving using NVIDIA AI Triton, stay tuned!

Last but not least, if you’re passionate about data science and looking for an exciting opportunity, we’re hiring! You can take a look at the job description here.

A social media image is worth a thousand fashion labels… was originally published in Heuritech, a Luxurynsight Company on Medium, where people are continuing the conversation by highlighting and responding to this story.