<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dani Solis on Medium]]></title>
        <description><![CDATA[Stories by Dani Solis on Medium]]></description>
        <link>https://medium.com/@danysolism?source=rss-34eebc052fa3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sgHz_hIkXSPQwI9JEQUSVg.jpeg</url>
            <title>Stories by Dani Solis on Medium</title>
            <link>https://medium.com/@danysolism?source=rss-34eebc052fa3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:01:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danysolism/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Under the hood of Picnic’s demand forecasting model: A Deep Dive into the Temporal Fusion…]]></title>
            <link>https://medium.com/picnic-engineering/under-the-hood-of-picnics-demand-forecasting-model-a-deep-dive-into-the-temporal-fusion-e281604d65a5?source=rss-34eebc052fa3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e281604d65a5</guid>
            <category><![CDATA[demand-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dani Solis]]></dc:creator>
            <pubDate>Tue, 23 Apr 2024 07:06:01 GMT</pubDate>
            <atom:updated>2024-04-23T07:06:01.554Z</atom:updated>
            <content:encoded><![CDATA[<h3>Under the hood of Picnic’s demand forecasting model: A Deep Dive into the Temporal Fusion Transformer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PmPUZhY5CTo-AP93PWSg-g.png" /></figure><p>Demand forecasting is critical for any retail company, but even more so for Picnic, an online supermarket selling perishable products with a just-in-time supply chain. This forecast is in the heartbeat of our operations. From ordering products from our suppliers to planning deliveries, every aspect of our app-only supermarket relies on the accuracy of our predictions. If we overestimate demand, we risk throwing away food. On the flip side, underestimating demand leads to disappointed customers who then need to drive to the nearest physical supermarket. Both scenarios we are determined to avoid.</p><p>Unfortunately, there is a catch: forecasting demand isn’t as straightforward as other regression problems. Our historical data isn’t just a timeline of past purchases; it’s a tangled web of actual demand and the constraints we faced in the past. Take Mondays, for example. If we did not have enough capacity to deliver to all customers, they might postpone their orders to Tuesday, skewing our perception of actual demand.</p><p>Yet, despite these difficulties, we’re up for the challenge of building the best forecasting models to ensure that our customers get exactly what they need when they need it. In this blog post, we will dive into the details of the model that is helping us achieve this goal: the Temporal Fusion Transformer (TFT).</p><h3>Our Journey Towards the TFT</h3><p>Before we dive into the technical details, let’s lay out the data and forecast requirements that guided our journey towards utilising TFTs.</p><p>When predicting demand for the next day, historical demand is one of the most common data sources we rely on. This data is a strong indicator of future demand, especially considering it’s a time series with visible trends over time. However, our forecasting ambitions extend beyond forecasting just one day ahead. We often need to predict demand for two, three, or even seven days into the future. This type of forecasting, where one predicts multiple time steps ahead, is known as multi-horizon forecasting.</p><p>Multi-horizon forecasting requires considering a broad set of variables of different nature. Apart from demand information, we wish to use relevant temporal information(variables that exhibit dynamic changes over time). It could be information unknown in the future, such as the weather, or known in both past and future, such as the day of the week or holidays (consider Christmas and Easter shopping). Lastly, we are also interested in non-temporal static variables, such as the city or number of households in the area, which offer valuable contextual information.</p><p>We initially employed gradient-boosting trees to forecast demand. These types of models show robust performance across various forecasting scenarios and allow using different types of features. However, they exhibited a significant drawback discovered during COVID-19: everyone suddenly started buying toilet paper! The model couldn’t extrapolate to values it hadn’t seen during training. As customer behaviour was radically changing, we couldn’t accurately predict demand. Because of this, we decided to look for a novel solution to address these challenges. Our exploration began with Long Short-Term Memory Networks (LSTMs), but we observed no significant improvement in performance. Thus, we delved into more sophisticated models, including <a href="https://arxiv.org/abs/1704.04110">DeepAR</a>. However, DeepAR’s architecture does not consider the wide variety of inputs commonly present in multi-horizon forecasting(static and time-varying variables). Therefore, we started looking into TFT.</p><h3>TFT Overview</h3><p>The Temporal Fusion Transformer model is a transformer-based neural network specifically designed for time-series forecasting. Google researchers introduced the network architecture in the paper <a href="https://arxiv.org/abs/1912.09363"><em>Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PsMPSZ9g5o76l4z6CBMTmA.gif" /><figcaption>Figure 1: Multi-horizon forecasting with static covariates and various time-dependent inputs. Figure from Google’s <a href="https://research.google/blog/interpretable-deep-learning-for-time-series-forecasting/">blogpost</a></figcaption></figure><p>The main characteristics of this model are the following:</p><ul><li><strong>Different feature types:</strong> It supports temporal and static features. It allows temporal data known only up to the present, such as demand, temporal data known into the future, such as weather forecast, number of confirmed customer orders, and static variables, such as city region and delivery period — morning or afternoon.</li><li><strong>Feature selection:</strong> A variable selection mechanism allows it to determine what variables are relevant at a specific moment in time. This enhances the mode’s ability to generalise by concentrating its learning capacity on the most significant features rather than overfitting to irrelevant ones.</li><li><strong>Temporal relationships:</strong> It has two different mechanisms to process temporal information. One focuses on short-term interactions, such as daily demand, special events, or unexpected supply chain disruptions, while the other identifies long-term dependencies, such as seasonal trends and payday patterns.</li><li><strong>Model Explainability:</strong> Common methods used to explain the behaviour of Deep Neural Networks have limitations with time series because they do not consider that the order of inputs matters. However, the TFT architecture allows us to interpret the reasons behind its predictions on a global scale.</li><li><strong>Prediction intervals:</strong> It can predict a range of target values instead of a point for each timestamp. It shows the output distribution and reflects the uncertainty at each timestamp.</li></ul><p>So, after seeing the most relevant aspects of this model, buckle up; let’s get technical and dive deep into its components.</p><h3>TFT Deep Dive</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*sSdtQZqNGdpQqvB4bfxZ1Q.png" /><figcaption>Figure 2: Top-level architecture of TFT, along with its main components. Figure from the <a href="https://arxiv.org/pdf/1912.09363.pdf">original paper</a> )</figcaption></figure><p>Figure 2 shows the architecture of the TFT. This network shares weights across the different components. Components that share weights have the same colour in the diagram. By sharing parameters across layers, one can reduce complexity and training time while still capturing complex patterns.</p><h4>Gated Residual Network (GRN)</h4><p>The gating mechanism is used throughout different components in the TFT. This adds adaptability to the model, allowing it to differentiate between inputs that require complex non-linear transformations and inputs that do not. With the activation function (Exponential Linear Units) the network efficiently selects inputs requiring either simple or complex transformations. Moreover, it has residual connections, meaning that the model can learn to skip inputs if needed, providing more flexibility.</p><h4>Static Covariate Encoders</h4><p>Unlike other time series forecasting architectures, the TFT is designed to integrate information from static metadata. It uses GRNs as static covariate encoders to generate four different context vectors, adding information across the following areas of the network:</p><ul><li>Temporal variable selection</li><li>Local processing of temporal representations in the Sequence-to-Sequence layer</li><li>Static enrichment of temporal representations</li></ul><h4>Variable Selection Networks</h4><p>In time series, we might have multiple features at hand, and their exact importance and contribution to the target are often unclear beforehand. TFT tackles this challenge by adding variable selection networks. These components not only provide information about which features are the most crucial for the prediction — <strong>explainability </strong>— but also help filter out any unnecessary noisy inputs that could potentially hinder performance.</p><p>There are three different selection networks implemented for each type of input: static covariates, past inputs (known and unknown) and known future inputs.</p><h4>Local Encoder Decoder</h4><p>When predicting demand, we work with time series; thus, the sequential order of our data matters. Previous information may provide context that helps us predict future steps. The context in forecasting is very sophisticated, as it involves understanding a wide range of factors, including short-term and long-term patterns. This layer focuses on understanding short-term/ local information.</p><p>The LSTM Encoder-Decoder component follows a Sequence-to-Sequence architecture. Such models generate an output sequence by predicting one token at a time. These models excel at encoding features to enhance short-term context awareness and reduce ambiguity. LSTMs have memory mechanisms to regulate the flow of information when processing sequences to facilitate “long-term memory”, yet they encounter challenges when processing lengthy sequences, leading to issues like the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing gradient problem</a>. Therefore, it’s better to rely on attention mechanisms for long-term dependencies, as explained in the next section.</p><h4>Temporal Fusion Decoder</h4><p>To learn temporal relationships, the temporal fusion decoder uses the output of three layers: the local encoder-decoder layer (short-term context), the Static Enrichment Layer (enrichment of temporal features with static information), and the Temporal Self-Attention Layer.</p><p>The self-attention layer employs an attention mechanism inspired by the visual attention observed in animals. This mechanism allows the model to focus on specific parts of the input sequence at each time step, similar to how animals focus on particular aspects of their visual inputs to generate appropriate responses (more on this <a href="https://medium.com/towards-data-science/transformer-models-how-did-it-all-start-2e5b385ddd93">here</a>). The self-attention layer enables the model to identify long-range temporal dependencies by learning where to allocate attention across the input sequence. Shared weights across all attention heads facilitate the tracing back of the most relevant values, ensuring interpretability. We can explain what the model is paying attention to when predicting. For instance, it might reveal that when forecasting demand, the same day of the week from the previous week holds greater significance than the day immediately preceding it. This understanding allows us to interpret and contextualise the model’s predictions, enhancing our confidence in its forecasts.</p><h4>Quantile Regression</h4><p>Instead of predicting a single estimate, the TFT employs quantile regression to generate prediction intervals. This involves simultaneously predicting various distribution percentiles (such as the 10th, 50th, and 90th percentiles) at each time step. Quantile regression is very useful in operational critical tasks since it provides a means to quantify the uncertainty associated with each predicted value, allowing for better decision-making in uncertain situations.</p><h3>How did we adopt the TFT?</h3><p>In our journey to predict demand accurately, we implemented two different TFT models. One predicts demand at an article level, for example, forecasting how many bananas our customers will need in the coming weeks. The other model operates at a delivery level, estimating the number of orders we’ll receive in the upcoming weeks.</p><p>We started out using <a href="https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html">Pytorch Forecasting</a> as it had a TFT implementation available. However, now we are considering changing to <a href="https://unit8co.github.io/darts/examples/13-TFT-examples.html?highlight=temporal%20fusion">Darts</a> as it is better maintained and it allows us to continue experimenting with other recent forecasting models.</p><p>During the adoption process, we faced several challenges. As the model was relatively new, there were limited online resources available to address our questions and guide us in effectively applying the model to our use case.</p><p>Moreover, when training the model, we faced challenges with memory usage. While the TFT allows you to provide extensive context from past timesteps in the training examples, it also results in significant increases in the size of your training data. To address this, we implemented sampling techniques.</p><p>Additionally, transitioning to a deep neural network requires more computational power. Unlike gradient boosting trees, which could be trained on CPUs, the TFT requires GPUs. Even with GPUs, training the TFT model takes several hours.</p><p>Despite the challenges we faced, we found the effort to be worthwhile. We saw a 20% increase in our forecast precision, particularly benefiting from improved performance in slow-moving articles, which are the most difficult to forecast. Furthermore, the model’s explainability helped us communicate the model’s behaviour to our stakeholders and guided our model development efforts. Thanks to it we could focus on improving the model in areas where we could deliver more value. Lastly, the prediction intervals allowed us to select different quantiles, effectively balancing waste and unavailability in our operations.</p><h3>Our learnings</h3><p>We did not want to finish this blog post without first sharing our key takeaways from our journey with the Temporal Fusion Transformer (TFT):</p><ul><li><strong>Embrace the Learning Curve</strong>: Implementing a complex model like TFT requires investing time to grasp its intricacies and tailor it to your needs, especially when it is driving critical operational tasks within your company; invest the necessary time to apply it to your use case properly.</li><li><strong>Consider Hardware Costs</strong>: Deep learning models like TFT demand significant hardware resources (GPUs), which can increase production costs. Make sure the benefits justify the investment for your particular use case.</li><li><strong>Start Simple</strong>: Begin with a strong baseline, potentially utilising simpler models like gradient boosting. In most cases, it may fulfil the majority of your requirements. Only pursue more advanced models if your use case really requires them. If you choose to do so, TFT can be a great option. It excels at understanding complex patterns while providing explainability.</li><li><strong>Stay Curious:</strong> The landscape of machine learning evolves rapidly, with newer models like <a href="https://arxiv.org/abs/2304.08424">TiDe</a> emerging. Continue exploring to find new ways of improving your forecasts.</li></ul><p>If you are interested in reading more about our demand forecasting models, check out our previous blog posts:</p><ul><li><a href="https://blog.picnic.nl/using-transformers-to-cut-waste-and-put-smiles-on-our-customers-faces-1ed2b638578f">Using Transformers to Cut Waste and Put Smiles on Our Customers’ Faces!</a></li><li><a href="https://blog.picnic.nl/running-demand-forecasting-machine-learning-models-at-scale-bd058c9d4aa7">Running demand forecasting machine learning models at scale</a></li></ul><h3>Join us</h3><p>If you share our passion and curiosity about how we plan to tackle this evolving challenge, we invite you to be a part of our team as a <a href="https://picnic.app/careers/all-jobs?search=machine+learning">Machine Learning Engineer</a>. Join us on this venture, contributing to the advancement of ML at Picnic by taking it to unprecedented heights!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e281604d65a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/picnic-engineering/under-the-hood-of-picnics-demand-forecasting-model-a-deep-dive-into-the-temporal-fusion-e281604d65a5">Under the hood of Picnic’s demand forecasting model: A Deep Dive into the Temporal Fusion…</a> was originally published in <a href="https://medium.com/picnic-engineering">Picnic Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformer models…How did it all start?]]></title>
            <link>https://medium.com/data-science/transformer-models-how-did-it-all-start-2e5b385ddd93?source=rss-34eebc052fa3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e5b385ddd93</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Dani Solis]]></dc:creator>
            <pubDate>Thu, 13 Aug 2020 08:48:36 GMT</pubDate>
            <atom:updated>2020-08-16T18:18:51.106Z</atom:updated>
            <content:encoded><![CDATA[<p>Transformer models have revolutionised the field of Natural Language Processing but, how did it all start? To understand current state-of-the-art architectures and genuinely appreciate why these models became a breakthrough in the field, we must go even further in time, where NLP as we know it started: when we first introduced Neural Networks in NLP.</p><p>The introduction of Neural models to NLP found ways to overcome challenges that traditional methods couldn’t solve. One of the most remarkable advances were Sequence-to-Sequence models: Such models generate an output sequence by predicting one word at a time. Sequence-to-Sequence models encode the source text to reduce ambiguity and achieve context-awareness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*duaY9-uSKYGprgtAK4qN2Q.png" /><figcaption>Sequence-to-Sequence models (image by author)</figcaption></figure><p>In any language task, context plays an essential role. To understand what words mean, we have to know something about the situation where they are used. Seq2Seq models achieve context by looking at a token level: previous word/sentences to generate the next words/sentences. The introduction to this representation of context embedded in space had multiple advantages such as avoiding data sparsity due to similar context data being mapped close to each other and providing a way to generate synthetic data.</p><p>However, context in language is very sophisticated. Most times, you can’t find context by only focusing on the previous sentence. There is a need for <strong>long range dependencies </strong>to achieve context awareness. Seq2Seq models work with Recurrent Neural Networks: LSTM or GRU. These networks have memory mechanisms to regulate the flow of information when processing sequences to achieve a “long term memory.” Despite this, if a sequence is long enough, they’ll have a hard time carrying information from earlier time steps to later ones.</p><p>RNNs will fall short when trying to process entire paragraphs of text. They suffer from the vanishing gradient problem. Gradients are values used to update the weights of a neural network and thus, learn. The vanishing gradient problem occurs when the gradient shrinks as it backpropagates through time. If a gradient value becomes extremely small, it doesn’t contribute much to learning. Moreover, RNNs’ topology is very time-consuming because, for every backpropagation step, the network needs to see the entire sequence of words.</p><figure><img alt="CNNs logarithmic path" src="https://cdn-images-1.medium.com/max/510/1*oKo25D45oKiAIIWvizLWaA.png" /><figcaption>CNN logarithmic path (image by author)</figcaption></figure><p>As a way to try to fix these problems, the use of Convolutional Neural Networks was introduced in NLP. Using convolutions to create a logarithmic path. The network can “observe” the entire sequence in a log number of convolutional layers. However, this raised a new challenge: positional bias. How do we make sure that the positions we are observing in the text are the ones that give more insights? Why focus on position X of the sequence and not X-1?</p><p>Besides, the challenge it&#39;s not only to find a way of encoding a large amount of text sequences but also to be able to determine which parts of that text are essential to gain context-awareness. Not all the text is equally important to gain understanding. To address this, the attention mechanism was introduced in Seq2Seq models.</p><p>Attention mechanism is inspired in the visual attention animals have were they focus on specific parts of their visual inputs to compute adequate responses. Attention used in Seq2Seq architectures seeks to give more contextual information to the decoder. At every decoding step, the decoder is informed how much “attention” it should give to each input word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*d9meOQXsxGUxtNotkgF86w.png" /><figcaption>Attention in Seq2Seq models (image by author)</figcaption></figure><p>Despite the improvements in context awareness, there was still a substantial opportunity to improve. The most significant drawback of these methods is the complexity of these architectures.</p><p>This is where the transformer model came into the picture. The transformer model introduces the idea of instead of adding another complex mechanism (attention) to an already complex Seq2Seq model; we can simplify the solution by forgetting about everything else and just focusing on attention.</p><p>This model removes recurrence, it only uses matrices multiplications. It processes all the inputs at once without having to process it in sequential manner. To avoid losing order, it uses positional embeddings that provide information about the position in the sequence of each element. And despite removing recurrence it still provides an encoder-decoder architecture such as the one seen in Seq2Seq models</p><p>So, after seeing all the challenges we face with previous models, let’s dive deep into what the transformer model solves in comparison to Seq2Seq models.</p><h3>Transformer technical deep dive</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*EGVB6fnFt4kikOR1gIXUOw.png" /><figcaption>Transformer architecture (image by author)</figcaption></figure><p>While RNN fell short when we needed to process entire paragraphs to gain context, transformers are able to identify long-range dependencies achieving context-awareness. We also saw that RNN by themselves struggle in determining which parts of the text give more information, to do so they needed to add an extra layer, a bidirectional RNN to implement the attention mechanism. On the contrary, the transformer works only with attention so that it can determine the essential parts of context at different levels</p><p>Another critical difference is that the transformer model removes recurrence. By eliminating the recurrence, the number of sequential operations is reduced, and the computational complexity is decreased. In RNNs for every backpropagation step, the network needs to see the entire sequence of words. In the transformer, all the input is processed at once decreasing computational complexity. This also brings a new advantage, we can now parallelise training. Being able to split training examples into several tasks to process independently boosts the training efficiency.</p><p>So how does the model keep the sequence order without using recurrence?</p><p>Using Positional embeddings. The model takes a sequence of n word embeddings. To model position information, a positional embedding is added to each word embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*GrzPC5qC1GtxgmPCDmYiCg.png" /><figcaption>Positional embeddings (image by author)</figcaption></figure><p>Positional embeddings are created using sine and cosine functions with different dimensions. Words are encoded with the pattern created by the combination of these functions; this results in a continuous binary encoding of positions in a sequence.</p><p>The transformer model uses multihead attention to encode the input embeddings, when doing so, it attends inputs in a forward and backward matter so the order in the sequence is lost. Because of this, it relies on positional embeddings that we just explained.</p><p>The transformer has three different attention mechanisms: the encoder attention, the encoder-decoder attention and the decoder attention. So how does the attention mechanism works? It is basically a vector multiplication, where depending on the angle of the vector one can determine the importance of each value. If the angles of the vectors are close to 90 degrees, then the dot product will be close to zero, but if the vectors point to the same direction, the dot product will return a greater value.</p><p>Each key has a value associated, and for every new input vector, we can determine how much does this vector relates to the value vectors, and select the closest term using a softmax function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GitcryJWzh6irenY95slAg.png" /><figcaption>Multi-Head Attention (image by author)</figcaption></figure><p>Transformers have a multihead attention; we can think of it as filters in CNN’s, each one learns to pay attention to a specific group of words. One can learn to identify short-range dependencies while others learn to identify long-range dependencies.This improves the context-awareness, we can understand what terms refer to when it’s not clear; for example, with words such as pronouns.</p><p>The transformer architecture facilitates the creation of powerful models trained on massive datasets. Even though it is not feasible for everyone to train these models. We can now leverage of transfer learning to use these pre-trained language modes and fine-tune them for our specific tasks.</p><p>Transformers models have revolutionised the field. They have excelled RNN-based architectures in a wide range of tasks, and they will continue to create tremendous impact in the area of NLP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e5b385ddd93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/transformer-models-how-did-it-all-start-2e5b385ddd93">Transformer models…How did it all start?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>