<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[DataNess.AI - Medium]]></title>
        <description><![CDATA[A space to share about data science related knowledge and expertise - Medium]]></description>
        <link>https://medium.com/dataness-ai?source=rss----b40f7eff31a4---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>DataNess.AI - Medium</title>
            <link>https://medium.com/dataness-ai?source=rss----b40f7eff31a4---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:25:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dataness-ai" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring BERT applications: BERTopic]]></title>
            <link>https://medium.com/dataness-ai/exploring-bert-applications-bertopic-dadd2714bc0c?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/dadd2714bc0c</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Asma Zgolli, PhD]]></dc:creator>
            <pubDate>Thu, 21 Aug 2025 10:14:17 GMT</pubDate>
            <atom:updated>2025-08-21T10:23:07.547Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Exploring BERT applications is the second part of our Transformers in NLP blog series. Read the first part <a href="https://medium.com/dataness-ai/exploring-bert-feature-extraction-fine-tuning-6d6ad7b829e7">here</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*TF_Ig3uUxAjYWeMcNS2u8Q.png" /></figure><p>Topic modelling is an NLP application that employs unsupervised machine learning techniques such as clustering to group similar words in a text. It uncovers semantic similarities in a document and extracts from them common themes [<a href="https://levity.ai/blog/what-is-topic-modeling#:~:text=Topic%20modeling%20is%20a%20type,predefined%20tags%20or%20training%20data.">1</a>]. These methods mainly help to categorize documents (such as comments and textual descriptions), discover hidden information or so-called themes and enable key-based search of these documents using those themes. Several techniques have been proposed to address the topic modeling challenge, such as LSA: Latent Semantic Analysis [<a href="https://towardsdatascience.com/latent-semantic-analysis-intuition-math-implementation-a194aff870f8">2</a>] and LDA: Latent Dirichlet Allocation [<a href="https://towardsdatascience.com/latent-dirichlet-allocation-lda-9d1cd064ffa2">3</a>]. With the rise of BERT as a powerful language model, BERTopic was developed to enhance and optimize topic modeling by leveraging its efficiency.</p><p>In the second part of our <strong>Transformers in NLP</strong> blog series, we explore one of BERT applications: BERTopic. We discuss this topic modeling technique and examin its practical use cases.</p><h3>BERTopic</h3><p>BERTopic [<a href="https://maartengr.github.io/BERTopic/index.html">4</a>] is a library that combines Transformers and a variation of TF-IDF to build clusters based on the most important words in a text and model them as topics.</p><p>Under the hood, BERTopic is a pipeline composed of the following building blocks (as shown in Figure 2 below).</p><h4><strong>Embedding extractor</strong></h4><p>This component transforms input text into a vector representation using a BERT model (e.g. SBERT). This technique is useful in topic modelling because it preserves the semantic relationships of words in a text. Read more about text embedding and the theories behind it in our <a href="https://medium.com/dataness-ai/exploring-bert-feature-extraction-fine-tuning-6d6ad7b829e7">previous blog</a>.</p><h4><strong>Dimensionality reduction with UMAP</strong></h4><p>Uniform Manifold Approximation and Projection (UMAP) is a dimensionality reduction technique that reduces the number of input variables to produce a more compact representation, which is useful for visualizing large and high-dimensional data. It is an alternative to algorithms such as t-SNE or PCA, and has been shown to produce better visualization quality and performance. Additionally, in the context of other machine learning applications, particularly in the BERTopic pipeline, UMAP can be used for feature selection, allowing the user to extract the most important and representative features from input data.</p><h4><strong>Clustering with HDBSCAN</strong></h4><p>HDBSCAN performs clustering on input data, represented as a vector array or a distance matrix, using the DBSCAN algorithm by varying the epsilon hyperparameter. It returns the clustering results that guarantee the best stability over epsilon, hence the name ‘hierarchical DBSCAN’. DBSCAN identifies clusters as dense, spatial regions. It first finds a dense area and then expands gradually to its neighbours until it builds a cluster. In this algorithm, epsilon represents the maximum distance between two points to consider them neighbours.</p><h4><strong>Bag-of-words extraction</strong></h4><p>This component extracts a representation of words from clusters of text generated by the previous model. It scores the presence of known words from a pre-defined vocabulary using either a count or frequency measure. The highly ranked words or phrases extracted from the input cluster are used to represent a cluster in this pipeline.</p><h4><strong>Topic representation with the class-based TF-IDF and MMR method</strong></h4><p>The C-tf-idf method aggregates all the documents belonging to one class together and calculates the tf-idf score of the terms belonging to those classes. The formula in Figure 1 is applied to calculate the tf-idf score of a term per class, where tf is the term frequency for a term x belonging to a class c, f is the frequency of x across all classes, and A is the average number of terms per class.</p><figure><img alt="Class-based tf-idf formula taken from BERTopic official documentation (https://maartengr.github.io/BERTopic/api/dimensionality/base.html)" src="https://cdn-images-1.medium.com/max/481/1*fzP6z1xTageKsYVqUkKcww.png" /><figcaption><em>Figure 1 Class-based tf-idf formula taken from BERTopic </em><a href="https://maartengr.github.io/BERTopic/index.html"><em>official documentation</em></a></figcaption></figure><p>This method extracts even more important words or phrases from the cluster. Combining clustering, bags of words and c-tf-idf allows us to obtain more compact topics. On the other hand, the Maximal Marginal Relevance (MMR) method calculates similarities of keywords from the vocabulary with the terms of the documents. This component in the BERTopic pipeline enables the generation of diverse topics that are semantically close to the information expressed in the documents.</p><p>This library is modular and aimed to be highly flexible and customizable. It’s possible to change the architecture of its pipeline by other building blocks that best suit the user’s use case.</p><figure><img alt="Modular architecture for BERTopic’s pipeline" src="https://cdn-images-1.medium.com/max/949/1*aT4jd7vcbdMUk9EB5v2hVA.png" /><figcaption><em>Figure 2 Modular architecture for </em>BERTopic<em>’s </em><a href="https://maartengr.github.io/BERTopic/index.html"><em>pipeline</em></a></figcaption></figure><p>Extracting topics from this library is as easy, as demonstrated in the following code snippet.</p><pre>from bertopic import BERTopic<br>topic_model = BERTopic()<br>topics, probs = topic_model.fit_transform(df[&quot;text&quot;])</pre><p>The hard work consists of cleaning and preprocessing the input text, fine-tuning the model and choosing the suitable parameters, and finally analysing the extracted topics (visually) and drawing insights from them. In the following section, we present a more advanced example of using BERTopic demonstrated on open-source data fetched from the following GitHub repository: <a href="https://github.com/linanqiu/reddit-dataset/tree/master">reddit-dataset</a>.</p><h3>Use case study</h3><p>This use case study aims to extract topics from Reddit comments and visualise those topics. The full code of this use case study is available on our GitHub repository: <a href="https://github.com/AsmaZgo/Transformers-in-NLP">Dataness.AI NLP repository</a>.</p><p>Before starting training the BERTopic we perform minimal text cleaning and preprocessing. For data collected from the net, this consists in removing useless HTML tags and removing bad language. The latter task is particularly important because our dataset is extracted from social media comments. The reason behind avoiding extensive text cleaning is that Bert needs the raw text to preserve the context. Lemmatization and other techniques are best performed in later stages of the BERTopic pipeline (during the vectorization step that follows the embedding, for example).</p><p>The first step in the topic modelling part is to initialize and train the BERTopic model using the train_transform method in BERTopic API. This method is responsible for the extraction of features from the documents and clustering them to identify representative topics.</p><p>The following code snippet builds a basic non-parametrised model.</p><pre>topic_model = BERTopic(calculate_probabilities=True,verbose=True)<br>topics, probs = topic_model.fit_transform(comments)</pre><p>The quality of the topics generated using this model is suboptimal! Many topics seem way too generic and non-representative (e.g. topic 0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*aot-AqzG2TfJRvTB9HfS9Q.jpeg" /><figcaption>Figure 3 Topics generated with the default BerTopic pipeline</figcaption></figure><p>The results obtained from generating topics for cleaned Reddit comments using the default configuration of BerTopic exhibit several issues. As shown in Figure 3, some topics consist of articles as keywords, and there are several non-representative keywords within the topics. This necessitates the selection of appropriate models and the fine-tuning of their hyperparameters.</p><p>In our efforts to enhance the results, we have incorporated additional configurations into the BerTopic pipeline. These include employing Keybert to refine and diversify the keywords within each topic, utilizing lemmatization techniques to eliminate redundant keywords, and experimenting with different models and hyperparameters in the pipeline. For example, we varied the number of neighbours in PCA and UMAP, the distance metrics used, the minimum cluster size in HDBSCAN, and the inclusion of n-grams in the count vectorizer.</p><p>To determine the optimal models and hyperparameters for the pipeline, we propose an iterative process involving various configurations and the evaluation of specific topic modelling metrics. These metrics serve as criteria for assessing the quality of the generated topics. They help identify the model that produces the most concise, human-understandable, and representative topics based on the analyzed input text. In other words, they aim to measure the coherence of the topics and their ability to encapsulate the meaning of various facts expressed in the source text within a single statement [<a href="https://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf">5</a>]. Examples of those metrics we find :</p><ul><li><strong>U_Mass Coherence</strong>: An intrinsic measure based on document co-occurrence. Scores are typically negative, and values closer to 0 (less negative) indicate better coherence.</li><li><strong>C_V Coherence</strong>: A composite measure based on sliding windows, word co-occurrence, and cosine similarity. Scores range from 0 to 1, with higher values indicating more coherent and interpretable topics.</li></ul><p>During the experimentation process that was performed in this use case study, we evaluated the following model configurations for the BerTopic pipeline.</p><ul><li>Feature extraction: for extracting embedding from the comments, we used (i) a sentence transformer model “all-MiniLM-L6-v2” which is typically used in BerTopic because of its capabilities to capture semantic similarity in documents and represent them in the embedding and (ii) TensorFlow universal sentence transformer which is optimized for generating embeddings from longer texts such as long sentences and paragraphs. Many other models can be used for this step out of the box like Spacy and Gensim models, further details are available on the <a href="https://maartengr.github.io/BERTopic/getting_started/embeddings/embeddings.html">official documentation</a>.</li><li>Dimensionality reduction: to represent the embeddings in a smaller vector space we evaluated the UMAP algorithm with a variation of the number of components (n_components), the number of neighbours (n_neighbours), the minimum distance between the vectors (min_dist) and the metric parameters as well as the PCA algorithm with a variation of the n_components parameter.</li><li>Clustering: similarly to dimensionality reduction, we varied different distances-related parameters compatible with the HDBSCAN and the K-Means algorithms.</li><li>Bag-of-words representation: for this step of the pipeline, we use the countvectoriser and vary the parameters: tokenizer and n_gram_range. Using the n_gram ranges (1,2) and (2,2) allows us to include in the keywords of the topics, longer composed words like :</li><li>Topic representation: we wish to diversify the keywords in each topic, for that reason, we use the KeyBERTInspired model.</li></ul><p>The next step consists of selecting the best BerTopic pipeline using the coherence metrics we presented earlier and human judgement after analyzing the visualizations produced by the top 5 models that had the maximum<strong> c_v_coherence </strong>and the <strong>u_mass_coherence </strong>closest to zero. The selected model produces the following results.</p><figure><img alt="Topic Word Scores" src="https://cdn-images-1.medium.com/max/1024/1*gjNmthLtRdsfTYArtKrW2Q.png" /><figcaption>Figure 4 Visualizing Topic Word Scores</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hZXjs8Z92Z6hDwmPZX4l6A.png" /><figcaption>Figure 5 Visualizing topic relationships with a similarity matrix</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YH-qPI_D3ODRkCtqZQCaCg.png" /><figcaption>Figure 6 Plotting hierarchical clustering of topics</figcaption></figure><p>The bar charts illustrate the top terms across different topics, helping us interpret the themes that emerged from the comments — for example, discussions about email, app folders, troubleshooting, or search queries. The score is calculated by the model showing how strongly each word contributes to a topic.</p><p>As for the heatmap, it shows how closely related different topics are in our improved BERTopic model using the previously described pipeline. The Darkest shades -along the diagonal- indicate self-similarity of the topics. On the other hand, the lighter and mid-range values reveal varying degrees of semantic overlap between them. Those shades help uncover clusters of related discussions and highlight potential redundancies across topics.</p><p>Finally, the dendrogram groups semantically similar topics discovered by BERTopic. By merging related branches, we can identify broader themes among our topics.</p><h4>Conclusion</h4><p>In this article, we explained the architecture of BERTopic and demonstrated its usage through a practical use case. To build a solid foundation in NLP and understand the technology that paved the way for large language models (LLMs), we recommend reading our previous article on BERT: <a href="https://medium.com/dataness-ai/exploring-bert-feature-extraction-fine-tuning-6d6ad7b829e7">Exploring BERT: Feature Extraction &amp; Fine-Tuning</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dadd2714bc0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/exploring-bert-applications-bertopic-dadd2714bc0c">Exploring BERT applications: BERTopic</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting book sales with Temporal Fusion Transformer]]></title>
            <link>https://medium.com/dataness-ai/forecasting-book-sales-with-temporal-fusion-transformer-dd482a7a257c?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/dd482a7a257c</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Mouna Labiadh]]></dc:creator>
            <pubDate>Tue, 19 Mar 2024 08:44:28 GMT</pubDate>
            <atom:updated>2024-03-19T08:44:33.652Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A step-by-step guide on how to use Temporal Fusion Transformer for book sales forecasting. We use the model implementation that is available in Pytorch Forecasting library along with Kaggle’s “tabular playground series” dataset.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F3a7poVccOIm3tpv" /><figcaption>Photo by <a href="https://unsplash.com/@luisabrimble?utm_source=medium&amp;utm_medium=referral">Luisa Brimble</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is Part 2 of my previous post about <em>Temporal Fusion Transformer </em>(TFT) [1]:</p><p><a href="https://medium.com/dataness-ai/understanding-temporal-fusion-transformer-9a7a4fcde74b">Understanding Temporal Fusion Transformer</a></p><p>In this part, we will be training and testing a TFT on Kaggle’s <a href="https://www.kaggle.com/competitions/tabular-playground-series-aug-2022/overview">Tabular playground series</a> (Sept 2022) dataset [2]. For the implementation of the model, we will use the open source python package <a href="https://pytorch-forecasting.readthedocs.io/"><em>Pytorch Forecasting</em></a> [3]. You can install it using pip:</p><pre>$ pip install pytorch-forecasting</pre><p><a href="https://pytorch-forecasting.readthedocs.io/">Pytorch Forecasting</a> is based on <em>Pytorch Lightning</em> and integrates <em>Optuna</em> for hyperparameters tuning.</p><h4>Data exploration</h4><p><a href="https://www.kaggle.com/competitions/tabular-playground-series-aug-2022/overview">Tabular playground series </a>dataset contains the total number of daily sales of 4 books from two stores located in six different countries. We have training data for over 4 years from 2017 to 2020. The goal is to predict the number of sales for 2021 for each book in each store and each country.</p><p>We will train our model on 2017~2019 and leave 2020 for validation. As you might assume, a major challenge in this use case is how to handle the COVID-19 pandemic effect on data during the year 2020.</p><p>Let’s start by taking a look on the data:</p><pre>DATAPATH = Path(&quot;./tabular-playground-series-sep-2022/&quot;)<br><br>train_df = pd.read_csv(DATAPATH / &quot;train.csv&quot;)<br>train_df[&#39;date&#39;]  = pd.to_datetime(train_df[&#39;date&#39;])<br>test_df = pd.read_csv(DATAPATH / &quot;test.csv&quot;)<br>test_df[&#39;date&#39;]  = pd.to_datetime(test_df[&#39;date&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*9GoL233gRn6MlLTanwoIJA.png" /></figure><pre>fig, ax = plt.subplots(1, 1, figsize = (23, 8))<br>sns.lineplot(x=&#39;date&#39;, y=&#39;num_sold&#39;, hue=&#39;product&#39;, data=(train_df.groupby([&#39;date&#39;, &#39;product&#39;]).num_sold.sum().to_frame()))<br>ax.set_title(&quot;sum of num_sold per product &quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lo3c8gtEHyYW1cPGET4zAA.png" /><figcaption>Daily total sales per book title</figcaption></figure><p>Similarly,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QVdoEFtK5rHmc8x_o_0vBQ.png" /><figcaption>Daily total sales per store</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ywqkcOa0WpzTGRgYYc8yqQ.png" /><figcaption>Daily total sales per country</figcaption></figure><p>What we have:</p><ul><li>The dataset does not contain missing values or missing timesteps,</li><li>The pattern of the data changes significantly in 2020 compared to previous years. The drop of sales around March~May is mostly due to COVID-19 lockdown and restrictions in Europe,</li><li>A significant increase of the sales average during 2020 compared to 2017~2019. Similarly, we notice an increased sales average around 11/01/2020~29/02/2020 compared to the second half of the same year,</li><li>In general, data seems to contain weekly and annual seasonalities, and an increase of sales in weekends. We also notice a peak during end-of-year holidays.</li></ul><p>We start by addressing the annual shift in average sales for each country by bringing 2017~2019 data to the same level as 2020. This will improve the generalization performance of our model by training and testing on like-to-like data.</p><pre># Add a year column<br>train_df_processed[&quot;year&quot;]=train_df_processed.index.year<br><br># Get average sales for each country in each year<br>mean_country_year = train_df_processed[[&#39;country&#39;, &#39;year&#39;, &quot;num_sold&quot;]].groupby([&#39;country&#39;, &#39;year&#39;], as_index=False).mean()<br><br># Shift the mean of 2017~2019 sales data w.r.t. 2020<br>for country in train_df_processed.country.unique():<br>    mean_2020 = mean_country_year.loc[(mean_country_year[&#39;year&#39;] == 2020) &amp; (mean_country_year[&#39;country&#39;] == country), &#39;num_sold&#39;].values[0]<br><br>    for year in train_df_processed.year.unique():<br>        if year==2020:<br>            break<br>        mean_year = mean_country_year.loc[(mean_country_year[&#39;year&#39;] == year) &amp; (mean_country_year[&#39;country&#39;] == country), &#39;num_sold&#39;].values[0]<br>        factor = mean_2020/mean_year<br>        train_df_processed.loc[(train_df_processed[&quot;country&quot;]==country) &amp; (train_df_processed[&quot;year&quot;]==year),&quot;num_sold&quot;]= train_df_processed.loc[(train_df_processed[&quot;country&quot;]==country) &amp; (train_df_processed[&quot;year&quot;]==year),&quot;num_sold&quot;]*factor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GmS4jdtKFleGBuH8a_-hxA.png" /><figcaption>Sales data after addressing annual shifts. An outlier value can be seen in 01/01/2020</figcaption></figure><p>Lockdown period from March~May 2020 does not reflect the general patterns that we want our model to learn, as they would not be generalizeable to following years (hopefully!). Including these months data would negatively impact the performance.</p><p>Different strategies can be considered to handle the impact of the pandemic on data for time series modeling use cases [4]. The most straightforward solution for this is to simply exclude the problematic span from our modeling as it represents an outlier. This could work given that after the recovery from the pandemic, the data patterns are similar to what they were before.</p><p>However, given the restricted amount of historical data we have, and the fact we’re using 2020&#39;s <em>pandemic-affected</em> data for validation (i.e. the model will be trained on <em>pre-pandemic</em> and tested on <em>post-pandemic</em>) . We rather try to drop/replace them by our estimation of <em>“what would have happened if it wasn’t for the pandemic”</em>. This yielded the best blind-test results on 2021.</p><p>The outlier seen in January 1st 2020 will be also dropped and replaced.</p><p>As an imputation strategy, we take the average sales over the previous years from 2017. We make sure to respect the weekly and annual seasonality for replacing March~May:</p><pre># handle 01-01-2020 outlier<br>train_df_processed.loc[pd.Timestamp(&#39;2020-01-01&#39;), &quot;num_sold&quot;]=np.nan<br><br>df_shifted = pd.concat(<br>        [train_df_processed[[&quot;num_sold&quot;]].shift(periods=365*48*x) for x in range(3)], axis=1<br>    )<br>train_df_processed[[&quot;num_sold&quot;]] = train_df_processed[[&quot;num_sold&quot;]].fillna(df_shifted.groupby(by=df_shifted.columns, axis=1).mean())<br><br># handle March~May lockdown outliers<br>train_df_processed.loc[(train_df_processed.year==2020)&amp;(train_df_processed.index.month.isin([3,4,5])), &quot;num_sold&quot;]=np.nan<br><br>df_shifted = pd.concat(<br>        [train_df_processed[[&quot;num_sold&quot;]].shift(periods=52*7*48*x) for x in range(3)], axis=1<br>    )<br>train_df_processed[[&quot;num_sold&quot;]] = train_df_processed[[&quot;num_sold&quot;]].fillna(df_shifted.groupby(by=df_shifted.columns, axis=1).mean())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mG5TyKmZXrZDQ_95VWt5YQ.png" /><figcaption>Daily total sales per country after pre-processing</figcaption></figure><p>To account for weekly and annual seasonalities exhibited by the data, we add the following calendar features: weekend flag, weekday and week of the year. For weekday and week features, we use cyclical encoding to reflect their cyclical nature.</p><pre>CALENDAR_CYCLES = {<br>    &quot;weekday&quot;: 7,<br>    &quot;week&quot;: 52,<br>    &quot;month&quot;: 12,<br>}<br><br>def add_cyclical_calendar_features(df: pd.DataFrame, features: Union[str, List[str]]):<br>    &quot;&quot;&quot;Cyclical encoding of calendar features&quot;&quot;&quot;<br>    if isinstance(features, str):multi-head attention<br>        features = [features]<br>    for feat in features:<br>        assert (<br>            feat in CALENDAR_CYCLES.keys()<br>        ), f&quot;Cyclical encoding is not available for {feat}&quot;<br>       <br>        values = getattr(df.index, feat)<br>        df[f&quot;{feat}_sin&quot;] = np.sin(2 * np.pi * values / CALENDAR_CYCLES[feat])<br>        df[f&quot;{feat}_cos&quot;] = np.cos(2 * np.pi * values / CALENDAR_CYCLES[feat])<br>    return df</pre><pre># add calendar features<br>train_df_processed = add_cyclical_calendar_features(train_df_processed.set_index(&quot;date&quot;), features=[&quot;weekday&quot;, &quot;week&quot;])<br>train_df_processed[&quot;weekend&quot;] = (train_df_processed.index.dayofweek &gt; 4).astype(int)</pre><p>We also add boolean flags for national holidays with respect to the country (we use <a href="https://pypi.org/project/holidays/"><em>holidays</em></a> package for this) and for end-of-year holidays (from December 24th to 31st).</p><pre># add holidays flag<br>holidays_dates_per_country = {}<br>for country in train_df_processed[&quot;country&quot;].unique():<br>    holidays_dates_per_country[country]=[tuple[0] for tuple in list(getattr(holidays, country)(years=set(train_df_processed.index.year)).items())]<br>    train_df_processed.loc[train_df_processed[&quot;country&quot;]==country, &quot;holidays&quot;]=train_df_processed.loc[train_df_processed[&quot;country&quot;]==country].index.isin(holidays_dates_per_country[country])<br><br>train_df_processed[&quot;holidays&quot;] = train_df_processed[&quot;holidays&quot;].astype(int)</pre><pre># add end-of-year holidays flag<br>train_df_processed[&quot;newyear&quot;]=0<br>for day in range(25,32):<br>    train_df_processed.loc[(train_df_processed.index.month == 12) &amp; (train_df_processed.index.day == day),&quot;newyear&quot;]=1</pre><h4>Model Training</h4><p>Pytorch Forecasting library requires <em>a TimeSeriesDataSet</em> object to encapsulate the data, features names (target, static covariates, time varying known and unknown variables, lagged values), feature encoders, scalers, the max and min lengths of the look-back window and the prediction horizon of the sequence-to-sequence block, etc.</p><p>As a reminder, static covariates in TFT consist in static metadata about the measured entities (i.e. books) that do no depend on time (i.e. country, store, product). Static covariates are used to define <em>groups</em> to separate individual times series contained in the dataset. The number of groups is the number of all possible combinations of static covariates values.</p><p>In our use case, we will have <em>48</em> individual time series : <em>number of countries * number of stores * number of products</em></p><p><em>TimeSeriesDataSet</em> expects the provided data to contain an integer column denoting the time index. It should start by 0 and increment by +1 for each individual time series if there are no missing time steps.</p><p>We replace our <em>date</em> column with <em>time_idx:</em></p><pre>train = train_df_processed.reset_index()<br>train = (train.merge((train[[&#39;date&#39;]].drop_duplicates(ignore_index=True).rename_axis(&#39;time_idx&#39;)).reset_index(), on = [&#39;date&#39;])).drop([&quot;date&quot;, &quot;row_id&quot;], axis=1)</pre><p>We use this time index to remove the last <em>max_prediction_length</em> timesteps from each individual time series, i.e. to make sure that the validation set is completely held out from the training.</p><pre>max_prediction_length = 365 # a whole year<br>min_encoder_length = 365<br>max_encoder_length = train.index.nunique()<br><br># keep the validation set held-out<br>training_cutoff = train[&quot;time_idx&quot;].max() - max_prediction_length # validation on 2020</pre><p>Given that our time series data are different in magnitude between different <em>groups</em> (country, store and product), we use a <em>GroupNormalizer</em> to normalize each time-series individually.</p><p>To account for the autocorrelation in the sales data, we explicitly add lagged features for t-7 (same day of previous week) and t-365 (previous year).</p><pre># Create training set<br>training_dataset = TimeSeriesDataSet(<br>    train[lambda x: x.time_idx &lt;= training_cutoff],<br>    time_idx=&quot;time_idx&quot;,<br>    target=&quot;num_sold&quot;, # target variable<br>    group_ids=[&quot;country&quot;, &quot;store&quot;, &quot;product&quot;], # static covariates<br>    max_encoder_length=max_encoder_length, # maximum size of lookup window<br>    min_encoder_length=max_encoder_length//2,  <br>    max_prediction_length=max_prediction_length, # maximum size of horizon window<br>    min_prediction_length=max_prediction_length,<br>    time_varying_known_reals=[<br>        &quot;time_idx&quot;, &#39;weekday_cos&#39;, &#39;weekday_sin&#39;, &#39;week_cos&#39;, &#39;week_sin&#39;, &#39;weekend&#39;, &#39;holidays&#39;, &#39;newyear&#39;],<br>    time_varying_unknown_categoricals=[],<br>    time_varying_unknown_reals=[&#39;num_sold&#39;],<br>    target_normalizer=GroupNormalizer(finalement<br>        groups=[&quot;country&quot;, &quot;store&quot;, &quot;product&quot;], transformation=&quot;softplus&quot;<br>    ),  # use softplus transformation and normalize by group<br>    lags={&#39;num_sold&#39;: [7,365]}, # add lagged values of target variable<br>    add_relative_time_idx=True,<br>    add_target_scales=True,<br>    add_encoder_length=True,<br>)</pre><p>We then create a validation <em>TimeSeriesDataSet</em> that has the same parameters as the training dataset object as follows:</p><pre># create validation set (predict=True)<br>validation_dataset = TimeSeriesDataSet.from_dataset(<br>  training_dataset, # dataset from which to copy parameters (encoders, scalers, ...)<br>  train, # data from which new dataset will be generated<br>  predict=True, # predict the decoder length on the last entries in the time index<br>  stop_randomization=True,<br>)</pre><p>Similarly, we use the training <em>TimeSeriesDataSet</em> object to create the TFT model:</p><pre># Create network from TimeSeriesDataSet<br>tft = TemporalFusionTransformer.from_dataset(<br>    training_dataset,<br>    learning_rate=0.03,<br>    hidden_size=16,<br>    attention_head_size=1,<br>    dropout=0.1,<br>    hidden_continuous_size=8,<br>    output_size=7,  # number of quantiles<br>    loss=QuantileLoss(),<br>    log_interval=10,  # logging every 10 batches<br>    reduce_on_plateau_patience=4,<br>)<br>print(f&quot;Number of parameters in network: {tft.size()/1e3:.1f}k&quot;)</pre><p><em>Pytorch Forecasting</em> implementation is based on <em>Pytorch Lightning</em>. Hence, to train the model, we need to define a <em>Trainer</em> object and <em>dataloaders</em>.</p><pre># define callbacks<br>early_stop_callback = EarlyStopping(monitor=&quot;val_loss&quot;, min_delta=1e-4, patience=10, verbose=False, mode=&quot;min&quot;)<br>lr_logger = LearningRateMonitor()  # log the learning rate<br>logger = TensorBoardLogger(save_dir=SAVE_DIR)  # log results to a tensorboard<br><br># create trainer<br>trainer = pl.Trainer(<br>    max_epochs=50,<br>    accelerator=&quot;gpu&quot; if torch.cuda.is_available() else &quot;cpu&quot;,<br>    devices=1,<br>    gradient_clip_val=0.1,<br>    limit_train_batches=30,  # run valiation every 30 batches<br>    log_every_n_steps=10,<br>    # fast_dev_run=True,  # comment in to check that networkor dataset has no serious bugs<br>    callbacks=[lr_logger, early_stop_callback],<br>    logger=logger,<br>)</pre><pre># create training and validation dataloaders<br>batch_size = 128 <br>train_dataloader = training_dataset.to_dataloader(train=True, batch_size=batch_size, num_workers=8)<br>val_dataloader = validation_dataset.to_dataloader(train=False, batch_size=batch_size * 10, num_workers=8)<br><br># fit network<br>trainer.fit(<br>    tft,<br>    train_dataloaders=train_dataloader,<br>    val_dataloaders=val_dataloader,<br>)</pre><h3>Hyper-parameters Tuning</h3><p>In the above code, we used pre-fixed hyper-parameters to train our model. <em>Pytorch Forecasting</em> proposes a wrapper function to tune them on the validation set using <em>Optuna</em> :</p><pre># Hyperparameters Tuning with Optuna<br># create study<br>study = optimize_hyperparameters(<br>    train_dataloader,<br>    val_dataloader,<br>    model_path=&quot;optuna_test&quot;,<br>    n_trials=200,<br>    max_epochs=50,<br>    gradient_clip_val_range=(0.01, 1.0),<br>    hidden_size_range=(8, 128),<br>    hidden_continuous_size_range=(8, 128),<br>    attention_head_size_range=(1, 4),<br>    learning_rate_range=(0.001, 0.1),<br>    dropout_range=(0.1, 0.3),<br>    trainer_kwargs=dict(limit_train_batches=30),<br>    reduce_on_plateau_patience=4,<br>    use_learning_rate_finder=False,  # use Optuna to find ideal learning rate or use in-built learning rate finder<br>)<br><br># pickle study results<br>with open(&quot;study.pkl&quot;, &quot;wb&quot;) as fout:<br>    pickle.dump(study, fout)<br><br># show best hyperparameters<br>study.best_trial.params</pre><h3>Validation</h3><p>Once trained, we load the model from the the checkpoint with the best validation loss.</p><pre># load the best model w.r.t. the validation loss<br>best_model_path = trainer.checkpoint_callback.best_model_path<br>best_tft = TemporalFusionTransformer.load_from_checkpoint(best_model_path)</pre><p>By default, the last <em>max_prediction_length</em> time steps (365 days) of each individual time series are used for validation. We set <em>return_x</em> to <em>True</em> to return input data that we plot later on the same plot as predictions.</p><pre># get prediction results as a dictionary<br>val_prediction_results = best_tft.predict(val_dataloader, mode=&quot;raw&quot;, return_x=True)</pre><pre># Plot actuals vs prediction and attention<br>for idx in range(val_predictions.prediction.shape[0]): # nb of groups combinations<br>    fig, ax = plt.subplots(figsize=(23,5))<br>    best_tft.plot_prediction(val_prediction_results.x, # network input<br>                            val_prediction_results.output, # network output<br>                            idx=idx,<br>                            add_loss_to_title=True,<br>                            ax=ax);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIDsi8e0TR9pEiwQ-rnz3Q.png" /><figcaption>Prediction results on the validation set for the first group</figcaption></figure><p>The gray line shows the attention weights for different points in time. Exploring these weights can be used to improve the model. For example, if attention peaks are located at the start of the look-back window, one should increase the size of this window to include all relevant past time-steps.</p><h3>Inferring 2021</h3><p>Now, we will infer the number of sales of 2021 for each individual time series. For this, we need to append the training dataset with static covariates and time-varying known values of 2021. In our case, this includes apriori known calendar features.</p><pre># add calendar features<br>test = add_cyclical_calendar_features(test_df.set_index(&quot;date&quot;), features=[&quot;weekday&quot;, &quot;week&quot;])<br>test[&quot;weekend&quot;] = (test.index.dayofweek &gt; 4).astype(int)<br><br># add holidays flag<br>holidays_dates_per_country = {}<br>for country in test[&quot;country&quot;].unique():<br>    holidays_dates_per_country[country]=[tuple[0] for tuple in list(getattr(holidays, country)(years=set(test.index.year)).items())]<br>    test.loc[test[&quot;country&quot;]==country, &quot;holidays&quot;]=test.loc[test[&quot;country&quot;]==country].index.isin(holidays_dates_per_country[country])<br><br>test[&quot;holidays&quot;] = test[&quot;holidays&quot;].astype(int)<br><br># add end-of-year holidays flag<br>test[&quot;newyear&quot;]=0<br>for day in range(25,32):<br>    test.loc[(test.index.month == 12) &amp; (test.index.day == day),&quot;newyear&quot;]=1<br><br># Add required time_idx column w.r.t to last time index of training df<br>test = test.reset_index()<br>test = (test.merge((test[[&#39;date&#39;]].drop_duplicates(ignore_index=True).rename_axis(&#39;time_idx&#39;)).reset_index(), on = [&#39;date&#39;]))<br>test[&quot;time_idx&quot;]+=train[&quot;time_idx&quot;].max()+1<br><br># Drop unused columns<br>test = test.drop([&quot;date&quot;, &quot;row_id&quot;], axis=1)<br><br># Vertically stack the test df at the end of the training df<br>test = pd.concat([train, test], ignore_index=True).fillna(0.0)</pre><p>We then create a <em>TimeSeriesDataSet</em> object, a <em>dataloader</em> for our test set and launch the prediction loop.</p><pre># Create test dataset<br>test_dataset = TimeSeriesDataSet.from_dataset(training_dataset,<br>                                        test, <br>                                        predict=True, <br>                                        stop_randomization=True)<br><br># Create test dataloader<br>test_dataloader = test_dataset.to_dataloader(train=False, batch_size=batch_size, num_workers=8)<br><br># Get prediction results<br>test_prediction_results = best_tft.predict(<br>    test_dataloader,<br>    mode=&quot;raw&quot;,<br>    return_index=True, # return the prediction index in the same order as the output<br>    return_x=True, # return network inputs in the same order as prediction output<br>    )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0kSc3i2ETo8CoKaxZpQY4Q.png" /><figcaption>Prediction results on the test set for the first group</figcaption></figure><p>2021 predictions are of shape <em>&lt;number of groups combinations, number of timesteps, number of quantiles&gt;</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*H6FXP984RKD-naU0cCj6cA.png" /></figure><p>To be able to construct the predictions dataframe and associate each predicted value with its corresponding store, country and product, we need to figure out the groups combination associated to the first dimension of the predictions Tensor. For this, we set <em>return_index</em> to <em>True</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*o-cp3oKbVf9qxS7LuPKQOA.png" /></figure><p>The <em>time_idx</em> in the index dataframe corresponds to the <em>time_idx</em> of the first prediction.</p><pre># Create predictions dataframe<br>predictions_df = test_df.copy()<br># Add num_sold column<br>predictions_df[&quot;num_sold&quot;]=np.nan<br><br># get 0.5 quantile (median) predictions<br>median_predictions = test_prediction_results.output.prediction.cpu().numpy()[:,:,4] <br><br># add sales predictions w.r.t to groups combination<br>for i, row in test_prediction_results.index.iterrows():<br>    predictions_df.loc[<br>        (predictions_df[&quot;country&quot;]==row[&quot;country&quot;]) &amp; <br>        (predictions_df[&quot;store&quot;]==row[&quot;store&quot;]) &amp; <br>        (predictions_df[&quot;product&quot;]==row[&quot;product&quot;]), <br>        &quot;num_sold&quot;] = median_predictions[i]</pre><h3>Interpretatability</h3><p><em>Interpretability </em>is a major property of the Temporal Fusion Transformer.<em> Pytorch Forecasting</em> proposes utility functions to get and plot interpretation results from the trained model.</p><p>Interpretability in TFT is established using :</p><ul><li>Shared weights in the multi-head attention mechanism that allow to trace back most relevant historical data points. Attention weights associated with each data point in the look-back window can be plotted as previously shown in the <em>Validation</em> section (gray line)</li><li>Special variable selection blocks that weigh the importance of each input variable. You can retrieve learned weights from <em>predictions</em> dictionary</li></ul><pre># plot variable importance<br>interpretation = best_tft.interpret_output(<br>  val_predictions, <br>  reduction=&quot;sum&quot;, # sum attentions<br>)<br>best_tft.plot_interpretation(interpretation)</pre><p><em>Pytorch forecasting</em> also provides a function to cross-plot predictions vs actual values of different variables to detect possible dependencies between residuals and input features. This might guide the explanatory features identification step.</p><pre>val_prediction_results = best_tft.predict(<br>    val_dataloader, <br>    mode=&quot;prediction&quot;, # get only median predictions<br>    return_x=True,<br>    )<br>predictions_vs_actuals = best_tft.calculate_prediction_actual_by_variable(val_prediction_results.x, val_prediction_results.output)<br><br># remove added lagged features<br>features = list(set(predictions_vs_actuals[&#39;support&#39;].keys())-set([&#39;num_sold_lagged_by_365&#39;, &#39;num_sold_lagged_by_7&#39;]))<br><br># plot cross_plots<br>for feature in features:<br>    best_tft.plot_prediction_actual_by_variable(predictions_vs_actuals, name=feature);</pre><h3>Key takeaways</h3><p><em>Pytorch Forecasting</em> is an open source Python library. It proposes an implementation of Temporal Fusion Transformer based on <em>Pytorch Lightning</em>. It provides support to hyperparameters tuning with <em>Optuna</em> and the visualization of inherent interpretability properties.</p><p>In addition to TFT, <em>Pytorch Forecasting</em> proposes implementations of other time series forecasting deep learning models like DeepAR, N-BEATS and N-HiTS.</p><p><em>Thank you for reading!</em></p><p>All the code is available as a Jupyter Notebook on my Github: <a href="https://github.com/mounalab/Temporal-fusion-transformer_Pytorch-Forecasting">https://github.com/mounalab/Temporal-fusion-transformer_Pytorch-ForecastingForecasting</a></p><h3>References</h3><p>[1] Lim, Bryan, et al. “<a href="https://arxiv.org/abs/1912.09363">Temporal fusion transformers for interpretable multi-horizon time series forecasting</a>.”<em> International Journal of Forecasting</em> 37.4 (2021): 1748–1764.</p><p>[2] “Tabular Playground Series” dataset: <a href="https://www.kaggle.com/competitions/tabular-playground-series-aug-2022/overview">https://www.kaggle.com/competitions/tabular-playground-series-aug-2022/overview</a></p><p>[3] Step by step tutorial : <a href="https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html">https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html</a></p><h3>Good Reads</h3><p>[4] A helpful Github discussion on how to handle <em>shocks</em> in time series modeling use cases : <a href="https://github.com/facebook/prophet/issues/1416">https://github.com/facebook/prophet/issues/1416</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd482a7a257c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/forecasting-book-sales-with-temporal-fusion-transformer-dd482a7a257c">Forecasting book sales with Temporal Fusion Transformer</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring BERT: Feature extraction & Fine-tuning]]></title>
            <link>https://medium.com/dataness-ai/exploring-bert-feature-extraction-fine-tuning-6d6ad7b829e7?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/6d6ad7b829e7</guid>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[question-answering]]></category>
            <category><![CDATA[text-embedding]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Mouna Labiadh]]></dc:creator>
            <pubDate>Tue, 13 Feb 2024 21:38:19 GMT</pubDate>
            <atom:updated>2024-03-15T14:47:25.921Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An introduction on BERT, one of the first Transformer-based large language models, and examples of how it can be used in common NLP applications.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*6Vsk-v6m1H4u4a9mLcYjJg.png" /></figure><blockquote><em>This post is part of an NLP blog series co-written with </em><a href="https://medium.com/@zgolliasma"><em>Asma Zgolli</em></a><em>.</em></blockquote><p>BERT (Bidirectional Encoder Representations from Transformers) [1] is a large language model developed by Google AI in 2018. It is :</p><ul><li><em>Pretrained: </em>with two unsupervised tasks: masked language modeling (i.e. prediction of randomly masking words from the input sentence) and next sentence prediction and using BooksCorpus and English Wikipedia (16GB)</li><li><em>Bidirectional</em>: deeply fuse the left and right context in the learned representation of each token.</li></ul><p>The architecture of BERT is composed of multiple encoder layers that apply self-attention to its input and pass it to the following layer. For instance, the smallest BERT model, BERT BASE, has an architecture composed of 12 encoder layers (cf. Figure 2), 768 hidden units in its feed-forward neural network block, and 12 attention heads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*YZQjWWCsqCCcR2-CiKOruA.png" /><figcaption><em>Figure 1: BERT BASE architecture (Figure adapted from [9])</em></figcaption></figure><h3>Input representations</h3><p>BERT takes as input sequences that are composed of sentences or pairs of sentences (&lt;question, answer&gt; for question-answering tasks). Input sequences are prepared before being fed to the model using <em>WordPiece Tokenizer </em>with a 30k vocabulary size. It works by splitting a word into several <em>subwords (Tokens)</em>. The use of subwords instead of words significantly reduces the total vocabulary size (only 30k) and the number of potential <em>out-of-vocabulary</em> (OOV) tokens.</p><p>Special tokens are:</p><ul><li>[CLS] used as the first token of each sequence. It is introduced in the original BERT implementation to use its corresponding final hidden state for classification tasks, where only one vector that represents the entire input sequence is needed to be fed to the classifier.</li><li>[SEP] used to separate the pair of sentences in the input sequence (e.g. question answering) or as the end token.</li><li>[PAD] used to represent paddings in the input sentences (empty tokens). The model expects fixed-length sentences as input. A maximum length is thus fixed depending on the dataset. Shorter sentences are padded, whereas longer sentences are truncated. To explicitly differentiate between real tokens and [PAD] tokens, we use an attention mask.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*f9hRTee8HoUjG85Phu_HzA.png" /><figcaption><em>Figure 2: Example of a sentence encoding using BERT (Figure by authors)</em></figcaption></figure><p>Each token will be then encoded by its ID which corresponds to its <em>index</em> in the vocabulary.</p><p>Eventually, for each token, the input embedding is the sum of the <em>token</em>, the <em>segmentation</em>, and the <em>positional embeddings</em>. To get the token embedding, a <em>lookup table</em> is used at the embedding layer (as illustrated in Figure 2), where rows represent all possible token IDs in the vocabulary (30k rows for instance) and columns represent the embedding size.</p><p><em>Segmentation embedding</em> is introduced to indicate if a given token belongs to the first or second sentence. <em>Positional embedding</em> indicates the position of tokens in a sentence. By contrast to the original Transformer, BERT learns positional embeddings from absolute ordinal position, instead of using trigonometric functions.</p><h3>Feature Extraction from Text</h3><p>BERT can be used out-of-the-box (i.e. at inference) to extract machine-readable data representations from text. Once this is done, applying traditional machine learning or deep learning techniques such as classification or regression becomes a straightforward process.</p><p>To create embeddings, we start by adding special tokens to the input text. This preprocessing step is required since the (transformer’s) model was trained on data that follows this organization.</p><p>In the code snippet below, we manually add the [CLS] and [SEP] tokens to respectively mark the start of a sentence and when a new sentence begins. After that, we create separate tokens by dividing a sentence into words or partial words using the <em>tokenize</em> method from a pre-trained BERT <em>tokenizer</em>. Finally, we add the [PAD] token to a fixed size equal to a pre-set maximum sentence length.</p><pre>from transformers import BertModel, AutoTokenizer<br>import torch<br><br># load &quot;bert-base-cased&quot; the pre-trained model<br>model = BertModel.from_pretrained(&#39;bert-base-uncased&#39;, output_hidden_states = False)<br><br># load the corresponding wordtokenizer<br>tokenizer = AutoTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)<br><br><br># example sentence<br>text = &quot;Let’s deep dive into BERT.&quot;<br><br># add [CLS] and [SEP] tokens<br>text = &quot;[CLS]&quot;+text.replace(&#39;\.&#39;, &#39;[SEP]&#39;)<br><br># get tokens from the sentence<br>tokens = tokenizer.tokenize(<br>  text=text,<br>)<br><br># set the maximum length of a sentence<br>max_length = 30<br><br># add [PAD] tokens to shorter sentences<br>padded_tokens = tokens + [&#39;[PAD]&#39; for _ in range(max_length-len(tokens))]<br><br># get token IDs<br>token_ids = tokenizer.convert_tokens_to_ids(padded_tokens)<br><br># generate the attention mask<br>attention_mask = [1 if token != &#39;[PAD]&#39; else 0 for token in padded_tokens]<br><br># whether each token belongs to sentence A (0) or sentence B (1)<br>segment_ids = [0 for _ in range(len(padded_tokens))]<br><br># convert lists to tensors<br>token_ids = torch.tensor(token_ids).unsqueeze(0)<br>attention_mask = torch.tensor(attention_mask).unsqueeze(0)<br>segment_ids = torch.tensor(segment_ids).unsqueeze(0)</pre><p>To retrieve the input embeddings extracted from the token IDs:</p><pre># get input embeddings<br>input_embeddings = model.embeddings(token_ids, token_type_ids=segment_ids)</pre><p>Two types of embeddings can be extracted from the input text.</p><ul><li><em>Token-level embeddings</em>: these are the embeddings generated by default by the last hidden states of BERT. It consists of representing every token in a separate embedding. Token-level embeddings are used for question-answering or named entity recognition.</li><li><em>Sequence-level embeddings</em>: This requires a pooling post-processing of generated token-level embeddings. We extract same-length embeddings for each sentence in the text [3]. Pooling can be performed by taking the mean of token embeddings (most used), the max, or by simply taking the output of the first token of each sequence (CLS token output) [4]. Sequence-level embeddings can be useful for text classification and sentiment analysis applications.</li></ul><pre># set the model in evaluation mode<br>model.eval()<br><br># get contextual embeddings<br>with torch.no_grad():<br>   # output of shape &lt;batch_size, max_length, embedding_size&gt;<br>   last_hidden_states = model(token_ids, attention_mask=attention_mask, token_type_ids=segment_ids)[&quot;last_hidden_state&quot;]<br><br># first token embedding of shape &lt;1, hidden_size&gt;<br>first_token_embedding = last_hidden_states[:,0,:]<br><br># pooled embedding of shape &lt;1, hidden_size&gt;<br>mean_pooled_embedding = last_hidden_states.mean(axis=1)</pre><p>In this section, we presented in detail how to generate text embeddings using BERT. For demonstration reasons, we manually added special tokens to our text. However, it’s possible to automate this pre-processing step in HuggingFace <em>transformers</em> library by making use of <em>encode_plus</em> function :</p><pre># encode the sentence<br>encoded = tokenizer.encode_plus(<br>  text=sentence,<br>  add_special_tokens=True,  # add [CLS] and [SEP] tokens<br>  max_length = 30,  # set the maximum length of a sentence<br>  truncation = True, # truncate longer sentences to max_length<br>  padding=&#39;max_length&#39;,  # add [PAD] tokens to shorter sentences<br>  return_attention_mask = True,  # generate the attention mask<br>  return_tensors = &#39;pt&#39;,  # return encoding results as PyTorch tensors<br>)<br><br># get the token IDs and attention mask<br>token_ids = encoded[&#39;input_ids&#39;]<br>attention_mask = encoded[&#39;attention_mask&#39;]</pre><h3>Fine-tuning for downstream tasks</h3><p>As mentioned earlier, BERT can be used out-of-the-box as a feature extractor to generate text embeddings. However, to achieve the best performance, it should be fine-tuned on the target dataset.</p><p>Fine-tuning is a transfer learning technique that allows one to solve target problems by applying a model trained on different source tasks or datasets. In the context of NLP applications, the models used are already trained on large text corpus thus they have general language understanding, and they capture complex semantic relationships in texts. Previous research initiatives have proven that fine-tuning existing LLMs leads to better results with fewer data and in less time than training from scratch a new neural network like a Transformer [5].</p><p>For task-specific fine-tuning, we add a <em>trainable</em> fully connected layer to the architecture and only a few epochs are needed to optimize it and obtain the desired result.</p><p>Hereafter, we briefly present fine-tuning strategies for the three most common use cases: <strong>sequence classification</strong>, <strong>token classification,</strong> and <strong>question answering</strong>.</p><ul><li><em>Sequence classification</em>: we simply add a classification layer that takes as input the sequence-level embedding and outputs the class label.<br>It can be used on a single text sequence (e.g. sentiment analysis and topic classification), or on a pair of text sequences (e.g. natural language inference and semantic textual similarity). The difference between the two cases is shown in Figures 3 and 4.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*q-bXA_KurJmUXH5eu6TGkw.png" /><figcaption><em>Figure 3: Fine-tuning BERT for sequence classification. As pooled sequence-level embedding, we take the embedding of the [CLS] token.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*0TYo_W1ahBgcf_wi0GXlhg.png" /><figcaption><em>Figure 4: Fine-tuning BERT for sequence-pair classification.</em></figcaption></figure><ul><li><em>Token classification</em>: the added classification layer takes as inputs token embeddings and outputs each token’s class label (cf. Figure 5). Typical applications of token classification are named entity resolution and part-of-speech tagging.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*VsmW7XaCZq9gZ9bCzX-mjQ.png" /><figcaption><em>Figure 5: Fine-tuning BERT for token classification.</em></figcaption></figure><ul><li><em>Question answering</em>: takes as input two text sequences, where the first one is the question and the second one is the passage that the question is about and its context (cf. Figure 6). The answer is a segment (text span) of this passage. The output layer will learn the probabilities of each token in the passage of being the start and the end index of the answer. The predicted answer is the one spanning between the start/end tokens with respectively the largest start/end probabilities.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*AT4_tXimoO7nfX2zfr-A3g.png" /><figcaption><em>Figure 6: Fine-tuning BERT for question-answering</em></figcaption></figure><h3>BERT variants</h3><p>Several BERT variations exist. Hereafter, we summarize some examples.</p><p><em>RoBERTa </em>is a largely used BERT-like model. RoBERTa uses a larger corpus for pretraining and adds dynamic masking of sentences at training time instead of processing time as in the original BERT for language masking pretraining. Several Pretrained RoBERTa models are available such as <em>CamemBERT </em>for<em> </em>French, <em>RuBERTa</em> for Russian and <em>XLM-RoBERTa</em> for multilingual.</p><p><em>DistilBERT </em>[6] and <em>distilRoBERTa</em> are applications of knowledge distillation mechanism for BERT models, i.e. a compression technique to effectively reproduce the bahavior of a large model in a smaller and hence faster and cheaper one. This makes them more suitable to be deployed on edge devices (mobile phones, …)</p><p><em>Sentence-BERT</em> (<em>sBERT</em>) [7] alleviates the need to pass two sentences simultaneously for each semantic similarity evaluation using Siamese/Triplet networks. <em>BERTopic</em> [8] is used for topic modeling tasks. Broadly, It uses sBERT to embed documents, and clusters semantically similar ones together. Given its modularity, BERTopic can use other document-embedding language models than sBERT to generate embeddings.</p><h3>Key Takeaways</h3><p><em>BERT</em> uses the encoder part of the Transformer architecture to generate contextualized embeddings. The bidirectional pre-training allows it to capture both the left and right context of words for language understanding. In general, encoder-based models are used for predictive modeling tasks like text classification.</p><p>Other popular LLMs for tackling related problems are <strong><em>decoder</em></strong>-based models like <em>GPT</em> and <strong><em>encoder-decoder-</em></strong>based models like <em>BART</em>. Decoder-based models are optimized for next-word prediction using causal/autoregressive attention. Hence, they are more adapted for text generation tasks. The third category, which combines the encoder and the decoder parts, has a more generalist approach and seeks to combine the advantages of both categories.</p><p><em>Thank you for reading!</em></p><h3>References</h3><p>[1] Devlin, Jacob, et al. “<a href="https://arxiv.org/abs/1810.04805">BERT: Pre-training of deep bidirectional transformers for language understanding.</a>” <em>arXiv preprint arXiv:1810.04805</em> (2018).</p><p>[2] Vaswani, Ashish, et al. “<a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>.” <em>Advances in neural information processing systems</em> 30 (2017).</p><p>[3] Choi, Hyunjin, et al. “<a href="https://arxiv.org/abs/2101.10642">Evaluation of bert and albert sentence embedding performance on downstream nlp tasks.</a>” <em>2020 25th International conference on pattern recognition (ICPR)</em>. IEEE, 2021.</p><p>[4] Reimers, Nils, and Iryna Gurevych. “<a href="https://arxiv.org/abs/1908.10084">Sentence-BERT: Sentence embeddings using siamese BERT-networks</a>.” <em>arXiv preprint arXiv:1908.10084</em> (2019).</p><p>[5] Chris McCormick and Nick Ryan. (2019, July 22). “BERT Fine-Tuning Tutorial with PyTorch”, from ”<a href="http://www.mccormickml.com/">http://www.mccormickml.com</a>&quot;</p><p>[6] Sanh, Victor, et al. “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.” <em>arXiv preprint arXiv:1910.01108</em> (2019).</p><p>[7] Reimers, Nils, and Iryna Gurevych. “Sentence-bert: Sentence embeddings using siamese bert-networks.” <em>arXiv preprint arXiv:1908.10084</em> (2019).<br><em>Python library</em>: <a href="https://www.sbert.net/">https://www.sbert.net/</a></p><p>[8] Grootendorst, Maarten. “BERTopic: Neural topic modeling with a class-based TF-IDF procedure.” <em>arXiv preprint arXiv:2203.05794</em> (2022). <br><em>Python library</em>: <a href="https://maartengr.github.io/BERTopic/index.html">https://maartengr.github.io/BERTopic/index.html</a></p><p>[9] Khalid, Usama &amp; Beg, Mirza &amp; Arshad, Muhammad. <a href="https://www.researchgate.net/publication/349546860_RUBERT_A_Bilingual_Roman_Urdu_BERT_Using_Cross_Lingual_Transfer_Learning">RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning. </a>(2021).</p><h3>Good Reads</h3><ul><li><a href="https://huggingface.co/course">Hugging Face NLP Course</a></li><li><a href="https://huggingface.co/transformers/v3.2.0/custom_datasets.html#question-answering-with-squad-2-0">Fine-tuning with custom datasets — transformers 3.2.0 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d6ad7b829e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/exploring-bert-feature-extraction-fine-tuning-6d6ad7b829e7">Exploring BERT: Feature extraction &amp; Fine-tuning</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Notes about Spark internals]]></title>
            <link>https://medium.com/dataness-ai/notes-about-spark-internals-5d186d02115f?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/5d186d02115f</guid>
            <category><![CDATA[spark-dataframe]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[spark-sql]]></category>
            <dc:creator><![CDATA[Asma Zgolli, PhD]]></dc:creator>
            <pubDate>Tue, 19 Dec 2023 13:56:43 GMT</pubDate>
            <atom:updated>2023-12-19T07:01:25.599Z</atom:updated>
            <content:encoded><![CDATA[<p>This article presents a summary of Apache Spark concepts and internal characteristics. I also include a brief overview of its different APIs.</p><p><em>Note: For the sake of reading simplicity, we will refer to Apache Spark as ‘Spark’ throughout the rest of the article.</em></p><h4>Introduction</h4><p>Spark is a cluster computing framework designed for managing and processing big data. It leverages commodity hardware to scale horizontally, enabling the processing of larger datasets. It also offers a rich API for large-scale distributed data processing and scalable analytics.</p><p>Spark employs parallel processing and distributed computing techniques to ensure the efficiency of its data-driven applications. It was developed as a successor to MapReduce, specifically designed to overcome latency and efficiency limitations. Indeed, MapReduce tends to be slow due to its practice of storing intermediate results on disk. In contrast, Spark offers an API with concepts and data structures that enable it to execute data flows in memory, effectively minimizing resource-intensive I/O operations.</p><h4>Spark Internals and characteristics</h4><p>Multiple concepts make Spark an excellent choice for efficient big data workload execution:</p><ul><li><strong>Distributed processing:</strong> The figure below depicts Spark’s architecture, comprising a master node and worker nodes. The master node receives the workload from the driver program (the client) via the SparkContext defined in the application and orchestrates the processing: It decomposes the workload (which is called in Spark a job) into tasks and sends them to the worker nodes, who will take care of the execution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*EzZs4uEuO30lV51KV07_RA.png" /><figcaption>Cluster overview [<a href="https://spark.apache.org/docs/latest/cluster-overview.html">1</a>]</figcaption></figure><ul><li><strong>Parallelism</strong>: Under the hood, Spark uses a collection of immutable (in-memory read-only) objects that are distributed across the cluster characterized by their fault tolerance called RDDs: Resilient distributed datasets. Those RDDs are built via parallel transformation like map, filter, etc. The splitting of the data across the worker nodes and the parallel execution using these transformations make Spark a fast and scalable processing engine. RDDs are the physical representation of data in Spark. Using the RDD API makes it more challenging to develop optimized applications, as it relies solely on the developer’s expertise. Utilizing higher-level APIs like the DataFrame API and SparkSQL enables us to benefit from the query optimizer, making the development of efficient applications easier.</li><li><strong>Lazy evaluation</strong>: Spark tasks fall into two primary categories: actions and transformations. The core concept here is that one can generate as many transformations on an RDD as necessary and subsequently invoke an action to initiate the computation. Every transformation creates a new RDD (immutability). There are two kinds of transformation : (i) narrow transformations that execute the process where the data resides and do not need any data movement e.g. map, flatMap, filter, union, etc. and wide transformations that need data shuffling (very expensive) of the partitions across the nodes e.g. reduceByKey, join, aggregate, repartition, etc. Actions, on the other hand, are functions that return anything but an RDD (e.g. a value, displays data or a collection of data). Examples of actions are: collect, show, count, countByValue, max, etc.</li><li><strong>DAG</strong>: Processing in Spark is modelled as a directed acyclic graph (DAG). Indeed, when Spark executes tasks in memory, they are structured within a graph having RDDs as vertices and the transformations producing them as edges [<a href="https://medium.com/plumbersofdatascience/understanding-spark-dags-b82020503444">2</a>]. The flow of execution follows only one direction (directed) and without cycles (acyclic). This structure helps prevent redundant task execution. Moreover, DAGs can automatically reconstruct RDDs in the event of a failure, as they preserve the lineage of the transformations. DAGs are composed of tasks that are grouped into stages. The RDDs of those stages are computed in memory and stored in a cache, but the resulting RDD of each stage is materialized (to enable failure recovery). On the other hand, storing these intermediary results on disk can introduce additional latency. More modern MPP systems like Impala, which execute data processing entirely in memory, can be more efficient but less resilient to failure. These systems are better suited for less critical analytics applications. The choice between Spark and Impala, for example, will certainly involve a trade-off between fault tolerance and efficiency and should depend on the user’s requirements regarding these characteristics.</li></ul><h4>Spark APIs:</h4><p>In the previous section, we explained the different characteristics that make Spark a <strong>scalable</strong>, <strong>distributed</strong>, <strong>highly available</strong> and <strong>reliable</strong> general-purpose processing engine. As follows, we present its internal APIs and explain some standard examples using them.</p><p><strong>Spark RDD API:</strong> As we explained earlier, this API is low-level and can be challenging for beginners to develop optimised data pipelines. A classic example of a Spark RDD application is the word count example.</p><pre># Read the input file and Calculating words count<br>text_file = sc.textFile(&quot;firstprogram.txt&quot;)<br>countsRDD = text_file.flatMap(lambda line: line.split(&quot; &quot;)) \<br>                            .map(lambda word: (word, 1)) \<br>                           .reduceByKey(lambda x, y: x + y)<br><br><br># Printing each word with its respective count<br>output = countsRDD.collect()<br>for (word, count) in output:<br>    print(&quot;%s: %i&quot; % (word, count))</pre><p>In the above code snippet, I illustrate the word count example using PySpark. The first line of code reads a text file from the specified path in the input argument, creating an RDD as the output. The second line depicts a sequence of transformations aimed at creating an RDD composed of key-value pairs. Here, the key represents a unique word, and the value signifies the count of its occurrences. These transformations include (i) ‘flatMap,’ which creates a new collection from the input by splitting the words separated by spaces and ‘flattening’ them into a single collection; (ii) ‘map,’ which transforms the input (a collection of words) into a collection of key-value pairs, where each key is a word from the input, and the value is the number 1; and (iii) ‘reduceByKey,’ which matches the keys from the key-value collection obtained from the input and executes a lambda function (in our example, a sum) that merges the values.</p><p><strong>SparkSQL and DataFrame API: </strong>Spark provides the concept of a DataFrame to facilitate structured data processing. This concept is equivalent to a table in the relational data model or a dataframe in the library pandas in Python. DataFrames can be manipulated using a Domain Specific Language API (DSL). This DSL API is composed of functions that resemble SQL Operators, making it intuitive for SQL engineers to develop distributed and parallel data processing applications. For instance, in a Spark DataFrame transformation application, one can filter data using conditions, aggregate it using predefined aggregation functions, join it with another DataFrame, and transform it using User Defined Functions (UDFs).</p><p>On the other hand, SparkSQL allows the users to query those DataFrames using SQL (or more precisely, an SQL-like query language. Starting from version 3, Spark even supports ANSI-compliant SQL). In the latest versions of Spark, both SparkSQL queries and DataFrame API programs can be optimised by the query optimizer using statistics and cost models. They also integrate out-of-the-box with tables defined in Apache Hive.</p><pre>val df2 = spark.read.format(file_type) <br>.option(&quot;inferSchema&quot;, infer_schema) <br>.option(&quot;header&quot;, first_row_is_header) <br>.option(&quot;sep&quot;, delimiter) <br>.load(file2_location)<br><br><br>//Creating a temporary table that will be deleted when the notebook (Databricks) session is shut down.<br>val temp_table2_name = &quot;gdp_over_hours_worked&quot;<br>df2.createOrReplaceTempView(temp_table2_name)<br><br>//Viewing the schema inferred by Spark for the DataFrame.<br>df2.printSchema()<br><br>//Casting of some columns types (code available in the Github repository)<br><br>//Transforming the data and enriching the Dataframe with a new column<br>var enrichedDF = transformedDF.withColumn(&quot;unemployment_category&quot;, <br>    when(col(&quot;unemployment_r&quot;) &lt;= 4, &quot;Low&quot;)<br>    .when((col(&quot;unemployment_r&quot;) &gt; 4) &amp;&amp; (col(&quot;unemployment_r&quot;) &lt;= 11), &quot;Moderate&quot;)<br>    .otherwise(&quot;High&quot;)<br>)<br><br>//Saving data to a Hive table<br>permanent_table_name = &quot;enriched_gdp_over_hours_worked&quot;<br>enrichedDF.write.format(&quot;parquet&quot;).saveAsTable(permanent_table_name)</pre><p>In the code example above, we use two datasets df and df2 loaded from a distributed file system (Databricks DBFS). These datasets are retrieved from Kaggle [<a href="https://www.kaggle.com/datasets/joebeachcapital/gdp-per-hour-worked/data">3</a>] and they represent the data used to compare the productivity growth and the economy in Europe and the USA in the following article [<a href="https://www.economist.com/graphic-detail/2023/10/04/productivity-has-grown-faster-in-western-europe-than-in-america">4</a>]. The full code that includes the data loading lines is provided in Dataness.AI’s <a href="https://github.com/dataness-ai/Meduim_Spark_Scala">GitHub</a>. In our example, we employed the Spark DataFrame DSL, which offers a concise and elegant approach to writing data pipelines, rather than using the alternative solution to developing those pipelines by mixing Spark Scala code with SQL query strings.</p><p>The proposed data pipeline executes a series of transformations on df and df2 to enrich df2 with new descriptive columns that will be helpful in the data analysis of populations in the world. It executes :</p><ul><li>Classification of the data into different categories based on the unemployment rate value.</li><li>Aggregation of the existing data using various grouping combinations.</li><li>Transformation of date columns to enhance clearer comparisons with existing records.</li><li>Filtering, joining and ordering the data based on specific columns.</li></ul><p>The resulting aggregated data is stored in Hive for future use (e.g. in a dashboard).</p><p><strong>Conclusion:</strong></p><p>In this article, I’ve explored the internal workings of Spark and its components, which enable the creation of scalable and efficient data processing applications, leveraging its distribution and parallelism capabilities. I also showcased an example of an ETL application that reads and loads data from external sources. Please feel free to post any questions you have about Spark and the libraries discussed here in the comment section below. Also, I welcome your suggestions for the next data engineering tools that you would like to see explained.</p><p><strong>Bibliography:</strong></p><p>[1] Apache Spark. (n.d.). Cluster Overview. Spark Official Documentation. <a href="https://spark.apache.org/docs/latest/cluster-overview.html">https://spark.apache.org/docs/latest/cluster-overview.html</a></p><p>[2] Chaudhari, V. (2023, Sept 12). Understanding Spark DAGs. Medium. <a href="https://medium.com/plumbersofdatascience/understanding-spark-dags-b82020503444">https://medium.com/plumbersofdatascience/understanding-spark-dags-b82020503444</a></p><p>[3] The Economist and Solstad, Sondre, 2023. “All work and no play”, The Economist, October 4th issue, 2023.</p><p>[4] The Economist and Solstad, Sondre. (2023, Oct 04).GDP per hour worked. Kaggle. <a href="https://www.kaggle.com/datasets/joebeachcapital/gdp-per-hour-worked/data">https://www.kaggle.com/datasets/joebeachcapital/gdp-per-hour-worked/data</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d186d02115f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/notes-about-spark-internals-5d186d02115f">Notes about Spark internals</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Anomaly Detection application Architecture Design]]></title>
            <link>https://medium.com/dataness-ai/an-anomaly-detection-application-architecture-design-ccc8720aafd1?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/ccc8720aafd1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Asma Zgolli, PhD]]></dc:creator>
            <pubDate>Fri, 08 Sep 2023 15:47:09 GMT</pubDate>
            <atom:updated>2023-09-08T13:44:25.113Z</atom:updated>
            <content:encoded><![CDATA[<p>As the first blog post for a series about system design for machine learning applications, we present a theoretical solution for an anomaly detection application architecture. The architecture proposes near real-time data ingestion and model training using isolation trees for accurate anomaly detection.</p><h3>Problem definition:</h3><p>For an anomaly detection use case, we wish to design a platform (i) for near real-time data ingestion from a data stream and (ii) for model training (online and batch) to solve the problem. The online (re)training should be executed at the rate of data arrival, and the batch (re)training should be triggered at the end of the day, integrating new data into the existing model.</p><h3>Model selection for the anomaly detection problem:</h3><p>The model that should be used for this problem is a decision tree called isolation trees. Outlier data is easier to detect in the tree. Additionally, an anomaly score can be used to detect data points that are located far away from the other data points.</p><h3>Architecture design for the model training (online and batch)</h3><p>High-level project setup:</p><ul><li>Streaming engine that supports near real-time events like Apache Spark Streaming.</li><li>Distributed machine learning techniques to be able to handle all the incoming batch data at the end of the day, like Spark ML or deep learning. The incoming data frequency is 10 MB per second, which will create a voluminous amount of data to process by the end of the day.</li><li>Spark Dataframe API can be used to access the batch training data at the end of each day.</li><li>Jupyter Notebook and Python can be used for development.</li></ul><p>The final project should be composed of the following components:</p><ul><li>Stream data access layer</li><li>Batch data access layer</li><li>Model training and evaluation layer</li><li>Feature selection and engineering layer</li><li>Model adapting layer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*Mwfhmk8swqQFyXvc.png" /><figcaption><em>Figure 1- System architecture design</em></figcaption></figure><p>According to the architecture, the online training method that we chose is model retraining. In order to update the model, we need to update the hyperparameters. For example, we should use lower values for learning rates to avoid retraining the model and, instead, only update the existing model with the new data. In this case, the new and old data should be combined to create a train and test set.</p><p>One approach to integrating online learning and offline model updates is to continuously update the model with each new data arrival. In this case, the learning rate should be updated at every new type of retrain, and this hyperparameter should be studied at the proof of concept (POC) step accordingly. Another approach would be to store all updates in one data store and retrain the model only once a day.</p><p>In order to offer near-real-time visualizations (live dashboards) for model monitoring and prediction, we propose using a streaming engine such as Apache Spark Streaming or Apache Flink to collect the streams. Spark streams produce micro-batches, and Spark Streaming offers this functionality with a streaming delay of just 1 second.</p><p>For the dashboard, we propose using Jupyter Notebook and designing it with libraries such as Seaborn or using the Dash library in Python.</p><p>The implementation needed for the given scenarios is as follows:</p><ul><li>Summary statistics of the data stream: Use aggregation methods in the streaming engine.</li><li>Anomaly scores: Extract anomaly scores by interrogating the stored model.</li><li>Other metrics and KPIs can be computed on the fly while learning anomalies and streaming data, and collected in an insight database (key-value store to ensure very fast data access).</li></ul><p>The final architecture becomes when we add the dashboard :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*2IEALW12Xu7i9kW5.png" /><figcaption><em>Figure 2- System architecture design updated</em></figcaption></figure><p><em>Originally published at The Analytics Gazette, </em><a href="https://asmazgo.github.io/machine-learning/feature/2023/05/18/anomaly-detection.html"><em>https://asmazgo.github.io</em></a><em> on May 18, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccc8720aafd1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/an-anomaly-detection-application-architecture-design-ccc8720aafd1">An Anomaly Detection application Architecture Design</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Recommender System Architecture Design]]></title>
            <link>https://medium.com/dataness-ai/a-recommender-system-architecture-design-cf0dea7c1792?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf0dea7c1792</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Asma Zgolli, PhD]]></dc:creator>
            <pubDate>Fri, 08 Sep 2023 15:46:41 GMT</pubDate>
            <atom:updated>2023-09-08T13:46:37.779Z</atom:updated>
            <content:encoded><![CDATA[<p>As the second installment in a blog series on Machine Learning system design, we present a straightforward architecture for training and maintaining a recommender system tailored to video-sharing platforms such as YouTube.</p><p>This problem of ML system design consists of two major building blocks:</p><p>1- Offline component: This component focuses on training and validating the recommender model.</p><p>2- Online component: This component is responsible for generating (inferring) recommendations in real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/0*xlF7GWlRb4EprIVE.png" /><figcaption><em>Figure 1- System architecture design</em></figcaption></figure><p>The initial offline training process follows the classic Data Science lifecycle, beginning with data collection and transformation into features, and concluding with model building and validation. The outcome of these steps is a dataset that captures the features and is stored in a feature store, along with a model that is persisted in a blob storage medium, such as a distributed file system like S3.</p><p>This article focuses specifically on the online component. In this subsystem, a user submits a request through a REST API, for instance. The request is then forwarded to an application server that encompasses the recommender functions. In our architecture, we propose utilizing a load balancer to provide increased scalability, performance, reliability, and redundancy to the service layer. Scaling this layer horizontally enables us to handle requests from billions of users and deliver improved response times by minimizing throughput.</p><p>As depicted in the architecture, the recommender module consists of two components: (i) a content filtering (CF) algorithm that identifies relevant videos for each user based on their profile and historical viewing data, and (ii) a ranking algorithm that assigns scores to the recommended videos.</p><p>The techniques employed in content filtering (CF) can range from basic approaches, such as simple similarity matching, to more advanced methods involving deep neural network models. In this particular example, we focus on utilizing a model as hinted in the above paragraphs. At inference, the CF algorithm reads the model from the model store and other relevant features (e.g. user profile data) from the features store and predicts the best movies that match the user, and its viewing history. In this architecture, we utilize an in-memory caching system to reduce data retrieval latencies that may arise when accessing a slower storage layer.</p><p>After generating the recommendation, the algorithm sends business metrics like turnover and click rates, and system-level metadata like the inference timestamp and its response time to the model store. These metrics help in (i) assisting the decision-making process behind triggering the model retraining and (ii) feeding the monitoring dashboard. Triggering the retraining of the model can occur in a timely manner (for instance, daily model retraining). It can also result from reaching a certain threshold due to data drift and concept drift events.</p><p><em>Originally published at The Analytics Gazette, </em><a href="https://asmazgo.github.io/machine-learning/feature/2023/06/01/recommender-system.html"><em>https://asmazgo.github.io</em></a><em> on June 1, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf0dea7c1792" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/a-recommender-system-architecture-design-cf0dea7c1792">A Recommender System Architecture Design</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Welcome to Dataness.AI !]]></title>
            <link>https://medium.com/dataness-ai/testing-newsletter-8bd96ad1fb93?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/8bd96ad1fb93</guid>
            <dc:creator><![CDATA[Asma Zgolli, PhD]]></dc:creator>
            <pubDate>Tue, 04 Jul 2023 20:41:59 GMT</pubDate>
            <atom:updated>2023-07-04T20:57:17.731Z</atom:updated>
            <content:encoded><![CDATA[<p>Our goal as a community is to facilitate knowledge sharing and collaboration in the development of <strong>non-profit </strong>projects related to various <strong>data</strong> topics. Discover our blogs on <strong>data science</strong>, <strong>AI</strong>, and <strong>machine learning</strong> published at our platform: <a href="https://medium.com/dataness-ai">https://medium.com/dataness-ai</a>. If you are interested in collaborating and proposing a blog post or a project idea, drop a message to: <a href="https://www.linkedin.com/in/dataness-ai-333810274/">https://www.linkedin.com/in/dataness-ai-333810274/</a> or a tweet to: <a href="https://twitter.com/DataNess_AI">https://twitter.com/DataNess_AI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bd96ad1fb93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/testing-newsletter-8bd96ad1fb93">Welcome to Dataness.AI !</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TigerGraph Overview]]></title>
            <link>https://medium.com/dataness-ai/tigergraph-overview-50c949272a5d?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/50c949272a5d</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[graph-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[graph-algorithms]]></category>
            <dc:creator><![CDATA[Asma Zgolli, PhD]]></dc:creator>
            <pubDate>Sat, 29 Apr 2023 23:07:15 GMT</pubDate>
            <atom:updated>2023-05-01T09:35:11.457Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A comprehensive overview of TigerGraph, highlighting its scalable system architecture and describing its unique data science capabilities.</blockquote><p>In this article, we are going to present a general overview of TigerGraph. We will describe its architecture to showcase its capabilities and the problems it is designed to solve. We will also present its data model to understand better how the data is stored and organized within the system. Finally, we will present an example of using TigerGraph in the industry, highlighting its machine-learning support.</p><h4>System Architecture</h4><p>TigerGraph is a distributed, graph-oriented datastore with a massively parallel processing query engine. It differs from other similar datastores on the market due to its horizontally scalable architecture, which allows it to support the execution of workloads on big graphs of up to hundreds of billions of entities and 1 trillion relationships [<a href="https://www.youtube.com/watch?v=ylqO-lOP9pE&amp;ab_channel=TigerGraph">1</a>]. It supports native parallel query execution, enabling faster deep exploration of the data stored in TigerGraph. Its query language, GSQL, is a declarative, SQL-like query language with a lower learning curve for new developers in the field. It is also Turing complete, meaning that any query can be expressed using GSQL, unlike other similar query languages. In the following figure (figure 1), we present two types of GSQL queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/0*SXSAaeVZJkdKE5Fk" /><figcaption>Figure 1- Examples of GSQL queries</figcaption></figure><p>The first query (Q1) is a simple filtering operation performed on all the graph vertices. The second query (Q2), however, lists some information about the edges of a set of paths filtered by a particular condition. The scanning of the paths in query Q2 is expressed using the graph pattern:</p><blockquote>source-vertex-name-(edge-name)-&gt;target-vertex-name.</blockquote><p>The figure below (Figure 2) represents the internal architecture of TigerGraph. The main building blocks of this architecture are (i) the graph storage engine (GSE), responsible for the physical storage and compression of graph data, and (ii) the graph processing engine (GPE), which offers parallel processing and graph partitioning capabilities, which are fundamental for the distribution and scalability of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/0*EGqFbQ5TWwLP957c" /><figcaption>Figure 2: TigerGraph internal architecture [<a href="https://docs.tigergraph.com/tigergraph-server/current/intro/internal-architecture">2</a>]</figcaption></figure><p>As shown in the same figure, <a href="https://zookeeper.apache.org/doc/r3.8.0/index.html">Apache Zookeeper</a> is used to coordinate the system&#39;s different components and manage the distribution. It guarantees that the data is consistently replicated across the servers. TigerGraph also uses <a href="https://kafka.apache.org/documentation/">Apache Kafka</a> for distributed message passing and queuing. Kafka communicates messages containing graph data, query results, and other information between the graph storage engine (GSE) and different instances of the graph processing engine (GPE). These two tools are fundamental in ensuring the reliability and scalability of the graph data store.</p><p>TigerGraph can be deployed on the cloud, on-premises, or on a hybrid infrastructure and can integrate different data sources, such as those from the Hadoop ecosystem or RDBMS solutions. It is also characterized by its high availability thanks to its replication and partitioning capabilities[<a href="https://docs.tigergraph.com/tigergraph-server/current/intro/continuous-availability-overview">3</a>]. How will TigerGraph’s distribution features benefit the user in their graph-based data analytics? This point will be addressed in the last section (Applications).</p><h3>TigerGraph Data Model</h3><p>The data model in TigerGraph is graph-oriented. It is useful for representing connected data. It provides a data structure that helps to model complex relationships between different entities (real-world objects). In a graph, these entities are called vertices (or nodes) and are interconnected by edges. Edges are links that represent the relationship between these nodes. In TigerGraph, these data structures have attributes that can be atomic, structured (having a tuple type or container type [<a href="https://docs.tigergraph.com/">4</a>]), or an accumulator. Accumulators are a type of attribute that stores the count or sum of a particular value. They are used to calculate metrics or statistics about the graph or to store intermediate results in a query. Examples of statistics that can be performed using accumulators include counting the occurrences of a value, comparing attributes with values, and calculating the average of an attribute. TigerGraph supports directed and undirected graphs, as well as weighted graphs.</p><h3>Applications</h3><p>Graph algorithms are often used in data analytics for tasks such as search, pathfinding, centrality or ranking, clustering, community detection, and similarity and classification. TigerGraph can support all these applications for large graphs thanks to its distributed nature and parallel processing query engine.</p><h3>Search and pathfinding</h3><p>These algorithms can help to search through large datasets quickly and efficiently in order to find specific information or patterns.</p><p>Graph search looks for vertices from a starting point by visiting its neighbours one hop at a time. Breadth-first search (BFS) and depth-first search (DFS) are examples of these search algorithms. The first searches the graph by visiting the neighbouring vertices first, and the second searches the path from a selected vertex until meeting the searched criteria and hops to another neighbour in case the search goal is not found. TigerGraph uses BFS in GSQL for graph-oriented queries thanks to its parallel processing capabilities.</p><p>Search algorithms allow the user to conduct deep link analysis in TigerGraph. Its query engine can search up to 10 hops away from the starting point, which is very efficient compared to earlier generation graph data stores that hardly achieve 3 or 2 hops search.</p><p>Another important search application specific to the graph data model is finding the “best” path between two vertices. A popular path-finding algorithm is the shortest path algorithm. Its goal is to minimize the distance between two vertices, and the distance can be measured as the number of hops or using weights assigned to the edges. Another popular path-finding algorithm is the A* algorithm, which is applicable to weighted graphs. Use cases for this type of algorithm include network traffic optimization and process optimization.</p><h3>Graph centrality</h3><p>Graph centrality (also known as influence, priority, or importance) techniques allow us to find the vertex that has the most connections in the graph: the closest vertex to the other neighbours or the most central vertex in the graph between other vertices (betweenness). It is a measure of how important a vertex is compared to its neighbours. The score used to calculate the centrality measure can also be used to rank the vertices of the graph, for example in the algorithm PageRank. Various scores are used to calculate centrality: (i) degree centrality, a measure of the number of neighbours for each vertex and the simplest centrality measure; (ii) betweenness centrality, the vertex with the number of paths between all the other vertices. These vertices are critical for connecting the different clusters in the graph and also belong to the shortest paths between the different vertices of the graph; (iii) closeness centrality, this score calculates all the shortest paths in the graph, then assigns the sum of the shortest path as a closeness degree for each vertex; (iv) eigen centrality, calculates the number of edges a vertex has as well as those of all of its neighbours one hop at a time and aggregates all of these measures; and finally, (v) PageRank, calculates the measure by counting the number of indegrees (incoming links from neighbours) beyond direct neighbours.</p><h3>Community detection and clustering</h3><p>In graph theory, communities or clusters are vertices that have common attributes or share a graph pattern. In the figure below (Figure 3), we have an example of two graph communities. The nodes in both communities share a common pattern of the following relationship: (person)-friends-(person).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/0*eNn0QXjLvLgQGHSs" /><figcaption>Figure 3- Example of a graph having two communities</figcaption></figure><p>Community (cluster) detection uses similarity algorithms and graph patterns to group the vertices.</p><h3>Recommendation</h3><p>In TigerGraph, recommendation engines are built using clustering, similarity, and classification techniques. Similarity in graph algorithms applies to attributes as well as links and graph (/paths) patterns. In TigerGraph, similarity scores are defined as user-defined functions (UDFs). This allows the user to customize the algorithms and adapt them to his application-specific needs. There are various predefined similarity functions in the standard UDF library of TigerGraph, such as [<a href="https://docs.tigergraph.com/graph-ml/current/similarity-algorithms/">5</a>]: (i) the Euclidean similarity, which measures the distance between two points in n-dimensional space using the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/0*fZQ3leoYiFhKO45C" /></figure><p>Where Xvi and Xvj are the attributes of two points Xi and Xj; (ii) the Pearson similarity, calculated using the Pearson correlation formula. The latter is calculated for two variables X and Y as the ratio between their covariance and the product of their standard deviation; and (iii) the cosine similarity, which is the measure of the cosine of the angle between two points represented in an n-dimensional space. It has the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/0*D22K39GtdHP8BJJt" /></figure><p>Where Ai and Bi are the attributes of two points A and B. In TigerGraph, it is possible to calculate three different variants of cosine similarity: single-sourced cosine similarity, calculated for two vertices using the attributes of their edges and those of the edges of their neighbours; batch cosine similarity, another variation of this algorithm that calculates the similarity between multiple pair of points in a single operation which optimizes the performance of the execution; and all pairs cosine similarity, which allows the user to calculate the cosine similarity for all the possible pairs in the graph. For most of these similarities, the attributes of the nodes/edges need to be transformed into numerical values and represented as a vector.</p><h3>Conclusion</h3><p>In this article, we presented a brief overview of TigerGraph and a summary of its graph algorithms for machine learning. For a deeper understanding of this tool, it is possible to explore multiple resources such as webinars and blogs about real-world use cases on their official website [<a href="https://www.tigergraph.com/">6</a>]. This blog post is the first of a two-blog series about Graph Analytics, and in the next one, we will showcase a practical example of using graph data science algorithms to solve problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50c949272a5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/tigergraph-overview-50c949272a5d">TigerGraph Overview</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Temporal Fusion Transformer]]></title>
            <link>https://medium.com/dataness-ai/understanding-temporal-fusion-transformer-9a7a4fcde74b?source=rss----b40f7eff31a4---4</link>
            <guid isPermaLink="false">https://medium.com/p/9a7a4fcde74b</guid>
            <category><![CDATA[model-interpretability]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Mouna Labiadh]]></dc:creator>
            <pubDate>Sat, 29 Apr 2023 22:53:31 GMT</pubDate>
            <atom:updated>2024-03-14T19:51:29.384Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Breakdown of Google’s Temporal Fusion Transformer (2021) for interpretable multi-horizon and multivariate time series forecasting.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*odRUDGDbpf773A_T" /><figcaption>Photo by <a href="https://unsplash.com/@yiorgosntrahas?utm_source=medium&amp;utm_medium=referral">Yiorgos Ntrahas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Temporal Fusion Transformer (TFT) [1] is a powerful model for multi-horizon and multivariate time series forecasting use cases.</p><p>TFT predicts the future by taking as input :</p><ol><li><strong>Past target </strong>values <em>y</em> within a look-back window of length <em>k</em></li><li><strong>Time-dependent exogenous </strong>input features which are composed of apriori <em>unknown inputs</em> <strong><em>z</em></strong> and <em>known inputs</em> <strong><em>x</em></strong></li><li><strong>Static covariates</strong> <strong><em>s</em> </strong>which provide contextual metadata about measured entities that does not depend on time</li></ol><p>Instead of just a single value, TFT outputs prediction intervals via quantiles. Each quantile <em>q </em>forecast of <em>τ</em>-step-ahead at time <em>t</em> takes the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*ymbCB4Kza7xlwU4rSq9Q6Q.png" /></figure><p><em>As an example, to predict future energy consumption in buildings, we can characterize location as a static covariate, weather and data as time-dependent unknown features, and calendar data like holidays, day of week, season, … as time-dependent known features.</em></p><p>Hereafter, an overview of the TFT model architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*OXba8Hi61f6TPQL8yl75Rw.png" /><figcaption>Figure from original paper <a href="https://arxiv.org/pdf/1912.09363.pdf">[1]</a></figcaption></figure><p>Now let’s dive in each of its blocks!</p><h4>Gated residual networks</h4><p>GRN are implemented at different levels of the TFT architecture. They ensure its<em> flexibility</em> by introducing skip/residual connections which feed the output of a particular layer to upper layers in the network that are not directly adjacent.</p><p>This way, the model can learn that some non-linear processing layers are unnecessary and skip them. GRN improve the<em> generalization</em> capabilities of the model across different application scenarios (e.g. noisy or small datasets) and helps to significantly reduce the number of needed parameters and operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*dxjYp3Y-vn9HW-owJUhh6Q.png" /><figcaption>Figure from original paper <a href="https://arxiv.org/pdf/1912.09363.pdf">[1]</a>. ELU stands for Exponential Linear Unit activation function</figcaption></figure><h4>Static covariate encoders</h4><p>Static covariate encoders learn context vectors from static metadata and inject them at different locations of the TFT network:</p><ol><li>Temporal variable selection</li><li>Local processing of temporal representations in the Sequence-to-Sequence layer</li><li>Static enrichment of temporal representations</li></ol><p>This allows to condition temporal representation learning with static information.</p><h4>Variable selection</h4><p>A separate variable selection block is implemented for each type of input : <em>static covariates</em>, <em>past inputs</em> (time-dependent <em>known</em> and <em>unknown</em>) and <em>known future inputs.</em></p><p>These blocks learn to weigh the importance of each input feature. This way, the subsequent Sequence-to-Sequence layer will take as input the re-weighted sums of transformed inputs for each time step. Here, transformed inputs refer to learned linear transformations of continuous features and entity embeddings of categorical ones.</p><p>The external context vector consists in the output of the static covariate encoder block. It is therefore omitted for the variable selection block of static covariates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*D4aPXR39A62uSbz32qIM1g.png" /><figcaption>Figure from original paper <a href="https://arxiv.org/pdf/1912.09363.pdf">[1]</a></figcaption></figure><h4>Sequence-to-Sequence</h4><p>TFT network replaces positional encoding that is found in Transformers <a href="https://arxiv.org/pdf/1706.03762.pdf">[2]</a> by using a Sequence-to-Sequence layer. Such layer is more adapted for time series data, as it allows to capture local temporal patterns via recurrent connections.</p><p>Context vectors are used in this block to initialize the cell state and hidden state of the first LSTM unit. They are also employed in what the authors call <em>static enrichment layer</em> to enrich the learned temporal representation from the Sequence-to-Sequence layer with static information.</p><h4>Interpretable Multi-head attention</h4><p>Attention mechanism weighs the importance of <em>values</em> based on the relationships between <em>keys</em> and <em>queries</em>. This is by analogy with information retrieval that would evaluate a search query (query) against document embeddings (keys) to retrieve most relevant documents (values).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*xp67QEMZKo3Giu4-mBSbVA.png" /></figure><p>where <em>α(Q,K)</em> are attention weights. A common choice for <em>α</em> is the scaled dot-product attention.</p><p>Original multi-attention mechanism, proposed in <a href="https://arxiv.org/pdf/1706.03762.pdf">[2]</a>, consists in using multiple attention heads to re-weigh the <em>values</em> based on the relevance between <em>keys</em> and <em>queries</em>. The outputs of different heads are then combined via concatenation, as follows :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*9RPInucwp6E36afkB5bDNQ.png" /></figure><p>where <em>W_H, W_Q, W_K, W_V</em> are projection weight matrices and <em>h</em> is the number of attention heads.</p><p>In <em>self-attention</em>, <em>queries</em>, <em>keys</em> and <em>values</em> come from the same input. This allows to learn the relevance of each time step with respect to the rest of the input sequence, and therefore to capture <em>long-range temporal dependencies</em>. Note that at the decoder part, subsequent time steps at each decoding step are masked to avoid information <em>leakage</em> from future to past data points.</p><p>TFT adjusts this definition to ensure <em>interpretability</em>. As such, instead of having multiple head-specific weights for <em>values</em>, these are shared across all attention heads. This allows to easily trace back most relevant <em>values</em>. The outputs of all heads are then additively aggregated :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*yTh_gsLwDuRTyXRJjgTudQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*m8QcaIPXO2pn4U_kv0hTAA.png" /><figcaption>Adapted from [2]</figcaption></figure><h4>Quantile regression</h4><p>Instead of just a single value, TFT predicts quantiles of the distribution of target ŷ using a special quantile loss function, also known as <em>pinball loss</em> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*5CyX0ZGrvJXGQHOH_oJ2aQ.png" /></figure><p>Intuitively, the first term of this loss function is activated for under-estimations and is highly weighted for upper quantiles, whereas the second term is activated for over-estimations and is highly weighted for lower quantiles. This way, the optimization process is forcing the model to provide reasonable over-estimations for upper quantiles and under-estimations for lower quantiles. Notice that for the median prediction (0.5 quantile), optimizing quantile loss function is equivalent to that of the MAE loss.</p><p>TFT is then trained by minimizing an aggregate of all quantile losses across all quantile outputs.</p><p>Quantile regression is very useful for high-stakes applications to have some kind of a quantification of uncertainty of predicted values at each time step.</p><h4>Interpretability</h4><p>As discussed in earlier sections, TFT enables a new form of<em> interpretability</em> via:</p><ul><li><em>Variable selection</em> blocks that explicitly learn global importance weights of input features</li><li>Adjusting the standard <em>multi-head attention</em> definition to have shared weights for <em>values</em> across all attention heads</li></ul><p>Interpretable attention weights are useful for 2 cases :</p><ol><li>Easily trace back most relevant past time-steps to predict each forecast. Such insight is traditionally gained using preliminary seasonality and autocorrelation analysis</li><li>Identify significant changes in temporal patterns. This is done by computing an average attention pattern per forecast horizon and evaluate the distance between it and attention weights at each point.</li></ol><h3>Existing implementations</h3><p>Some available TFT model implementations are:</p><ul><li><em>Pytorch Forecasting </em>: <a href="https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html">https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html</a></li><li><em>Darts</em> : (adopted from <em>Pytorch Forecasting</em> implementation) <a href="https://unit8co.github.io/darts/examples/13-TFT-examples.html?highlight=temporal%20fusion">https://unit8co.github.io/darts/examples/13-TFT-examples.html?highlight=temporal%20fusion</a></li></ul><p>Both implementations are based on Pytorch Lightning.</p><p>You can refer to this blog post about using TFT for book sales forecasting:</p><p><a href="https://medium.com/@mouna.labiadh/forecasting-book-sales-with-temporal-fusion-transformer-dd482a7a257c">Forecasting book sales with Temporal Fusion Transformer</a></p><h3>Key Takeaways</h3><p>TFT is a transformer for time series data. It supports:</p><ul><li>Multi-horizon forecasting</li><li>Multivariate time series with heterogeneous features (support for static covariates, time varying known and unknown variables)</li><li>Prediction intervals to quantify uncertainty</li><li>Interpretablity of results</li></ul><p>TFT is mainly able (1) to capture temporal dependencies at different time scales by a combination of the LSTM Sequence-to-Sequence and the Transformer’s Self-Attention mechanism, and (2) to enrich learned temporal representations with static information about measured entities.</p><p><em>Thank you for reading!</em></p><h3>References</h3><p><a href="https://arxiv.org/pdf/1912.09363.pdf">[1] Lim, Bryan, et al. “Temporal fusion transformers for interpretable multi-horizon time series forecasting.”<em> International Journal of Forecasting</em> 37.4 (2021): 1748–1764.</a></p><p><a href="https://arxiv.org/pdf/1706.03762v5.pdf">[2] Vaswani, Ashish, et al. “Attention is all you need.” <em>Advances in neural information processing systems</em> 30 (201</a>7).</p><p>[3] <a href="https://data-science-blog.com/blog/author/yasuto/">Yasuto Tamura</a>, Multi-head attention mechanism, <a href="https://data-science-blog.com/blog/2021/04/07/multi-head-attention-mechanism/">https://data-science-blog.com/blog/2021/04/07/multi-head-attention-mechanism/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a7a4fcde74b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataness-ai/understanding-temporal-fusion-transformer-9a7a4fcde74b">Understanding Temporal Fusion Transformer</a> was originally published in <a href="https://medium.com/dataness-ai">DataNess.AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>