<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nakul Upadhya on Medium]]></title>
        <description><![CDATA[Stories by Nakul Upadhya on Medium]]></description>
        <link>https://medium.com/@upadhyan?source=rss-4d9dddc62a80------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W-J3yFPrfgauYcSYeyEubg.jpeg</url>
            <title>Stories by Nakul Upadhya on Medium</title>
            <link>https://medium.com/@upadhyan?source=rss-4d9dddc62a80------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:52:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@upadhyan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introduction to Interpretable Clustering]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/introduction-to-interpretable-clustering-d0e07fbd2c99?source=rss-4d9dddc62a80------2"><img src="https://cdn-images-1.medium.com/max/2600/0*OoO4kpDovCHGNXSe" width="6000"></a></p><p class="medium-feed-snippet">What is interpretable clustering and why is it important.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/introduction-to-interpretable-clustering-d0e07fbd2c99?source=rss-4d9dddc62a80------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/introduction-to-interpretable-clustering-d0e07fbd2c99?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0e07fbd2c99</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Thu, 01 Aug 2024 16:00:28 GMT</pubDate>
            <atom:updated>2024-08-05T20:21:44.728Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Neural Basis Models for Interpretability]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/neural-basis-models-for-interpretability-fd04ac958ff2?source=rss-4d9dddc62a80------2"><img src="https://cdn-images-1.medium.com/max/1888/1*nZ6UitvWW-P7mOUiD9tQZA.png" width="1888"></a></p><p class="medium-feed-snippet">Unpacking the new interpretable model proposed by Meta AI</p><p class="medium-feed-link"><a href="https://medium.com/data-science/neural-basis-models-for-interpretability-fd04ac958ff2?source=rss-4d9dddc62a80------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/neural-basis-models-for-interpretability-fd04ac958ff2?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd04ac958ff2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Wed, 11 Oct 2023 04:45:19 GMT</pubDate>
            <atom:updated>2023-10-11T04:45:19.266Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Critical Tools for Ethical and Explainable AI]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/critical-tools-for-ethical-and-explainable-ai-ed0e336d82a?source=rss-4d9dddc62a80------2"><img src="https://cdn-images-1.medium.com/max/2600/0*1chlcYr9HTjtbhAp" width="5622"></a></p><p class="medium-feed-snippet">A guide to essential libraries and toolkits that can help you create trustworthy yet robust models</p><p class="medium-feed-link"><a href="https://medium.com/data-science/critical-tools-for-ethical-and-explainable-ai-ed0e336d82a?source=rss-4d9dddc62a80------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/critical-tools-for-ethical-and-explainable-ai-ed0e336d82a?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed0e336d82a</guid>
            <category><![CDATA[fairness]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Wed, 19 Jul 2023 16:54:41 GMT</pubDate>
            <atom:updated>2023-07-20T14:15:26.205Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[NODE: Tabular-Focused Neural Trees]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/node-tabular-focused-neural-trees-ee08c752fcd2?source=rss-4d9dddc62a80------2"><img src="https://cdn-images-1.medium.com/max/627/0*dN_cCC8xwwKLlSkO" width="627"></a></p><p class="medium-feed-snippet">Exploring NODE: A Neural Decision Tree architecture for tabular data</p><p class="medium-feed-link"><a href="https://medium.com/data-science/node-tabular-focused-neural-trees-ee08c752fcd2?source=rss-4d9dddc62a80------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/node-tabular-focused-neural-trees-ee08c752fcd2?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee08c752fcd2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Tue, 04 Jul 2023 14:40:17 GMT</pubDate>
            <atom:updated>2023-07-04T14:40:17.701Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Neural Prototype Trees]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/neural-prototype-trees-f7bac36437a9?source=rss-4d9dddc62a80------2"><img src="https://cdn-images-1.medium.com/max/2120/1*ieH-Saaj9qpbWEKEQoXXOQ.png" width="2120"></a></p><p class="medium-feed-snippet">Explainable image classification through mimicking human reasoning.</p><p class="medium-feed-link"><a href="https://medium.com/data-science/neural-prototype-trees-f7bac36437a9?source=rss-4d9dddc62a80------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/neural-prototype-trees-f7bac36437a9?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7bac36437a9</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Fri, 02 Jun 2023 15:41:36 GMT</pubDate>
            <atom:updated>2023-06-02T15:47:57.606Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Return of the Fallen: Transformers for Forecasting]]></title>
            <link>https://medium.com/data-science/the-return-of-the-fallen-transformers-for-forecasting-24f6fec5bc30?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/24f6fec5bc30</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Wed, 24 May 2023 21:24:36 GMT</pubDate>
            <atom:updated>2023-05-24T21:24:36.786Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5Z05wNqb9xmJYc9N" /><figcaption>Photo by <a href="https://unsplash.com/@aditya1702?utm_source=medium&amp;utm_medium=referral">Aditya Vyas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Introducing a new transformer model: PatchTST</h4><p>Lately, there has been a significant surge in the adoption of Transformer-based approaches. The remarkable achievements of models like BERT and ChatGPT have encouraged researchers to explore the application of this architecture in various areas, including time series forecasting. However, recent work by researchers at the Chinese University of Hong Kong and the International Digital Economy Activity showed that the implementations developed for this task were less than optimal and could be beaten by a simple linear model on various benchmarks [1].</p><p>In response, researchers at Princeton and IBM proposed PatchTST (Patched Time Series Transformer) in their paper<a href="https://arxiv.org/abs/2211.14730"> A Time Series is Worth 64 Words </a>[2]. In this paper, Nie et. al introduces 2 key mechanisms that bring transformers back to the forecasting arena:</p><ol><li>Patched attention: Their attention takes in large parts of the time series as tokens instead of a point-wise attention</li><li>Channel Independence: different target series in a time series are processed independently of each other with different attention weights.</li></ol><p>In this post, I aim to summarize how these two mechanisms work and discuss the implications of the results discovered by Nie et. al [2].</p><h3>Background: Are Transformers Effective</h3><p>Before we dive into PatchTST, we need to first understand the problems that Zeng et. al discovered with self-attention in the forecasting space. For those interested in a detailed summary, I highly encourage reading the original paper or the summary I have written on their work:</p><ul><li><a href="https://arxiv.org/abs/2205.13504">Are Transformers Effective for Time Series Forecasting?</a></li><li><a href="https://towardsdatascience.com/transformers-lose-to-linear-models-902164ca5974">Do Transformers Lose to Linear Models?</a></li></ul><p>In summary, self-attention has a few key problems when applied to the forecasting space. More specifically, prior time-series transformers had point-wise self-attention mechanisms where each individual time-stamp was treated as a token. However, this has two main issues. For one, this causes the attention to be permutation-invariant and the same attention values would be observed if you were to flip points around. Additionally, a single timestamp doesn’t have a lot of information contained by itself and gets its importance from the timestamps around it. A parallel in language processing is if we focused on individual characters instead of words.</p><p>These problems led to a few interesting results when testing forecasting transformers:</p><ol><li>Transformers seemed incredibly prone to overfitting on random patterns since adding noise to the data did not decrease the performance of the transformers significantly.</li><li>Longer lookback periods did not help with the accuracy, indicating that the transformers were unable to pick up on significant temporal patterns.</li></ol><h3>Here Comes PatchTST</h3><p>In an attempt to address the issues that transformers have in this space, Nie et. al [2] introduced two main mechanisms that differentiate PatchTST from prior models: Channel Independence and Patching.</p><p>In previous works, all target time series would be concatenated together into a matrix where each row of the matrix is a single series and the columns are the input tokens (one for each timestamp). These input tokens would then be projected into the embedding space, and these embeddings were passed into a single attention layer</p><p>PatchTST instead opts for Channel Independence where each series is passed independently into the transformer backbone (Figure 1.b). This means that every series has its own set of attention weights, allowing the model to specialize better. This approach is commonly used in convolution networks and has been shown to significantly improve the accuracy of networks [2]. Channel independence also enables the use of the second mechanism: patching.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKRgQIFXt0Rrff5DZYTf4g.png" /><figcaption>Figure 1: PatchTST Model Overview (Figure from Nie et. al 2023 [2])</figcaption></figure><p>As mentioned before, attending on single time-steps is like attending on a single character. You end up losing a sense of order (“dog ”and “god” would have the same attention values with character-wise attention) and also increase the memory usage of the model at hand. So what’s the solution? We need to attend to words obviously!</p><p>Or more accurately, the authors propose to split each input time series up into fixed-length patches [2]. These patches are then passed through dedicated channels as the input tokens to the main model (the length of the patch is the token size) [2]. The model then adds a positional encoding to each patch and runs it through a vanilla transformer [3] encoder.</p><p>This approach naturally allows PatchTST to take local semantic information into account that is lost with point-wise tokens. Additionally segmenting the time series significantly reduces the number of input tokens needed, allowing the model to capture information from longer sequences and dramatically reducing the amount of memory needed to train and predict [2]. Additionally, the patching mechanism also makes it viable to do representational learning on time series, making PatchTST an even more flexible model [2].</p><h4>Forecasting Experimental Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/0*eS8bYTITU1KJ2TgK.png" /><figcaption>Figure 2: MSE &amp; MAE results on benchmarks. First place is bolded, second place is underlined. (Figure from Nie et. al 2023)</figcaption></figure><p>In their paper, the authors test two different variations of PatchTST: one with 64 patches inputted into the model (hence the title) and one with 42 patches. The 42 patch variant has the same lookback window as the other models, therefore it can be thought of as a fair comparison to the other models. For both variants, a patch length of 16 and a stride of 8 were used to construct the input tokens [2]. As seen in Figure 2, the PatchTST variants dominate the results, with DLinear [1] winning in a very small number of cases. On average, PatchTST/64 achieved a 21% reduction in MSE and a 16.7% reduction in MAE. PatchTST/42 achieved a 20.2% reduction in MSE and a 16.4% reduction in MAE.</p><h3>Conclusion</h3><p>PatchTST represents a promising future for Transformer architectures in the time-series forecasting task, especially as patching is a simple and effective operator that can easily be implemented in future models. Additionally, the accuracy success of PatchTST indicates that time-series forecasting is indeed a complex task that could benefit from capturing complex non-linear interactions.</p><h3>Resources and References</h3><ol><li>PatchTST Github repository: <a href="https://github.com/yuqinie98/PatchTST">https://github.com/yuqinie98/PatchTST</a></li><li>An implementation of PatchTST can be found in NeuralForecast: <a href="https://nixtla.github.io/neuralforecast/">https://nixtla.github.io/neuralforecast/</a></li><li>If you are interested in neural forecasting architectures that aren’t transformers, consider reading my previous article on Neural Basis Analysis Networks: <a href="https://towardsdatascience.com/xai-for-forecasting-basis-expansion-17a16655b6e4">https://towardsdatascience.com/xai-for-forecasting-basis-expansion-17a16655b6e4</a></li></ol><p>If you are interested in Forecasting, Deep Learning, and Explainable AI, consider supporting my writing by giving me a follow!</p><h4>References</h4><p>[1] A. Zeng, M. Chen, L. Zhang, Q. Xu. <a href="https://arxiv.org/abs/2205.13504">Are Transformers Effective for Time Series Forecasting?</a> (2022). Thirty-Seventh AAAI Conference on Artificial Intelligence.</p><p>[2] Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2023). International Conference on Learning Representations, 2023.</p><p>[3]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, I. Polosukhin. <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a> (2017). 31st Conference on Neural Information Processing Systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24f6fec5bc30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-return-of-the-fallen-transformers-for-forecasting-24f6fec5bc30">The Return of the Fallen: Transformers for Forecasting</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks as Decision Trees]]></title>
            <link>https://medium.com/data-science/neural-networks-as-decision-trees-89cd9fdcdf6a?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/89cd9fdcdf6a</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Mon, 03 Apr 2023 18:22:22 GMT</pubDate>
            <atom:updated>2023-06-20T15:18:56.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*stQ6HuERA6IQ5B3k" /><figcaption>Photo by <a href="https://unsplash.com/@madebyjens?utm_source=medium&amp;utm_medium=referral">Jens Lelie</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Get the power of a Neural Network with the interpretable structure of a Decision Tree</h4><p>The recent boom in AI has clearly shown the power of deep neural networks in various tasks, especially in the field of classification problems where the data is high-dimensional and has complex, non-linear relationships with the target variables. However, explaining the decisions of any neural classifier is an incredibly hard problem. While many post-hoc methods such as DeepLift [2] and Layer-Wise Relevance Propagation [3] can help with explaining individual decisions, explaining the global decision mechanisms (or what the model generally looks for) is much more difficult.</p><p>Because of this, many practitioners in high-stakes fields instead opt for more interpretable models like basic Decision Trees since the decision hierarchy can be clearly visualized and understood by stakeholders. However, basic trees by themselves often do not provide enough accuracy for the task at hand and often ensemble methods like Bagging or Boosting are used to improve the model’s performance. This however again sacrifices some interpretability as in order for one to understand a single decision, a practitioner would need to look through hundreds of trees. However, these are still preferred over deep networks as at least feature importance (on both a local and global scale) can be easily retrieved and displayed.</p><p>So the problem at hand is that we want the discriminative power of a neural network, but with the interpretability of a decision tree. So why don’t we just structure our network as a tree? Well, that is the main approach taken by Fross and Hinton (2017) in their paper “Distilling a Neural Network into a Soft Decision Tree” [1]. In this article, I will break down the key mechanisms behind a Neural Decision Tree and explain some of the benefits of their approach as well as some factors one may need to consider when implementing this methodology in practice. While we will mainly be discussing classification trees, the approaches detailed can also be applied to regression trees with a relatively small number of tweaks.</p><h3>Methodology</h3><h4>Soft Vs. Hard Decision Trees</h4><p>Before diving into how to construct a neural network into a soft decision tree, let&#39;s first define what a soft decision tree is.</p><p>When people think of decision trees (such as the ones implemented in sklearn), they are thinking about hard decision trees where every decision is deterministic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PYzOixfTCLa0JK2DC7pnNQ.png" /><figcaption>Example of a Hard Decision Tree (Image by Author)</figcaption></figure><p>If a condition is met, we go towards the left branch, otherwise, we go right. Each leaf node has a class and a prediction is made by simply going through the tree and picking the class we end up in. The large we allow the tree to grow, the more paths we can take to achieve the final decision.</p><p>Soft decision trees have many similarities, but work slightly differently</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kN7SeGDwlSXZJR0CkYoYmw.png" /><figcaption>Example of a soft decision tree (Image by Author)</figcaption></figure><p>While in hard decision trees, each branching is deterministic, soft decision trees define the probability of going into a certain branch if the condition is met. So while a hard decision tree outputs a single value, <strong>soft decision trees instead output a probability distribution for all possible classes where the probability of a class is the product of the probabilities that we travel through to reach the leaves. </strong>For example, the probability of approval in the tree above is equal to <em>P(b1|X)(1-P(b2|X) + (1-P(b2|X))(P(b3|X)). </em>Classification decisions are then just the class with the highest probability.</p><p>This structure has many benefits. For one, having non-deterministic decisions provides users with an idea of the uncertainty present in a given classification. Additionally, technically hard trees are just special variants of soft trees where all branching probabilities are all equal to 1.</p><p>One downside of these trees is the slight reduction in interpretability. From a stakeholder standpoint, its easier to understand “we approved a loan because the individual made $100k a year and had less than $400k in debt” compared to:</p><blockquote>If the income is $110k we have a .7 probability to go right and if the debt is under 400k we have a .8 probability to approve which results in in a .56 probability plus whatever happens in the left branch</blockquote><p>That doesn’t mean that these are not interpretable (as one can still see exactly what the model is looking at) but just require a bit more help from the model developer.</p><h4>Oblique Decision Trees</h4><p>The second concept that is required before getting into Neural Decision Trees is the concept of “Oblique” Decision trees.</p><p>Traditional decision trees are considered “Orthogonal” trees in that their decisions are made orthogonal to a given axis. Simply put, only one variable is used in any given decision. Oblique trees on the other hand use multiple variables in their decision-making, usually in a linear combination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*paEQneylDFlxNWQs.png" /><figcaption>Example of an oblique decision boundary (Figure from Zhang et. al 2017 [4])</figcaption></figure><p>Some examples of the values in decision nodes could be “Income — Debt &gt; 0”. This can result in stronger decision boundaries. One downside is that without proper regularization, these boundaries can get increasingly complex.</p><h4>Putting it together</h4><p>Now that we understand Soft and Oblique decision trees, we can put these together to understand the neural formulation.</p><p>The first component is the decision nodes. For each node what we need is some probability based on the input value. To achieve this, we can use the bread and butter of neural networks: Weights and activations. In each decision node, we will first take the linear combination of the input variables and then apply a sigmoid function on the sum, resulting in the branching probability.</p><p>To prevent extremely soft decisions (and make the decision tree more like a hard decision tree), <a href="https://ojs.aaai.org/index.php/AAAI/article/view/17123">a tempered sigmoid</a> (or a multiplication of the linear combination before applying the sigmoid) can be used instead.</p><p>Each leaf node contains an N-dimensional tensor where N is the number of classes. This tensor represents the probability distribution of samples being in a class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wHpMhr0xTiS-UMRtcqItiw.png" /><figcaption>Neural Network as a Decision Tree (Replication of a figure from Frosst &amp; Hinton 2017 [1])</figcaption></figure><p>Like in a soft decision tree, the output of this neural tree is a probability distribution of the classes. The output distribution is equal to the sum of the distributions multiplied by the path probability to reach that distribution.</p><h4>Training the Tree</h4><p>One benefit of neural trees is that they can be trained through continuous optimization algorithms like Gradient Descent instead of the greedy algorithms normal decision trees need to be constructed with. All we need to do is define the loss function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*LPhpuXaLtx2Mwu1RRZUjuQ.png" /><figcaption>The loss function for the neural tree (Image from Frosst &amp; Hinton 2017 [1])</figcaption></figure><p>The loss function for this tree is similar to cross-entropy loss. In this equation, <em>P^l(</em><strong>x</strong><em>)</em> is the probability of reaching the leaf node <em>l</em> given the data point <strong>x, </strong><em>T_k</em> is the probability of the target being class <em>k</em> (1 or 0), and Q_k^l is the element of the tensor (probability distribution) in the leaf node <em>l</em> that corresponds to class <em>k</em>.</p><p>One important note about this structure is that <strong>the tree shape is fixed</strong>. Unlike normal decision trees which use a greedy algorithm to split one node at a time and grow the tree, with this soft decision tree we first set the size of the tree and then use gradient descent to update all parameters simultaneously. One benefit of this approach is that it is much easier to constrain the size of the tree without losing much discriminative power.</p><p>One potential pitfall we may run into during the training process is that the model can heavily favor a single branch and not leverage the power of a tree. To avoid getting stuck on poor solutions, it is recommended to introduce a penalty to the loss function that encourages the tree to leverage both left and right sub-trees.</p><p>The penalty is the cross-entropy between the desired averaged distribution (50/50 for both left and right trees) and the actual average distribution defined as alpha.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*tukwvzT3TTJjTmaZhBrlPQ.png" /><figcaption>Definition of alpha for a given node i (Image from Frosst &amp; Hinton 2017 [1])</figcaption></figure><p>In this equation, <em>P^i</em>(<strong>x</strong>) is the path probability from the root node to node <em>i</em> for a given data point <strong>x</strong>. We then sum the penalty across all internal nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zcd1xzrkT-KaK_uQxtT6fA.png" /><figcaption>Definition of the penalty (Image from Frosst &amp; Hinton 2017 [1])</figcaption></figure><p>In this equation, lambda is a hyperparameter that determines the strength of the penalty. However, this may cause some problems as we descend the tree, the data has less of a chance to fall into a 50/50 split, so therefore it is encouraged to instead use an adaptive lambda that changes depending on the depth of the tree. This would modify the penalty to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*fcqBXKEv7MnoL7Da76toug.png" /><figcaption>Modified Penalty function (Image by author, taken from text in Frosst &amp; Hinton 2017 [1])</figcaption></figure><p>As we go deeper into the tree, it is recommended to decay the lambda proportional to 2^-d.</p><h4>Visualizing Results</h4><p>While the reformulation of a neural network into a tree is interesting, the main reason one would pursue this avenue is to provide more model interpretability.</p><p>Let’s first see the interpretation on a classic problem—digit classification in MNIST:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uEt3V60Gp81XLjDV" /><figcaption>Example on MNIST (Image from Frosst &amp; Hinton 2017 [1])</figcaption></figure><p>In the figure above, the images at the inner nodes are the learned filters, and the images at the leaves are visualizations of the learned probability distribution over classes. For each leaf and node, the most likely classification is annotated in blue.</p><p>Looking at this tree, we can see some interesting features. For example, if we look at the right-most internal node the potential classifications are between 3 and 8. In fact, we can actually see the outline of the 3 in the decision node visualization. The white areas seem to indicate that the model looks for lines that close out the inner loops of a 3 to make it an 8. We can also see that the model looks for the shape of a zero in the 3rd node from the left in the bottom set of inner nodes.</p><p>Another interesting example is predicting wins in Connect4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*mt1ruYZqJg5vIrNws0fuqA.png" /><figcaption>Visualization of the first 2 Layers of a neural decision tree predicting who would win a Connect4 game (Image from Frosst &amp; Hinton 2017)</figcaption></figure><p>The learned filters in this example show that the game can be split into two distinct types of games: ones where the players have primarily focused on the edge of the board versus the ones where players have placed pieces in the center of the board.</p><h3>Conclusion</h3><p>Constructing Neural Networks as a Soft-Decision Tree allows us to leverage the power of neural networks while still preserving some interpretability. As shown by the results on the MNIST dataset, the learned filters can provide both local and global explainability, a trait that is welcome and helpful for higher stake tasks as well. Additionally, the training method (optimizing and updating the entire tree at a time) allows us to get more discriminative power while keeping the size of the tree fixed, something we are not able to achieve with normal decision trees.</p><p>Despite this, Neural Trees are not perfect. The soft nature of the tree does mean that the data scientist using these needs to “preprocess” the tree before presenting it to non-technical stakeholders, while normal decision trees can be shown as is (as they are pretty self-explanatory). Additionally, while the oblique nature of the tree helps with accuracy, having too many variables in a given node can again make it harder to explain. This means that regularization is not only recommended but to a certain extent it is necessary. Also, no matter how interpretable a model is, there is still <a href="https://towardsdatascience.com/defining-interpretable-features-ebd7ed94897">the need for interpretable features that stakeholders can understand</a>.</p><p>All of these drawbacks however do not detract from the potential of these models to grow the interpretability-performance frontier. I highly encourage everyone to try these models in their next data science task. I also recommend that everyone <a href="https://arxiv.org/abs/1711.09784#">read the original paper</a>.</p><h3>Resources and References</h3><ol><li><a href="https://github.com/kimhc6028/soft-decision-tree">Implementation of the Soft Decision Tree in PyTorch</a></li><li>For more information on XAI and time-series forecasting, follow</li></ol><h4>References</h4><p>[1] N. Frosst, G. Hinton. <a href="https://arxiv.org/abs/1711.09784#">Distilling a Neural Network Into a Soft Decision Tree</a> (2017). 2017 Artificial Intelligence in Action Conference</p><p>[2] A. Shrikumar, P. Greenside, A. Jundjae. <a href="https://arxiv.org/abs/1704.02685">Learning Important Features Through Propagating Activation Differences</a> (2017). International Conference on Machine Learning PMLR 2017.</p><p>[3] S.Bach, A. Binder, G. Montavon, F. Klauschen, K-R. Muller, W. Samek. <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140">On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation</a> (2015). PloS one, 10(7), e0130140</p><p>[4] L. Zhang, J. Varadarajan, P. N. Suganthan, N. Ahuja, P. Moulin. <a href="https://ieeexplore.ieee.org/abstract/document/8100100">Robust Visual Tracking Using Oblique Random Forests</a> (2017). Conference on Computer Vision and Pattern Recognition 2017.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89cd9fdcdf6a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/neural-networks-as-decision-trees-89cd9fdcdf6a">Neural Networks as Decision Trees</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[XAI for Forecasting: Basis Expansion]]></title>
            <link>https://medium.com/data-science/xai-for-forecasting-basis-expansion-17a16655b6e4?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/17a16655b6e4</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Fri, 24 Mar 2023 14:24:02 GMT</pubDate>
            <atom:updated>2023-03-24T14:24:02.948Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yns63Z1PqSl7TP1z" /><figcaption>Photo by <a href="https://unsplash.com/@orwhat?utm_source=medium&amp;utm_medium=referral">Richard Horvath</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>NBEATS and other Interpretable Deep Forecasting Models</h4><p>Forecasting is a critical aspect of many industries, from finance to supply chain management. Over the years, researchers have explored various techniques for forecasting, ranging from traditional time-series methods to machine learning-based models.</p><p>In recent years, forecasters have turned to deep learning and have gotten promising results with models such as Long Short-Term Memory (LSTM) networks and Temporal Convolution Networks (CNNs) showing great potential. Before 2019, the primary approach to the forecasting problem was combining traditional statistical methods (like ARIMA) with deep learning [1]. However, the forecasting literature broke away from this in 2020 and diverged in two different directions.</p><p>The first direction involves developing means of using the attention mechanism and the transformer architecture for forecasting. This approach was pioneered with the introduction of the LogSparse Transformer by Li et. al [5] which adjusted the traditional self-attention mechanism used in NLP tasks to be more sensitive to locality and use less memory [5]. This work was then later extended and improved by models such as Autoformer [6], Informer [7], FEDFormer [8], and more. This is a thriving field, but recent work has put this approach into question. Zeng et. al has recently questioned the effectiveness of the self-attention mechanism for the forecasting task and did a slew of experiments to show that the attention mechanisms may not be useful for temporal representations [9] (a summary of these experiments can be found in <a href="https://medium.com/towards-data-science/transformers-lose-to-linear-models-902164ca5974">this article</a>). Additionally, this approach also falls a bit short from an Explainable AI (XAI) perspective. While these models all have attention mechanisms that can be visualized, many academics have argued that this may not be explainable and this is an active field of debate. Denis Vorotyntsev has made a<a href="https://towardsdatascience.com/is-attention-explanation-b609a7b0925c"> great article summarizing the debate</a> and I highly encourage checking his article out as well [10].</p><p>In contrast to the attention-based approach of transformers, the other primary direction of tackling the forecasting problem is the neural basis expansion analysis approach first proposed by Oreshkin et. al [1] in 2020 with the N-BEATS architecture. This methodology involves using multiple stacks of deep fully-connected networks to iteratively build the forecast. Instead of directly predicting the time series, the networks in each stack instead predict the weights of a basis. This architecture allows users to specify the components of the time series they want to extract and the reconstructive nature of the forecast adds an extra layer of interpretability.</p><p>In this article, I aim to summarize the full mechanisms behind the basis analysis architecture and also showcase the improvements to the basis expansion approach since the original N-BEATS paper. Specifically, I aim to cover how N-BEATS works [1] and the two improvements to NBEATS: N-HiTS [2] and DEPTS [4].</p><h3>N-BEATS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X9ayTSFuc1Aj2oHG.png" /><figcaption>Figure 1: N-BEATS Architecture from Oreshkin et. al 2020 [1]</figcaption></figure><p>N-BEATS stands for <strong>N</strong>eural <strong>B</strong>asis <strong>E</strong>xpansion <strong>A</strong>nalysis for <strong>T</strong>ime <strong>S</strong>eries and is the origin model for the basis expansion architecture. Because of this, we will spend a bit more time explaining this architecture compared to the others.</p><p>When developing N-BEATS, the author’s goal was to showcase the power of deep learning by creating a forecasting model that did not leverage any statistical concepts such as maximum likelihood estimation or traditional autoregressive models (ex. ARIMA) but still provided some level of interpretability that the traditional time-series approaches provided [1]. In striving for these qualities, Oreshkin et. al not only met their goals but also proved that their model could do better than the state-of-the-art at the time. Some qualities of N-BEATS that make it a strong model are:</p><ol><li>Great performance: On the M4 dataset (a dataset containing 100,000 different time series), N-BEATS was able to beat the top performer at the time, ES-RNN [11], by an average of 3%.</li><li>Large and Multi-Horizon forecasting: The methods N-BEATS uses to generate its forecasts can perform direct multi-step forecasting avoiding the forecast drift issues brought about by iterative forecast generation (passing predictions back into the model)</li><li>Relatively Fast Training: Compared to its Transformer counterparts, N-BEATS only contains MLP stacks and does not have any recurrent networks in its architecture, making the training much simpler.</li><li>Interpretability: When using the interpretable version of N-BEATS (N-BEATSi), users can see a clear breakdown of time series components such as trend and seasonality. other time-series datasets with astounding success.</li></ol><p>To achieve all these goals, N-BEATS uses a few clever architectural components and tricks.</p><h4>Blocks &amp; Basis</h4><p>The block is probably the most important piece of the N-BEATS architecture.</p><p>As showcased in blue in Figure 1 above, the block contains 1 fully-connected network stack which passes its output into 2 different linear layers. We can think of the fully connected layers as generating an encoding of the input. The two linear layers then take in this encoding and project it into 2 sets of weights: one set for the forecast basis and one for the backcast basis. Mathematically, the set of operations can be described as follows (this has a 4-layer fully connected network):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWCoknTuMF5a6RXLfkvRpg.png" /><figcaption>Image from Oreshkin et. al 2022</figcaption></figure><p>In this, <em>x_l</em> is the input into the <em>l-th</em> block and the thetas are the basis weights.</p><h4><strong>Basis</strong></h4><p>Now it&#39;s important to note that when we say basis, we mean in the linear algebra sense. In other words, a basis is a set of linearly independent vectors that we can linearly combine to form any other vector in a vector space. In the case of N-BEATS, the weights of the linear combination are defined by the neural network.</p><p>The generic architecture follows this exact definition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*sC9y-hQ45WhA7soKLaZ0hA.png" /><figcaption>Image from Oreshkin et. al 2022</figcaption></figure><p>Where y is the piece of the time series predicted by block 1. The end output is just the weighted sum of a set of basis vectors defined by the user or set as a trainable parameter. However, this version of the basis is not interpretable and doesn’t attempt to explicitly capture specific components of a time series. However, it is really easy to modify N-BEATS into a more interpretable version by changing the basis equations used. The authors call this new configuration N-BEATSi [1].</p><p>The first interpretable basis is the trend basis. This changes the basis to be a series of low-power polynomials (ex. at+bt²+ct³) and our network is predicting the weights for each polynomial term.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*BoXzEVplbNOULoNe.png" /><figcaption>Image from Oreshkin et. al 2022</figcaption></figure><p>The second interpretable basis is the seasonality basis. Similar to the trend basis, the network predicts the weights for a set of equations, but this time it is predicting the weights for a Fourier series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*TqhmQK1R9OWwR_vs.png" /><figcaption>Image from Oreshkin et. al 2022</figcaption></figure><p>In the end, the network is effectively providing us with a regression equation for our time series, allowing the user to understand the exact workings of the time series [1].</p><h4><strong>Backcasts &amp; Forecasts</strong></h4><p>Each block does not only produce a forecast, but it also produces “backcasts” — a prediction of the input window. The backcast can be thought of as a filtering mechanism. When a block produces a forecast, the backcast it produces is the component of the time series that has already been analyzed and should not be captured by subsequent blocks as the current block has already captured that information [1].</p><p>In the generic architecture, the forecast and backcasts will have different sets of weights produced by different networks and we rely on the training process to make them align with each other. This is unavoidable since the dimensions of the basis vectors for the forecast and backcast are different. However, since the interpretable version uses equations that are dependent on time steps, it is highly recommended to have the backcast and forecast share basis weights when using the interpretable basis [1].</p><h4>Doubly Residual Stacking</h4><p>N-BEATS is not just made up of 1 block, but rather multiple blocks chained together that each interpret a signal of the time series. The key mechanism that allows for this behavior is doubly residual stacking. <strong>Simply put: the input of the current block is equivalent to the input to the previous block minus the backcast of the previous block. Additionally, the final forecast is the sum of all the forecasts from all the blocks [1].</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/0*cFgl37pjggZRsvjM.png" /><figcaption>Image from Oreshkin et. al 2022</figcaption></figure><p>In the generic case, this doubly residual stacking mechanism allows for smoother gradient flows and faster training. Additionally “the aggregation of meaningful partial forecasts” also provides a large degree of interpretability as it helps users identify the key signals present in the time series they are trying to forecast[1] Additionally, by examining the backcasts of all the blocks, the users can see if any signals or patterns were ignored or if any were overfit, helping the debugging process greatly.</p><h4>Stacks</h4><p>To provide more organization to the network, the architecture also organizes the individual blocks into stacks. In each stack, the blocks all share the same type of basis (generic, trend, or seasonal). This organization not only helps the signal decomposition procedure but also allows for more interpretability as now we can simply view the stack-level output instead of examining each block.</p><p>For how these interpretations look, check out <a href="https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/ar.html#Interpret-model">the great example found in the PyTorch Forecasting documentation</a>.</p><h4>Covariates</h4><p>While the original N-BEATS architecture only worked with univariate time series, Challu et. al [3] introduced methods to allow for covariates.</p><p>The first way was to simply flatten and append the covariates to the inputs of the fully connected stack in each block. Each backcast and forecast would still only be for the target time series, and the residual stacking mechanism would only apply to the target, not the covariates.</p><p>The second method was to generate an encoding of the covariates first, and then use that as a basis for a block:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*xGItj3HVfOqBOhoxj6LDYw.png" /><figcaption>Image from Challu et. al 2023</figcaption></figure><h3>DEPTS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bs7KWdIcVHor8gTY.png" /><figcaption>DEPTS Architecture from Fan et. al 2022[4]</figcaption></figure><p>DEPTS stands for <strong>D</strong>eep <strong>E</strong>xpansion <strong>L</strong>earning for <strong>P</strong>eriodic <strong>T</strong>ime <strong>S</strong>eries <strong>F</strong>orecasting and as the name suggests, this is an improvement to NBEATS that focuses on upgrading the periodic/seasonality forecasting capabilities of the original model [4]. Specifically, DEPTS changes the functionalities of NBEATS in two ways:</p><ol><li>While NBEATS/NBEATSi could handle periodicity via the Fourier basis, this method struggles to take larger periodic behaviors that stretch beyond our lookback window into account, something that DEPTS is better equipped to handle [4].</li><li>NBEATS/NBEATSi only can take additive seasonality into account (since it builds the forecasts additively. DEPTS can handle multiplicative seasonalities as well [4].</li></ol><h4>Periodicity Module &amp; Periodic Context</h4><p>The first mechanism DEPTS uses to achieve the two features above is its dedicated periodicity module that takes in the timesteps for both the lookback window and the forecast horizon and generates a seasonality context vector defined by a cosine series [4]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*Jv5B8pPeId21540rl9Nk4w.png" /><figcaption>Image from Fan et. al 2022 [4]</figcaption></figure><p>In this module, <em>A</em>, <em>F</em>, and <em>P </em>are all trainable parameters. <strong>By allowing the network to learn these parameters and share them across predictions, the model can directly learn global periodic patterns instead of having to re-infer the patterns based on the input data as an NBEATS model has to do </strong>[4].</p><p>One tricky piece of this learning process however is that while attempting to learn <em>A</em>, <em>F</em>, and <em>P</em>, the model can easily get stuck in local minima. To get around this, a pre-learning optimization is performed to initialize the parameter values. Note: in this equation, <em>phi </em>represents the set of learnable parameters and <em>M</em> is a binary vector that gets multiplied with every term in the generator function (1 if the term is chosen, 0 else) [4]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g22q3Xp_C_xj29W9BfQ2Qg.png" /><figcaption>Image from Fan et. al 2022 [4]</figcaption></figure><p>To put this simply, before training the network do a pre-fit of parameters on a training set, then choose the <em>J</em> best frequencies (the user chooses <em>J</em>). To avoid solving a whole optimization problem, the authors suggest performing a discrete cosine transform and choosing the <em>J </em>terms with the highest amplitudes [4].</p><h4>Periodicity Block and Periodicity Residuals</h4><p>The whole purpose of the periodicity module is to generate the initial periodic context vector <em>z</em>. These context vectors are passed into periodic blocks [4]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*MfWvH0E5JCm-d4OxjrTQnw.png" /><figcaption>DEPTS Periodic Block Architecture from Fan et. al 2022[4]</figcaption></figure><p>Simply put, the periodic block takes in the periodic context vector (which is effectively the seasonal components of the time series) and passes it through a very small sub-network producing a backcast and a forecast like in the traditional NBEATS block. <strong>This takes the additive seasonalities that the periodicity module handles and processes them to deal with non-linear seasonalities </strong>[4]<strong>.</strong></p><p>How are these periodic backcasts and forecasts used? Diving closer into a layer of DEPTS gives us more insights into what&#39;s happening. Note in this figure <em>x</em> is the target series [4]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0q0AUt1Qtfcx3xsfzlCGQ.png" /><figcaption>DEPTS Layer Architecture from Fan et. al 2022[4]</figcaption></figure><p>Once the model generates the forecast vector backcast and forecast, the backcast is then subtracted from the input of the local block, and the forecast is added to the overall model forecast. The local block is the same as the block in NBEATS, performs the same operations, and uses the same doubly residual stacking mechanism. DEPTS also uses the doubly residual stacking mechanism for the context vector and we remove the explained seasonalities from the periodic context vector and both the local block output and periodic context vector as passed on to the next layer [4].</p><p>All of these mechanisms put together make DEPTS a periodic forecasting beast [4].</p><h4>Interpretability</h4><p>Like NBEATS, DEPTS maintains many of the interpretation benefits of NBEATS, namely the aggregation of forecasts. However, DEPTS also offers a look into global periodic patterns, something NBEATS does not have. It is easy to visualize the global periodic patterns by just running the periodicity module. Additionally, one can examine the final periodic context vector to see how a local prediction may deviate from global patterns by looking at the “leftover” patterns.</p><h3>N-HiTS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WDRxvqV3PnS2zEG9.png" /><figcaption>N-HiTS Architecture from Challu et. al 2022 [2]</figcaption></figure><p>The second major improvement from NBEATS is N-HiTS or <strong>N</strong>eural <strong>H</strong>ierarchical <strong>I</strong>nterpolation for <strong>T</strong>ime Series <strong>F</strong>orecasting [2]. While DEPTS introduced dedicated periodic forecasting mechanisms, N-HiTS focuses on better structuring the forecast aggregation process and offers two main benefits:</p><ol><li>Through a hierarchical method of forecast aggregation, N-HiTS is better at identifying and extracting signals than N-BEATS is in most cases [2].</li><li>Through interpolation mechanisms, N-HiTS’s memory usage is magnitudes smaller than NBEATS’s, making it even easier to train [2].</li></ol><p>All of this is possible due to the Hierarchical Interpolation mechanism which is made up of 2 parts: Multi-Rate Signal sampling and Interpolation [2].</p><h4>Multi-Rate Signal Sampling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*u-Y-Dbb7BnDYmf0L5yybbA.png" /><figcaption>N-HiTS Block Architecture from Challu et. al 2022 [2]</figcaption></figure><p>The first primary mechanism is Multi-Rate signal sampling. This is extremely simple (which makes it even more beautiful): Add a MaxPool layer at the start of each block that pre-processes the block input and effectively smooths it [2]. Why is this useful? Well, w<strong>e can vary the kernel size of the MaxPool layers across each of the blocks and effectively capture different magnitudes of signal sizes [2].</strong> Additionally, the MaxPool also makes the model less susceptible to noise due to the smoothing it provides and by itself is a powerful, but simple tool [2].</p><h4>Interpolation</h4><p>The multi-rate sampling is powerful, but it works even better when combined with an interpolation mechanism.</p><p>Going back to the original generic N-BEATS block architecture, each block generates 2 sets of weights, a backcast set, and a forecast set. Normally, the number of weights generated is equal to the length of the window <em>L</em> (for the backcast) or the length of the horizon <em>H</em> (for the forecast) [1]. However, this approach has a few problems. For one, predicting all the weights can cause the generated forecasts to be more volatile and noisy [2]. Additionally, for larger horizons, this can cause extremely high memory usage [2].</p><p>N-HiTS instead chose to apply interpolation where the number of predicted parameters was instead equal to <em>r*H</em> (and <em>r*L</em> for the backcast) where r is the expressiveness ratio [2]. A higher expressiveness ratio results in more parameters predicted. Then one can fill in the missing weights through any interpolation method. The authors test linear interpolation, nearest neighbor interpolation, and cubic interpolation, but any custom methods can also be used.</p><p>By predicting only a few of the weights, the memory usage of N-HiTS is extremely small when compared to N-BEATS, making it a much more lightweight model.</p><blockquote>NOTE: To my knowledge, the authors have not provided interpolation mechanisms for the interpretable basis functions, so N-HiTS is limited to the generic basis.</blockquote><h4>Hierarchical Interpolation</h4><p>The authors then combine this weight interpolation as well as the multi-rate signal sampling mentioned above to create the hierarchical interpolation mechanism. I<strong>n hierarchical interpolation, both the expressiveness ratio and the MaxPool kernel size of the blocks are inversely varied across the blocks </strong>[2]<strong>.</strong> Any block with a high expressiveness ratio also uses a small kernel size. Any block with a low expressiveness ratio uses a large kernel size. The synchronization of these two parameters is really what allows N-HiTS to capture different magnitude signals as shown in the figure below [2]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9lN2rcUE498UmE3taLrMaw.png" /><figcaption>Linear Interpolation (Left) vs. No Interpolation (Right). Figure from Challu et. al 2022 [2]</figcaption></figure><p>Intuitively this also makes sense: if aggressive smoothing is applied to a function (AKA a large kernel size MaxPool), fewer parameters are needed to capture the behavior of the input and vice versa.</p><p>Hierarchical interpolation also provides more interpretability to the generic form of NBEATS. Users can look at a stack output and understand which one is modeling larger patterns, and which ones are trying to capture small noise. While N-HiTS cannot use the interpretable basis equations, one can make the argument that N-HiTS does not need those to still be interpretable.</p><h3>Conclusion</h3><p>When N-BEATS came out, it changed the deep forecasting space and spawned a new branch of forecasting. Suddenly we could get highly accurate forecasts from a relatively simple model that could be trained end-to-end. N-HiTS and DEPTS were then able to build on NBEATS and add even more functionalities that enhance specific aspects of basis expansion methods. It&#39;s important to note that the hierarchical interpolation of N-HiTS and the periodicity modules/blocks/context vectors of DEPTS can be used together as well, creating an even more powerful model.</p><p>There is also some work in applying attention to these basis expansion methods, mainly the DeepFS model by Jiang et. al [12] which uses an attention layer (adapted from [7]) to generate an encoding, and then pass that encoding into a module that resembles a periodic N-BEATS block as well as into a feed-forward stack. While this can be considered an improvement, I did not include a full detailed description of this model in this article as the benefit of attention in time-series forecasting is still up for debate as shown by [9].</p><p>There is also some debate on whether or not basis expansion is interpretable. If in the end, our models produce 40 different equations, is it interpretable? Additionally, while we can identify what signals all of these models are identifying, the mechanisms for determining these are still relatively black box and we may not know why a model chooses to pick up or ignore a given signal. The interpretability also reduces when adapting these to use covariate time series.</p><p>Despite these flaws, however, all the models discussed here have consistently shown strong performance and have paved the way for more innovation in the forecasting space.</p><h3>Resources and References</h3><h3>Resources and References</h3><ol><li>N-BEATS and N-HiTS can be used through <a href="https://unit8co.github.io/darts/index.html">Darts</a> and <a href="https://pytorch-forecasting.readthedocs.io/en/stable/models.html">Pytorch Forecasting</a>, two high-quality forecasting packages for Python.</li><li><a href="https://github.com/weifantt/DEPTS">Source code for DEPTS</a></li><li><a href="https://github.com/cchallu/n-hits">Source code for N-HiTS</a></li><li><a href="https://github.com/cchallu/nbeatsx">Source code for NBEATSx</a></li><li><a href="https://github.com/ServiceNow/N-BEATS">Source code for NBEATS</a></li></ol><h4>References</h4><p>[1] B.N. Oreshkin, D.Carpov, N. Chapados, Y. Bengio. <a href="https://arxiv.org/abs/1905.10437">N-BEATS: Neural basis expansion analysis for interpretable time series forecasting</a> (2020). Eighth International Conference on Learning Representations.</p><p>[2] C. Challu, K.G. Olivares, B.N. Oreshkin, F. Garza, M. Mergenthaler-Canseco, A. Dubrawski. <a href="https://arxiv.org/abs/2201.12886">N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting</a> (2022). Thirty-Seventh AAAI Conference on Artificial Intelligence</p><p>[3] K.G. Olivares, C. Challu, G. Marcjasz, R. Weron, A. Dubrawski. <a href="https://www.sciencedirect.com/science/article/pii/S0169207022000413">Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with NBEATSx</a> (2023). International Journal of Forecasting.</p><p>[4] W. Fan, S. Zheng, X. Yi, W. Cao, Y. Fu, J. Bian, T-Y. Liu. <a href="https://arxiv.org/abs/2203.07681">DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting</a> (2022). Tenth International Conference on Learning Representations.</p><p>[5] S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y. Wang, X. Yan. <a href="https://proceedings.neurips.cc/paper/2019/hash/6775a0635c302542da2c32aa19d86be0-Abstract.html">Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting</a> (2019). Advances in Neural Information Processing systems 32.</p><p>[6] H. Wu, J. Xu, J. Wang, M. Long. <a href="https://arxiv.org/abs/2106.13008">Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting</a> (2021). Advances in Neural Information Processing Systems 34.</p><p>[7] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, W. Zhang. <a href="https://arxiv.org/abs/2012.07436">Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting</a> (2021). The Thirty-Fifth AAAI Conference on Artificial Intelligence, Virtual Conference.</p><p>[8] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, R. Jin. F<a href="https://arxiv.org/abs/2201.12740">EDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting</a> (2022). 39th International Conference on Machine Learning.</p><p>[9] A. Zeng, M. Chen, L. Zhang, Q. Xu. <a href="https://arxiv.org/abs/2205.13504">Are Transformers Effective for Time Series Forecasting?</a> (2022). Thirty-Seventh AAAI Conference on Artificial Intelligence.</p><p>[10] D. Vorotyntsev. <a href="https://towardsdatascience.com/is-attention-explanation-b609a7b0925c">Is Attention Explanation?</a> (2022). Towards Data Science.</p><p>[12] S. Jiang, T. Syed, X. Zhy, J. Levy, B. Aronchik, Y. Sun. <a href="https://www.amazon.science/publications/bridging-self-attention-and-time-series-decomposition-for-periodic-forecasting">Bridging self-attention and time series decomposition for periodic forecasting</a> (2022). 31st ACM International Conference on Information and Knowledge Management</p><p>[11]S. Smyl, J. Ranganathan, A Pasqua. <a href="https://eng.uber.com/m4-forecasting-competition/">M4 Forecasting Competition: Introducing a New Hybrid ES-RNN Model</a> (2018). Uber Research</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17a16655b6e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/xai-for-forecasting-basis-expansion-17a16655b6e4">XAI for Forecasting: Basis Expansion</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explainable AI and ChatGPT Detection]]></title>
            <link>https://medium.com/@upadhyan/explainable-ai-and-chatgpt-detection-4057e0386fc4?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/4057e0386fc4</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Mon, 06 Feb 2023 20:42:05 GMT</pubDate>
            <atom:updated>2023-02-06T20:42:05.042Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*El_R1QCi8NVN2CulvYBFrQ.png" /><figcaption>Image generated by <a href="https://openai.com/dall-e-2/">DALLE-2</a></figcaption></figure><h4>The measures to counter AI-Generated text in education and other fields need their own transparency measures.</h4><p>When ChatGPT was last November, it took the world by storm. Every influencer on LinkedIn started posting about how to use it for coding and productivity. Entrepreneurs began to think of potential business ideas around generative text models. Even Google was spooked and issued a ‘code red’ due to the chatbot’s ability to provide comprehensive answers.</p><p>But despite this hype, educators around the world immediately saw a huge problem: students using ChatGPT for their homework and essays. According to a Study.com survey, over 89% of students have used ChatGPT for a homework assignment, 48% have used it for a test or quiz, and 53% have used it to write an essay [2].</p><p>OpenAI themselves have included some considerations for education in their ChatGPT documentation, acknowledging the chatbot’s use in academic dishonesty.</p><p>To combat these issues, OpenAI recently released an <a href="https://platform.openai.com/ai-text-classifier">AI Text Classifier</a> that predicts how likely it is that a piece of text was generated by AI from a variety of sources, such as ChatGPT. But this isn’t the only took. Companies like <a href="https://originality.ai/">Originality.ai</a> have been around even longer and perform the same functionality. Even more recently, Stanford University researchers published <a href="https://detectgpt.ericmitchell.ai/">DetectGPT</a>, a zero-shot classifier to detect GPT-2 generated text [4].</p><p>So the plagiarism issue is solved, right? People won&#39;t be able to cheat using chatbots with these tools around, right? The answer to both of these questions is…<em> it’s complicated</em>.</p><p><strong>While all of these tools report high detection accuracy, it is not perfect accuracy, and the false positives and false negatives can cause ethical issues that require their own set of solutions and research directions. </strong>In this article, I am to break down some of these issues around model-based chatbot detection.</p><p>First, I’ll address some of the issues that simply stem from model inadequacies and can be fixed with more training and tuning. These issues are localized to OpenAI’s Text Classifier specifically and may not generalize to production-ready AI-Detectors in general. Many of these issues actually come from OpenAI in their blog post about their text classifier [3], so I highly encourage everyone to give that a read as well</p><p>I then aim to elaborate on some of the more complex issues involved with <em>any </em>AI Text Classifier, possible ways to try and explain their predictions, and the extra work and research that is needed to make these chatbot detectors viable in high-stakes use.</p><h3>Training &amp; Tuning Problems</h3><p>The first set of problems are ones that can simply be solved with more training and tuning. These are:</p><ol><li><strong>The classifier is unreliable on short texts [3].</strong> <em>In my opinion, this limitation could potentially be solved in the future through more tuning and data sets that include smaller texts.</em></li><li><strong>The classifier currently only works on English text, but not on other languages or on code [3].</strong> <em>Like issue 1, this could be solved over time through more tailored data sets and training.</em></li><li><strong>Classifiers based on neural networks are known to be poorly calibrated outside of their training data [3].</strong> <em>There are plenty of techniques to help reduce overfitting in ML models. Additionally, multiple different models could be trained to identify AI-Generated Text in different subject matters, reducing the need for generalization.</em></li><li><strong>The classifier is likely to get things wrong on text written by children because it was primarily trained on English content written by adults.</strong> <em>Providing the classifier with stories and essays written by children can help with this.</em></li></ol><p>While these issues are serious, they are all symptoms of OpenAI’s model <em>still being in production</em>. It is unknown if other companies’ models have these specific issues, but any model still under development will run into these roadblocks.</p><h3>The Hard Problems</h3><p>The previous set of problems could be tackled with some data collection, tuning, and extra training. Unfortunately, there are some problems with AI-detecting tools that more training and development cannot bridge.</p><p>The first problem is when someone simply takes inspiration from a chatbot or heavily paraphrases the chatbot [3]. A Chatbot Detector could pick up on the writing style of a human (since a human re-wrote the chatbot answer) and classify the document as human-written. In these cases, is the detection model wrong? Is this different from summarizing articles online? It is not clear to humans whether or not someone paraphrasing a chatbot is still “AI-writing,” therefore it is impossible for someone to make a model to understand this distinction.</p><p>In a similar vein to the previous problem, the model cannot classify what OpenAI called “predictable text,” text generated from prompts where the answer is very straightforward. If I ask ChatGPT and a human “When did the US, Canada, and Mexico sign NAFTA?”, both of them could give the identical answer of “January 1st, 1994.” If you ask a chatbot detector to classify this as AI-generated vs. human-generated, it can only make a wild guess. While this is a problem, it is not a problem unique to chatbots. Students have been googling answers to questions ever since search engines became popular, and in an examination context, instructors and institutions need to adopt good proctoring practices or change their exams in such a way that the questions require critical thinking instead of regurgitation.</p><h3>The Ethical Problems</h3><p>The above problems, while serious, were all due to various edge cases in the text input. However, even with a non-problematic input, <strong><em>chatbot-detecting models can and will be wrong in many cases. </em></strong>No model can have perfect accuracy in real-world data, but we still need these models, so what do we do?</p><p>Let&#39;s look at a plausible scenario and walk through the various issues we may encounter when using chatbot writing detection.</p><h4>College Admissions</h4><blockquote>Imagine Bill is a senior in high school and is applying to Stanford. He spends 2 and a half months thinking, outlining, and writing all the various essays required for admissions. At the end of this process, he is excited and believes that the admissions officer at his dream school will see his passion… only for him to get a rejection email on the grounds that they believe the essay is ChatGPT generated. He disputes this claim knowing that he never touched that tool and asks what about his essay made the detector flag his essay, but the admissions officers are unable to give an answer, but trust the detector since it has a 99.5% accuracy.</blockquote><p>In this story, the chatbot detectors misclassified Bill’s essay as being ChatGPT generated (a false positive) and people took the prediction at face value due to the high accuracy. However, this completely disregards the law of large numbers. Each college gets thousands of applications. For example, Stanford received around 55,471 applications in 2021 [5]. Even a 0.5% error rate can result in around 280 applications being incorrectly flagged, a huge number considering they only admitted 2,190 students [5]. With these statistics, a dispute process may be needed, but <strong>how would disputes be resolved if even the admissions officers don’t know why the model made a prediction</strong>? This is why we need Explainable AI (XAI).</p><h4>The “Obvious” Solution</h4><p>One potential solution to the above conundrum is to identify word importance. Since these prediction models are often large and complicated, users can leverage post-hoc (after training and prediction), model-agnostic methods like <a href="https://shap.readthedocs.io/en/latest/index.html">SHAP</a> to get an approximation of each word’s attribution to the end prediction. This methodology has been used to provide explanations for sentiment classification, topic tagging, and other NLP tasks and could potentially work for chatbot-writing detection as well.</p><p>Let&#39;s say the admissions officers did have an explanation for why Bill’s essay was flagged and it looked something like this:</p><blockquote>One topic I am deeply passionate about is the application of <strong>graph theory in social network analysis</strong>. I have always been fascinated by the potential of <strong>graph theory</strong> to provide insights into complex systems, and I believe<strong> </strong>that its application in <strong>social network analysis</strong> offers a unique opportunity to understand and make sense of the interconnected world around us. From uncovering<strong> </strong>patterns of human behavior to predicting the spread of ideas and information, the <strong>application of graph theory in social network analysis</strong> holds immense promise. As I pursue my studies, I am eager to delve deeper into this field and contribute to its development, and I believe that attending Stanford University would provide me with the ideal environment to do so.</blockquote><p>The bolded words here are what SHAP says the model picked up on to say that Bill’s essay was AI-written. Looking at this, it seems that the detection model just can’t believe that Bill knows anything about Graph Theory and Social Network Analysis, and if I was the admissions officer I would have more insights into the decision and I could make the judgment that Bill’s essay may be a false positive. Even if the admissions officer blindly trusted the detection model, Bill could dispute this decision because he does know what graph theory is and that the AI incorrectly misclassified his essay.</p><h4>Is this enough?</h4><p>In a perfect world, one could get a cut-and-dry explanation from SHAP like the one above and everyone is happy, but unfortunately, we do not live in a perfect world and these perfect explanations will probably be rare. One problem with post-hoc methods is that they can only provide an <strong><em>estimate </em></strong>of what the model looked at, but they do not actually know what the model is doing internally. Depending on some of the parameters used for the SHAP calculations, we could get an explanation like this:</p><blockquote>One topic I am <strong>deeply passionate</strong> about is the <strong>application </strong>of graph theory in social network analysis. I have always been fascinated by the <strong>potential of graph theory</strong> to provide insights into complex systems, and I believe<strong> </strong>that its <strong>application </strong>in <strong>social network analysis</strong> offers a unique opportunity to understand and make sense of the interconnected world around us. From <strong>uncovering patterns of human behavior to predicting the spread of ideas and information</strong>, the application of graph theory in social network analysis holds immense promise. As I pursue my studies, I am eager to <strong>delve deeper into this field and contribute to its development</strong>, and I believe that attending Stanford University would provide me with the ideal environment to do so.</blockquote><p>This explanation is very different from the previous explanation but came from the same input and the same model. Additionally, which version of the explanation should we trust? It is in this case where post-hoc methods fall short.</p><p>At this point, individuals familiar with the architecture of ChatGPT are probably protesting. Yes SHAP may not be perfect, but we don’t need SHAP! Can’t we just <a href="https://medium.com/nlplanet/two-minutes-nlp-visualizing-global-vs-local-attention-c61b42758019">visualize the model’s attention weights</a>? That is a word importance explanation measure that comes directly from the model. And I agree to an extent. Attention mechanisms have often been touted as an in-built explanation mechanism, allowing any Transformer to be inherently explainable. By visualizing attention weights, we may get a more accurate representation than post-hoc methods like SHAP.</p><h4>Downsides of Attention and Word Importance</h4><p>While attention is an in-built part of the model, it is debated as to whether it actually provides accurate explanations, especially in classification tasks. Some researchers have shown that attention in general does not agree with other explanation methods such as gradient * input and input perturbation [6]. While some of these results have been disputed by other researchers [7], they do showcase some of the unreliability of attention. Additionally, other pieces of literature show that while attention can generate accurate explanations, these may not always be right and attention can only noisily showcase word importance in classification tasks [8].</p><p>Even if attention does accurately show word importance, is the word importance (AKA input saliency) truly the best way to conduct these types of explanations? Can a human look at a saliency map and understand how the highlighted words signal AI-generated content? I don’t think so. Saliency maps also have the unfortunate impact of sometimes providing the same explanation for all possible predictions. Quoting Cynthia Rudin [9]:</p><blockquote>Even if both models are correct (the original black box is correct in its prediction and the explanation model is correct in its approximation of the black box’s prediction), it is possible that the explanation leaves out so much information that it makes no sense.</blockquote><p>Dr. Rudin then showcases a funny example from image classification (one that could translate well to text classification as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/0*8clJ_aFqKzfvfhut.png" /><figcaption>A demonstration of the drawbacks of saliency maps (Image created by C. Rudin [9])</figcaption></figure><p>Imagine if the explanation for why Billy’s essay is GPT generated is:</p><blockquote>One topic I am <strong>deeply passionate</strong> about is the <strong>application </strong>of graph theory in social network analysis. I have always been fascinated by the <strong>potential of graph theory</strong> to provide insights into complex systems, and I believe<strong> </strong>that its <strong>application </strong>in <strong>social network analysis</strong> offers a unique opportunity to understand and make sense of the interconnected world around us. From <strong>uncovering patterns of human behavior to predicting the spread of ideas and information</strong>, the application of graph theory in social network analysis holds immense promise. As I pursue my studies, I am eager to <strong>delve deeper into this field and contribute to its development</strong>, and I believe that attending Stanford University would provide me with the ideal environment to do so.</blockquote><p>But the explanation for why it isn’t AI-written is:</p><blockquote>One topic I am deeply <strong>passionate</strong> about is the <strong>application </strong>of graph theory in social network analysis. I have always been fascinated by the <strong>potential of graph theory</strong> to provide insights into complex systems, and I believe<strong> </strong>that its <strong>application </strong>in <strong>social network analysis</strong> offers a unique opportunity to understand and make sense of the interconnected world around us. From <strong>uncovering patterns of human behavior to predicting the spread of ideas and information</strong>, the application of graph theory in social network analysis holds immense promise. As I pursue my studies, I am eager to delve <strong>deeper into this field and contribute to its development</strong>, <strong>and </strong>I believe that attending Stanford University would provide me with the ideal environment to do so.</blockquote><p>This would cause an incredible amount of confusion and distrust in the chatbot-detection system.</p><h3>So what do we do?</h3><p>Unfortunately, I don’t have a fool-proof answer, but there are a few possible suggestions for development directions that data science researchers, developers, and stakeholders could take:</p><ol><li><strong>Don’t rely on a single model</strong>: One model could be wrong, but if 3 detectors agree, the chance of all of them being wrong is much lower. This isn’t foolproof as maybe somebody’s writing is just very similar to a chatbot’s. This also does not bridge the interpretability gap, but it&#39;s better than nothing as at least the predictions are a bit more trustworthy.</li><li><strong>Chatbot Watermarks</strong>: One suggestion made by OpenAI researcher Scott Aaronson is to add a “watermark.” This would involve tweaking how the model generates outputs so that there is a hidden pattern, one that detection models would try to find. Saliency measures would then work much better here as the “word importance” would simply be a highlight of the watermark. Unfortunately, this relies on the chatbot developers being altruistic. If someone developed their own version of ChatGPT and did not add these watermarks, the watermark detection would fail.</li><li><strong>Use Interpretable Models</strong>: Instead of relying on large Transformers, a possible research direction would instead to shift the focus to attention-powered interpretable models. One such model could be Neural Prototype Trees [11], a model architecture that makes a decision tree off of “prototypes,” or interpretable representations of patterns in data. For example, prototypes in bird image recognition could be “Red Throat” and “Elongated Beak.” For chatbot detection, these prototypes could potentially be things like “repeated long phrases, unusual grammar, and no citations.”</li><li><strong>Change the questions and prompts</strong>: In an educational context, if your assignments or questions can be answered by a chatbot, it’s worth considering whether or not the question itself is valid. It might be worthwhile to instead write prompts and questions that require more logical reasoning and justifications.</li></ol><p>All of these suggestions have their own upsides and downsides, but they are a starting point. As a society, we will be dealing with more AI-related issues, and research and development into XAI needs to be a priority in this ever-changing landscape.</p><h3>Resources and References</h3><p>[1] A. Mok. <a href="https://www.businessinsider.com/google-management-issues-code-red-over-chatgpt-report-2022-12">Google’s management has reportedly issued a ‘code red’ amid the rising popularity of the ChatGPT AI</a>. (2022). Business Insider</p><p>[2] <a href="https://study.com/resources/perceptions-of-chatgpt-in-schools">Productive Teaching Tool or Innovative Cheating?</a> (2023). Study.com</p><p>[3] <a href="https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text/">New AI classifier for indicating AI-written text</a>. (2023). OpenAI</p><p>[4] E. Mitchell, Y. Lee, A. Khazatsky, C.D. Manning, C. Finn. <a href="https://arxiv.org/abs/2301.11305v1">DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature</a>. (2023). arXiv preprint arXiv:2301.11305.</p><p>[5] <a href="https://toptieradmissions.com/resources/college-admissions-statistics/stanford-acceptance-rates/">Stanford University: Acceptance Rates &amp; Statistics</a>. (2022). Top Tier Admissions</p><p>[6] S. Jain, B.C. Wallace. <a href="https://arxiv.org/abs/1902.10186">Attention is not Explanation</a>. (2019). 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics.</p><p>[7] S. Weigreffe, Y. Pinter. <a href="https://arxiv.org/abs/1908.04626">Attention is not not Explanation</a> (2019). The 2019 Conference on Empirical Methods in Natural Language Processing.</p><p>[8] S. Serrano, N. Smith. <a href="https://arxiv.org/abs/1906.03731">Is Attention Interpretable?</a> (2019). 57th Annual Meeting of the Association for Computational Linguistics</p><p>[9] C. Rudin. <a href="https://www.nature.com/articles/s42256-019-0048-x">Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead</a>. (2019). Nature Machine Intelligence.</p><p>[10] S. Anderson. <a href="https://scottaaronson.blog/?p=6823">My AI Safety Lecture for UT Effective Altruism</a>. (2022) Shtetl-Optimized: The Blog of Scott Aaronson.</p><p>[11] M. Nauta, R.v. Bree, C. Seifert. <a href="https://arxiv.org/abs/2012.02046">Neural Prototype Trees for Interpretable Fine-grained Image Recognition</a> (2021). IEEE Conference on Computer Vision and Pattern Recognition 2021.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4057e0386fc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SHAP for Time Series Event Detection]]></title>
            <link>https://medium.com/data-science/shap-for-time-series-event-detection-5b4d9d0f96f4?source=rss-4d9dddc62a80------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b4d9d0f96f4</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nakul Upadhya]]></dc:creator>
            <pubDate>Wed, 01 Feb 2023 14:02:49 GMT</pubDate>
            <atom:updated>2023-02-01T14:02:49.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PRr87CIHJne3IbiP" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Using a modified KernelSHAP for time-series event detection</h4><p>Feature importance is a widespread technique used to explain how machine learning models make their predictions. The technique assigns a score or weight to each feature, indicating how much that feature contributes to the prediction. The scores can be used to identify the most important features and to understand how the model is making its predictions. One frequently used version of this is Shapley values, a model-agnostic metric based on game theory that distributes the “payout” (the prediction) fairly among the features [1]. One extension for Shapley values is KernelSHAP which uses a kernel trick along with local-surrogate models to approximate the value of the Shapley values, which allows it to compute feature importance values for more complex models such as Neural Networks [2].</p><p>KernelSHAP is often applied to explain time-series predictions, but it does come with some significant constraints and drawbacks in this domain:</p><ol><li>Time series prediction often involves large windows of past data, and this can cause computation numerical underflow errors when applying KernelSHAP, especially in multivariate time-series prediction [3].</li><li>KernelSHAP <em>assumes feature independence</em>. This can often work in tabular data cases, but feature and time-step independence is an exception rather than the norm in time series [3].</li><li>KernelSHAP uses coefficients of a linear model that has been fit to perturbations of data. In the time-series case, however, a <a href="https://online.stat.psu.edu/stat510/lesson/11/11.2">Vector AutoRegressive model</a> (VAR) is often more apt to model a process instead of just a linear model [3].</li></ol><p>To fix these issues, researchers at J.P. Morgan’s AI Research division (Villani et al.) proposed variations of KernelSHAP that are more suited to time series data in <a href="https://arxiv.org/abs/2210.02176">their October 2022 paper</a> [3]:</p><ol><li>The researchers first created VARSHAP, a KernelSHAP alteration that uses VAR Models instead of a linear model. This modification makes it The researchers also calculated a closed-form method to calculate SHAP values for AR, MA, ARMA, and VARMAX models.</li><li>Along with VARSHAP as the basis, the researchers proposed <strong>Time-Consistent SHAP</strong> which leverages the temporal component of the problem to reduce the computation of SHAP values.</li></ol><p>Using the Time Consistent SHAP measure, the researchers showcased a promising method for event detection by capturing surges of feature importance.</p><p>In this post, I will first explain how KernelSHAP is calculated and how to modify it for VARSHAP. I will then also explain how to get Time-Consistent SHAP (TC-SHAP) and how to use TC-SHAP to identify events in time-series analysis.</p><h3>KernelSHAP and VARMAX</h3><p>The formula of the SHAP value as provided by [2] is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*M0Zu9vWfL6faYGAFn6-xJQ.png" /><figcaption>Equation 1: SHAP equation</figcaption></figure><p>Where the <em>phi </em>in the equation above is the SHAP value of feature <em>i</em> given the value function <em>v (</em>the value function is usually the model predictions)<em>. C </em>is the set of all features and <em>N</em> is the size of <em>C</em>, or the number of features. <em>P(C)</em> is the powerset of all features without feature <em>i. Delta(S, i)</em> is the change in prediction feature <em>i</em> causes when added to the feature coalition <em>S</em> (which is a set within the powerset C).</p><p>The equation summarizes down to “add the weighted marginal contribution of feature <em>i</em> to each possible coalition of features that doesn’t include <em>i</em>”.</p><p>The issue that KernelSHAP handles is that this computation can get incredibly large as the powerset size scales exponentially with the number of features. The KernelSHAP is calculated by solving the following problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*P9tYkPNnL5XeEBF2i_M0JA.png" /><figcaption>Equation 2: KernelSHAP Equation [3]</figcaption></figure><p>Where <em>h_x</em> is a masking function that is applied on <em>z</em>, a binary vector sampled from a set <em>Z</em> which represents the collection of all possible coalitions of features. This function maps the coalition represented by <em>z</em> to a masked data point which is then put into our model (<em>f_theta</em>). The goal is to find the best Linear Model (<em>g</em>) that estimates the model performance across all the masks. The weights in the linear model are the KernelSHAP values. This is all possible due to a combinatorial kernel defined by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*MMq_Pw7otYcKvNz33qzfUg.png" /><figcaption>Equation 3: Combinatorial Kernel [3]</figcaption></figure><p>To instead calculate VARSHAP, simply replace the linear representation of g with a VAR model. According to the authors, since both the coefficients of a linear model and a VAR model are estimated through Ordinary Least Squares, all the math for KernelSHAP holds, and becomes more representative for a time series [3].</p><h3>Time Consistent SHAP</h3><p>As mentioned before, SHAP is a method that interprets the features of a model as players in a game and uses Shapley values to find fair allocations of rewards. However, for games that develop over time, these allocations may not provide enough incentive for all parties to pursue the initial goal. To avoid this, game theorists use imputation schedules and the concept of time consistency to manage incentives across time [3].</p><p>This idea of competing interests through time extends to features as well, as the traditional SHAP methods consider the same feature at a different time step as a different player in the game. According to the authors, we can potentially bridge this gap by adding time consistency [3].</p><p>The time consistency of SHAP values can be presented as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*rcuVPNBjCQ_gDPL1ksDoDg.png" /><figcaption>Equation 4: The time consistency of Shapley Values [3]</figcaption></figure><p>In this equation, <em>beta </em>represents an imputation schedule of payments made to player (feature) <em>i</em> across <em>t</em> time steps and <em>phi(0,i)</em> is the total value that the player contributes to the game (prediction). Think of this as similar to a business partnership.</p><p>Each individual (AKA the feature) pays an initial amount into the startup fund(which is phi at time step 0). Then in future time steps, the individual is periodically paid returns as they contribute more to the business outcomes (AKA the end prediction). These payouts also disincentive any individual from acting against the business interests. By framing the problem this way, TC-SHAP works much better in the context of time series since now <strong>the different time steps of a feature are modeled as one entity instead of as separate players.</strong></p><p>To use these in practice, the following steps can be taken:</p><ol><li>Compute the total SHAP contribution of each feature (<em>phi </em>at time step 0) by masking the feature by replacing the feature with either zeros or the feature average. Repeat this for all features.</li><li>Then we need to compute the “subgame SHAPs” for each time step in our window (<em>t-w</em>). This is done by changing the masking mechanism in Equation 2 so that instead of only masking the time-step (<em>t-w</em>), we instead mask all the time steps between <em>t-w </em>and <em>t (</em>again with either zeros or the mean).</li><li>Then we simply calculate the imputation schedule using equations 4 and 5.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*9-GwWiyfgaYumVZjfTf8MA.png" /><figcaption>Equation 5: Imputation schedule [3]</figcaption></figure><p>Step 1 calculates the “initial investment”. Step 2 then enforces the idea that we have <em>N</em> features across multiple timesteps (<em>W</em>) instead of having <em>N*W</em> features. Step 3 wraps it all together by providing the imputation schedule (or the periodic returns of each “investor”).</p><p>This procedure also has the benefit of reducing the number of computations from 2<em>^(N*W)</em> to <em>W</em>*2^<em>N</em> where <em>W </em>is the number of timesteps used for predictions and <em>N </em>is the number of features [3].</p><p>Once calculated, we can interpret the TC SHAP values as “how, at a given time step, the evolution of features will affect the coalition of other feature trajectories.” In simpler terms, TC SHAP represents how a feature at a given time step changes how other features contribute together in future time steps. The feature-timestep points that heavily impact future collaboration will by definition heavily impact the end predictions.</p><h3>Event Detection with TC SHAP</h3><p>While getting the importance of a given time step for a single prediction is useful, time series analysis often involves analyzing multiple predictions and patterns and we may want to know what some influential time steps are overall in the model’s prediction [3].</p><p>According to the authors, we can find the influential time step by adding up the TC SHAP values for a given set of predictions (or all predictions if we want a global event detection mechanism). By plotting this out, we can then easily see which time steps are important and where some important events may have happened [3].</p><p>The authors demonstrate the effectiveness of this approach with the <a href="https://archive.ics.uci.edu/ml/datasets/individual+household+electric+power+consumption">Individual Household Electricity dataset</a>. The authors trained an LSTM network followed by a dense layer to predict power consumption. They then calculated the TC-SHAP values and summed them up to get the event detection convolutions. They then overlayed the convolutions onto the target time series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A73uDJzkHzub_S8KXrucEw.png" /><figcaption>Figure 1: Event Detection Convolution (Blue) for Sub-Metering 2 and 3 compared to the target (Figure from [3])</figcaption></figure><p>As shown in Figure 1, the large shifts in the target variables can be explained by large spikes in the event convolutions. For example, there is a large spike in the event convolution for sub_metering_2 right after time-step 25. This was then followed by a large drop in the target soon after. Similarly, a large drop in the event convolution was followed by a large drop in the target around time step 75 in sub_metering_3. Most of the large shifts can be explained by some sub-meter shifts.</p><h3>Conclusion</h3><p>The modifications to KernelSHAP fill a large hole in the current work. Apart from this, there is not a large amount of work in developing post-hoc interpretability methods that specifically address time series feature importance. TC-SHAP helps tackle this issue and is sorely needed.</p><p>There are some concerns and further work needed around this new method, however. One such concern (that the author also addresses) is the significant difference in the explanations between VARSHAP and TC-SHAP, indicating that more work is needed to examine the exact interpretation of these values. Additionally, while TC-SHAP in theory overcomes the independence issue, more experimentation is necessary to fully confirm this claim.</p><p>Additionally, model-agnostic methods in general can be misleading as they can only provide an estimation of importance, but not the true importance. However, for most use cases this rough evaluation is enough, and having a method that addresses temporal dependencies is incredibly useful.</p><h3>Resources and References</h3><ol><li>SHAP Package for Python: <a href="https://shap.readthedocs.io/en/latest/index.html">https://shap.readthedocs.io/en/latest/index.html</a></li><li>A more in-depth explanation of Kernel SHAP: <a href="https://christophm.github.io/interpretable-ml-book/shap.html#kernelshap">https://christophm.github.io/interpretable-ml-book/shap.html#kernelshap</a></li></ol><p><strong>References</strong></p><p>[1] L. Shapley. <a href="https://books.google.ca/books?hl=en&amp;lr=&amp;id=Pd3TCwAAQBAJ&amp;oi=fnd&amp;pg=PA307&amp;dq=A+value+for+n-person+games.&amp;ots=gtuWLb6iq-&amp;sig=dm0WHMP9kzTY8kx6J75CLsf82wk#v=onepage&amp;q=A%20value%20for%20n-person%20games.&amp;f=false">A value for n-person games.</a> (1953). Contributions to the Theory of Games 2.28.</p><p>[2] S.M. Lundberg, S-I. Lee. <a href="https://arxiv.org/abs/1705.07874">A Unified Approach to Interpreting Model Predictions.</a> (2017). Advances in Neural Information Processing Systems, 30.</p><p>[3] M. Villani, J. Lockhart, D. Magazzeni. <a href="https://arxiv.org/abs/2210.02176">Feature Importance for Time Series Data: Improving KernelSHAP</a> (2022). ICAIF Workshop on Explainable Artificial Intelligence in Finance</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b4d9d0f96f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/shap-for-time-series-event-detection-5b4d9d0f96f4">SHAP for Time Series Event Detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>