<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Reza Yazdanfar on Medium]]></title>
        <description><![CDATA[Stories by Reza Yazdanfar on Medium]]></description>
        <link>https://medium.com/@rezayazdanfar?source=rss-45e9c77210d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UIPX1o0taf2kz3N4BpNIaA.jpeg</url>
            <title>Stories by Reza Yazdanfar on Medium</title>
            <link>https://medium.com/@rezayazdanfar?source=rss-45e9c77210d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:31:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rezayazdanfar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://rezayazdanfar.medium.com/sparsetsf-modeling-long-term-time-series-forecasting-with-1k-parameters-22ab12ca0a0f?source=rss-45e9c77210d7------2"><img src="https://cdn-images-1.medium.com/max/1698/1*1WIGhauQ3DzCviK8aGri2A.png" width="1698"></a></p><p class="medium-feed-snippet">This article describes a time series forecasting architecture with less than 1000 parameters (incredibly low) that is showing efficient&#x2026;</p><p class="medium-feed-link"><a href="https://rezayazdanfar.medium.com/sparsetsf-modeling-long-term-time-series-forecasting-with-1k-parameters-22ab12ca0a0f?source=rss-45e9c77210d7------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://rezayazdanfar.medium.com/sparsetsf-modeling-long-term-time-series-forecasting-with-1k-parameters-22ab12ca0a0f?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/22ab12ca0a0f</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Sun, 03 Nov 2024 15:32:08 GMT</pubDate>
            <atom:updated>2024-11-03T15:32:08.198Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[LLM-Mixer: Multiscale Mixing in LLMs for Time Series Forecasting]]></title>
            <link>https://pub.towardsai.net/llm-mixer-multiscale-mixing-in-llms-for-time-series-forecasting-f4a230e9522e?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4a230e9522e</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Wed, 23 Oct 2024 18:02:11 GMT</pubDate>
            <atom:updated>2024-10-23T18:02:11.762Z</atom:updated>
            <content:encoded><![CDATA[<h4>Integrating multiscale time-series decomposition with pre-trained large language models (LLMs) to capture both short-term fluctuations and long-term trends.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unjfxAvZ8uyaIYvih9anxw.jpeg" /></figure><h3>Time Series</h3><p>Time series is crucial in different fields, for example, in finance, accurate time series forecasting helps in stock market predictions and risk management, enabling better investment decisions!</p><p>Time series forecasting in easy words is more of using the past data, and forecasting the future data; something that we humans, intelligent creatures, do it intuitively more than often (don’t believe me? go and talk with any financial analyst or trader…).</p><p>Time series forecasting is challenging because of it inherent complexities of real-world data, which often represents nonlinear, non-stationary, and multivariate characteristics. Moreover time series data features multiple time scales, with short-term fluctuations and long-term trends that traditional models struggle to capture simultaneously.</p><h3>LLMs in time series</h3><p>Openai showed the power of LLMs by introducing GPT3.5 and later GPT4, and ever since everything changed. Most researchers realized the power of scaling, and since then, the research in LLMs and AI in general multiplied.</p><p>One line of work has been using the power of LLMs in other domains of AI like time series forecasting. <strong>LLMs are particularly attractive for time series forecasting due to their abilities in few-shot or zero-shot transfer learning, multimodal knowledge integration, and complex reasoning.</strong></p><p>Time series data often represents continuous and irregular patterns, unlike the discrete tokens that LLMs are typically designed to process. Not to mention that time series usually come with multiple time scales, ranging from short term fluctuations to long term trends.</p><blockquote>The level of complexity is quite lucrative in my humble opinion, and well, consequently challenging! 🔥😄</blockquote><h3>Problem in short</h3><p>The challenge of accurately forecasting time series data, which often involves complex multiscale temporal patterns.</p><h3>Solution in short: LLM-Mixer</h3><p><strong>LLM-Mixer adapts LLMs for time series forecasting by decomposing the data into multiple temporal resolutions.</strong> This approach allows the LLM to better understand and model the complex patterns within the time series data, capturing both short-term and long-term dependencies effectively. <strong>Through multiscale time-series decomposition combined with LLMs, LLM-Mixer achieves competitive performance and improves forecasting accuracy across various datasets and forecasting horizons.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uobEXQDrx_i79sol" /><figcaption>Figure 1: The LLM-Mixer framework for time series forecasting. Time series data is downsampled to multiple scales and enriched with embeddings. These multiscale representations are processed by the Past-Decomposable-Mixing (PDM) module and then input into a pre-trained LLM, which, guided by a textual description, generates the forecast.</figcaption></figure><h3>LLM-Mixer in details: Architecture</h3><h4><strong>1) Data Downsampling and Embedding</strong>:</h4><p>We start by downsampling the time series data into <strong>multiple temporal resolutions </strong>to capture both short-term fluctuations and long-term trends.</p><p>These multiscale series are then enriched with three types of embeddings: <strong><em>token</em></strong>, <strong><em>temporal</em></strong>, and <strong><em>positional embeddings</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kTJAp9K_gjwBGAb20CI1A.png" /></figure><h4><strong>2) Token, Temporal, and Positional Embeddings</strong>:</h4><p>We calculate <strong>token embeddings </strong>by <strong>1D convolutions</strong>, temporal embeddings <strong>encode information </strong>such as day, week, and month, and positional embeddings <strong>encode the sequence positions</strong>. These embeddings transform the multiscale time series into <strong>deep feature representations</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*UsTk9rsWUoA7qT6KNtCIng.png" /></figure><h4><strong>3) Past-Decomposable-Mixing (PDM) Module</strong>:</h4><p>The multiscale representations are processed by the PDM module, which <strong>mixes past information across different scales</strong>. The PDM module breaks down complex time series data into <strong>separate seasonal and trend</strong> <strong>components</strong>, allowing for targeted processing of each component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*IYGNghQjnJ7kgM6YsvAmCw.png" /></figure><h4><strong>4) Pre-trained Large Language Model (LLM) Processing</strong>:</h4><p>The processed multiscale data, along with a textual prompt that provides <strong>task-specific information</strong>, is input into a frozen pre-trained LLM. The frozen LLM utilizes its <strong>semantic knowledge</strong> and the <strong>multiscale information</strong> to generate the forecast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*gUk2Lk4dHs9H7N-bq-mmoA.png" /></figure><h4><strong>5) Forecast Generation</strong>:</h4><p>Finally, a trainable decoder, which is<strong> a simple linear transformation</strong>, is applied to the last hidden layer of the LLM to predict the next set of future time steps. This step culminates in the output of forecasts, completing the LLM-Mixer framework processing pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZTwEmRvZ32pv-yAL5-ziyA.png" /></figure><h3>Datasets</h3><ul><li>For long-term forecasting, the datasets include the ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), as well as the Weather, Electricity, and Traffic datasets.</li><li>For short-term forecasting tasks, the framework uses the PeMS dataset, which consists of four public traffic network datasets (PEMS03, PEMS04, PEMS07, PEMS08), with time series data collected at various frequencies.</li></ul><blockquote>This is not that important for the aim of this article, since it’s the standard for every ml paper to evaluate on the same benchmark over and over again.</blockquote><h3>Results</h3><blockquote>In short LLM-Mixer is great, lol, if you don’t think so read this section :)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_r3bA_7ZLux2T8skmRtxRA.png" /><figcaption>Table 1: Long-term multivariate forecasting results in terms of MSE and MAE, the lower the better. Red: the best, Blue: the second best.</figcaption></figure><p>For long-term multivariate forecasting, LLM-Mixer shows competitive performance, particularly excelling on the ETTh1, ETTh2, and Electricity datasets. It consistently achieves low Mean Squared Error (MSE) and Mean Absolute Error (MAE) values over multiple forecasting horizons (96, 192, 384, and 720 time steps), outperforming models like TIME-LLM, TimeMixer, and PatchTST.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GVDMK3imCtPI_ziIZp_XXg.png" /><figcaption>Table 2: Short-term multivariate forecasting results.</figcaption></figure><p>For short-term multivariate forecasting, LLM-Mixer again exhibits strong performance, delivering low MSE and MAE values consistently across the PEMS datasets. It achieves competitive accuracy on datasets like PEMS03, PEMS04, and PEMS07, outperforming other models including TIME-LLM, TimeMixer, and PatchTST. On the PEMS08 dataset, LLM-Mixer delivers superior results compared to iTransformer and DLinear, emphasizing its <strong>effectiveness in capturing</strong> <strong>essential temporal dynamics</strong> for short-horizon forecasting tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Xqe1qs32sQVfonVVTHGjQ.png" /><figcaption>Table 3: Long-term univariate forecasting results.</figcaption></figure><p>Finally, for univariate long-term forecasting, LLM-Mixer achieves the <strong>lowest MSE and MAE</strong> values across datasets on the ETT benchmark, consistently outperforming methods such as Linear, NLinear, and FEDformer.</p><p>Thanks for reading this article, feel free to follow me on <a href="https://twitter.com/reza__yazdanfar">X platforms</a> or <a href="https://de.linkedin.com/in/rezayazdanfar">LinkedIn</a>. I have used <a href="https://nouswise.com/homepage">Nouswise.com</a> to write this article, you can find the original paper of this article and millions of other papers on it. (caution: you didn’t read an AI generated text 🔥😂)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bD0YS0oGVjFEWgw4gLRA_g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4a230e9522e" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/llm-mixer-multiscale-mixing-in-llms-for-time-series-forecasting-f4a230e9522e">LLM-Mixer: Multiscale Mixing in LLMs for Time Series Forecasting</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Diffusion Auto-Regressive Transformer For Effective Self-Supervised Time Series Forecasting]]></title>
            <link>https://pub.towardsai.net/diffusion-auto-regressive-transformer-for-effective-self-supervised-time-series-forecasting-31b7d7ba2062?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/31b7d7ba2062</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Sat, 19 Oct 2024 14:02:04 GMT</pubDate>
            <atom:updated>2024-10-19T16:01:29.281Z</atom:updated>
            <content:encoded><![CDATA[<h4>TimeDART, a generative self-supervised method that integrates diffusion and auto-regressive modeling for time series forecasting tasks.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3oKIPhmN5gOCOZyRK8%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2F3d-vr-3oKIPhmN5gOCOZyRK8&amp;image=https%3A%2F%2Fmedia3.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExYzB3djF5b21xbW1jaHhuZjB4ZnhkMmQ4eWU2bGhlMm5oanBtOWk0ZSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2F3oKIPhmN5gOCOZyRK8%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="289" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5fd824ed07cc2d621fcfaf6558d2dfc/href">https://medium.com/media/f5fd824ed07cc2d621fcfaf6558d2dfc/href</a></iframe><p><strong>Time series</strong> is important (though underappreciated!) in various domains due to its ability to provide accurate predictions of future data points, which in turn leads to better decision-making, resource allocation, and risk management. This capability leads to significant operational improvements and strategic advantages, particularly in fields such as finance, healthcare, and energy management.</p><p><strong>Deep neural networks</strong> have emerged as a popular and effective solution paradigm for time series forecasting, reflecting the growing interest in leveraging advanced machine learning techniques to tackle the complexities of sequential data.</p><h4>Self-supervised Learning</h4><p>A paradigm that models <strong>learn </strong>from <strong>unlabeled data </strong>by generating supervisory signals <strong>internally</strong>, typically through <strong>pretext tasks</strong>.</p><p>Unlike supervised learning, which requires labeled data, self-supervised learning <strong>leverages the inherent structure</strong> <strong>within the data</strong> to create the necessary labels for training.</p><h4>Self-supervised Learning for Time Series:</h4><p>In the context of <strong>self-supervised learning</strong>, time series offers unique abilities to develop models that can learn <strong>universal representations </strong>from <strong>unlabeled data</strong>.</p><p>This approach enhances time series forecasting by allowing models to capture both <strong>long-term dependencies</strong> and <strong>local detail features</strong>. However, effectively capturing these aspects remains challenging, prompting the need for innovative methods like <strong>TimeDART (</strong>this paper<strong>)</strong>, which <strong>integrates diffusion and auto-regressive modeling</strong> to address these challenges.</p><h4>Problem:</h4><p>The challenge for time series is <strong>capturing global sequence dependencies</strong> and <strong>local detail features</strong> effectively using self-supervised learning methods.</p><p>Traditional methods <strong>struggle </strong>with this dual task, impacting their ability to learn comprehensive and expressive representations of time series data.</p><p>The solution is TimeDarT:</p><h4>TimeDarT</h4><blockquote>In one word, it’s the “<strong>solution“ </strong>to time series forecasting problem! but well, that’s not enough! We gotta inspect and dig into it :)</blockquote><p>TimeDART, short for Diffusion Auto-regressive Transformer, is a self-<strong>supervised learning</strong> method designed for <strong>time series forecasting</strong>. It aims to improve the prediction of future data points by <strong>learning from patterns</strong> in past data within a time series. It’s like breaking down the time series data into smaller segments, <strong>patches</strong>, and uses these patches as <strong>the basic units </strong>for <strong>modeling.</strong></p><p>The researchers used <strong>Transformer encoder with self-attention mechanisms </strong>to understand <strong>dependencies </strong>between <strong>these patches</strong>, <strong>effectively capturing </strong>the <strong>overall sequence structure </strong>of the data.</p><p>Two processes of <strong>diffusion </strong>and <strong>denoising </strong>are used to address detailed features within each patch. These two help to <strong>capture local features</strong> by adding and removing noises from data (<em>pretty typical process in all diffusion models</em>). As a matter of fact, it helps the model to behave better on detailed patterns.</p><h3>TimeDART Architecture:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xk5bZhjStSct43JRKDgvLg.png" /><figcaption>Figure 1: The TimeDART architecture captures global dependencies using auto-regressive generation while handling local structures with a denoising diffusion model. The model introduces noise into input patches during the forward diffusion process, generating self-supervised signals. In the reverse process, the original sequence is restored auto-regressively. [<a href="https://arxiv.org/pdf/2410.05711">source</a>]</figcaption></figure><h4>Instance Normalization and Patching Embedding</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A92TTWthTdvnMJSDGATJmg.png" /></figure><p>The first step is applying <strong>instance normalization (normalization) </strong>to the input multivariate time series data to ensure each instance has a <strong>zero mean </strong>and <strong>unit standard</strong> <strong>deviation</strong>, which helps maintain <strong>consistency </strong>in the final forecast.</p><p>The time series data is divided into <strong>patches </strong>instead of individual points, this allows <strong>capturing </strong>more comprehensive <strong>local information</strong>.</p><p>The <strong>patch length </strong>is set <strong>equal </strong>to the <strong>stride </strong>to avoid information leakage, this helps us to make sure each patch contains <strong>only non-overlapping </strong>segments of the original sequence.</p><h4>Transformer Encoder for Inter-Patch Dependencies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pgzd_MOlCsKe9arRf2h68A.png" /></figure><p>We have a <strong>self-attention-based Transformer encoder </strong>in the architecture, this helps to model the <strong>dependencies between patches</strong>.</p><p>This approach helps in capturing the <strong>global sequence dependencies </strong>by considering the <strong>relationships </strong>between <strong>different patches of the time series data</strong>.</p><p>The use of a Transformer encoder allows TimeDART to learn <strong>meaningful</strong> <strong>inter-patch representations</strong>, which are crucial for <strong>understanding </strong>the high-<strong>level structure </strong>of the time series.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/190c711948aa826b3f20d64c7b638826/href">https://medium.com/media/190c711948aa826b3f20d64c7b638826/href</a></iframe><h4>Forward Diffusion Process</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTZfFBa-KD2Xq7oWf_b32A.png" /></figure><p>In the forward diffusion process, <strong>noise </strong>is applied on the input patches. This step is essential for generating self-supervised signals that enable the model to <strong>learn robust representations </strong>by <strong>reconstructing </strong>the original data from its <strong>noisy version</strong>.</p><p>This noise helps the model <strong>recognize </strong>and <strong>focus</strong> on the <strong>intrinsic patterns </strong>within <strong>the time series data</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2410d663f15b9757c2b551bff025e13/href">https://medium.com/media/e2410d663f15b9757c2b551bff025e13/href</a></iframe><h4>Cross-Attention-Based Denoising Decoder</h4><p>The denoising decoder employs a <strong>cross-attention mechanism </strong>to <strong>reconstruct </strong>the original, noise-free patches.</p><p>This allows for <strong>adjustable optimization difficulty</strong>, making the self-supervised task <strong>more effective </strong>and enabling the model to <strong>focus </strong>on <strong>capturing detailed intra-patch features</strong>. This design increases the model’s <strong>capability </strong>to <strong>learn </strong>both <strong>local</strong> and <strong>global features</strong> effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_0GXG6JuyXzziqXh5IcUg.png" /></figure><p>It takes the noises (as query) and the outputs of the encoder (keys and values) in, and we mask the decoder to make sure that the j-th input in noise added corresponds to the j-th output from the Transformer encoder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cfde200f585c3c4254fb4009f6f393e/href">https://medium.com/media/2cfde200f585c3c4254fb4009f6f393e/href</a></iframe><h4>Auto-Regressive Generation for Global Dependencies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*g4IPatOBEsKs1UBJijug2g.png" /></figure><p>The responsibility is capturing the <strong>high-level global dependencies </strong>in the time series. By restoring the original sequence auto-regressively, the model can <strong>understand </strong>the <strong>overall temporal patterns </strong>and <strong>dependencies</strong>, improving its forecasting ability.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cff40f8cbfef768d86399def3bf8438a/href">https://medium.com/media/cff40f8cbfef768d86399def3bf8438a/href</a></iframe><h4>Optimization and Fine-Tuning</h4><p>Finally, the entire model is optimized in an <strong>auto-regressive manner</strong> to obtain <strong>transferable representations </strong>that can be <strong>fine-tuned </strong>for specific forecasting tasks. This step ensures that the model’s learned representations are both <strong>comprehensive </strong>and <strong>adaptable </strong>to various downstream applications, enabling <strong>superior performance </strong>in time series forecasting.</p><h3>Evaluation:</h3><h4>Datasets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SDI-0I7YGjUeksIicwzfQ.png" /></figure><p>The TimeDART model was evaluated using eight popular datasets to test its effectiveness in time series forecasting. These datasets include four <strong>ETT </strong>datasets (ETTh1, ETTh2, ETTm1, ETTm2), as well as the <strong>Weather</strong>, <strong>Exchange</strong>, <strong>Electricity</strong>, and <strong>Traffic datasets</strong>.</p><p>These datasets include a range of application scenarios, such as power systems, transportation networks, and weather forecasting. (as I said, time series is important everywhere 👀🙂)</p><h4>Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWmQGXiqd4gp-TpwfrTzQg.png" /><figcaption>Table 1. Multivariate time series forecasting results comparing TimeDART with both SOTA selfsupervised approaches and supervised approaches. The best results are in bold and the second best are underlined. “#1 Counts” represents the number of times the method achieves the best results. [<a href="https://arxiv.org/pdf/2410.05711">source</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SP8dsvLgYv9qKFGQj_tL_g.png" /><figcaption>Table 2. Multivariate time series forecasting results comparing TimeDART, pretrained across five datasets and fine-tuned on specific ones. All results are averaged from 4 different predicted window of {96, 192, 336, 720}. The best results are in bold. [<a href="https://arxiv.org/pdf/2410.05711">source</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xilEChOiE58z78cZgnTvcQ.png" /><figcaption>The ablation study. All results are averaged from 4 different predicted window of {96, 192, 336, 720}. The best results are in bold. [<a href="https://arxiv.org/pdf/2410.05711">source</a>]</figcaption></figure><blockquote>Please note that the researchers mentioned more details of the work such as hyper parameters; however, aiming to prevent this article longer I did not mention them and herby refer you to read the <a href="https://arxiv.org/pdf/2410.05711">original paper</a>.</blockquote><p>I got some help from AI, specifically <a href="https://nouswise.com/homepage"><strong>Nouswise.com</strong></a><strong> </strong>to write this article, you can find this paper and millions of other AI papers on Nouswise:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dzu0wgh3un26BpEg0dItRg.png" /></figure><p>You can also contact me directly via <a href="https://de.linkedin.com/in/rezayazdanfar">LinkedIn</a> or <a href="https://twitter.com/reza__yazdanfar">X platform</a> (formerly twitter)🙂🔥🤗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31b7d7ba2062" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/diffusion-auto-regressive-transformer-for-effective-self-supervised-time-series-forecasting-31b7d7ba2062">Diffusion Auto-Regressive Transformer For Effective Self-Supervised Time Series Forecasting</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are Language Models Actually Useful for Time Series Forecasting?]]></title>
            <link>https://pub.towardsai.net/are-language-models-actually-useful-for-time-series-forecasting-81a099415702?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/81a099415702</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Sun, 21 Jul 2024 14:01:48 GMT</pubDate>
            <atom:updated>2024-07-21T14:01:48.383Z</atom:updated>
            <content:encoded><![CDATA[<h4>We’ve seen huge progress in language models but what about Time Series tasks like forecasting? This article covers whether We can benefit from LLMs for TS or not?</h4><h3>Time Series</h3><p>Time series is one of the most challenging lines of work in machine learning, and this has made researchers less reluctant to work on it. However, solving time series tasks like anomaly detection, time series forecasting, … are vital in a wide variety of industries and could save tons of money.</p><h3>What happened in Language processing?</h3><p>The laws of scaling, initiated by OpenAI, showed that models can generalize better on more raw data, and the result was <strong>ChatGPT</strong>. Revolutionary!! Since then, LLMs captured the attention of all, politicians to researchers.</p><h3>What is going on now?</h3><p>Since then researchers have been trying to use LLMs for time series! I mean it makes sense to an extent because both language data and time series are sequential, and researchers thought if the llms could generalize well on language data, then probably it can be for time series.</p><p>There are a bunch of cool works get done about it, you can read more <a href="https://arxiv.org/abs/2310.01728">here</a>, <a href="https://arxiv.org/abs/2302.11939">here</a>, <a href="https://arxiv.org/html/2403.07300v1">here</a>, <a href="https://arxiv.org/abs/2402.02713">here</a>, and <a href="https://arxiv.org/abs/2310.07820">here</a>.</p><h3>The question is “<strong>How many LLMs are really useful for time series tasks?</strong>”</h3><p>I’d argue some works are showing promising future for time series such as <a href="https://arxiv.org/abs/2404.11757">time series reasoning</a> and <a href="https://openreview.net/forum?id=s9z0HzWJJp">social understanding</a> (agents) that used LLMs to achieve what they intended.</p><h4><strong>Time series reasoning</strong>:</h4><p>Using Large Language Models (LLMs) for time series reasoning can enhance time series reasoning by integrating three key forms of analytical tasks: <strong>Etiological Reasoning</strong>, <strong>Question Answering</strong>, and <strong>Context-Aided</strong> <strong>Forecasting</strong>.</p><ol><li><strong>Etiological Reasoning</strong> involves hypothesizing potential causes behind observed time series patterns, enabling models to identify scenarios that most likely generated given time series data.</li><li><strong>Question Answering</strong> enables models to interpret and respond to factual queries about time series, such as identifying trends or making counterfactual inferences about changes in the data.</li><li><strong>Context-Aided Forecasting</strong> allows models to leverage additional textual information to enhance their predictions about future data points, integrating relevant context to improve forecast accuracy.</li></ol><p>However, current LLMs demonstrate limited proficiency in these tasks, performing marginally above random on etiological and question-answering tasks and showing modest improvements in context-aided forecasting.</p><h4><strong>Social understanding</strong>:</h4><p>Using Large Language Models (LLMs) for time series analysis can significantly enhance social understanding by enabling agents to systematically analyze and predict societal trends and behaviors. LLM-based agents utilize real-world time series data from various domains such as finance, economics, polls, and search trends to approximate the hidden world state of society. This approximation aids in the formulation and validation of hypotheses about societal behaviors by correlating time-series data with other information sources like news and social media.</p><p>By including these diverse data streams, LLMs can provide deep insights into multi-faceted and dynamic societal issues, facilitating complex and hybrid reasoning that holds both logical and numerical analyses.</p><p>Moreover, the hyper portfolio task within SocioDojo allows these agents to make investment decisions based on their understanding of societal dynamics, which serves as a proxy to measure their social comprehension and decision-making capabilities.</p><p>This method ensures that the agents are not merely performing historical data fitting but are actively engaging with and adapting to the continuous flow of real-world data, making their analyses and predictions relevant and applicable in real-life scenarios.</p><blockquote>Pretty mind blowing, ain’t it?</blockquote><p><strong>However, these new models do not use the natural reasoning abilities of pretrained LMs when it comes to time series.</strong></p><h3>Do LLMs really help time series tasks?</h3><p>A new study just showed that if we replace this language with attention layers, the performance will <strong>not </strong>change <strong>dramatically</strong>. Even if they get removed completely, the performance gets better. This even improves both training and inference speed by up to three orders of magnitude.</p><p>The researchers chose three ablations: deleting the llm-component or replacing it. The three modifications are as follow:</p><ol><li><strong>W/O LLM </strong>(Figure 1 (b)). We remove the language model entirely, instead passing the input tokens directly to the reference method’s final layer.</li><li><strong>LLM2Attn</strong> (Figure 1 (c)). We replace the language model with a single randomly-initialized multi-head attention layer.</li><li><strong>LLM2Trsf </strong>(Figure 1 (d)). We replace the language model with a single randomly-initialized transformer block.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RcNdaBKYQEE_fYQFF5T2A.png" /><figcaption>Figure 1: Overview of all LLM ablation methods. Figure (a) represents time series forecasting using an LLM as the base model. In some works, the LLM components are frozen, while in others, they undergo fine-tuning. Figure (b) shows the model with the LLM components removed, retaining only the remaining structure. Figure (c) replaces the LLM components with a single-layer self-attention mechanism. Figure (d) replaces the LLM components with a simple Transformer. [<a href="https://arxiv.org/abs/2406.16964">source</a>]</figcaption></figure><blockquote>The first left one (Figure (a)) is the model with LLM, as the baseline here.</blockquote><h3>Datasets</h3><p>The datasets are mainly the benchmark datasets in all other time series research: <strong>ETT</strong>, <strong>Illness</strong>, <strong>Weather</strong>, <strong>Traffic</strong>, <strong>Electricity</strong>, <strong>Exchange Rate</strong>, <strong>Covid Deaths</strong>, <strong>Taxi (30 min)</strong>, <strong>NN5 (Daily)</strong> and <strong>FRED-MD</strong>.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*jb7Swg4BUcs5TUPHl29sXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*LFZ39rsAsYXCzannitRMQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*9uuOdC-Y8AzZczg3Kr8dpA.png" /><figcaption>Forecasting performance of all models — Time-LLM, LLaTA, and OneFitsAll and results from our ablations. All results are averaged across different prediction lengths. Results in <strong>Red </strong>denote the best-performing model. # Wins refers to the number of times the method performed best, and # Params is the number of model parameters. “-” means the dataset is not included in the original paper [<a href="https://arxiv.org/abs/2406.16964">source</a>]</figcaption></figure><p>As you can see the ablations are superior to Time-LLM in <strong>all cases</strong>, LLaTA in <strong>22 </strong>out of 26 and OneFitsAll in <strong>19 </strong>out 26 cases. The metrics used here are MAE and MSE, mean-absolute-error and mean-square-error, respectively.</p><p>It can be concluded that LLMs don’t improve the performance on time series forecasting tasks in a meaningful way.</p><p>Now let’s take a look at the computation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*DFKAaztzy6eRZoXBCtArQQ.png" /><figcaption>In time series tasks, LLM (LLaMA and GPT-2) significantly increases training time. The table shows the number of model parameters (in millions) and total training time (in minutes) for three methods predicting over a length of 96 on ETTh1 and Weather data. Compared with original method “w/ LLM” are “w/o LLM”, “LLM2Attn” and “LLM2Trsf”. [<a href="https://arxiv.org/abs/2406.16964">source</a>]</figcaption></figure><p>Time-LLM, OneFitsAll, and LLaTA take, on average, 28.2, 2.3 and 1.2 times longer than the modified models. That says, the trade off from the computation of LLMs for time series doesn’t not worth it.</p><p>Now, we can have a look at <strong>whether pretraining with language datasets could result in better time series forecasting or not?</strong></p><p>The research took four different combinations: <strong>Pretrain + Finetune</strong>, <strong>Random Initialization + Finetune</strong>, <strong>Pretrain + No Finetuning</strong> and <strong>Random Initialization + No Finetuning</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*_mWhG_r0FvPHzRd-cvE52w.png" /><figcaption>Randomly initializing LLM parameters and training from scratch (woPre) achieved better results than using a pretrained (Pre) model. “woFT” and “FT” refer to whether the LLM parameters are frozen or trainable. [<a href="https://arxiv.org/abs/2406.16964">source</a>]</figcaption></figure><p>As you can see, <strong>language knowledge</strong> offers <strong>very limited improvement</strong> for forecasting. However, “Pretrain + No Finetuning” and the baseline “Random Initialization + No Finetuning” performed the best 5 times and 0 times, respectively, insinuating that Language knowledge does not help during the finetuning process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*v3wdu_EJZoj-YsPsuFRO6w.png" /><figcaption>For the input shuffling/masking experiments on ETTh1 (predict length is 96) and Illness (predict length is 24), the impact of shuffling the input on the degradation of time series forecasting performance does not change significantly before and after model modifications. [<a href="https://arxiv.org/abs/2406.16964">source</a>]</figcaption></figure><p>In this experiment, three types of shuffling are used: <strong>shuffling the entire</strong> <strong>sequence randomly</strong> (“sf-all”), <strong>shuffling only the first half of the sequence</strong> (“sf-half”), and <strong>swapping the first and second halves of the sequence</strong> (“ex-half”).</p><p>As the results show that LLM-based models are no more vulnerable to input shuffling than their ablations.</p><h3>Conclusion</h3><p>This research showed that it’s better to leave traditional time series forecasting to what they’re used to, instead of trying to use Large Language models for time series tasks.</p><p>It doesn’t mean not to do anything; there are new things that could be interesting to pursue in the intersection of Time Series and Large Language Models.</p><h3>I’m working on a new search engine, nouswise, would love if you check it out and let me know your thoughts. You can contact me through any social media, X platform or LinkedIn.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81a099415702" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/are-language-models-actually-useful-for-time-series-forecasting-81a099415702">Are Language Models Actually Useful for Time Series Forecasting?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Token-wise Influential Training Data Retrieval for Large Language Models]]></title>
            <link>https://pub.towardsai.net/token-wise-influential-training-data-retrieval-for-large-language-models-7476037d6fc2?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7476037d6fc2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Tue, 04 Jun 2024 14:01:27 GMT</pubDate>
            <atom:updated>2024-06-04T14:01:27.438Z</atom:updated>
            <content:encoded><![CDATA[<h4>This paper presents RapidIn, a framework designed to estimate the influence of specific training data on the outputs of Large Language Models (LLMs) by compressing gradient vectors and enabling efficient retrieval.</h4><blockquote>I’m working on a powerful product, <a href="https://nouswise.com/home">nouswise</a>, go and check it out. It’s in the beta phase, if it interests you hit me up on <a href="https://twitter.com/reza__yazdanfar?lang=de">X</a>.</blockquote><p>It’s important to know which response corresponds to which data point in the training dataset, but as a matter of fact, AI models are considered as black boxes, which means we don’t know how the model generates a response.</p><h4>Influence Estimation</h4><p>Influence estimation aims to determine which training data points are most responsible for specific outputs generated by large language models (LLMs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjYnh-IM5bMmpZziHG5BXQ.png" /><figcaption>Figure 1: Influence estimation for a given generation. [<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><h4>What makes others not effective?</h4><p>Scalability, computation and efficiency have been avoiding the prior works to be used at scale, while this paper solves these two challenges, RapidIn.</p><h4>What are the challenges?</h4><p>There are several new challenges in the field of large language models:</p><ul><li>The new llms have been trained on massive datasets (e.g., 2 trillion tokens for llama-2)</li><li>They’re so so big, like 70 billion parameters</li><li>Almost all prior works have been around classification tasks</li></ul><h4>what’s the solution?</h4><p>RapidIn is a framework designed to estimate the influence of training data on large language models (LLMs) by compressing gradient vectors and efficiently passing over cached gradients.</p><blockquote>Sounds amazing, doesn’t it? but how actually it work?</blockquote><p>It has two stages:</p><h4>1) caching</h4><p>This involves compressing the gradient vectors of each training data into a low-dimensional representation known as RapidGrad, significantly reducing their size to MBs or even KBs, making them suitable for storage. The caching operations are independent, allowing for multi-GPU parallelization which further speeds up the process .</p><blockquote>Now the data is prepared for the next stage, retrieval.</blockquote><h4>2) retrieval</h4><p>During retrieval, for any given test generation, its gradient vector is also converted into a RapidGrad. The influence is then estimated by computing the inner products between this RapidGrad and the cached RapidGrads of all training data points.</p><p>Let’s get into more details:</p><h3>1) Caching Stage:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LTx0B3Cd780tJBS_RQM1Hg.png" /><figcaption>[<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><h4>Layer-wise Normalization.</h4><p>Problem: Influence functions are fragile and could result in inaccurate results, this could stem from the model’s weights and gradients.</p><p>Solution: Applying Layer-wise L2-nomalization to the original gradients before conversion for trained models without weight decay or other regularization.</p><h4>Gradient Compression.</h4><p>Problem: Directly using equations for the influence of training data could reduce the speed of calculation and not memory efficient.</p><p>Solution: Implementing a vector compression based on the count-sketch data structure, Min-Max Hash, and random projection, by mixing random shuffling and random projection to compress the gradient vector.</p><h4>RandomShuffling.</h4><p>Problem: Though random permutation is common to use this method, it is not a suitable choice for llms. Gradient vectors have extremely large dimensionality (for e.g. the length of 7B for lamma-2 7b), it results in hectic as it scales.</p><p>solution: Random Shuffling. It’s suitable for large vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*GpzyD1JRgvZMYUKcCGtY5A.png" /><figcaption>[<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><p>This method enables efficient vector shuffling by repeatedly applying randomized permutations to rows and columns, avoiding the creation of a complete permutation vector</p><h4>Random Projection</h4><p>The researchers employed Random Projection, this technique involves generating a random vector 𝜌 following the Rademacher distribution, where each element 𝜌𝑖∈{−1,1} with equal probability. The original gradient vector 𝑣 is then element-wise multiplied by 𝜌, and the resulting values are summed in groups to produce a much smaller, lower-dimensional vector called <strong>RapidGrad.</strong></p><h4>Multi-GPU Parallelization</h4><p>Multi-GPU parallelization significantly enhances the efficiency and scalability of both the RapidIn’s caching and retrieval stages.</p><h3>2) Retrieval Stage:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AGnTICm7Kln97WZpRPBjPg.png" /><figcaption>[<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><p>The retrieval stage in RapidIn involves estimating the influence of training data on a specific generation by leveraging pre-cached, compressed gradient representations (RapidGrads).</p><p>Once the RapidGrads for all training data have been cached, the gradient vector of a given test generation is converted into a RapidGrad in the same way as during the caching stage.</p><p>The influence is then estimated by computing inner products between this RapidGrad and the cached RapidGrads of each training data point, significantly speeding up the process by operating on low-dimensional vectors instead of the original high-dimensional ones.</p><p>This stage is further accelerated by parallelizing the influence estimation across multiple GPUs and processes, with results gathered on the CPU, enhancing both speed and scalability</p><h3>Experimental Evaluation and Results</h3><p>The evaluations were conducted on the alpaca dataset and compared RapidIn against five baselines: random selection, embedding similarity, BM25, influence function, and TracIn.</p><p>The experiments also showed that RapidIn can handle large models like l 70b without encountering out-of-memory issues, unlike the influence function and TracIn methods.</p><p>The experimental evaluation of RapidIn, as shown in Table 4, reveals several key insights. Random selection and BM25 both achieve poor results in the influence estimation tasks.</p><p>However, the embedding similarity method performs reasonably well. This performance is attributed to the fact that the test generation and the poisoned data share the same trigger and similar content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9zaOtVqgXj3Xpynim4FfA.png" /><figcaption>Table 4: The result of verifying by backdoor attack. (LN denotes the layer-wise normalization.) [<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><h4>Memory and Time Consumption</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xy0Gx649YJRlI8e_p4bRQ.png" /><figcaption>Table1. length and memory usage of gradient vector for each training data. [<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*yQnWqNHyP0-GnBHhS-XXlw.png" /><figcaption>Table 2. Time (hour) of influence estimation for a single test generation on the entire alpaca 52 K. RapidIn used 2 GPUs, and T=1 for caching, T = 8 for retrieval. [<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><h4>Qualitative Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDJ325UbY3_kCrnfc89CJA.png" /><figcaption>Figure 4: Token-wise visualization of top-3 influential training data for the generation of asking the flu symptoms. [<a href="https://arxiv.org/pdf/2405.11724">source</a>]</figcaption></figure><p>I’m working on a powerful product, <a href="https://nouswise.com/home">nouswise</a>, go and check it out. It’s in the beta phase, if it interests you hit me up on <a href="https://twitter.com/reza__yazdanfar?lang=de">X</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7476037d6fc2" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/token-wise-influential-training-data-retrieval-for-large-language-models-7476037d6fc2">Token-wise Influential Training Data Retrieval for Large Language Models</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion]]></title>
            <link>https://pub.towardsai.net/softs-efficient-multivariate-time-series-forecasting-with-series-core-fusion-0ac40d2adcd2?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ac40d2adcd2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Sun, 19 May 2024 12:02:04 GMT</pubDate>
            <atom:updated>2024-05-19T12:02:04.809Z</atom:updated>
            <content:encoded><![CDATA[<h4>How SOFTS, a new MLP-based model with the innovative STar Aggregate-Dispatch (STAD) module, achieves state-of-the-art performance in multivariate time series forecasting with remarkable efficiency and scalability by reducing computational complexity from quadratic to linear.</h4><blockquote>Yes I know the subtitle seems too catchy 😅</blockquote><p>I’m skipping the part to say “yay, time series is important but challenging! and …” which means I assume the reader knows the delicacy of Time Series forecasting and wants to absorb the core!</p><p>What does this paper focus on?</p><h3>The contribution</h3><p>There are two main contributions in this paper: SOFT and STAD</p><h4>SOFT</h4><p><strong>SOFT</strong>: <strong>S</strong>eries-c<strong>O</strong>re <strong>F</strong>used <strong>T</strong>ime Series forecaster</p><p>It is designed for multivariate time series forecasting that uses STAD module to balance channel independence and correlation. This helped to achieve superior performance with linear complexity by centralizing channel interactions into a global core representation.</p><h4>STAD</h4><p><strong>STAD</strong>: the <strong>ST</strong>ar <strong>A</strong>ggregate <strong>D</strong>ispatch module</p><p>STAD is the foundation of SOFT, where SOFT is a simple MLP-based model. STAD is a centralized structure that captures the dependencies between channels in multivariate time series. The results show that this method is both <strong>effective </strong>and <strong>scalable</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o5_TTDeD2M7qaJvu.png" /><figcaption>Figure 1:Overview of our SOFTS method. The multivariate time series is first embedded along the temporal dimension to get the series representation for each channel. Then the channel correlation is captured by multiple layers of simple and efficient STAD modules. The STAD module utilizes a centralized structure which first aggregates the series representation to obtain a global core representation, and then dispatches and fuses the global information with each series. [<a href="https://arxiv.org/pdf/2404.14197">source</a>]</figcaption></figure><h4>Reversible Instance Normalization</h4><p>The researchers used this method as it is adapted from <a href="https://arxiv.org/pdf/2310.06625">ITRANSFORMER</a> paper, to consider normalization as a hyperparameter.</p><p><strong><em>Reversible Instance Normalization</em></strong> in iTransformer simply applies attention and feed-forward networks on the inverted dimensions, enabling the model to capture multivariate correlations and learn nonlinear representations effectively.</p><h4>Series Embedding</h4><p>Series embedding is <strong>less complicated</strong> than patch embedding, we can say it’s like setting the patch length to the length of the whole series. The researchers used a linear projection to embed the series of each channel to So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*eZS_RdqH09-9nu51TR-VTg.png" /></figure><h4>STar Aggregate Dispatch (STAD) module</h4><p>We refine the series embedding by several STAD modules:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*mVfUVtj0DgkvYDgm1KuUVg.png" /></figure><h4>Linear Predictor</h4><p>After N layer of STAD, there is a linear predictor for our task (forecasting), if S_<em>N </em>is the output representation of layer n, the prediction will be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*XN6YLps6pN6-g7CbFPdDuw.png" /></figure><blockquote>Let’s get into more details of STAD</blockquote><h3>Star Aggregate Dispatch Module</h3><p><em>The STar Aggregate Dispatch</em> (<strong>STAD</strong>) module is a centralized mechanism designed to <strong>capture dependencies </strong>between <strong>channels </strong>in multivariate time series forecasting. Unlike traditional distributed structures such as attention mechanisms, which directly compare characteristics of each channel pair and result in quadratic complexity, STAD reduces this <strong>complexity to linear </strong>by employing a centralized strategy. It aggregates information from all series into a global core representation and then dispatches this core information back to individual series representations, enabling efficient channel interactions with improved robustness against abnormal channels.</p><p>This centralized structure is inspired by the star-shaped systems in software engineering, where a <strong>central server</strong> aggregates and exchanges information instead of direct peer-to-peer communication between clients. This design allows STAD to maintain the benefits of channel independence while <strong>capturing necessary correlations to improve predictive accuracy</strong>. By aggregating channel statistics into a single core representation, STAD mitigates the risk of relying on potentially untrustworthy correlations in non-stationary time series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MCLDCbVrSSx8nFRZ.png" /><figcaption>Figure 2:The comparison of the STAD module and several common modules, like attention, GNN and mixer. These modules employ a distributed structure to perform the interaction, which relies on the quality of each channel. On the contrary, our STAD module utilizes a centralized structure that first aggregates the information from all the series to obtain a comprehensive core representation. Then the core information is dispatched to each channel. This kind of interaction pattern reduces not only the complexity of interaction but also the reliance on the channel quality. [<a href="https://arxiv.org/pdf/2404.14197">source</a>]</figcaption></figure><p>Empirical results show that the STAD module not only achieves superior performance over existing state-of-the-art methods but does so with significantly lower computational demands. This makes it scalable for datasets with a large number of channels or long lookback windows, a challenge for many other transformer-based models. Additionally, the STAD module’s universality allows it to be used as a replacement for attention mechanisms in various transformer-based models, further validating its efficiency and effectiveness.</p><p>The input for STAD is the series representation for each channel, processing it through an MLP and then pooling (here it is <strong>stochastic pooling</strong>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*-j-C7J2yp7wdcyY9cjl7wQ.png" /></figure><p>Now we calculated the core representation (O), we fuse the representations for the core and all the series:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*6MByo60pqlMIKIxE58qWIA.png" /></figure><p>The Repeat_Concat concatenates the core representation O to each series representation to get Fi. Then we give this Fi to another MLP and add the output to the previous hidden dimension to calculate the next one.</p><ul><li>Note that there’s also a residual connection from the input to the output.</li></ul><h3>Results</h3><p>Though the method looks simple, it reduced the complexity significantly (from quadratic to linear) which is great😅😉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*wuV8f84llFgSBGoZlgJCoQ.png" /><figcaption>Complexity comparison between popular time series forecasters concerning window length L, number of channels C and forecasting horizon H. Our method achieves only linear complexity.</figcaption></figure><p>The researchers experimented with a wide range of datasets and compared to most predecessors, as you can see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*6l9U7azXCAJZGNDYz8QSJw.png" /><figcaption>Multivariate forecasting results with horizon H ∈ {12,24,48,96} for PEMS and H ∈ {96,192,336,720} for others and fixed lookback length L = 96. Results are averaged from all prediction horizons. [<a href="https://arxiv.org/pdf/2404.14197">source</a>]</figcaption></figure><p>They did other experiments, but in order to prevent lengthening this article, I recommend reading the original research paper.</p><p>Disclaimer: I used <a href="https://nouswise.com/">Nouswise</a> to write this article, it’s like a search engine that you can find the information through your documents. It’s not generally available, but you can contact me directly to give access, it could be on <a href="https://twitter.com/reza__yazdanfar">X (formerly Twitter)</a> or our <a href="https://discord.gg/6Dsw9weFjs">Discord Server</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cWlV0t_1Wn8RsjFWVaHebA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ac40d2adcd2" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/softs-efficient-multivariate-time-series-forecasting-with-series-core-fusion-0ac40d2adcd2">SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nouswise: How to Be Your Best]]></title>
            <link>https://rezayazdanfar.medium.com/nouswise-how-to-be-your-best-a8b0ae33fe1a?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8b0ae33fe1a</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[study]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Thu, 16 May 2024 15:28:58 GMT</pubDate>
            <atom:updated>2024-05-16T15:28:58.997Z</atom:updated>
            <content:encoded><![CDATA[<p>Nouswise is a new search engine that searches through the sources you want to focus on, empowered with a stack of AI, nouswise unlocks the power of knowledge with information discovery with the least hallucination rate.</p><h4>I am pleased to write about the most amazing AI product that I have ever been involved in my entire life. And it is Nouswise!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZ5CagTAMG1SQ1SjoecUBg.png" /></figure><h3>Personal Search engine in the information you trust the most</h3><p>Nouswise is a new knowledge-ground platform that provides the most suitable environment to have full control to focus on your specific domain of expertise. It is built, designed, and serviced just for you, as a student of life. 😉</p><h3>Why am I so excited about it?</h3><p>My WHY is the fact that it unlocked my ability to say I can commemorate whatever I’ve learned from books/papers I’ve read, both physical and digital. On the other hand, I’m creating my library of resources that I read, take notes efficiently, and have an extended mind that is super smart.</p><blockquote>It’s not generally available right now, but if you want to get early access just hit me up on <a href="https://twitter.com/reza__yazdanfar"><strong>X (formerly Twitter)</strong></a>. 😉🤗</blockquote><h3>Why this name: Nouswise</h3><p>“Nous” is a Greek philosophical term meaning “mind and intellect” and “wise” means “having or showing experience, knowledge”.</p><h3>Here I am providing an overview of how you can enjoy and be your best:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g212Z4McQKzTMYt_tho7aw.png" /></figure><h3>Creating Nous (Notes)</h3><p>Currently, Nouswise has two ways of taking notes: Written Notes and Stored Responses.</p><h4>1. You can start writing from scratch like this:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SquCtttz3RB2EGduStTHYA.png" /></figure><p>You have a note editor that enables you to write as you wish. Anytime you start a new note, it saves it automatically as you write:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2MP1SdiUsi_GvpHWhVzu9g.png" /></figure><p>Or</p><h4>2. You can Ground and Ask:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z3Yf1AMb9v9WwpphTGW8Bw.png" /></figure><p>It’s like you select a source(s) and make Nouswise focus on it (them):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CEhEbozSqEaQ-ILO3Awa2A.png" /></figure><blockquote>Note that I uploaded the most important scientific paper in the recent decade for AI community “Attention Is All You Need’ since I can assume that almost most tech savvies are aware of it.</blockquote><p>It provides two suggested questions as suggestions for exploration, or you can simply ask what you&#39;re looking for to know:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vAGPcDxdTzTnLORchdspRw.png" /></figure><blockquote>We will talk about those arrows in each paragraph (citations)</blockquote><p>You can change the prompt without rewriting it from scratch, or just regenerate to get a different response. Whenever you get your answer, you can save it; it will be saved as a small box, nous (note).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ccsg5jWf41Su_JrUjsCTw.png" /></figure><p>You can open it to full screen or side by side if you need Nouswise during your research:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9xMubsSyD5HG_TlJwt-dQ.png" /><figcaption>Full-screen view</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2dlXxB3DM2gho-vzfC-vSA.png" /><figcaption>Side-by-side view</figcaption></figure><h3>Citation:</h3><p>When you query a specific information, Nouswise gives you answer with fine-grained citations from the sources you set. They’re shown in the form of arrows, and when you hover over them; the information of where the answer is coming from and a highlighted preview of that part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBG6GWZs66SGpjFxrut3ug.png" /></figure><p>I believe this is a great way to verify the trustfulness and reliability of the information given to you, or just get back to the source and read that part if you feel a need for that.</p><p>It’s amazing since it not only gives you where the answer comes from but also tells you the paper&#39;s name, page number, and the specific line it is referring to.</p><h3>Organizing and searching through your nous (notes)</h3><p>In the traditional approach of note-taking, the process has been always fun in the beginning and then morphing into hectic. Nouswise has solved this problem!</p><p>You can tag your notes (nous) and filter them based on your tags:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MxHgEEOBxtQPmn6AkmWtxA.png" /></figure><p>Or semantically search:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1zvqa9H1-DJQCXEWk7Nkg.png" /></figure><p><em>Here I just searched for the word “RNN”.</em></p><p>Or even, you can simply ask:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5i0HZglkHnCB1F9O1ccTrA.png" /></figure><p>The answer comes from the notes you have saved!</p><blockquote>Spoiler: soon this kind of answer will also include citations from your notes. 🤗😁🫡</blockquote><p>This was a quick overview of some features we’ve been working on. You can contact me directly here or my <a href="https://twitter.com/reza__yazdanfar">X (formerly Twitter)</a> to get early access. Also, I’m building a <a href="https://discord.gg/6Dsw9weFjs">community </a>of enthusiasts on Discord.</p><p><a href="https://nouswise.com/home"><strong>Nouswise.com</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8b0ae33fe1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ATFNet: Adaptive Time-Frequency Ensembled Network for Long-term Time Series Forecasting]]></title>
            <link>https://pub.towardsai.net/atfnet-adaptive-time-frequency-ensembled-network-for-long-term-time-series-forecasting-b7532d6cd204?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7532d6cd204</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Mon, 29 Apr 2024 16:01:36 GMT</pubDate>
            <atom:updated>2024-04-29T16:01:36.220Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is a simple but comprehensive explanation of ATFNet, a new deep-learning model for long-term time forecasting. Enjoy!</h4><p>ATFNet is a deep learning model that combines time and frequency domain modules to capture dependencies in time series data. It introduces a novel weighting mechanism to adjust weights based on periodicity, enhances the Discrete Fourier Transform, and includes a Complex-valued Spectrum Attention mechanism for intricate relationship discernment, and of course, outperforming current methods in long-term time series forecasting (<em>This is the fixed part of any new paper, I mean being better than others</em>😂).</p><blockquote>Oh dear, it sounds a lot!! (at least to myself) But don’t worry I’m going to break it down into digestible😉</blockquote><h3>The challenge of Time Series data:</h3><p>One of the challenges with time series comes from two concepts: periodicity and non-periodicity.</p><p>Periodicity means that at any given time, the value is dependent on a <strong>certain data point at a certain time ago </strong>and is more <strong>global</strong>, while the latter is about the dependency with the <strong>near data point </strong>and is more <strong>local</strong>. Taking both into account when it comes to time-series forecasting is challenging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxxO80JzJX8w4zF_fvx9jw.png" /><figcaption>Figure 1. Real-world time series with distinct periodic pattern</figcaption></figure><p>Also, there are two types of domain for time series (TS) analysis, namely <strong>time domain </strong>and <strong>frequency</strong>.</p><p>The <strong>time domain </strong>is about changes in the intensity of a signal as time progresses while the <strong>frequency domain</strong> analyzes time series from the frequency perspective.</p><p>The first one helps to understand local dependencies and the latter global dependencies.</p><p>Mixing both is a great approach but needs to get done efficiently unless we cannot take advantage of both at the same time.</p><p>ATFNet framework (<em>the proposed model here</em>) aims to address the challenge of dealing with the mixing of distinct periodic properties in real-world time series data. This combination allows for a comprehensive analysis that leverages the strengths of both the time and frequency domain representations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MYCUsdtfWu26QmBw.png" /><figcaption>Figure 2: Model architecture of ATFNet. ATFNet is mainly composed of three sub-parts: 1) T-Block to capture local dependency from time domain; 2) F-Block to capture global dependency from frequency domain. The Extended DFT is used to generate frequency-aligned spectrum of input series. 3) The Dominant Harmonic Series Energy Weighting to allocate appropriate weights for F-Block and T-Block according to the periodic property of input series. [<a href="https://arxiv.org/pdf/2404.05192">source</a>]</figcaption></figure><p>ATFNet is mainly composed of three sub-parts:</p><ol><li>T-Block to capture local dependency from the time domain.</li><li>F-Block to capture global dependency from the frequency domain. The Extended DFT is used to generate a frequency-aligned spectrum of input series.</li><li>The Dominant Harmonic Series Energy Weighting allocates appropriate weights for F-Block and T-Block according to the periodic property of the input series.</li></ol><blockquote>Seems a lot, doesn’t it?</blockquote><blockquote>Well let’s break it down to understand it better ;)</blockquote><h3>Extended DFT:</h3><p>Here we use Extended DFT in ATFNet to align the frequency spectrum of the input series, allowing for a more comprehensive analysis of the time series data. We decrease the cost by considering just the first half of the output (the second is removed).</p><p>DFT basis of the complete series is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*NM92pgS0BdIcBA4tdeTIkg.png" /></figure><p>In this way, we achieve a spectrum of length L + T that aligns with the DFT spectrum of the complete series.</p><blockquote>Now, let’s dive in the model architecture🫡</blockquote><h3>Architecture in general:</h3><ol><li>T-Block</li><li>F-Block</li><li>The Dominant Harmonic Series Energy Weighting</li></ol><p><em>In general, the input goes in two directions, firstly T-Block, and secondly Extended DFT and then F-Block.</em></p><h3><strong>F-Block</strong></h3><p>Note that F-Block is based on the original attention (<a href="https://arxiv.org/abs/1706.03762"><em>attention is all you need</em></a>) with a bit of modification. In a nutshell, the attention mechanism is the next level of neural network, routing everything to everything to learn and generalize on the data. It’s not efficient but powerful, though there are lots of modifications to make it more efficient.</p><p>This block takes in F, a univariate spectrum (the output of Extended DFT) with the length of L. Then we normalize F for RevIN method to handle frequency domain spectra (we denormalize them at the end).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*si5cjmVdK0u5kjRk8mgUaA.png" /></figure><p>Then the final output of Complex-valued Spectrum Attention is calculated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*qtSB2LUVYWCvBNDNA9sPZQ.png" /></figure><p><a href="https://openreview.net/forum?id=cGDAkQo1C0p">Reversible Instance Normalization (RevIN)</a>, is a normalization-and-denormalization method with learnable affine transformation. RevIN can be applied to any deep neural network, being a model-agnostic method.</p><p>Note that the positional embedding is disabled here (not used).</p><h3>T-Block</h3><p>This part is responsible for the local dependency of the data, in other words, the time domain. The data is patched into N smaller patches with the sequence length of P. We embed each small patch into the encoder transformer and at the end we use a linear projection to generate the output. Note that RevIN is used here as well (for distribution shift).</p><h4>Dominant Harmonic Series Energy Weighting</h4><p>The Dominant Harmonic Series Energy Weighting mechanism dynamically adjusts the weights between the time and frequency domain modules based on the periodic property of the input series</p><h3>Results:</h3><blockquote>It’s part of any paper to say that we’re the best, and this paper is not and exception here😅</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-eH-JhZRC_OharWehu3KA.png" /><figcaption>Multi-variate long-term time series forecasting results on 8 datasets. The best results are in bold and the second best results are underlined. We only display the average results of all prediction lengths T ∈ {96, 192, 336, 720} here, see Table 9 for full results. [<a href="https://arxiv.org/pdf/2404.05192">source</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIUShDjE76r1L1LOVlTUtA.png" /><figcaption>Uni-variate long-term time series forecasting results on ETT datasets. The ETT datasets have a target feature ‘Oil Temperature’, and we take it as the uni-variate time series for forecasting. The best results are in bold and the second best results are underlined. We only display the average results of all prediction lengths T ∈ {96, 192, 336, 720} here, see Table 10 in Appendix for full results. [<a href="https://arxiv.org/pdf/2404.05192">source</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_oqYWI5aEEu6p9eKL0b_fQ.png" /><figcaption>Ablation study results. The best results are in bold and the second best results are underlined. We only display the average results of all prediction lengths T ∈ {96, 192, 336, 720} here, see Table 11 in Appendix for full results. [<a href="https://arxiv.org/pdf/2404.05192">source</a>]</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7532d6cd204" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/atfnet-adaptive-time-frequency-ensembled-network-for-long-term-time-series-forecasting-b7532d6cd204">ATFNet: Adaptive Time-Frequency Ensembled Network for Long-term Time Series Forecasting</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SimVP: Simpler yet Better Video Prediction]]></title>
            <link>https://rezayazdanfar.medium.com/simvp-simpler-yet-better-video-prediction-7401b4e443e6?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7401b4e443e6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Mon, 23 Oct 2023 21:03:55 GMT</pubDate>
            <atom:updated>2024-04-09T12:51:56.209Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is going to be a short, not a long story. The story of a simpler model architecture but efficient for video prediction.</h4><blockquote>I read every day, my book, stories, and papers. Here you see those AI-related research that captures interests. Doesn’t include everything, but enough to know.</blockquote><h3>I am building nouswise🤗!! check it out and get on the waitlist😉 If you want early access hmu on X (Twitter). Let me know your thoughts.</h3><h3>WHY DO WE NEED TO PREDICT VIDEO’S FUTURE?</h3><p>For a variety of tasks, it’d be awesome to have at least an intuition of what would happen in the future, the next bubble interests economists, the next fashion trend interests fashion designers, the next Google interests Venture Capitals, etc.</p><p>A way of forecasting the future is using video data, exactly like we humans, whenever we can speculate a goal in soccer in a few seconds, or even less, beforehand and then there’s an unbelievable and bright excitement in our eyes, etc.</p><p>In the arena of Artificial Intelligence (computer vision to be precise), researchers in the community are developing algorithms with diverse architectures to predict the new few frames. A growing focus for all, researchers, companies, and investors.</p><p>There are numerous applications now, including simulating the behavior of dynamic systems such as proteins in biology or self-driving vehicles on the streets and so on.</p><p>With the rise of Attention-base Transformers, attentions are applied to a variety of tasks. On the other hand, as we can say they are kind of ANNs (Artificial Neural Networks) for <strong>generalizing </strong>data but weigh more <strong>powerful</strong>, but also high energy consuming for computation. Consequently, a lot of improved variations have been developed.</p><p>This article, simply, is an attempt to respond to the question “<strong>Is there a simple method that can perform comparably well?</strong>”</p><blockquote>Is there a simple method that can perform comparably well?</blockquote><p>Well, the results say yes: <strong>SimVP</strong></p><h3>HOW CAN WE DEVELOP A SIMPLE MODEL BUT ROBUST?</h3><h4>First, we must look into what has been done, and then we can find how to do the job</h4><p>What makes SimVP different is not using an attention mechanism in its architecture but fully convolutional operators.</p><p>We can categorize key methods into four branches:</p><p>a) RNN-RNN-RNN</p><p>b) CNN-RNN-CNN</p><p>c) CNN-ViT-CNN</p><p>d) CNN-CNN-CNN</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lr7Orw6_PpCIaPoUCKXnRg.png" /><figcaption>Fig 1. Red and blue lines help to learn the temporal evolution and spatial dependency. [<a href="https://arxiv.org/pdf/2206.05099.pdf">source</a>]</figcaption></figure><p>All four various architectures are awesome for video prediction, have gained a lot of attention, and worked on. The paper has mentioned developed architectures in these four categories with their limitations.</p><p>For now, it is enough to know that SimVP is a CNN-CNN-CNN architecture that has less complexity than other RNN or ViT-based architectures.</p><blockquote>The results have been astonishing at the time of the research on various datasets. However, contrary to the more complicated the better, here we aim to simplify.</blockquote><p>The reason is obvious, the more complicated takes a huge portion of time, energy, and resources for a long time. And then, we want to use it in the real-time, deployment stage, with less time to predict, throughput.</p><p>So now we know it has convolutional operators. Now let’s dive into the architecture itself.</p><h3>WHAT?</h3><h4>So now, we know that SimVP is a CNN-CNN-CNN, what is the architecture look like?</h4><p>The SimVP in one view looks like Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TKbLpH88Txg7i00BZwVGMg.png" /><figcaption>Fig 2. The architecture in one view [<a href="https://arxiv.org/pdf/2206.05099.pdf">source</a>]</figcaption></figure><p>In a nutshell, it has three parts:</p><ol><li>An encoder to understand and encode spatial features of the frame.</li><li>Then, a translator to understand the dynamical behavior (temporal evolution),</li><li>And finally, a decoder to decode and blend both temporal and spatial features</li></ol><blockquote>Yeah there are preprocessing and postprocessing as well, but we don’t consider them as architecture.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_Q4zPrr1fZXHob_k7faaA.png" /><figcaption>Fig 3. The architecture in detail. The encoder stacks Ns ConvNormReLU block to extract spatial features, i.e., convoluting C channels on (H, W). The translator employs Nt Inception modules to learn temporal evolution, i.e., convoluting T ×C channels on (H, W). The decoder utilizes Ns unConvNormReLU blocks to reconstruct the ground truth frames, which convolutes C channels on (H, W). [<a href="https://arxiv.org/pdf/2206.05099.pdf">source</a>]</figcaption></figure><p>The encoder is a simple convolution and normalization layer and the decoder is unConvNormReLU blocks (ConvTranspose2d + GroupNorm + LeakyReLU) which all are the established block and operations.</p><p>For the Translator, it uses a number of modules called Inceptions for learning the temporal features. And each Inception is designed like Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*MXpuXwGzZcrovznqKbCY1Q.png" /><figcaption>Fig 4. Inception [<a href="https://arxiv.org/pdf/2206.05099.pdf">source</a>]</figcaption></figure><h4>Results</h4><p>I am not going to discuss the results, generally, SimVP has achieved SOTA results on lightweight benchmarks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omTFbG2PzTfpyLsO9mxuLw.png" /><figcaption>Table 1. SimVP vs SOTA. The optimal(or suboptimal) results are marked by bold(or underlined). [<a href="https://arxiv.org/pdf/2206.05099.pdf">source</a>]</figcaption></figure><blockquote>The main research is SimVP: <a href="https://arxiv.org/pdf/2206.05099.pdf">Simpler yet Better Video Prediction</a>, the code is also publicly available on <a href="https://github.com/A4Bio/SimVP-Simpler-yet-Better-Video-Prediction">Github</a>. The code is an easy going one, contrary to other repos I’ve checked. 😉</blockquote><p>This was a short story, and I believe it’d be more productive for me sometimes to know a problem and how it’s been solved, and only if I face the issue I’d start knowing it inside out. What you think? Lemme know.</p><p>Where all are tired and usually skip:<br>I write more about AI models and their architectures, if you liked it follow me on <a href="https://rezayazdanfar.medium.com/"><strong>Medium</strong></a>.<br>I got fascinated in business and markets, so just in case you might be interested, hit me up and/or subscribe my <a href="https://rezayazdanfar.substack.com/"><strong>Substack</strong></a>; I write on behalf of industries and startups.<br>By the way I love networking and getting to know various people with various backgrounds, so why not? Do not hesitate to book a time on <a href="https://calendly.com/reza_yazdanfar/30min"><strong>Calandly</strong></a><strong> </strong>or contacting me directly on <a href="https://de.linkedin.com/in/rezayazdanfar"><strong>Linkedin</strong></a>.</p><h4><a href="https://mlearning.substack.com/about">WRITER at MLearning.ai</a> / <a href="https://open.substack.com/pub/mlearning/p/ai-art-meets-heart-start-your-augmented-reality?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">Augmented AI</a> / <a href="https://open.substack.com/pub/mlearning/p/how-to-create-your-own-ai-generated?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">AI movie</a> / <a href="https://open.substack.com/pub/mlearning/p/list-of-80-gpt4-v-use-cases-free-alternative?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">80+ GPT4 V</a></h4><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7401b4e443e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Can We Foresee Our Relationship With AI?]]></title>
            <link>https://pub.towardsai.net/how-can-we-foresee-our-relationship-with-ai-9aa0f746dcec?source=rss-45e9c77210d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9aa0f746dcec</guid>
            <category><![CDATA[future]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[society]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Reza Yazdanfar]]></dc:creator>
            <pubDate>Fri, 06 Oct 2023 14:01:50 GMT</pubDate>
            <atom:updated>2023-10-06T14:01:50.647Z</atom:updated>
            <content:encoded><![CDATA[<h4>Does AI make us dumber or smarter? How can we see our relationship with AI?</h4><p>Human-AI interaction at the current stage or post-AGI is always a matter of debate. Whether you are on the next negative side or position side, I would like to disentangle the aftermath of AI on our minds. Will humans become more intelligent in a world where AI is proliferated or unintelligent?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/0*8WGl7-C1uIAXPBKp.jpg" /><figcaption><a href="https://www.techlicious.com/blog/does-technology-make-us-smart-or-dumb/">source</a></figcaption></figure><p>Firstly, AI is not a brand-new part of life, we already get used to it while we listen to Amazon recommendations or scroll the indefinite reels of our TikTok, etc. They know us pretty well, our interests, and sometimes dictate what to do by recommending us (the modern version of advertising — Recommendation Systems). Some people, on this, consider “conspiracy” while tech-savvies argue this helps us avoid burdening our minds with thinking about meager things. I have seen this debate personally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*KDW9mDjiyIpIEZFA.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Big_Tech">source</a></figcaption></figure><p>It is true that tech has improved productivity but eliminated a lot of tasks by automating them.</p><p>There was a time when people were hired to do calculations on paper, then they were given the calculator to help them lest putting too much time and energy into calculating on paper, now they have computers and Excel to do the math for them.</p><p>There was a time when people were hired to be responsible for manually stopping lifts for timing to stop the lift at a specific floor, now they are automated and just need some repairment when they are positioned to operate. Currently, machines are responsible for 30% of tasks, while the rest is for people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fu1T4jPi4emWTQwfLWUpew.png" /><figcaption><a href="https://www.shrm.org/hr-today/news/all-things-work/pages/jobs-of-the-future.aspx">Jobs of the Future (shrm.org)</a></figcaption></figure><p>A calculator has not made an accountant dumb to calculate on air but more productive and precise. Automated lift has made some expert technicians capable of doing their job, but not everyone is capable of doing their job. AI is the next revolution. Speculations say that this is about to be 50–50 blend of humans and machines, a dramatic shift. That is why many people worry about it, lest they can’t keep pace and equip themselves before AI does their job, even better and cheaper and more productive.</p><p>In any way, this article is not about whether AI is taking our jobs or not. This will cost, a labor cost, for those who fell behind. More than 120 million workers globally will need retraining in the next three years due to artificial intelligence’s impact on jobs, according to an <a href="https://newsroom.ibm.com/2019-09-06-IBM-Study-The-Skills-Gap-is-Not-a-Myth-But-Can-Be-Addressed-with-Real-Solutions">IBM survey</a>.</p><blockquote>After all, the subject here is our intelligence; shall AI expand our minds to be more creative, innovative, and productive??</blockquote><p>We are giving our intelligence to the data. Yes, it is true to a significant extent, the data that most AI tools have been trained on have been produced by us, people. The more data we produce, the more intelligence they get. That&#39;s why European countries are concerned about their citizens’ data, as well as the European people themselves as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3EGNwRK2bEeuhuBK.jpg" /><figcaption><a href="https://www.basecamp.digital/kuenstliche-intelligenz-wie-steht-es-um-den-ai-act-der-eu/">source</a></figcaption></figure><p>It has been that we draw a line, a border, between our minds and AI. We as creators and they (machines) as created. We handed over our intellectual tasks to them. We are superior to them. Let’s reorganize this viewpoint; on the contrary, we both, human and AI, are coupling and merging, blending and combining, forming units.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F6srLZJ0wSszhzW3.jpg" /><figcaption><a href="https://tycoonsuccess.com/what-is-the-difference-between-data-science-artificial-intelligence-and-machine-learning-how-data-science-can-be-a-good-career-option-for-freshers/">source</a></figcaption></figure><p>The concept of extended mind is defined by two philosophers, Andy Clark and David Chalmers, that says that “cognition” doesn’t just happen in our heads, in other words, the mind is not the residence of the brain, exclusively, but it could extends into outer world, physical objects. For example, not long after Apple’s release of its smartphones, it was capable of doing tasks that we used to do by ourselves, such as memorizing telephone numbers, navigation, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7Q210p26rk_plcsu" /><figcaption><a href="https://nypost.com/2023/01/09/on-this-day-in-history-jan-9-2007-steve-jobs-introduces-apple-iphone-at-macworld-in-san-francisco/">source</a></figcaption></figure><p>Likewise, we can see “extended body”, for example, implanting chips by Neuralink into our heads, or the external suit in Iron Man, etc. These are exemplars of the above-mentioned combining tech in general and humans. And we can generalize the idea to capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*5ffgkCKgmJxkSmyC" /><figcaption><a href="https://creator.nightcafe.studio/creation/5X27Um1ekstO1Wr90QNu">source</a></figcaption></figure><p>In fact, this has been good for humankind in the 21st century. Smartphones, help blind or low-vision people to navigate their last few steps by detecting doors or live captions for deaf people, and so on and on. (<a href="https://www.apple.com/newsroom/2022/05/apple-previews-innovative-accessibility-features/#:~:text=Using%20advancements%20across%20hardware%2C%20software,Voice%20Control%20and%20Switch%20Control">Apple previews innovative accessibility features — Apple</a>) Though tech has progressed a lot in its journey so far, it still needs to be improved more but has the potential. (<a href="https://www.thevaluable500.com/spotlight/lets-discuss-how-can-technology-empower-people-with-disabilities">Let’s discuss: How can technology empower people with disabilities? — The Valuable 500</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUPgXyIz9odCKIB8YSM9dA.png" /><figcaption><a href="https://www.thevaluable500.com/spotlight/lets-discuss-how-can-technology-empower-people-with-disabilities">source</a></figcaption></figure><p>There are three conditions that this coupling must meet to be in harmony:</p><ol><li>Each part has its own casual responsibility (or responsibilities) while they are well connected and construct a thorough system</li><li>All systems are operating just exactly as the mind thinks to do</li><li>If external systems (not all but at least one) are lost, the overall behavioral capacities should be decreased</li></ol><p>Up to now, we’ve been utilizing tech to enhance our productivity, which worked quite well if we want to be honest with ourselves. According to another philosopher, <a href="https://twitter.com/JohnDanaher">John Danaher</a>, there is a distinct but eliminating difference between “external” and “internal” forms of the combination of human and tech for enhancement reasons. He believes if tech can improve the mental health or behavior of people in new environments, it acts towards human enhancement.</p><blockquote><strong><em>If the [extended mind hypothesis] is true, then we are always enhancing the human mind through the use of technology.</em></strong></blockquote><blockquote><strong>John Danaher -</strong>Why internal moral enhancement might be politically better than external moral enhancement.</blockquote><p>Thus we can apply the same analogy to AI as well for those AI tools (e.g. ChatGPT, etc.) that we humans can take advantage of directly. Yes, we cannot easily apply it to all AI tools. Why? Because some of them are in our control, by control I mean we could trigger their action, such as prompting an AI chatbot, asking Alexa/Siri a favor, etc. On the other hand, some of them are partially in our control, such as Instagram/TikTok algorithms, Amazon recommendation systems, etc. And some are out of our control.</p><p>For this, David Chalmers gave a talk on whether we can consider Large Language Models, LLMs, as extended minds or not. He questioned whether we could consider AI tools, like ChatGPT, as tools to extend our minds, because given the right prompt, the chatbot responds to us with useful information. To the amazement of all, he says that a technology like ChatGPT might be operating too much independently to be an extension of the mind.</p><p>The theory of extended mind applies only to a number of AI tools, and as they get more autonomous the less they’re able to be considered as a mind extension.</p><p>He reasoned and discussed the criterion of why and when we consider a new technology as an extended mind; at the end, he concludes with a question to think about “Is this process extended cognition?”. A funny thing he did and we can ask the same question from GPT-4:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*RvUOaCmEHA5-nzYt10xodg.png" /></figure><p>Interestingly there is a paper, published titled “Do Large Language Models Know What Humans Know?” <a href="https://arxiv.org/abs/2209.01515">[2209.01515] Do Large Language Models know what humans know? (arxiv.org) </a>They did a False Belief Task, humans were correct 82.7% of the time while LLM was correct 74.5% of the time, which is pretty impressive; but the large language model did not describe human behavior completely. That says that language statistics solely are enough to generate something to false beliefs, but not fully account for humans.</p><p>Also, on the other hand, relying more and more on AI, to recommend (or on extreme dictate) us what to do, some argue that it makes our brains dumber and less creative. It has not been a new subject to debate, it has been discussed on Google ( <a href="https://www.theatlantic.com/magazine/archive/2008/07/is-google-making-us-stupid/306868/">Is GOOGLE Making Us Stupid?</a>) in the past or now on new advances of AI (<a href="https://www.forbes.com/sites/forbestechcouncil/2023/09/15/will-generative-ai-make-us-dumber/">Will Generative AI Make Us Dumber?</a>)</p><p>One of the very first stories of facing human and artificial intelligence goes to the late 90s, the story of Deep Blue and the world champion chess player, Garry Kasparov. He won and lost several games with the IBM machine, Deep Blue. It is out of the scope of this article to articulate the story in detail, so we only suffice to one of the lessons that Kasparov mentioned in his book, <a href="https://www.amazon.de/-/en/Garry-Kasparov/dp/1473653517">Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins</a>.</p><p>He believes that anything we, us humans can do, machines can do better. Contrary to great minds and computer scientists like Alan Turing, who thought chess is the ultimate test of intelligence. He is not against AI, actually, he believes in a simple equation, a person plus machines can beat a genius man in his area of expertise. You also can watch <a href="https://www.youtube.com/watch?v=zhkTHkIZJEc">Deep Thinking | Garry Kasparov | Talks at Google</a></p><p>In fact, it’s become a reality, in 2016, a series of Go games took place between DeepMind’s AI, AlphaGo, and the world champion of Go, Lee Sedol. The human, Lee Sedol, won only one out of five games and lost the rest. The same story on chess; we should consider that Go is much harder than chess because of its gigantic freedom of movement, which leads to numerous strategies.</p><p>Later on, an employee of DeepMind with no knowledge of the game, and only with recommendations of the AI was able to beat the Go world champion. If you are interested in the subject of human plus machine, watch the video by <a href="https://hbr.org/webinar/2018/08/human-plus-machine-reimagining-work-in-the-age-of-ai">HBR</a> or read the book “<a href="https://www.amazon.com/Human-Machine-Reimagining-Work-Age/dp/1633693864">Human + Machine: Reimagining Work in the Age of AI</a>”</p><p>Some talks and books would be useful in comprehending this article</p><p><a href="https://www.youtube.com/watch?v=ksasPjrYFTg">Is your phone part of your mind? | David Chalmers | TEDxSydney</a></p><p><a href="https://www.youtube.com/watch?v=tETOCvTzKEQ">David Chalmers: Do large language models extend the mind?</a></p><p><strong>Thanks</strong> to <a href="https://medium.com/u/4979de61f5b6">Ali Moezzi</a> for reading drafts of this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9aa0f746dcec" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-can-we-foresee-our-relationship-with-ai-9aa0f746dcec">How Can We Foresee Our Relationship With AI?</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>