<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Benedikt Heidrich on Medium]]></title>
        <description><![CDATA[Stories by Benedikt Heidrich on Medium]]></description>
        <link>https://medium.com/@benedikt_heidrich?source=rss-ddaa84f9958b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*oZyiTxuZtRmcsz0U</url>
            <title>Stories by Benedikt Heidrich on Medium</title>
            <link>https://medium.com/@benedikt_heidrich?source=rss-ddaa84f9958b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:31:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@benedikt_heidrich/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What an agent saved me on 496 bug reports]]></title>
            <link>https://medium.com/@benedikt_heidrich/what-an-agent-saved-me-on-496-bug-reports-65bd32afb090?source=rss-ddaa84f9958b------2</link>
            <guid isPermaLink="false">https://medium.com/p/65bd32afb090</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ai-coding]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Benedikt Heidrich]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 22:13:12 GMT</pubDate>
            <atom:updated>2026-10-11T19:29:34.152Z</atom:updated>
            <content:encoded><![CDATA[<p>Software projects often accumulate a backlog of bug reports. Many are old, and it is unclear whether they have already been silently fixed or can still be reproduced. Each report has to be checked manually. This sounded like a task for an agent, so I built a small system and ran it on the sktime bug backlog. As a disclosure: I am an sktime core developer, so these results are not an outsider’s view.</p><h3>Solving the fear of running random code from the internet</h3><p>This plan requires reading public bug reports and running code written by strangers on my laptop. That is scary. Two risks in particular:</p><ul><li>The issue text is attacker-controlled. An issue can therefore contain harmful code or a prompt injection.</li><li>The code generated by the model can do whatever it wants, since the system is supposed to act autonomously.</li></ul><p>To mitigate these risks, I made two decisions.</p><ol><li>I gave my agent no tools at all. That is, I started Claude with `claude -p — tools “”`. The model can therefore only read the text I provide and return text.</li><li>The code generated by the agent never runs on the host. Instead, I run each snippet in a fresh Podman container without network access.</li></ol><h3><strong>The pipeline</strong></h3><p>The pipeline consists of two stages. First, I fetched all issues labelled as bugs, including their descriptions and comments, and stored them as JSON. Second, I used an LLM to decide whether an issue is verifiable and, if so, whether it is still a bug. To make this decision, the agent repeated the following steps until it reached a decision.</p><ol><li>If the LLM decides the bug is automatically verifiable, it generates code. Automatically verifiable means that the bug is clearly enough defined to write code that raises an exception if the bug still exists.</li><li>The code is executed in a Podman container that has sktime and all its dependencies installed. Since the container has no internet access, the LLM is prompted to generate code that does not require network access.</li><li>If no exception is raised (exit 0), I assume the bug is gone. If an AssertionError is triggered by one of the assertions in the generated code, I assume the bug is still present. If any other exception occurs, I assume the code itself is broken and the LLM has to generate new code.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6vCw1o1YUZuvMEMnGWcJIg.png" /></figure><p>Note that other errors can occur as well, such as missing dependencies. Since the LLM cannot fix those — the container has no internet access — I skip the repair step in these cases.</p><h3><strong>Is such an approach worth running?</strong></h3><p>To assess whether it is worth it, I need to know how often the LLM gives a helpful answer (bug reproduced or no bug) and how often that answer is right.</p><p>The following diagram shows the share of the different answers. In more than 40% of the cases, the issue was not automatically verifiable. In 11%, some other error occurred, such as missing dependencies or timeouts. Regarding the helpful answers: in 28% of the cases the LLM reported that the bug is still present, and in 21% that there is no bug. In total, 49% of the cases produced an answer that was helpful for triaging. Note that the not automatically verifiable option is explicitly offered in the prompt, to avoid hallucinations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjVcXgD5dwX8eRF_A9OCNA.png" /><figcaption>The share of how many bugs are labelled as reproduced bugs, no bugs, not verifiable or other.</figcaption></figure><h3><strong>Accuracy — 100 hand-adjudicated verdicts</strong></h3><p>To asses if it is worth, we need to examine the accuracy on the helpful answers. Thus, I manually checked the oldest 100.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6UPxzfTIOH-0xIKgrtYWQ.png" /><figcaption>The number of issues triaged correctly and incorrectly</figcaption></figure><p>73% of the 100 answers were correct, 20% were wrong, and 7% I could not verify. Not verified means that I am not familiar enough with the relevant parts of the sktime code and could therefore not judge whether the answer was correct.</p><p>20% wrong is a lot. So I checked whether there is an asymmetry between the two kinds of answer. The following plot shows that there is one: answers labelled bug reproduced were correct in 79% of the cases, while answers labelled no bug were correct in only 68%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lXNe3WiWagycn3hzBDZETg.png" /></figure><p>My explanation for this asymmetry is that an AssertionError is a clear signal that a bug is there, and not something you hit by accident. A green test, in contrast, can mean either that the bug is fixed or that the test is not testing the right thing. It is therefore much easier to end up with a green test by accident.</p><p>Overall, these numbers show that full autonomy is not possible with this system.</p><h3><strong>Was it actually worth it? — It depends</strong></h3><p>That question is hard to answer, since I would need to compare how much time I spend triaging issues without AI against how much I spend with it. I only have numbers for the latter. I measured the time on a subset of the 100 hand-checked issues. The mean was 6.3 minutes per issue.</p><p>My sense is that I would generally need more time to triage an issue without AI, since I would have to understand it completely on my own and write the reproduction code myself. On top of that, the labelling by the LLM helped me focus on the issues that are easier to triage, which makes the comparison even harder.</p><p>So overall I would say it saved me some time and helped me triage the backlog. But the setup I tested requires a human in the loop who checks the results carefully.</p><h3><strong>What would I change next time?</strong></h3><p>I would give the agent more freedom. Probably, instead of running only the generated code in a container, I would run the agent itself autonomously inside the container.</p><p>This might cost more tokens, since the agent would investigate more, but hopefully it would also produce more correct answers. I would also run it on a server, so that nothing happens on my laptop and the run does not stop when I close it.</p><p>I hope this post gives some insight into how to use an agent to triage a bug backlog. I would be glad to hear your thoughts on the approach.</p><h3><strong>Notes</strong></h3><p>The system was run on the 18 August 2026. The run used the Claude Code default model; it was not pinned to a specific version. The results may therefore change if the system is run again at a later date.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65bd32afb090" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Driven AutoForecasting with sktime’s `craft()`]]></title>
            <link>https://pub.towardsai.net/llm-driven-autoforecasting-with-sktimes-craft-0355f5c720e8?source=rss-ddaa84f9958b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0355f5c720e8</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Benedikt Heidrich]]></dc:creator>
            <pubDate>Mon, 25 May 2026 20:01:00 GMT</pubDate>
            <atom:updated>2026-07-27T16:19:21.339Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Jg7lbDlHpD3KG4vOAkn7w.png" /></figure><p>AutoML is relying in many cases on some kind of grid searches. This is expensive. However, if humans are selecting hyperparameters, they often have some feeling about a good choice based on previous experiences. So I wondered, if LLM can act like an human expert and determine the parameters by considering the problem and using the knowledge stored in its weights. By reading about the autoresearch package from Andrew Karpathy, I got inspired to use something similar for time series forecasting. I wrote an sktime forecaster that uses an LLM to propose and refine forecasting pipelines. It uses the `craft()` method from sktime to turn LLM-generated blueprints into sktime forecasters.</p><p>In this post, I shortly explain the `craft()`method, and how it is used in the `AutoResearchForecaster` to create an agentic AutoML loop. I also discuss current limitations and potential improvements.</p><h3><strong>`craft()`ING Pipelines in sktime</strong></h3><p>The `craft()` method in sktime is a version of `eval()`. It takes a string that is formatted like a Python constructor call of an sktime estimator. This string can reference any class that is in the scope of sktime’s `all_estimators()`. When you call `craft()` on that string, it returns a fully constructed sktime object. Internally, `craft()` is doing all the imports required to create the estimator. Besides creating single estimators, `craft()` can also create compositors, such as pipelines: <em>“TransformedTargetForecaster([Deseasonalizer(sp=12), Detrender(), AutoARIMA()])”</em></p><pre>from sktime.registry import craft<br># Bare estimator<br>forecaster = craft(&quot;NaiveForecaster(strategy=&#39;last&#39;)&quot;)<br># Full pipeline - no imports needed<br>pipeline = craft(<br>  &quot;TransformedTargetForecaster([Deseasonalizer(sp=12), Detrender(), AutoARIMA()])&quot;<br>)</pre><p><strong>Why is `craft()` so useful for agentic systems?</strong></p><p>As you can see, `craft()` is basically accepting a text string that looks like python code. This is a perfect fit for LLMs, since they are generating text and are trained also on a lot of code (obviously also python code).</p><p>I.e., when using `craft()`, no complex parsing logic or tool calls has to be defined. The LLM can directly generate the spec string that `craft()` expects, and then we can directly evaluate the resulting object.</p><p>Note, since `craft()` can perform imports it can also execute malicious code.</p><p><strong>How `craft()` powers the `AutoResearchForecaster`</strong></p><p>The `AutoResearchForecaster` is inspired by `autoresearch`. However, instead of iteratively improving the training of a neural network. Here, the goal is to iteratively find the best sktime forecaster for a given time series. To do this, the `AutoResearchForecaster` gets information about time series and a list of all available forecasters in sktime. To provide information about time series, I implemented three methods:</p><ol><li>simple statistics about the time series,</li><li>an LLM provided description of the time series. Therefore, the time series is plotted and the plot is provided to the LLM,</li><li>and, providing the plot of the time series directly to the LLM.</li></ol><p>Then, the LLM performs the following loop</p><ul><li>Based on the results of previous iterations, the LLM is prompted to propose `n_blueprints` new forecasters. Each blueprint has a name and a spec string that can be passed to `craft()`.</li></ul><pre>{<br>&quot;name&quot;: &quot;Deseason + AutoARIMA&quot;,<br>&quot;spec&quot;: &quot;TransformedTargetForecaster([Deseasonalizer(sp=12), AutoARIMA()])&quot;<br>},<br>{<br>&quot;name&quot;: &quot;BoxCox + ETS&quot;,<br>&quot;spec&quot;: &quot;TransformedTargetForecaster([BoxCoxTransformer(), ExponentialSmoothing(trend=&#39;add&#39;)])&quot;<br>}<br>]</pre><ul><li>`craft()` is used to instantiate each spec, and the resulting forecasters are evaluated on the data using the provided cross-validation splitter. The evaluation results are collected and if a there is a new best blueprint, it is saved.<br> Note, in case of an error, the LLM has the option to fix this forecaster.</li><li>All scores and potential errors are fed back to the LLM as part of the next prompt. Based on this information, the LLM can learn and propose an improved set of new forecasters.</li></ul><p>This loop is repeated for `n_iterations`, and at the end the best blueprint is refit on the full data and is used as the final model.</p><p><strong>Example with Ollama</strong></p><p>This example shows how you can use this forecaster to do forecasting. As an example, I have chosen the Airline Passengers dataset. It consists of an univariate time series with a increasing trend and a yearly seasonality. As training dataset, I used the first 132 months, and hold out the last 12 for testing.</p><p>I instantiate the `AutoResearchForecaster` with a cross-validation splitter to provide and evaluation scheme to the forecaster to evaluate the blueprints. Besides that, I also passed the parameters n_iterations, n_blueprints, api_params, and the model that should be used. Since the forecaster is using `litellm`, it is sufficient to just pass a string specifying model provider and model name. I have chosen to use the Gemma 4 model that I hos locally using ollama. On a Mac Book Air, the runtimes for 5 iterations with 3 blueprints is around 4 minutes.</p><pre>from sktime.split import ExpandingWindowSplitter<br>cv = ExpandingWindowSplitter(initial_window=12 * 5, step_length=12, fh=fh)<br># Create the LLM Blueprint Forecaster<br>forecaster = AutoResearchForecaster(<br>  cv = cv,<br>  # LLM to use for blueprint generation<br>  model=&quot;ollama/gemma4:e4b&quot;,<br>  # number of generate-evaluate-refine cycles<br>  n_iteratipons=5,<br>  # blueprints per iteration<br>  n_blueprints=3,<br>  # LLM sampling temperature<br>  api_params={&quot;temperature&quot;: 0.7},<br>)<br># Fit - this triggers the LLM blueprint search<br>forecaster.fit(y_train, fh=fh)<br>forecaster.predict()</pre><p>Below (Figure 1) you see the resulted prediction of the forecaster, as well as the ground truth and the training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DIk-PPFdf2_GV-cSH2GuJg.png" /><figcaption>Figure 1: Input time series, test data, and the forecast of the AutoResearchForecaster.</figcaption></figure><p>In this example the best blueprint was a pipeline consisting of a log transformation and Prophet.</p><h3><strong>Insights</strong></h3><p>Besides just applying it, I would also like to report some interesting insights. First, I would like to focus on how the three different ways to provide the time series to the LLM impacts the performance. Second, I would like to dig a bit into the trace of the LLM. I.e., seeing how often the forecaster gets improved if the model is iterating over multiple rounds etc. Third, how does the LLM knows which estimator exist.</p><p>Note, for this analysis, I decided not to use a model hosted via Ollama. Instead, I used Mistral-Small, Mistral-Medium, and Mistral-Large. As dataset, I still used the airline dataset since with longer time series the proposed estimators took far more time to be fitted.</p><p><strong>What Image Description work best?</strong></p><p>The first question was does it matter how I pass the information about the time series to the LLM. As mentioned above, the AutoResearchForecaster has the following three methods:</p><ul><li>Provide only simple statistics about the time series (basic)</li><li>Provide a plot of the time (image)</li><li>Provide a description of the plotted time series. This description is created by an VLM (described_plot)</li></ul><p>In the results below (Figure 2) you see, that the overall best result was achieved by Mistral-Small with the basic statistical description. However, the results are very fluctuating, and no clear pattern is visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*M2khMnBn4HdC8cOJAN1M8w.png" /><figcaption>Figure 2: The average Mean Squared Errors across the different folds.</figcaption></figure><p>To explain why there are no clear pattern, I performed the following small analysis. I analysed which estimators are picked how often by the LLM across five different runs (Figure 3). The estimator picked most often was the TransformedTargetForecaster, which is unsurprising, as it serves as the pipeline wrapper rather than a standalone model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pT9KRGI-kEUIrLJrfBbdTg.png" /><figcaption>Figure 3: Average occurrences of each estimator by the three different image description methods. The number of occurrences are average across five runs</figcaption></figure><p>When comparing how often the other estimators are selected when using different description methods for the time series, we see that there are minor fluctuations but no systematic differences across methods. This might indicate that the LLM is not really able to get a proper understanding of the time series.</p><p>Furthermore, we also observe that the LLM is only selecting a very small subset of the available transformers and estimators. The cause of this might be either a bias in the system prompt or in the training data, since methods like differentiation, ARIMA, or Prophet are widely discussed methods in the forecasting literature.</p><p>Wrapping up, a possible reason why the description method does not seem to impact the performance might be that the LLM is not really understanding time series. However, I think there are also further valid hypotheses, such as:</p><ul><li>The airline dataset is so simple that no differences in the description methods occur.</li><li>The LLM is not really reasoning about the provided the time series. Thus, the provided information is not really used. Instead, the LLM is more or less selecting forecasters in a random order.</li></ul><p>Thus to finally assess the reason why the description method does not seem to impact the performance, a more extensive analysis must be performed. This might also include some benchmarking tasks that assess the time series understanding capabilities of LLMs.</p><p><strong>Evaluation of best forecasters</strong></p><p>I also tracked how the best forecaster found by the LLM so far evolves through time. And what the best model per round is. See Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNlI0VjhotqWVhUFI0nfHw.png" /><figcaption>Figure 4: The best forecaster found so far (left), and the best forecaster found in each round (right).</figcaption></figure><p>Interestingly, for the used dataset, the best score drops very fast in the beginning but does not really change in later steps. When checking the best model per round, we see that the results are strongly fluctuating without any trends. This suggests that the LLM is not really learning from previous evaluation results in a meaningful way. It might be that the LLM picked rather arbitrary some forecaster than reasoned about the time series. If this is the case, this might explain the fluctuations of the best forecaster per round. Regarding the fast convergence, the lack of understanding time series might also be an explanation here. E.g., if the model is better understanding the time series, it might create better models.</p><p><strong>Discussion</strong></p><p>In this blogpost, I evaluated on one time series, thus, the insights are limited. However, I think we can still learn some things here:</p><ul><li>While it seems natural that the documentation is helpful for the LLM to select the estimator. The LLM can not handle the complete documentation at once. Thus, it might be useful to use other approaches like tool calling to provide the LLM information about certain estimators.</li><li>I had the feeling that the used LLM cannot really understand time series. Thus, using and test frontier models might lead to different results. Also activating reasoning can also improve the results here.</li><li>Finally, depending on the selected forecaster and the length of the time series. Fitting the forecaster can take quite long. Thus, introducing some time budget for the estimator could help to avoid long running forecasters.</li></ul><p>Wrapping up, I see potential in using LLMs for automated time series forecasting. However, there is a lot room for improvements at the side of the LLMs but also at the side of the harness. And we might see more in the future here.</p><p><strong>Conclusion</strong></p><p>`craft()` is a small function with outsized impact for agentic ML systems. It solves the problem of bridging LLM text output to library objects cleanly, without requiring custom parsers.</p><p>The `AutoResearchForecaster` demonstrates that a complete AutoML search loop — propose, evaluate, refine, select — can be built on this single primitive. As LLMs improve at structured and constrained output, the combination becomes even more capable.</p><p><strong>References:</strong></p><ul><li>The AutoResearchForecaster is currently under review in the sktime repo. <a href="https://github.com/sktime/sktime/pull/9855">https://github.com/sktime/sktime/pull/9855</a></li><li><a href="https://github.com/karpathy/autoresearch">https://github.com/karpathy/autoresearch</a></li></ul><p><strong>Further Readings:</strong></p><p>If you are interested in further reading regarding Agentic or LLM based forecasting, you might check out:</p><ul><li><a href="https://medium.com/@benedikt_heidrich/can-we-do-time-series-analysis-with-llm-powered-workflows-using-sktime-12b19cf39376">https://medium.com/@benedikt_heidrich/can-we-do-time-series-analysis-with-llm-powered-workflows-using-sktime-12b19cf39376</a></li><li><a href="https://github.com/TimeCopilot/timecopilot">https://github.com/TimeCopilot/timecopilot</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0355f5c720e8" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/llm-driven-autoforecasting-with-sktimes-craft-0355f5c720e8">LLM Driven AutoForecasting with sktime’s `craft()`</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can we do Time Series Analysis with LLM-powered Workflows using sktime?]]></title>
            <link>https://medium.com/@benedikt_heidrich/can-we-do-time-series-analysis-with-llm-powered-workflows-using-sktime-12b19cf39376?source=rss-ddaa84f9958b------2</link>
            <guid isPermaLink="false">https://medium.com/p/12b19cf39376</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm-agent]]></category>
            <dc:creator><![CDATA[Benedikt Heidrich]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 17:40:33 GMT</pubDate>
            <atom:updated>2025-08-26T17:40:33.244Z</atom:updated>
            <content:encoded><![CDATA[<p>Want to see how we can build LLM-powered time series analysis workflows? In this blog post, we implement such a workflow using sktime, LlamaIndex, and Mistral. The main focus is on figuring out how such systems can be built rather than their performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3p6xgTzol08HCfZoix-iTg.png" /><figcaption>The goal is to ask an LLM to do anomaly detection or forecasting for a provided time-series. The LLM should then select an appropriate estimator from sktime and do the prediction.</figcaption></figure><p>First, we provide some background on sktime and what workflows are. Second, we focus on the implementation and shows several results of the resulting workflow. Finally, we will discuss the findings and the limitations of this blog post.</p><p>Let’s explore this topic together.</p><h3>Background</h3><p>In this section, I’ll provide a brief overview of sktime, agents, and the tech stack I used. Please also check other publications for more comprehensive explanations on these topics.</p><h4>sktime</h4><p>sktime is an open-source Python machine learning toolbox for time series data. Built with a unified, scikit-learn inspired interface, it supports a wide range of estimators and tasks, such as forecasting, anomaly detection, and classification. Key features of sktime include composability and modularity allowing users to construct flexible ML pipelines through composition and reduction techniques.</p><p>sktime also provides a unified interface for all estimators for a specific task. For example, all forecasters share the same signatures for <em>fit, predict</em>, and similar functions. This consistency makes it easy for users to switch between different estimators.</p><p>This unified interface, which makes estimators easily interchangeable, also makes sktime an ideal choice for this blog post and experiment. Since the LLM should decide which estimator to use for a given task, the estimators must be easily swappable.</p><h4>Workflows</h4><p>In this blog post, a workflow refers to a system that manages the execution of an application using events. The application is divided into several steps. Each step has a dedicated task ranging from simple tasks (such as determining the user’s intent) to complex tasks (such as analyzing a provided time series). This modular approach makes it easy to develop workflows that are driven by GenAI.</p><p>In contrast to pure agent-based approaches, the main control flow in this blog post is largely determined by the defintion of the individual steps themselves. However, the LLM still plays a key role by deciding the parameters that are passed on to subsequent steps.</p><p>To implement such workflows, I used the library LlamaIndex. Llama Index is an open source orchestration framework developed for building applications with LLMs. The reason why I used LlamaIndex was my mainly curiosity to try a new framework.</p><h3>Time Series Analysis Workflow</h3><p>To implement the workflow, we follow a simple data science analysis process that includes the following steps::</p><ol><li>Identify the tasks that should be solved!</li><li>Visualize and analyze the data. Try to identify important aspects of the time series and get a feel for it.</li><li>Get an overview of the available models. Make a preselection of the estimators based on the time series analysis from step 2 and your experience with the estimators.</li><li>Read the estimator documentation and check if they are really suitable for the time series and your task.</li><li>Read the documentation again and select the hyperparameters for your estimator.</li><li>Fit the model and analyze the results. In case of issues, repeat some of the previous steps.</li></ol><h4>Implementation</h4><p>Before defining, the different steps, I implemented the events that are triggering the steps of the workflow in LlamaIndex:</p><ul><li>TSAEvent</li><li>SelectEstimatorEvent</li><li>ParameterSelectionEvent</li><li>EvaluateEvent</li></ul><p>These events are pydantic objects containing arguments relevant to the workflow step they trigger. In the following, there is the implementation of the TSAEvent as an example:</p><pre>class TSAEvent(Event):<br>    query: str<br>    estimator_type: Literal[&quot;forecaster&quot;, &quot;detector&quot;]<br>    filter_tags: dict = {}  </pre><p>The workflow itself is implemented as a Python class, where each step is a method annotated with <em>@step</em>. Each step receives a context object and the event that triggers it as input. The return value is always an event that triggers the next step.</p><p>The <strong>determine_task </strong>step identifies whether the user wants to do forecasting or anomaly detection. To do this, it sends a prompt together with the user query to an LLM (for the exact prompt, check out my Github <a href="https://github.com/benHeid/TSA-Workflow-with-LLMs">repo</a>). Based on the model’s response, it triggers the <em>TSAEvent</em> with different <em>estimator_types</em> (forecaster or detector).</p><pre>@step<br>async def determine_task(<br>    self, ctx: Context, ev: StartEvent<br>) -&gt; TSAEvent:<br>    # initialize<br>    content = DETERMINE_CONTEXT_PROMPT.format(query=ev.query)<br>        <br>    response = await self.get_response(ctx, content, response_format={<br>        &quot;type&quot;: &quot;json_object&quot;,<br>    })<br><br>    response = json.loads(response)[&quot;task&quot;]<br>    <br>    if response == &quot;forecast&quot;:<br>        # try again<br>        return TSAEvent(query=ev.query, estimator_type=&quot;forecaster&quot;)<br>    elif response == &quot;anomaly&quot;:<br>        return TSAEvent(query=ev.query, estimator_type=&quot;detector&quot;,<br>                        filter_tags={&quot;task&quot;: &quot;anomaly_detection&quot;})<br>    else:<br>        StopEvent(<br>            result=&quot;The task could not be determined.&quot;)</pre><p>This event triggers the <strong>analyse_time_series</strong> step. In this step, the model is asked to analyse the time series and provide an analysis. To do this, the time series is first fetched from the context, visualized, and plotted as an image. This image is then base64-encoded before being sent, along with the analysis prompt, to Pixtral. The report returned by Pixtral is then passed to the resulting SelectEstimatorEvent.</p><pre>@step<br>async def analyse_time_series(<br>    self, ctx: Context, ev: TSAEvent<br>) -&gt; SelectEstimatorEvent:<br>    time.sleep(1.1)  # Simulate a delay for the function<br>    y = await ctx.get(&quot;time_series&quot;)<br>    plot_series(y, labels=[&quot;time series&quot;])<br>    plt.savefig(f&quot;time_series_analysis.png&quot;)<br><br>    image_path = &quot;time_series_analysis.png&quot;<br>    with open(image_path, &quot;rb&quot;) as image_file:<br>        image_data = image_file.read()<br>    image_base64 = base64.b64encode(image_data).decode(&#39;utf-8&#39;)<br>    content = [<br>                {<br>                    &quot;type&quot;: &quot;text&quot;,<br>                    &quot;text&quot;: ANALYSIS_PROMPT<br>                    &quot;type&quot;: &quot;image_url&quot;,<br>                    &quot;image_url&quot;: f&quot;data:image/png;base64,{image_base64}&quot;<br>                }<br>            ]<br><br>    response = await self.get_response(ctx, content, model=&quot;pixtral-12b-2409&quot;)<br>    return SelectEstimatorEvent(query=ev.query,<br>                                estimator_type=ev.estimator_type, <br>                                tsa_properties=response)</pre><p>The SelectEstimatorEvent triggers the <strong>get_estimator</strong> step. The goal of this step is to choose an estimator to be used in the subsequent steps. First all available sktime estimators for a given task need to be determined. To do this, we use sktime’s convenient <em>all_estimators </em>method, which retrieves all estimators for a specific task and allows additional filtering based on sktime’s tag system. Once we have the list of estimators, we ask the LLM to make an initial selection.</p><p>After this initial selection, we further filter the list using <em>filter_estimators.</em> This method provides the docstring of each selected estimator to the LLM and ask if this estimator is truly suitable for the given time series (for more details check out the Github repo). From the remaining estimators, we select the first one and pass it to the SelectParameterEvent that is returned.</p><pre>@step<br>async def get_estimator(<br>        self, ctx:Context, ev: SelectEstimatorEvent<br>) -&gt; SelectParametersEvent:<br>    time.sleep(1.1)  # Simulate a delay for the function<br><br>    # 1. get all sktime estimtators.<br>    estimators =  all_estimators(estimator_types=[ev.estimator_type],<br>                                 filter_tags=ev.filter_tags)<br><br>    # 2. Let the LLM select the suitable estimators based on the time series<br>    #  properties and user query.<br>    content = GET_ESTIMATORS_PROMPT.format(<br>        query=ev.query,<br>        tsa_properties=ev.tsa_properties,<br>        estimator_type=ev.estimator_type,<br>        estimators=&quot;\n&quot;.join([f&quot;{est[0]}: {est[1].__doc__}&quot; for est in estimators])<br>    )<br><br>    response = await self.get_response(ctx, content, response_format={<br>        &quot;type&quot;: &quot;json_object&quot;,<br>    })<br>    <br>    response = json.loads(response)[&quot;estimators&quot;]<br><br>    # 3. refine selection, by letting the LLM read the docstrings<br>    selected_estimators = await self.filter_estimators(ev, ctx, response, json)<br>    <br>    # 4. Go to select parameter step.<br>    if not selected_estimators:<br>        return StopEvent(<br>            result=&quot;No suitable estimator was found.&quot;<br>        )<br>    return SelectParametersEvent(<br>        query=ev.query, <br>        estimator_type=ev.estimator_type,<br>        estimator=selected_estimators[0],  <br>        tsa_properties=ev.tsa_properties<br>    ) </pre><p>The SelectParametersEvent triggers the <strong>get_parameters</strong> step, which determines the parameters that should be used for the given estimator. To do this, it provides the estimator’s docstring, the report from the time series analysis step, and the user query to the LLM. The selected parameters are then checked to ensure they actually exist. Once verified, the parameters are passed on to the EvaluateEvent.</p><pre>@step<br>async def get_parameters(<br>    self, ctx:Context, ev: SelectParametersEvent<br>) -&gt; EvaluateEvent:<br>    time.sleep(1.1)  # Simulate a delay for the function<br>    est, docstring = self.get_docstring(ev)<br>    content = GET_PARAMETERS_PROMPT.format(<br>        estimator=ev.estimator,<br>        docstring=docstring,<br>        tsa_properties=ev.tsa_properties,<br>        query=ev.query,<br>    )<br>    response = await self.get_response(ctx, content, response_format={<br>        &quot;type&quot;: &quot;json_object&quot;,<br>    })<br>    parameters = json.loads(response)<br><br>    # Keep only the parameter that are actually parameters of the estimator.<br>    parameters = self.check_and_filter_parameters(ev, est, parameters)<br><br>    return EvaluateEvent(<br>        query=ev.query,<br>        estimator_type= ev.estimator_type,<br>        estimator=ev.estimator,<br>        parameters=json.loads(parameters),<br>        tsa_properties=ev.tsa_properties,<br>    )</pre><p>Finally, the <strong>evaluate</strong> step is triggered by the EvaluateEvent. This step attempts to instantiate and fit the models as well as call predict. If this is successful, the result is visualized and the StopEvent triggered. Otherwise, the the SelectEstimatorEvent is triggered again. Note that the context keeps track of estimators that should not be tried again.</p><pre>@step<br>async def evaluate_result(<br>    self, ctx:Context, ev: EvaluateEvent<br>) -&gt; StopEvent | SelectEstimatorEvent:<br>    logger.info(&quot;Evaluate the results.&quot;)<br>    y = await ctx.get(&quot;time_series&quot;)<br>    y = deepcopy(y)<br>    <br>    try:<br>        # 1. Instantiate the model with the parameters, store <br>        # it in the context.<br>        estimator, parameters, model = self.instantiate_estimator(ev)<br>        # 2. Fit the model<br>        await self.fit_estimator(ctx, ev, model, y)<br>        # 3. Predict<br>        y_pred = self.call_predict(ev, model, y)<br><br>    except Exception as e:<br>        logging.error(&quot;Error when executing the model &quot; + <br>                      f&quot;{ev.estimator} with parameters {ev.parameters}&quot;)<br>        await ctx.set(&quot;forbidden_estimators&quot;, <br>                      await ctx.get(&quot;forbidden_estimators&quot;, []) + <br>                                    [ev.estimator])<br>        return SelectEstimatorEvent(<br>            query=ev.query,<br>            filter_tags={&quot;task&quot;: &quot;anomaly_detection&quot;} if <br>                         ev.estimator_type == &quot;detector&quot; else {},<br>            tsa_properties=ev.tsa_properties,<br>            estimator_type=ev.estimator_type,<br>        )<br>    <br>    await self.visualise_results(ctx, ev, estimator, y, y_pred)</pre><p>The above methods are all part of the <em>DataAnalysisWorkflow</em> class. With LlamaIndex, we can visualise the workflow implemented by this class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*064fKOy9c9U5FCCafc7DjA.png" /><figcaption>The implemented workflow for doing time series analysis with Mistral and sktime.</figcaption></figure><h3>Forecasting</h3><p>In the first example, we ask the model to forecast the load_airline time series. Therefore, we instantiate the <em>DataAnalysisWorkflow</em>, we built earlier, set the LLM and the time series to be analyzed in the context, and then run the workflow with the following prompt: “”Provide a forecast of the time series”.</p><pre>from llama_index.core.workflow import Context<br>from mistralai import Mistral<br>from sktime.datasets import load_airline<br><br>workflow = DataAnalysisWorkflow(timeout=15,)<br>ctx = Context(workflow)<br><br>await ctx.set(&quot;llm&quot;,  Mistral(api_key=os.environ.get(&quot;MISTRAL_API_KEY&quot;)))<br>await ctx.set(&quot;time_series&quot;, load_airline()) <br><br>await workflow.run(ctx=ctx, query=&quot;Provide a forecast of  the time series.&quot;)</pre><p>Below, you can see the actual time series alongside the predicted one. For this toy example, the workflow was able to produce a forecast that looks quite reasonable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wGhvcGjMxb1seLBmpJwriw.png" /></figure><h4><strong>Insights in the Load Airline Forecasting Task — Analyzing LLM Responses</strong></h4><p>To better understand what happened, let’s check the answers the LLM provided at the different steps in this particular run.</p><p>In the time series analysis step, the model focused on trend, seasonality, and anomalies. It made the following observations:</p><ul><li>Regarding the trend, the LLM correctly noted, <em>“The time series shows a general upward trend from 1948 to 1962.”</em></li><li>Similarly, for seasonality, it correctly observed, <em>“The peaks and troughs suggest that there might be seasonal factors influencing the number of airline passengers”</em> but it failed to identify the frequency as yearly.</li></ul><p>While the LLM successfully detected trend and seasonality, it made an incorrect statement regarding anomalies:</p><blockquote>- There are several points where the number of passengers significantly deviates from the general trend. For example, around 1956 and 1960, there are noticeable peaks that are higher than the surrounding data points.</blockquote><blockquote>- These anomalies could be due to specific events or changes in the industry that caused a temporary increase in passenger numbers.</blockquote><p>In the get_estimator step, the model selected ARIMA, SARIMAX, and AutoARIMA, and justified the selection mainly with the seasonality and tend of the time series. For example, here are the reasons given for selecting the ARIMA model:</p><blockquote>The ARIMA model is suitable for forecasting time series data with trends and seasonality, which are present in the airline passenger data. The model can capture the overall upward trend and the seasonal patterns observed in the data. Additionally, the SARIMA variant of ARIMA can handle the seasonal component explicitly, making it a good choice for this type of data. The presence of anomalies can be addressed by preprocessing the data or using the model’s ability to handle such variations. Therefore, the ARIMA model is appropriate for providing a forecast of the time series.</blockquote><p>Since the workflow takes the first estimator, only ARIMA parameters were generated. The selected parameters were:</p><p><em>{ “order”: [1, 1, 0], “seasonal_order”: [0, 1, 0, 12], “suppress_warnings”: true}</em></p><p>This selection was justified:</p><blockquote>“The time series exhibits a clear upward trend, strong seasonality (likely annual, given the data frequency), and some anomalies. Based on the observations, a SARIMA model with differencing to handle the trend and seasonal components is appropriate. The parameters are chosen to capture the trend (d=1), the seasonal pattern (seasonal_order=(0,1,0,12)), and a simple autoregressive component (order=(1,1,0)) to account for the overall trend and seasonality.”</blockquote><p>Note, the LLM’s justification mentioned d=1, but this parameter was missing in the output.</p><h4>Insights in the Load Airline Forecasting Task — Consistency</h4><p>When using LLMs for such tasks, it is useful to figure out how consistent the results are over multiple runs. Thus, I made a very simple study where I analysed which forecasters in <em>get_estimators</em> are selected by the LLM in multiple runs and what the average position was.</p><p>Out of 15 runs, 14 were successful. The following plot shows how often each estimator was selected and at which position. Note that AutoARIMA was selected 15 times in 14 runs, meaning that in at least one run it was selected multiple times. Overall, the LLM tended to prefer AutoARIMA and AutoETS, while other forecasters, such as AutoEnsembleForecaster, Prophet, and others, were rarely chosen. Regarding the selection position, AutoARIMA was most often chosen first. However, this might also be influenced by the alphabetical order of the estimators provided to the LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*Vu4T7wm0odJQEi0RX6BVqA.png" /></figure><h4><strong>Energy Time Series Forecasting Task</strong></h4><p>I applied the same approach, without any additional code changes, to a longer time series covering eight weeks of electricity demand in New South Wales. The following plot clearly shows that the model struggles to predict the time series accurately. The LLM selected an AutoARIMA model, but although it detected seasonality, it chose the wrong value (12 instead of 48).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MjiVQePnZAxwm7xvUYTfEg.png" /></figure><h3>Anomaly Detection</h3><p>In the following plot, I tested anomaly detection on a very simple time series with an obvious anomaly. For this time series, the LLM was able to select an anomaly detection method that correctly identified the anomaly, as shown in the plot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*w2-ImJvfazvdnL095l9OjQ.png" /></figure><p>Future work should apply this workflow to real-world time series with anomalies to better understand how well agentic workflows perform for anomaly detection.</p><h3>Discussion</h3><p>After analyzing the results of the workflow, let’s discuss the results, the limitations of the current implementation, and a few disclaimers for this blog post.</p><h4>Results</h4><p>If you pull the workflow from GitHub and run it yourself, you will likely see different results. While the final prediction for the <em>load_airline</em> series was mostly consistent, the forecast for the energy time series varied more frequently. Tracking down the exact cause is not easy, as each call to the LLM can change the outcome. In particular, the data analysis step likely has a large impact on the final results.</p><ul><li>For forecasting, the LLM primarily selects models like ARIMA or other statistical models. An open question is whether these models appear more frequently in the training data, which could create a bias toward them — especially since, in the get_estimator step, the LLM only receives the names of the available forecasters</li><li>Running the workflow multiple times leads to different results not only in the final prediction but also intermediate steps differs. For example, the model may sometimes detect seasonality in the time series and sometimes not. A potential reason of this behaviour might be that the used model is too small.</li></ul><h4>Limitations of the Implementation</h4><p>As mentioned in the introduction, this blog post is more about exploring what could work with current models in LLM-based workflows for time series analysis. Thus, the implementation presented here has several issues that could be addressed in the future. Below, I highlight four of these key issues.</p><ul><li>The time series analysis step is very limited. The LLM only receives a single plot of the entire time series along with one prompt. In contrast, data scientists typically work is more complex. E.g, they are using multiple visualizations at different resolutions. Therefore, there is significant room to improve this step and enhance the overall workflow.</li><li>Currently, the workflow is using only the time series itself and ignores all exogenous features. However, in forecasting exogenous information often improving the results of forecasts.</li><li>This workflow is also limited in the sktime models it can use. While the LLM might select any model, the <em>select_parameter</em> step currently works only with a limited set of argument types, such as lists, dictionaries, integers, floats, strings, and booleans. If an object needs to be passed as an argument, it does not work . As a result, currently, the workflow cannot use composite models such asConformalIntervals.</li></ul><p>Since I have made the code available on GitHub, you are invited to fork it and implement fixes for the limitations mentioned above</p><h4>Why I haven’t used an Agent with Tools?</h4><p>I think this is a valid question. I also experimented with pure agent-based approaches by providing the LLM with the methods described above as tools. My experience was that this approach was less robust and more often led to incomplete results. For example, the LLM sometimes stopped after selecting a forecaster instead of actually executing it. With more prompt engineering, these issues could probably be addressed.</p><p>If there’s interest, I might write a separate blog post on a purely agent-based approach.</p><h4>Disclaimer</h4><p>As mentioned, the implemented workflow is just a prototype and can be improved in many ways. Here are a few disclaimers:</p><ul><li>This blog post only demonstrates how such workflows could look. It does not compare them to AutoML approaches, as such evaluations were beyond the scope of this post.</li><li>I have not performed any sophisticated prompt engineering or optimization steps, which could potentially improve performance.</li><li>I mainly used Mistral Small, Pixtral, and Mistral Medium in the various steps. Using larger models could lead to better results, particularly in the analysis step, where larger Vision Language Models might provide more accurate insights.</li></ul><h3>Conclusion</h3><p>As seen before, this workflow works for toy examples in forecasting and anomaly detection. However, it struggles when the time series become longer or more complex. We also observed several flaws in the intermediate steps — for example, incorrectly identifying outliers in the load_airline data. Such mistakes can negatively impact the output of subsequent steps. Therefore, I believe that agentic workflows for more complex time series analysis tasks will require additional engineering effort. I am excited to see how these workflows might be improved in the future.</p><h4>Potential Next Steps</h4><p>If you are interested in developing this further, I see multiple very promising ways:</p><ul><li>Let the LLM produce the string of the model specification and use the craft feature of sktime. This craft feature might even enable the workflow to create composite estimators and pipelines. See the following link: <a href="https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.registry.craft.html">https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.registry.craft.html</a></li><li>Improve the time series analysis step. E.g., make it more iteratively. E.g., let the LLM zoom in or out in specific aspects of the time series. Maybe, you can use also an agent for this task.</li><li>Implement a feedback loop. E.g., let the LLM also evaluate the final prediction and reiterate over the selected estimators.</li></ul><h3>References and Links</h3><ul><li>sktime: <a href="https://www.sktime.net/en/stable/">https://www.sktime.net/en/stable/</a></li><li>LlamaIndex: <a href="https://docs.llamaindex.ai/en/stable/">https://docs.llamaindex.ai/en/stable/</a></li><li>Australian Electricity Dataset: <a href="https://zenodo.org/records/4659727">https://zenodo.org/records/4659727</a></li><li>Github Code: <a href="https://github.com/benHeid/TSA-Workflow-with-LLMs">https://github.com/benHeid/TSA-Workflow-with-LLMs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12b19cf39376" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting using Foundation Models and sktime]]></title>
            <link>https://medium.com/@benedikt_heidrich/forecasting-using-foundation-models-and-sktime-4d5a09909742?source=rss-ddaa84f9958b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d5a09909742</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[foundation-models]]></category>
            <category><![CDATA[forecasting]]></category>
            <dc:creator><![CDATA[Benedikt Heidrich]]></dc:creator>
            <pubDate>Fri, 01 Nov 2024 18:03:26 GMT</pubDate>
            <atom:updated>2024-11-03T05:57:54.130Z</atom:updated>
            <content:encoded><![CDATA[<p>Time series foundation models promise to change time series forecasting since they can do zero-shot forecasting and overcome the cold-start problem. In this blog post, I show how to use such models with sktime.</p><h4>Time Series Foundation Models</h4><p>To understand the promise of time series foundation models, let us recap how classical time series forecasting works. In classical time series forecasting, a separate model is trained for each considered time series and retrained if new data arrives. In contrast, time series foundation are pretrained on many different time series. This training on many different time series allows them to forecast arbitrary time series without fitting the model by just providing the historical time series as a context. This results in having one model for all time series, and a user can directly use it without training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jzVe9xyOeFOncyEC8htQ-Q.png" /></figure><h3>Foundation Models in sktime</h3><p>sktime is an open-source toolbox for time series analysis. It provides unified algorithms for various tasks, such as time series classification, clustering, anomaly detection, and forecasting. For forecasting, sktime provides different algorithms like theta forecaster, ARIMA models, and time series foundation models. sktime is (to the best of the author’s knowledge) the only time series analysis library providing unified access to many forecasting foundation models (state 17.10.2024):</p><ul><li>MOIRAI</li><li>TinyTimesMixer</li><li>TimesFM</li><li>Chronos</li></ul><p>The mentioned unified interfaces in sktime enable us to start forecasting with foundation models very fast since they follow the same API as the other forecasting algorithms. The following code shows how to forecast using MOIRAI:</p><pre># Do imports<br>from sktime.forecasting.moirai_forecaster import MOIRAIForecaster<br>from sktime.datasets import load_forecastingdata<br>from sktime.utils.plotting import plot_series<br><br>import pandas as pd<br><br># Load data<br>data = load_forecastingdata(&quot;australian_electricity_demand_dataset&quot;)[0]<br>data = data.set_index(&quot;series_name&quot;)<br>series = data.loc[&quot;T1&quot;][&quot;series_value&quot;]<br>y = pd.DataFrame(series, index=pd.date_range(&quot;2006-01-01&quot;, periods=len(series), freq=&quot;30min&quot;))<br>y = y.resample(&quot;H&quot;).mean()<br>series2 = data.loc[&quot;T2&quot;][&quot;series_value&quot;]<br>y2 = pd.DataFrame(series2, index=pd.date_range(&quot;2006-01-01&quot;, periods=len(series), freq=&quot;30min&quot;))<br>y2 = y2.resample(&quot;H&quot;).mean()<br><br># Initialise the forecasters<br>morai_forecaster = MOIRAIForecaster(<br>    checkpoint_path=f&quot;Salesforce/moirai-1.0-R-small&quot;<br>)<br><br># Fit the model (in most cases no fitting happens, it is about API consistency)<br>morai_forecaster.fit(y=y, fh=range(1, 96))<br><br># Call predict<br>pred_moirai = morai_forecaster.predict(y=y2[:-96])<br></pre><p>Due to the unified interface of sktime, you can also easily exchange the forecaster without changing any fit or predict calls.</p><pre># Do imports<br>from sktime.forecasting.ttm import TinyTimeMixerForecaster<br>from sktime.forecasting.chronos import ChronosForecaster<br><br># Initialise models<br>chronos = ChronosForecaster(&quot;amazon/chronos-t5-tiny&quot;)<br>ttm = TinyTimeMixerForecaster()<br><br># Fit<br>chronos.fit(y=y, fh=range(1, 96))<br>ttm.fit(y=y, fh=range(1, 96))<br><br># Predict<br>pred_chronos = chronos.predict(y=y2[:-96])<br>pred_ttm = ttm.predict(y=y2[:-96])</pre><blockquote>Note that time series foundation models are global forecasters, which means the forecaster can be fitted on time series that differ from the target time series. Thus, the predict method accepts time series (y) as input forming the target time series’s context (historical values). Furthermore, fit needs to be called on the foundation models to be API consistent. However, in most cases, calling fit does not trigger a training.</blockquote><p>As shown above, you can generate forecasts from three different time series foundation models with just a few lines of code. The figure below shows these three forecasts and the actual time series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LLAiLicO7O16K14qtEUufQ.png" /></figure><h3>Composing Foundation Models</h3><p>In addition to interfacing and implementing various time series algorithms, sktime also provides a wide range of composed forecasters. Composed forecasters mean a new forecaster is created from different algorithms. For example, forecasting pipelines are forecasters composed of times series transformations (e.g. Differentiation or De-seasonalization) and a forecaster. Other composed forecasters are ensemble models or probabilistic wrappers such as BaggingForecasteror ConformalIntervals.</p><h4>Pipelines and foundation models</h4><p>Pipelines enable the easy creation of powerful forecasters, including feature extraction or time series pre- and postprocessing. Below, we create a pipeline that adds a differencer step with a seasonality parameter of 24 to the forecaster.</p><pre># Import<br>from sktime.transformations.series.difference import Differencer<br><br># Build the pipeline<br>moirai_pipe = Differencer(24) *  MOIRAIForecaster(<br>    checkpoint_path=f&quot;Salesforce/moirai-1.0-R-small&quot;<br>)<br><br># Fit<br>moirai_pipe.fit(y2[:-96], fh=range(1, 96))<br><br># Predict<br>pred_moirai_diff = moirai_pipe.predict()<br></pre><blockquote>In contrast to the previous example, predict has no parameter y. I.e., the target time series is the time series passed in fit. The reason is that the transformations in the pipeline need to be on the actual time series in sktime.</blockquote><p>In the following visualisation, we plot the Moirai forecaster, the Moirai with the differentiation, and the actual time series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sr3Pun02CUT0F9xj-s_xIQ.png" /></figure><h4>Probabilistic Wrappers and foundation models</h4><p>As mentioned, probabilistic wrappers also exist as compositions in sktime. Probabilistic forecasters add support for probabilistic forecasts to point forecasters. sktime provides multiple probabilistic wrappers, such as the BaggingForecaster and the ConformalIntervals.</p><p>The BaggingForecaster uses a bootstrapping algorithm and a forecaster as arguments. The bootstrapping algorithm bootstraps the original time series when the fit and predict methods are called. The forecaster is applied to each of the bootstrapped time series, resulting in a set of forecasts. Out of this set of forecasts, the probabilistic forecasts are derived.</p><pre># Import<br>from sktime.forecasting.compose import BaggingForecaster<br>from sktime.transformations.bootstrap import STLBootstrapTransformer<br><br>morai_forecaster = MOIRAIForecaster(<br>    checkpoint_path=f&quot;Salesforce/moirai-1.0-R-small&quot;<br>)<br><br># Create the BaggingForecaster using STLBootstrapTransformer and moirai_forecaster<br>moirai_bagging = BaggingForecaster(STLBootstrapTransformer(), morai_forecaster)<br><br># Fit this BaggingForecaster<br>moirai_bagging.fit(y2[:-96], fh=range(1, 96))<br><br># Create a point forecast with predict and prediction intervals with predict_interval<br>pred_moirai_bagging = moirai_bagging.predict()<br>pred_moirai_bagging_int = moirai_bagging.predict_interval()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2Tc_MsIaZgzLL0ms9Qj-Q.png" /></figure><p>An alternative probabilistic wrapper is ConformalIntervals , which guarantee calibrated forecasts. ConformalIntervals fit a model on a training data and use a separate calibration dataset to calculate the residuals. During prediction these residuals are used to determine an prediction interval.</p><p>Similar to the BootstrapForecaster , we can again create probabilistic forecasts.</p><pre># Import Conformal Intervals<br>from sktime.forecasting.conformal import ConformalIntervals<br><br># Create a ConformalIntervals forecaster with moirai internally. <br># Note to save compute time, we set sample_frac to 0.2<br>ci = ConformalIntervals(morai_forecaster, sample_frac=0.2)<br><br># Fit on the most recent observations.<br># Note fitting on the complete time series is not possible due to high memory consumption of ConformalIntervals.<br>ci.fit(y2[-512:-96], fh=range(1, 96))<br><br># Predict<br>moirai_pred_int = ci.predict_interval()<br>moirai_pred = ci.predict()<br><br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AoiYyEBXGTh5dZsqBFUXAw.png" /></figure><h3>Outlook</h3><p>While this blogpost shows that time series foundation models are easily usable in sktime to produce forecasts. It is unclear how well those models perform compared to more classical approaches. Thus, we aim to perform benchmarks in the future using the sktime benchmarking module.</p><h3>References</h3><ul><li>sktime: <a href="https://github.com/sktime/sktime">https://github.com/sktime/sktime</a></li><li>LagLlama: Rasul, Kashif, et al. “Lag-llama: Towards foundation models for time series forecasting.” <em>arXiv preprint arXiv:2310.08278</em></li><li>TimesFM: Das, Abhimanyu, et al. “A decoder-only foundation model for time-series forecasting.” <em>arXiv preprint arXiv:2310.10688</em></li><li>MOIRAI: Woo, Gerald, et al. “Unified training of universal time series forecasting transformers.” <em>arXiv preprint arXiv:2402.02592</em></li><li>Chronos: Ansari, Abdul Fatir, et al. “Chronos: Learning the language of time series.” <em>arXiv preprint arXiv:2403.07815</em></li><li>TTM: Ekambaram, Vijay, et al. “TTMs: Fast Multi-level Tiny Time Mixers for Improved Zero-shot and Few-shot Forecasting of Multivariate Time Series. “arXiv preprint arXiv:2401.03955</li><li>MomentFM: Goswami, Mononito, et al. “Moment: A family of open time-series foundation models.” <em>arXiv preprint arXiv:2402.03885</em></li><li>Bagging Forecaster: Bergmeir, C., et al. (2016). “Bagging exponential smoothing methods using STL decomposition and Box-Cox transformation.” International Journal of Forecasting, 32(2), 303–312</li><li>Conformal Intervals: Kamile Stankeviciute et al. “Conformal Time Series Forecasting.” NeurIPS 2021.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d5a09909742" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>