<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Rowe on Medium]]></title>
        <description><![CDATA[Stories by Michael Rowe on Medium]]></description>
        <link>https://medium.com/@mike.roweprediger?source=rss-b8348664ec71------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5GDlv_FOwtCpoApPM-FtKg.png</url>
            <title>Stories by Michael Rowe on Medium</title>
            <link>https://medium.com/@mike.roweprediger?source=rss-b8348664ec71------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:13:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mike.roweprediger/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fast, Online Forecasting Packages]]></title>
            <link>https://medium.com/@mike.roweprediger/fast-online-forecasting-packages-850343737c04?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/850343737c04</guid>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Wed, 01 Nov 2023 20:32:55 GMT</pubDate>
            <atom:updated>2023-11-01T20:32:55.052Z</atom:updated>
            <content:encoded><![CDATA[<p>Here is how to use incremental, online time-series prediction algorithms or glance at their relative speed in the Elo ratings. It is based largely on materials from the Microprediction site with permission.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hZta7f_9jDnrqID1ln95Lg.png" /><figcaption>Popular batch-oriented (offline) time-series forecasting Packages and speedy alternatives. Acknowledgment to 30000007961 at <a href="https://es.lovepik.com/image-401785361/robot-running.html">https://es.lovepik.com/image-401785361/robot-running.html</a></figcaption></figure><p>I help contribute to <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/univariate-k_013.html">Elo ratings</a> which rank a hundred different approaches for accuracy on an ongoing basis. They are drawn from packages such as <a href="https://pypi.org/project/statsmodels/">statsmodels</a>, <a href="https://pypi.org/project/pmdarima/">pmdarima</a>, <a href="https://pypi.org/project/neuralprophet/">neuralprophet</a>, <a href="https://pypi.org/project/pydlm/">pydlm</a>, <a href="https://pypi.org/project/tbats/">tbats</a> and <a href="https://pypi.org/project/sktime/#description">sktime</a>, and they are exposed as simple forecasting functions <a href="https://github.com/microprediction/timemachines/tree/main/timemachines/skaters">here</a>.</p><p>The <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/univariate-k_013.html">leaderboards</a> now report computation times for the models, and there is quite a stark contrast between the quick and the slow.</p><h3>Large speed discrepancies</h3><p>For example, TBATS is a great library and has some solid accuracy results, but it may simply be impractical if you are processing data in real-time. The table below shows that its CPU consumption is 962.8 units (row 3), versus 0.6 for an incremental alternative (row 1). At least for now, that alternative is at least as accurate, though it might be lucky. Here’s a snapshot, but check the <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/overall.html">live table</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jUboee9u_jOetF4xwdFihg.png" /></figure><h3>A Recommender</h3><p>The timemachines package has a <a href="https://github.com/microprediction/timeseries-elo-ratings/blob/main/time_series_recommendations.ipynb">colab notebook</a> that recommends time-series models based on how many steps ahead you wish to forecast. Or you can use the <strong>timemachines</strong> package directly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-h38pWhgKEugLAECuG7Nw.png" /></figure><p>In keeping with our theme, you can apply a <strong>max_seconds</strong> parameter, to only choose really fast models only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*uHjVNNJqOXz5ARcahgfNXw.png" /><figcaption>Example of listing the top-rated, but speedy, time-series models</figcaption></figure><p>The time limit applies to fifty predictions. So this means the model should produce about 500 separate forecasts per second <em>including fitting, estimation, and everything else. </em>Sorry about the bizarre units. I’ll fix that sooner or later.</p><h3>The TimeMachines Quickstart</h3><p>The <a href="https://github.com/microprediction/timeseries-elo-ratings/blob/main/time_series_recommendations.ipynb">colab notebook</a> also illustrates how you can use these speedy models.</p><p>You might care to peruse the <a href="https://github.com/microprediction/timemachines/tree/main/examples/basic_usage">basic usage</a> of the timemachines package. For example, to use <strong>thinking_slow_and_fast </strong>one first locates it in the <a href="https://github.com/microprediction/timemachines/tree/main/timemachines/skaters">skaters directory</a> (the hardest step until I populate a few __init__.py’s), and then we know how to import it. Like all skaters, you feed it one data point at a time and it spits out a forecast vector of length <strong>k</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNzkUPPQbmhlyXMg4iKw3A.png" /></figure><p>Most of the home-spun methods are in the <a href="https://github.com/microprediction/timemachines/tree/main/timemachines/skaters/simple">skaters/simple</a> modules but if you really get stuck, you can instantiate a skater from its name. Come to think of it, the module <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skaters/allskaters.py">allskaters</a> has a utility function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*B8CXfc4EEaxrhcZRxzKSlQ.png" /></figure><p>If you are looking to build ensembles of top-performing models, knock yourself out. Be aware that there already are some (see <a href="https://github.com/microprediction/timemachines/tree/main/timemachines/skaters/elo">skaters/elo</a>) and feel free to use that template.</p><h3>Why Incremental?</h3><p>The alternative involves re-fitting a model every time a data point arrives.</p><p>Uber recently announced <a href="https://github.com/uber/orbit">Orbit</a> and Linked-In touted <a href="https://engineering.linkedin.com/blog/2021/greykite--a-flexible--intuitive--and-fast-forecasting-library">Greykite</a> as a response to Facebook Prophet (<a href="https://www.microprediction.com/blog/prophet">review</a> here), which itself has had a facelift. But none of these will be breaking speed records, that’s for sure.</p><p>Measured on the time-scale of something like <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skaters/simple/thinking.py">thinking_slow_and_fast</a>, batch-style methods usually are 1,000 to 100,000 times slower. In fairness, the latter is merely shrinking of moving averages, but that’s all you need sometimes for superior performance.</p><p>Nixtla folks are trying hard to optimize batch-style methods, and we thanks them for that. StatsForecast speeds up <a href="https://www.sktime.org/en/latest/roadmap.html#project-aims">sktime</a> which is one of the popular frameworks for time-series. The statsforecast <a href="https://www.sktime.org/en/latest/api_reference.html#sktime-forecasting-time-series-forecasting">API</a> is batch-style, almost exclusively.</p><p>Accuracy aside, we can agree that there’ll be something of a fundamental speed limit for approaches that are not explicitly designed for online use. It will take a while for popularity to reflect that, as more streaming applications arise and more companies deploy models, and come up against the practical challenges of doing so, as compared with merely talking about artificial intelligence.</p><h3>The Speedster List</h3><p>Even if you don’t care to run the notebook, you’ll find a growing list of “green” fast models <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/fastest.html">here</a>. And on any of the leaderboards, such as the <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/overall.html">overall</a> leaderboard, you can see the relative speeds. You can judge for yourself the tradeoff between speed and accuracy — if indeed there is one.</p><p>At the time of writing, the table below lists the top-performing time-series algorithms amongst those that are blazingly fast. The reported Elo is an average Elo rating across all the sub-categories, assuming ten matchups or more in each category. I can’t guarantee they are close to equilibrium as you read this, but they are getting there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPEQ6QT5c337_Zz0h1_Bdg.png" /><figcaption>Example of an Elo ratings table showing performance time in second in the second to last column</figcaption></figure><p>(See the <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/fastest.html">live table</a>s)</p><h3>Online, Incremental Time-Series Packages</h3><p>There will be more <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/composition/residualcomposition.py">stacking</a>, <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/composition/residualshypocratic.py">shrinking</a>, <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/ensembling/precisionweightedskater.py">ensembling</a>, and <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/composition/conjugation.py">conjugating</a> too. Are you interested in helping, or just becoming more familiar with what is out there? Some ideas:</p><ul><li>I’m always on the lookout for underpromoted packages and see a little activity in Michael Nowotny’s <a href="https://github.com/michaelnowotny/state_space">state_space</a> package. You can see what he’s up to <a href="https://github.com/michaelnowotny/state_space/blob/master/state_space/examples/conditional_univariate_linear_factor_model.py">here</a> and perhaps the compilation of state-space models from a symbolic specification is an interesting way to go.</li><li>State-space models can be fast, assuming they support speedy estimation (a hard problem). I would draw your attention to packages such as <a href="https://github.com/oseiskar/simdkalman">simdkalman</a>, from Otto Seiskari, which provides single-instruction multiple-data vectorized Kalman filtering — something of a mouthful but also inspiring if you are looking for fast ensembles.</li><li>Or maybe you can utilize <a href="https://pypi.org/project/onelearn/">onelearn</a>, developed by Stephane Gaiffas, in a sneaky way to improve ensembles of time-series forecasts — though I’m yet to try that myself.</li><li><a href="https://pypi.org/project/proust/">Proust</a> might have been abandoned by Daniel Suo. Hey Daniel: help us fill the void.</li><li>Or did this become the <a href="https://github.com/google/timecast">timecast</a> package? That also aspires to performant, online learning. (It is on the Google github, incidentally, and thus an exception to the remark I made earlier about large companies).</li><li>Time-series isn’t the primary focus for Max Halford and company, who are quite <a href="https://github.com/online-ml/river/graphs/contributors">actively</a> developing <a href="https://github.com/online-ml/river">river</a> (good name: a merger of creme and scikit-multiflow by the way). But that doesn’t mean river isn’t very useful for time-series already.</li></ul><p>You can scroll through the <a href="https://www.microprediction.com/blog/popular-timeseries-packages">listing of packages</a> too.</p><h3>Online feature generation?</h3><p>Please comment if you have leads.</p><ul><li>A package like <a href="https://tsfresh.readthedocs.io/en/latest/">tsfresh</a>, developed by Nils Braun, Maximilian Christ, and others, could be used periodically without killing the amortized cost of computation.</li><li>Then there’s the seemingly brute force approach provided by reservoir computing approaches (like <a href="https://github.com/cknd/pyESN">pyESN</a>, from Clemens Korndorfer). Here the features are intermediate activations in a neural network, and thus extremely fast to compute.</li><li>It’s also more than plausible that anomaly detection packages (such as <a href="https://github.com/linkedin/luminol">luminol</a> by Brennan Vincello et al) can reduce errors too, without paying too high a computational price.</li><li>Classification packages can be used to assist the ensembles, and there are many of them as you can see. Here’s <a href="https://github.com/spaghettix/DB_OC_TSC">one</a> by Mauceri Stefano you’re unlikely to have stumbled upon, just to illustrate the diversity.</li></ul><p>Stacking residual predictors seems promising too. If you are looking for inspiration, maybe use the residual category in the model recommender:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9UA-eQqbYyIQn2nb7-s0Xg.png" /><figcaption>Example of selecting fast models that ought to do an okay job of prediction residuals</figcaption></figure><p>The question seems not to be “what is the best package?” but “how can they be combined in clever ways”. The faster each package is, the more leeway we have in creating combinations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=850343737c04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Clean Way to Deal With Ordinal Values in SkLearn]]></title>
            <link>https://medium.com/@mike.roweprediger/the-clean-way-to-deal-with-ordinal-values-in-sklearn-b6e9ff5ec588?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6e9ff5ec588</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ordinal-encoding]]></category>
            <category><![CDATA[ordinal]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[fit-transform]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Sun, 15 Oct 2023 17:32:37 GMT</pubDate>
            <atom:updated>2023-10-15T17:32:37.486Z</atom:updated>
            <content:encoded><![CDATA[<p>Ordinal data is a little annoying, but less so with sklearn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTaUppxDjQiN7rRb6oppLg.png" /><figcaption>Pretty much all you need</figcaption></figure><p>Let’s mock some data up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FT11moP6oYvQwq6jqXUoBA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*54JpIcTgaZaE0mLtSZvovA.png" /></figure><p>It’s a little inconvenient to have those categorical and ordinal values. But just use fit_transform() of the OrdinalEncoder like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*xQ7dnWkl91irISgDzdbB_Q.png" /></figure><p>Admittedly we might want to do something else with the categorical data, such as one-hot encoding, but let’s proceed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*hobA6wQImsLxxE4SxmRuWg.png" /><figcaption>Encoded ordinal data</figcaption></figure><p>And then you can proceed as usual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*vwE5UvINk6VUVD1ax0wDsw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6e9ff5ec588" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Presidential Poetry Appreciation]]></title>
            <link>https://medium.com/@mike.roweprediger/presidential-poetry-appreciation-f6ea45fb8c83?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6ea45fb8c83</guid>
            <category><![CDATA[poetry]]></category>
            <category><![CDATA[history]]></category>
            <category><![CDATA[poetry-appreciation]]></category>
            <category><![CDATA[poetry-writing]]></category>
            <category><![CDATA[us-president]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Tue, 19 Sep 2023 19:12:22 GMT</pubDate>
            <atom:updated>2023-09-19T19:12:22.446Z</atom:updated>
            <content:encoded><![CDATA[<p>Because why not? I stole this fun example from the <a href="https://github.com/microprediction/chattychattybangbang/tree/main/examples">examples</a> directory of the <strong>chattychattybangbang</strong> package. Here are the presidents that ChatGPT thinks liked poetry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*JY-VtQoNkpO7rF66S2jvOw.png" /></figure><p>The rest not so much! Obviously this is just an example of data harvesting and you can see how this is done with just a single line of code <a href="https://github.com/microprediction/chattychattybangbang/blob/main/examples/presidents.py">here</a> — though admittedly the function itself is about as verbose as ChatGPT! This uses <em>castigate_until_numeric_dict_with_known_keys_iteratively </em>which is a utility function taking an arbitrary question to guide scoring of any kind applied to a long list of keys.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6ea45fb8c83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Use the Huber Pseudo-Mean]]></title>
            <link>https://medium.com/@mike.roweprediger/how-to-use-the-huber-pseudo-mean-75a7f9baaa75?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/75a7f9baaa75</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[central-tendency]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Sat, 02 Sep 2023 20:06:39 GMT</pubDate>
            <atom:updated>2023-09-02T20:06:39.574Z</atom:updated>
            <content:encoded><![CDATA[<p>A note on the accuracy of bootstrapped forecasts and Huber means.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vihyAgiv1GOvclNSNYkRlg.png" /><figcaption>Different types of means might be viewed as different types of springs.</figcaption></figure><h3>The Huber mean</h3><p>There is a little <a href="https://github.com/microprediction/microprediction/blob/master/examples/notebooks/huber_loss.ipynb">notebook</a> of using a “Huber mean”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_zf8rBk5OByRXTEFIVSMPw.png" /></figure><p>If that’s all you need, have a nice day!</p><h3>Details</h3><p>Do you care about the details? See the <a href="https://github.com/microprediction/timemachines/tree/main/timemachines/skatertools/combining">combining</a> utilities of the timemachines package, specifically the <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/combining/hubermean.py">huber_mean</a> function that minimizes a smoothed Huber loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PXFUlb-oyejv-N98sBu-ww.png" /><figcaption>Modified Huber Loss</figcaption></figure><p>This comes from a <a href="https://arxiv.org/pdf/2108.12627.pdf">paper</a> by Kaan Gokcesu, Hakan Gokcesu. And actually, it is a modified version that <em>scales the parameters to take into account the standard deviation of the data</em> — a minor detail perhaps. You may prefer a more standard Huber — say from <a href="https://github.com/keras-team/keras/blob/v2.10.0/keras/losses.py#L1381-L1448">tf.keras.losses.Huber</a> or <a href="https://www.statsmodels.org/dev/_modules/statsmodels/robust/scale.html">StatsModels.robust.scale.Huber</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*jSsRDe_tjBqviuIYRjMatw.png" /><figcaption>The standard Huber loss you will find in Keras. This article uses a different one.</figcaption></figure><h3>Motivation</h3><p>Huber loss doesn’t require a lengthy a priori justification. It approximates squared error for small values, and approximate absolute error for large — thus providing a compromise between mean and median. The question is one of efficacy: where is this likely to help?</p><p>Intuitively we’d expect it to do so in the presence outliers or observations with fat-tailed measurement error distributions.</p><h3>Using Huber pseudo-means to assess model combinations</h3><p>Here’s something we can try:</p><ol><li>Use an Auto-ARIMA model then apply a ‘<a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/smoothing/wiggling.py">wiggler</a>’ (which was introduced in the post “<a href="https://microprediction.medium.com/smooth-move-does-wiggling-make-time-series-models-less-accurate-8544e675873">Smooth Move</a>”) to create an ensemble of models.</li><li>Combine using different measures of recent accuracy.</li></ol><p>If you’d like to dive further into that example, see the script <a href="https://github.com/microprediction/timemachines/blob/main/examples/performance/sk_sf_wiggly_performance.py">here</a>. By all means discuss in the microprediction <a href="https://microprediction.github.io/microprediction/slack.html">slack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75a7f9baaa75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Memory-Frugal Torch Dataset from a Collection of CSVs]]></title>
            <link>https://medium.com/@mike.roweprediger/creating-a-memory-frugal-torch-dataset-from-a-collection-of-csvs-a924ee090eee?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/a924ee090eee</guid>
            <category><![CDATA[datasets-for-ml]]></category>
            <category><![CDATA[dataloader]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 06:11:55 GMT</pubDate>
            <atom:updated>2023-08-08T06:11:55.365Z</atom:updated>
            <content:encoded><![CDATA[<p>The Python <a href="https://pypi.org/project/csvsdataset/0.0.5/">csvsdataset</a> package does just one thing, and if you have a large collection of CSV data files that you don’t want to put in memory all at once, it is one of your options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*g9lNkJYK06_UlCGRYDB1HA.png" /><figcaption>Data for machine learning spread across several CSV files</figcaption></figure><h3>File collection requirements</h3><p>To use this you’ll need a directory containing a plurality of CSV files. They should have regressor and target columns in common. Suppose our regressors lie in columns like <em>col_15 </em>and they are scattered across files called <em>test_data_117.csv,</em> et cetera. Anything will do so long as the columns can be discerned using regular expressions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*4-RzVlCQn9tCbyTELxLV9g.png" /><figcaption>Specifying a distributed dataset</figcaption></figure><p>Or one can specify the columns directly.</p><h3>Constructor</h3><p>Either though these regular expressions, or directly specifying the column names, we create a CsvsDataset object, viz:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYwGxFRRKHQ6-SGNnvjjeg.png" /><figcaption>Instantiating a CsvsDataset object. The contents of the files are not read into the dataset. However an intial scan of the files is performed to determine how many rows are in each.</figcaption></figure><p>The CsvsDataset class takes several arguments:</p><ul><li>folder_path: a string representing the path to the folder containing the CSV files</li><li>file_pattern: a string representing a pattern for matching the CSV files in the folder</li><li>x_columns: a list of strings representing the names of the columns to be used as features (X) for the dataset</li><li>x_columns_pattern: a string representing a regular expression pattern for matching column names to be used as features (X) for the dataset</li><li>y_column: a string representing the name of the column to be used as the target (Y) for the dataset, or…</li><li>y_columns: a list of strings representing the names of the columns to be used as targets (Y) for the dataset, or …</li><li>y_columns_pattern: a string representing a regular expression pattern for matching column names to be used as targets (Y) for the dataset</li><li>cache_capacity: an integer representing the maximum number of CSV files to cache in memory</li></ul><p>Note that there are different ways to specify the columns.</p><h3>Usage</h3><p>Not much to see here. The indexing operator returns the X and y values pertaining to a given row. It is <em>as if</em> the files have been loaded and concatenated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DGo_pu70kv-u9cFPR2UUgA.png" /></figure><h3>How it works</h3><p>Inside the constructor method, the glob module is used to find all the CSV files in the folder that match the file_pattern argument. The number of rows in each file is also calculated using the _get_num_rows() method.</p><p>The column names for the dataset are determined from the first CSV file found, using the pd.read_csv() function with nrows=0 to read only the header row. If x_columns is not provided, the column names for features (X) are determined using x_columns_pattern, which is compiled into a regular expression object and matched against all the column names. If y_columns is not provided, y_columns_pattern is used in a similar way to determine the column names for targets (Y).</p><p>Finally, an OrderedDict is used as a cache for the CSV files, with cache_capacity representing the maximum number of files to cache in memory. A simple last-used eviction policy is enforced.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a924ee090eee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clustering for Efficient Computation of Contest Winning Probabilities]]></title>
            <link>https://medium.com/@mike.roweprediger/clustering-for-efficient-computation-of-contest-winning-probabilities-10f6d4e2cc4e?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/10f6d4e2cc4e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[probability-distributions]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Tue, 01 Aug 2023 17:46:31 GMT</pubDate>
            <atom:updated>2023-08-01T17:46:31.551Z</atom:updated>
            <content:encoded><![CDATA[<p>Here’s a contest with three “fast” horses and several slow ones. How shall we quickly estimate winning probabilities? Could we do it efficiently using a lattice size that is, say length 151 instead of 301 as shown?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNHhEE4vbsGbjyuSZg28VQ.png" /></figure><p>This is the guts of a Python function you can find <a href="https://github.com/microprediction/winning/blob/main/winning/lattice.py">here</a> that is used to calculate the state prices of a horse race using a clustered offsets method that deals with lattice limitations. The idea:</p><ol><li>Create a “blurry” race to get a rough idea of everyone’s winning probabilities (even if the offsets are a bit too big to fit on the lattice)</li><li>Compute more precise winning probabilities for sub-races</li><li>Approximate with conditional probabilities, making a mildly courageous assumption somewhat akin to Harville.</li></ol><p>To wit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCIIm5ki20KkgpQzxelX2A.png" /><figcaption>Splitting a race and recombining.</figcaption></figure><p>The function takes four arguments:</p><ul><li>density: The density function of the horse race.</li><li>offsets: The offsets of each horse in the race.</li><li>fast_ndxs: A list of indexes of the fast horses in the race.</li><li>unit_ratio: The unit ratio used to dilate the density function.</li></ul><p>If there is only one slow horse, the function sets its relative state price to 1.0. Otherwise, the function calculates the relative state prices of the slow horses using the state_prices_from_offsets function.</p><p>Next, the function calculates the relative state prices of the fast horses using the state_prices_from_offsets function.</p><p>Now the crucial step. It then creates an approximate race involving all the horses by dilating the density function using the dilate_density function and the unit_ratio argument that I explained in this <a href="https://medium.com/@mike.roweprediger/how-to-move-a-discrete-density-from-one-unit-size-to-another-27d4ffeab036">article</a>. You’ll notice that the dilated offsets are calculated by dividing the original offsets by the unit_ratio so that the interpretation carries over to the new lattice.</p><p>The function then calculates the share of the slow horses in the approximate race by summing up the state prices of the slow horses. The share of the fast horses is then calculated as 1 minus the share of the slow horses.</p><p>Now the brave assumption, which isn’t really that brave. The function then calculates the state prices of the slow and fast horses by multiplying their relative state prices by their respective shares. Finally, the function combines the state prices of the slow and fast horses into a single list of state prices and returns it.</p><p>Play with it yourself in this <a href="https://github.com/microprediction/winning/blob/main/splitting_race.ipynb">notebook</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10f6d4e2cc4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Doesn’t PyCaret Work Better for Time-Series?]]></title>
            <link>https://medium.com/@mike.roweprediger/why-doesnt-pycaret-work-better-for-time-series-a6f168d41dc0?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6f168d41dc0</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[pycaret]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Sun, 23 Jul 2023 15:21:38 GMT</pubDate>
            <atom:updated>2023-07-23T15:21:38.430Z</atom:updated>
            <content:encoded><![CDATA[<p>PyCaret doesn’t have a very good <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/univariate-k_008.html">Elo rating</a> for univariate timeseries. It is also very computationally expensive. Worst of both worlds. Perhaps it is doomed by overfitting — but this article encourages someone to try to fix the situation and shows you that most of the work has been done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q2F4-U3d-DNshxHUWPyw3g.png" /></figure><p>First, here’s a wrapper:</p><h3>PyCaret Wrapper</h3><p>We just want to take floats and produce a prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIRO8NGV43irnSuTI2w6nA.png" /></figure><p>This code defines a function called pycrt_iskater that uses the PyCaret library to perform time series forecasting on a univariate time series. The input parameters are:</p><ul><li>y: A list of lists or a 2D numpy array containing the time series data.</li><li>k: The number of time steps to forecast into the future.</li><li>a: A list of exogenous variables. This parameter is not used in the function.</li><li>t: A list of timestamps corresponding to each time step in the time series. This parameter is not used in the function.</li><li>e: A list of error terms. This parameter is not used in the function.</li><li>n_select: The number of top-performing models to select.</li><li>blend_method: The method used to blend the top-performing models.</li><li>turbo: A boolean indicating whether to use a fast (but potentially less accurate) version of the algorithm.</li></ul><p>It creates a Pandas Series object with the time series data and sets the index to a PeriodIndex object created from a date range starting at “2021–01” and having a length equal to the length of the time series.</p><p>The function then creates a TimeSeriesExperiment object from the PyCaret library (by all means update this code!) and sets it up with the time series data, the number of time steps to forecast into the future, and the number of cross-validation folds.</p><p>It then compares a set of baseline models using the root mean squared error metric and selects the top n_select models. The function then tunes the hyperparameters of each of the top-performing models and blends them using the specified method. Finally, the function generates forecasts using the blended model and returns the forecasted values and standard deviations (which are all set to 1 for this implementation).</p><h3>Stop reading</h3><p>Now, if you just want to make that use of PyCaret better, you don’t need to know anything else. But, if you wish to know how it is benchmarked, I shall explain further.</p><h4>How it is benchmarked: Step 1</h4><p>Any batch-style forecasting function can be turned into an incremental style using the “batch skater factory”: See code <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skaters/pycrt/pycaretblend.py">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fv0JFLGZ_VacEU6cE-qeXA.png" /></figure><p>The batch skater factory (code <a href="https://github.com/microprediction/timemachines/blob/main/timemachines/skatertools/batch/batchskater.py">here</a>) does this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESD-vuPmKUzUKgBLMExtEw.png" /></figure><p>I won’t try to copy all the code here. But here’s what it does.</p><p>The function takes as input a time series y, a number of steps to forecast k, and several optional parameters including a, t, e, and min_e. It also takes a dictionary s that maintains state between function calls, and two parameters emp_mass and emp_std_mass that specify the weight to be given to empirical predictions and their standard deviation in the final prediction, respectively (this doesn’t impact the benchmarking though).</p><p>The code maintains history. It appends the input time series y to a list s[&#39;y&#39;]. If a is not None, it is appended to a list s[&#39;a&#39;], and likewise if t is not None, it is appended to a list s[&#39;t&#39;] after checking that it is a float.</p><p>The code then checks whether a batch forecast should be called based on the values of n_warm, k, a, t, e, and min_e. If the conditions are met, the batch forecast is called by passing a subset of s[&#39;y&#39;], s[&#39;a&#39;], and s[&#39;t&#39;] to the iskater function along with some additional keyword arguments specified in iskater_kwargs. If the batch forecast is not called, a default naive forecast is returned.</p><p>(It’s not a great idea to call pycaret until the history is warmed up)</p><p>After obtaining the forecast from the batch or naive method, the code updates the prediction parade s[&#39;p&#39;] using the current forecast and the actual value of the next step in the time series. It then calculates the empirical center of the forecast by combining the naive forecast and the empirical prediction based on the prediction parade using a weighted average, and returns the empirical forecast and its standard deviation.</p><h4>Sound reasonable?</h4><p>Well it isn’t performing as well as other things in the <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/univariate-k_008.html">timeseries Elo ratings</a>. So please help figure out this connundrum. It might simply be the case that there isn’t currently enough defensiveness in PyCaret for this particular use case. Other methods are better at avoiding overfitting, to put it another way. By all means discuss in the microprediction <a href="https://microprediction.github.io/microprediction/slack.html">slack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6f168d41dc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Volatility Using PyKalman’s Implementation of the Expectation Maximization (EM)…]]></title>
            <link>https://medium.com/@mike.roweprediger/predicting-volatility-using-pykalmans-implementation-of-the-expectation-maximization-em-6fb4b7db3ce6?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fb4b7db3ce6</guid>
            <category><![CDATA[data-science-training]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Fri, 16 Jun 2023 20:21:48 GMT</pubDate>
            <atom:updated>2023-06-16T20:21:48.617Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predicting Volatility Using PyKalman’s Implementation of the Expectation Maximization (EM) Algorithm</h3><p>I present a colab <a href="https://github.com/microprediction/timeseries-notebooks/blob/main/pykalman_hello.ipynb">notebook</a> example of using the Expectation Maximization algorithm to predict noisy data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*cUMO-wC5Tz8SNvTF5KL_7g.png" /><figcaption>The Kalman Filter is a classic means of sailing through noise</figcaption></figure><h3>Packages</h3><p>We need a kalman filter, and some time-series data. So:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*kFC1d1Ooku6mf5bnSqFq1w.png" /></figure><p>I use microprediction here so the data is live. It isn’t essential to the example really, but it is good practice.</p><p>Testing time series packages on live data is useful because it allows for more accurate assessment of the package’s performance and behavior in real-world situations. This is because live data is constantly changing and evolving, which presents a more realistic test scenario compared to a single stale source that does not change.</p><ol><li>Real-time data changes: Live data reflects the constant changes and fluctuations that occur in real-world time series data. Testing on live data can help identify issues related to data changes that may not be captured by a static data source.</li><li>Realistic testing: Testing on live data provides a more realistic and representative test scenario, as the data is constantly evolving and the package needs to adapt to these changes.</li><li>Dynamic testing: Live data allows for dynamic testing of the package’s functionality and performance, as the package needs to handle incoming data streams and process them in real-time.</li><li>Quality assurance: Testing on live data can help ensure the quality and reliability of the time series package, as it is tested in real-world conditions.</li><li>Overfitting: There’s no point if your model just memorizes the past!</li></ol><p>With that elaborate justification:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HlJv3SNltHw3Hs-p1TDvPA.png" /></figure><h3>Using a Kalman Filter</h3><p>PyKalman makes this really easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QLVMtAooYuYjK4AHV0B3bw.png" /></figure><p>But notice a few things.</p><p>The second line of code in the function is creating an instance of a KalmanFilter object from the KalmanFilter class. The initial_state_mean parameter specifies the mean of the initial state of the system, which is assumed to be normally distributed. In this case, the initial state mean is set to 0, which means that the Kalman filter assumes the initial state of the system is at zero.</p><p>The n_dim_obs parameter specifies the dimensionality of the observation vector, which is the number of observations made at each time step. In this case, the dimensionality is set to 1, which means that the Kalman filter is expecting a single observation at each time step.</p><p>Once the Kalman filter object is created, it can be used to estimate the state of a system based on a series of noisy observations. The Kalman filter works by recursively updating its estimate of the state of the system based on new observations, while also taking into account the uncertainty of the observations and the dynamics of the system.</p><h3>Expectation Maximization</h3><p>Now let’s talk about one more line of code.</p><p>The KalmanFilter object kf is being trained on a set of training data represented by ys_train. The .em() method is called on the KalmanFilter object to perform the training using the Expectation-Maximization (EM) algorithm.</p><p>The EM algorithm is an iterative method used to estimate the parameters of a statistical model in situations where some of the variables are not observed. In the context of time series analysis with the Kalman filter, the EM algorithm can be used to estimate the parameters of the state transition model and observation model based on the available data.</p><p>When the .em() method is called on the KalmanFilter object with the training data ys_train, the algorithm iteratively estimates the parameters of the Kalman filter model until convergence is reached. Specifically, the EM algorithm alternates between two steps:</p><ol><li>The E-step (Expectation step): In this step, the algorithm estimates the expected values of the latent variables, which are the unobserved variables in the model, based on the current estimates of the model parameters.</li><li>The M-step (Maximization step): In this step, the algorithm updates the estimates of the model parameters based on the expected values of the latent variables computed in the E-step.</li></ol><p>Do we have to do this every single data point? Not really, but as you can see it is fast enough on this occasion. With some more standard code we can see the results, and zoom in as we like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*4WSQQO2UtAdpnAOIizC4Kw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*N5qcaNP6uKHOedKrEB4FjQ.png" /><figcaption>Kalman Filter prediction (prior) versus noisy observations</figcaption></figure><h3>Understanding the Kalman Filter?</h3><p>There are many roads. My personal favorite is a physical analog and I’ll borrow from this <a href="https://www.microprediction.com/blog/analog">blog article</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gPzWKCqHVn3of4OemkamMA.png" /><figcaption>Physical interpretation of Kalman filter</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fb4b7db3ce6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making a Safe, Simple Online Version of NeuralProphet]]></title>
            <link>https://medium.com/@mike.roweprediger/making-a-safe-simple-online-version-of-neuralprophet-5a3e76d1bec7?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a3e76d1bec7</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Sat, 03 Jun 2023 16:51:27 GMT</pubDate>
            <atom:updated>2023-06-03T16:51:27.210Z</atom:updated>
            <content:encoded><![CDATA[<p>NeuralProphet is a Python package for time-series forecasting that is built on the principles of Prophet, an open-source forecasting library developed by Facebook. NeuralProphet extends the Prophet model by incorporating neural network components, which allows it to handle more complex time-series data and capture nonlinear patterns. In this note we present a little wrapper that allows it to be used in one line of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XSL5UPmkBgXirD9BOmG9Cw.png" /></figure><p>NeuralProphet also includes several tools for model evaluation and visualization, such as time-series plots and forecast accuracy metrics. The package provides an easy-to-use interface for generating forecasts and visualizing the results, making it accessible to both data scientists and researchers. See the <a href="https://neuralprophet.com/model-overview.html">documentation</a>.</p><p>Let’s use it!</p><h3>Install glitch?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cIU221aLEPM0uJskaGshqQ.png" /></figure><p>The second import is for some data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56k4Iy9L-imhLIlBs09pdw.png" /><figcaption>Grabbing time-series data</figcaption></figure><h3>Using the forecaster</h3><p>Notice the similarity in style to Facebook prophet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4WRuVu80o_8DsD8I5jGLQ.png" /></figure><p>Let’s unwrap this. First, a NeuralProphet object is created with the desired model parameters, such as the number of lags, changepoints range, and number of changepoints. These parameters control the complexity and flexibility of the forecasting model.</p><p>Next, the fit method of the NeuralProphet object is called on the input dataframe DF, which contains the time-series data. This method trains the model on the input data and returns a set of forecast metrics, such as the mean squared error and mean absolute error.</p><p>After fitting the model, the make_future_dataframe method is called to create a dataframe of future time points for which we want to generate forecasts. This method takes as input the input dataframe DF and the number of future time points periods.</p><p>(This is the part that comes from prophet. Personally I find it a bit ceremonial but I guess it is a matter of taste)</p><p>Finally, the predict method is called to generate forecasts for the future time points. The yhat1 column of the resulting forecast dataframe contains the point forecast for the last future time point, which is extracted using the .values[-1] notation.</p><h3>Testing without any possibility of data leakage</h3><p>Testing time-series models out-of-sample is important because it allows us to evaluate the performance of the model on unseen data. If we evaluate the model on the same data that we used to train it, the performance metrics will be overly optimistic and may not generalize well to new data. By testing the model on a separate dataset, we can get a more realistic estimate of its performance and make better decisions about model selection and deployment.</p><p>Data leakage is another important consideration in time-series modeling. Data leakage occurs when information from the future is inadvertently used to make predictions for the past. This can lead to overly optimistic performance metrics and a false sense of confidence in the model.</p><p>To avoid data leakage, we could split the data into training and testing sets before conducting any analysis, and to ensure that the testing set only includes data from the future that the model has not seen during training. This ensures that the model is evaluated only on its ability to make accurate predictions for new, unseen data.</p><p>But that is a bit to “batchy” for our liking. Maybe it is better to just wrap it in an online style first, so then there can be no data leakage issues at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eS-AE253YTkzJHArLpvtg.png" /></figure><p>This Python code defines a function prophet_next that uses the NeuralProphet package to predict the next point in a time-series sequence. The function takes as input a list of float values ys representing the historical time-series data and an optional model object that can be used for making predictions.</p><p>The first set of conditions in the function check whether the input ys list is empty, has fewer than 400 values, or whether a new model needs to be fitted. If any of these conditions are true, the function returns the last value in the ys list as the predicted value and a None object for the model. That’s because NeuralProphet really isn’t designed for use on short time-series.</p><p>But … that aside if there is data and a new model needs to be fitted, a new NeuralProphet object is created with the specified parameters, and the historical time-series data is used to train the model. After fitting the model, the function creates a future dataframe containing the next time point and uses the predict method of the model to generate a forecast for this point.</p><p>Finally, the function returns the predicted value for the next time point, which is extracted from the forecast dataframe, and the updated model object. This seems like a lot of work just to predict the next point, doesn’t it? Well, don’t blame me. Blame the precedent set by the original prophet.</p><p>We can now predict in perfectly safe forward fashion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*whCV9rIIkpHJSO1dAKa7KQ.png" /></figure><p>And the result is pretty. We can also be completely sure that there is no cheating going on under the covers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*RXQv4CiRq_rvWIpzhJErtA.png" /></figure><p>If you’d like to help create more simple forecasting functions that can be compared against many others, join the <a href="https://microprediction.github.io/microprediction/slack.html">slack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a3e76d1bec7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fractional Differencing in PyTorch, with Reconstruction]]></title>
            <link>https://medium.com/@mike.roweprediger/fractional-differencing-in-pytorch-with-reconstruction-5e6c9f2759b9?source=rss-b8348664ec71------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e6c9f2759b9</guid>
            <category><![CDATA[timeseries-forecasting]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Michael Rowe]]></dc:creator>
            <pubDate>Thu, 01 Jun 2023 14:17:12 GMT</pubDate>
            <atom:updated>2023-06-01T14:17:12.177Z</atom:updated>
            <content:encoded><![CDATA[<p>Gottfried Wilhelm Leibniz first suggested the possibility of fractional derivatives in the 17th Century. It allows for a nuanced approach to differencing compared to integer-order differencing. It blends the concepts of integration and differentiation, enabling the retention of memory from past observations while still removing non-stationarity in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDTvtveSK0dI_97mOBDkJQ.png" /><figcaption>Fractional differencing of a time-series using order 0.25, with reconstruction and reconstruction errors.</figcaption></figure><h3>Acknowledgements</h3><p>I’ll be working from a <a href="https://github.com/microprediction/timeseries-notebooks/blob/main/pytorch_fractional_differencer.ipynb">colab notebook</a> in a Microprediction repository (interesting <a href="https://mitpress.mit.edu/9780262047326/microprediction/">book</a> btw). The notebook acknowledges the <a href="https://github.com/adamvvu/tsfracdiff/blob/master/tsfracdiff/tsfracdiff.py">tsfracdiff</a> implementation by GitHub user <a href="http://adamvvu/">adamvu</a>. I’ll also mention in passing that there is also a <a href="https://github.com/fracdiff/fracdiff">fracdiff</a> package which has a very fast implementation and also a pytorch <a href="https://github.com/fracdiff/fracdiff/blob/main/fracdiff/torch/functional.py">version</a> although not, at the time of writing, an inverse.</p><h3>Motivation</h3><p>The motivation behind fractional differencing lies in its ability to address the trade-off between retaining memory in the data and ensuring stationarity. In many real-world applications, data may exhibit long-range dependence, which means that current observations are influenced by past observations over an extended period.</p><p>Integer-order differencing can remove non-stationarity, but in the process it might ruin your forecasting because it can eliminate much of this long-range dependence, leading to a weaker representation of the underlying data-generating process. Fractional differencing provides a more flexible approach, allowing for a balance between retaining memory and achieving stationarity — that’s the idea, anyway.</p><p>Fractional differencing has found applications in various domains, including finance, economics, and geophysics. In finance, for example, it is used for modeling and predicting stock prices, volatility, and other financial time series. One of the early applications of fractional differencing can be found in the work of <a href="https://www.jstor.org/stable/2335817">Hosking (1981)</a>, who used the technique to model hydrological time.</p><p>The goal here is to create a pytorch-friendly version of the transform and its inverse transform. As we’ll see the inverse transform is in theory and practice possible if we have enough lags and the order is not too high — though one has to be a bit careful and I’ll get to that when we discuss the parameters.</p><p>Finally a quick remark that not everybody feels the need for the inverse transform. It is perfectly fine if you wish to apply the forward transform and then just use it as a feature like any other for prediction. However I think it is nice to pair any transform with its inverse where possible, and that’s what we’ll be validating here.</p><h3>Requirements</h3><p>No surprises here although we’ll needscipy.special.comb which computes the number of combinations of k items that can be chosen from a set of n items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*nwsbpKwDc99cQIasISnnqw.png" /></figure><h3>Memory weights</h3><p>I’m going to build up to a full class in steps. First we have a helper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RHycBShCIlAoLMP8rlVsKw.png" /></figure><p>The purpose of this method is to compute a sequence of weights that can be used to compute a linear combination of previous input values (just as with an “autoregressive” model in time series analysis). It iteratively computes the next weight in the sequence by multiplying the previous weight by a factor that depends on the order of the autoregressive model and the lag of the input value.</p><p>The coefficients themselves require a mathematical justification I will skip at least for now. The purpose of reversing the sequence of memory weights is to match the convention used by numpy.correlate and other time-series analysis functions, which expect the weights to be in reverse order.</p><h3>The forward transform</h3><p>We are ready to perform the transform now. This is merely the application of the trailing autoregressive coefficients as suggested above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8GdXXsLA6GewW3v_ywFjAA.png" /></figure><p>The method first computes the length K of the memory_weights list. It then applies a sliding window of size K to the input time series ts, computing the weighted sum of the previous K values at each window position. (Notice that the list comprehension that iterates over the range of window positions from K to the length of the time series).</p><p>That’s how we get the output array of fractional differences. We can make this torch-ready as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwwH9T5uuXon-iI3o5dJNA.png" /></figure><h3>Inverse transform</h3><p>Now to the fun part … can be put the pieces back together successfully? I would encourage you to play with the <a href="https://github.com/microprediction/timeseries-notebooks/blob/main/pytorch_fractional_differencer.ipynb">notebook</a> here and not take my word for it. In particular as you vary the length of the time-series, the order and the <strong>max_lags</strong> parameter you will get a sense of the reconstruction error. You might also look closely at whether the reconstrution error <em>differences near the end of the time-series are well behaved.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AQz64CkNfJqnvnvnuLKujw.png" /></figure><p>Obviously the purpose of this method is to “undo” a fractional difference transformation that was applied to a time series, in order to obtain the original (un-differenced) time series. But notice that you need the lag data tensor. Indeed the method first checks that the lag_data_tensor parameter has the correct shape, which should be (num_lags, num_features) where num_lags is the number of lags used in the differencing and num_features is the number of features in the time series.</p><p>The method then calls the _get_memory_weights method that we discussed to compute the memory weights for the specified order of differencing. Notice also how the result is put together. The method extracts the first K lagged data points from the lag_data_tensor, where K is the number of lags used in the differencing. It also extracts the remaining differenced data points from the frac_diffed_tensor.</p><p>There is a special case worthy of note for if K is zero, then the original time series is equal to the differenced data, so the method simply returns x_tilde. But setting that edge case aside the method applies a loop over the remaining differenced data points, computing the corresponding original time series values using the memory weights and the previous K lagged data points. tensor.</p><h3>Verification</h3><p>The <a href="https://github.com/microprediction/timeseries-notebooks/blob/main/pytorch_fractional_differencer.ipynb">notebook</a> contains a test script to see if the reconstruction actually works. I won’t go through all of it. Suffice to say that the important ingredients you might want to play with are the input data and the fractional differencing order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*kQOKesjtmbTeyyzhr19vcg.png" /></figure><p>Here is an example of the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U1ZUl7Myw8fSB5jSXJ-UXw.png" /></figure><p>And notice the error in the reconstruction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*yDazwfTZzprTdGKl-N2-kg.png" /></figure><p>Quite typical. You need to be careful with use of this inverse transform. Make sure you are only using values where the reconstruction error it likely to be small.</p><h3>Reading</h3><p>The <a href="https://en.wikipedia.org/wiki/Fractional_calculus">fractional calculus</a> Wikipedia page is pretty incredible. Nor will you have trouble finding papers focussed on applications. For example.</p><ol><li>Granger, C. W. J., &amp; Joyeux, R. (1980). An introduction to long-memory time series models and fractional differencing. Journal of Time Series Analysis, 1(1), 15–29. <a href="https://doi.org/10.1111/j.1467-9892.1980.tb00297.x">https://doi.org/10.1111/j.1467-9892.1980.tb00297.x</a></li><li>Hosking, J. R. M. (1981). Fractional differencing. Biometrika, 68(1), 165–176. <a href="https://doi.org/10.1093/biomet/68.1.165">https://doi.org/10.1093/biomet/68.1.165</a></li><li>Geweke, J. (1986). Measurement of linear dependence and feedback between multiple time series. Journal of the American Statistical Association, 81(394), 1149–1164. <a href="https://www.jstor.org/stable/2287238">https://doi.org/10.2307/2289016</a></li><li>Robinson, P. M. (1994). Efficient tests of nonparametric hypotheses against long range dependent alternatives. Econometrica, 62(4), 881–890. <a href="https://www.jstor.org/stable/2291004">https://doi.org/10.2307/2951578</a></li></ol><p>There are also entire books, such as:</p><p><em>Fractional calculus and fractional processes with applications to financial economics: Theory and practice</em>. Edited by Hassan Fallahgoul, Sergio Focardi, Frank Fabozzi <a href="https://www.elsevier.com/books/fractional-calculus-and-fractional-processes-with-applications-to-financial-economics/fallahgoul/978-0-12-804248-9">https://doi.org/10.1142/9789812813746</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e6c9f2759b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>