<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andrey Alekseev on Medium]]></title>
        <description><![CDATA[Stories by Andrey Alekseev on Medium]]></description>
        <link>https://medium.com/@ilekseev?source=rss-27cd59fddc81------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5TxDN7V8D1hC0WLDUtDrrA.jpeg</url>
            <title>Stories by Andrey Alekseev on Medium</title>
            <link>https://medium.com/@ilekseev?source=rss-27cd59fddc81------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:27:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ilekseev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Doing DE job as DS]]></title>
            <link>https://medium.com/@ilekseev/doing-de-job-as-ds-1f2291705d9b?source=rss-27cd59fddc81------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f2291705d9b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-beam]]></category>
            <dc:creator><![CDATA[Andrey Alekseev]]></dc:creator>
            <pubDate>Fri, 14 Mar 2025 07:32:31 GMT</pubDate>
            <atom:updated>2025-03-14T07:32:31.255Z</atom:updated>
            <content:encoded><![CDATA[<p>As a Machine Learning Engineer I’ve seen how data pipelines become crucial for effective ML systems (even more than the models itself). So instead of refusing this job, I suggest to embrace it. In a daily DS job, you inevitably end up doing DE work. Creating data pipelines to move data between systems isn’t just a one-time thing, and data rarely cooperates by fitting neatly in memory. After the model development phase, you’re stuck in the pipeline maintenance period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5VKisopaF9iOwC778JmsuQ.jpeg" /></figure><p>So what are our options? I’ve tried a few approaches…</p><h3>Python (Micro)service (FastAPI or CronJob in container)</h3><p>Works nicely for all types of data. With <a href="https://github.com/dbader/schedule">schedule package</a> that’s quite easy to set up for regular jobs and with FastAPI you can even go for streaming data. Most pipelines run hourly or daily anyway. I would not recommend doing it for big(-ger) data though. Sure, you can process in batches, but now you’re writing that orchestration logic yourself. Plus if you ever need to speed up the process you are kinda left without scaling ability.</p><h3>Scheduled Jobs (Cloud Run, K8s Jobs and CronJobs)</h3><p>This has become my go-to for medium datasets (especially with cloud run ease of setup). Being just a Docker container means I can use whatever packages and architecture I want. Startup times are quick, and it can be manually scaled.</p><p>K8s Jobs and Cloud Run Jobs both let you parameterize executions. Batch numbers, date ranges, or partition keys are parameters that can be passed to the job without rebuilding the docker image. Here you only need to handle starting up the job. That can be done via Airflow or any microservice that can send http requests or execute kubectl commands. With CronJobs you can simply set up multiple cronjobs with different parameters and the startup will be handled by k8s itself. Another upside is that you pay only for uptime. However there is an annoying part as well. You build everything by hand: integration, batching, orchestration, failure scenarios.</p><h3>Specialized Services (Apache Beam with Dataflow, Spark, Flink)</h3><p>Apache Beam allows to describe your workload and then execute it on different runners. If you have these runners available then you are in luck because Beam and its runners come with a lot of pre-built connections for pushing data around and handle batching, autoscaling and throttling out of the box. It works with streaming data too. But Apache Beam is more expensive and has a steeper learning curve even with runners available. But as a DS you will love tweaking the configs to find the perfect batch size interval. This system works really good after you spend time setting it up and configuring and have all necessary components nearby to create new or tweak existing pipeline.</p><p>Despite these options, consistency triumphs theoretical perfection. If your team has invested in Apache Beam, stick with it rather than introducing yet another pipeline approach.</p><p>Anyway these DE skills aren’t optional extras anymore — they’re becoming core to our toolkit. The data scientists are expected to manage the full DS lifecycle.</p><p>What approaches have worked for you?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f2291705d9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Why Your ML Service Needs Rust + CatBoost: A Setup Guide That Actually Works]]></title>
            <link>https://medium.com/@ilekseev/why-your-ml-service-needs-rust-catboost-a-setup-guide-that-actually-works-60a7303ded8b?source=rss-27cd59fddc81------2</link>
            <guid isPermaLink="false">https://medium.com/p/60a7303ded8b</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[catboost]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Andrey Alekseev]]></dc:creator>
            <pubDate>Mon, 13 Jan 2025 17:40:09 GMT</pubDate>
            <atom:updated>2025-01-23T20:21:13.566Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZ_uFv_X4tU3g4TQ8iNy-g.jpeg" /><figcaption>Festival of light, Eindhoven 2024 (word play intended: Rust lightning fast)</figcaption></figure><p>Let’s talk about a problem that’s been bothering ML teams for a while now. When we’re working with batch processing, Python does the job just fine. But real-time serving? That’s where things get interesting.</p><p>I ran into this myself when building a service that needed to stay under 50ms latency. Even without strict latency requirements, everything might seem fine at first, but as the service grow, Python starts showing its limitations. Variables would randomly turn into None, and without type checks, tracking down these issues becomes a real headache.</p><p>Right now, we don’t have a go-to solution for real-time model serving. Teams often turn to tools like KServe or BentoML, but this means dealing with more moving parts — more pods to watch, extra network calls slowing things down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*oteIl0IsP7UJ8sTw2NZBaQ.png" /><figcaption>Possible ways of serving the model</figcaption></figure><p>What about other languages? C++ is fast and works with pretty much every ML library out there, but let’s be real — building and maintaining a C++ backend service is not something most teams want to take on.</p><p>It would be great if we could build models in Python but serve them in a faster language. ONNX tries to solve this, and it works relatively great for neural networks. But when I tried using it with CatBoost, handling categorical features turned into a challenge — the support just isn’t there yet.</p><p>This brings us to Rust, which offers an interesting middle ground:</p><ol><li>It’s just as fast as C++, but easier to my liking</li><li>The type system keeps your business logic clean and predictable</li><li>The compiler actually helps you write better code instead of just pointing out errors</li><li>The ML ecosystem is growing, with support from big names like Microsoft</li></ol><h3>Working with Official Catboost in Rust</h3><p>Good news — there’s actually an official Catboost crate for Rust! But before you get too excited, let me tell you about quirks that I discovered along the way.</p><p>The tricky part isn’t the Rust code itself — it’s getting the underlying C++ libraries in place. You’ll need to compile Catboost from source, and getting the environment right for this is most difficult part.</p><p>Catboost team provides their own Ubuntu-based image for building it from source, which sounds great. But what if you’re planning to run your service on Debian to keep things light? Then you better build Catboost on the same version of Debian you’ll use for serving, otherwise you might run into compatibility issues.</p><p>Let’s talk about why this matters in practice. The Ubuntu build image needs a hefty 4+ GB of memory to work with. But if you set up a custom Debian build correctly, you can bring that down to just 1 GB. And when you’re running lots of services in the cloud, these numbers of extra memory usage start adding up in your monthly bill.</p><h3>Setting Up Your Rust + Catboost Build Environment</h3><p>Let me walk you through setting up a Debian-based environment for Catboost. I’ll explain not just what to do, but why each step matters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*BXTgI-KIHqlNUsk2QDk19w.png" /><figcaption>Possible (and recommended) file structure</figcaption></figure><h4>Installing Catboost</h4><p>On the Rust side it happens as simple as with other crates in Rust:</p><pre>[package]<br>name = &quot;MLApp&quot;<br>version = &quot;0.1.0&quot;<br>edition = &quot;2021&quot;<br><br>[dependencies]<br>catboost = { git = &quot;https://github.com/catboost/catboost&quot;, rev = &quot;0bfdc35&quot;}</pre><p>However Catboost crate does not have precompiled C/C++ bindings and during installation (`cargo build`) will try to compile it from sources specifically for your environment. So let’s set up our environment.</p><h4>Starting with the Right Base Image</h4><p>First, we’re going with `debian:bookworm-slim` as our base image. Why? It comes with CMake 3.24+, which we need for our build process. The ‘slim’ variant keeps our image size down, which is always nice.</p><h4>Setting Up the C++ Build Environment</h4><p>We need a bunch of C++ packages, and while I’m using version 16 in our setup, you actually have flexibility here. Any version that supports `-mno-outline-atomics` will work fine.</p><p>Let’s break down our package installation into logical groups.</p><p><strong>Setting Up Package Sources</strong><br>First, we need to get our package sources in order. This part is crucial for getting the right LLVM tools:</p><pre>RUN apt-get update &amp;&amp; DEBIAN_FRONTEND=noninteractive apt-get install -y - no-install-recommends \<br> # will use it to download packages<br> wget \ <br> # cryptographic package to verify LLVM sources <br> gnupg \<br> # check Debian version to get correct LLVM package<br> lsb-release \<br> # package management helper<br> software-properties-common</pre><p>Then we need to add LLVM’s repository.</p><pre>RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - \<br> &amp;&amp; echo &quot;deb http://apt.llvm.org/$(lsb_release -sc)/ llvm-toolchain-$(lsb_release -sc)-16 main&quot; \<br> &gt;&gt; /etc/apt/sources.list.d/llvm.list</pre><p><strong>The main step of installing packages</strong><br>We need quite a few packages, and I recommend to organize them by purpose — it makes maintenance so much easier:</p><pre>RUN apt-get update &amp;&amp; DEBIAN_FRONTEND=noninteractive apt-get install -y - no-install-recommends \<br> # Basic build essentials<br> build-essential \<br> pkg-config \<br> <br> # Core development packages<br> libssl-dev \<br> cmake \<br> ninja-build \<br> python3-pip \<br> <br> # LLVM toolchain - version 16 works great, but any version with <br> # -mno-outline-atomics support will do<br> clang-16 \<br> libc++-16-dev \<br> libc++abi-16-dev \<br> lld-16 \<br> <br> # Don&#39;t forget git!<br> git</pre><p>Next step cost me some time to figure out. Catboost expects to find clang in `/usr/bin/clang`, but our installation puts it in `/usr/bin/clang-16`. That’s why we have this bit:</p><pre>RUN ln -sf /usr/bin/clang-16 /usr/bin/clang &amp;&amp; \<br> ln -sf /usr/bin/clang++-16 /usr/bin/clang++ &amp;&amp; \<br> ln -sf /usr/bin/lld-16 /usr/bin/lld</pre><p>And do not forget to set up environment variables</p><pre>ENV CC=/usr/bin/clang<br>ENV CXX=/usr/bin/clang++<br>ENV LIBCLANG_PATH=/usr/lib/llvm-16/lib<br>ENV LLVM_CONFIG_PATH=/usr/bin/llvm-config-16</pre><h4>Managing Dependencies</h4><p>We need Conan (version 2.4.1+) for handling C++ dependencies. A word of caution about the installation:</p><pre>RUN pip3 install --break-system-packages &quot;conan==2.11.0&quot;</pre><p>That ` — break-system-packages` flag might look scary, but it’s actually the easiest way I found to install Python packages system-wide in newer Debian versions. Besides we won’t be using much Python anyway in our build image.</p><h4>Smart Build Strategy</h4><p>Here’s a trick that’ll save you tons of build time during active development stage. Split your build into two steps:</p><ol><li>First, build just the dependencies:</li></ol><pre>COPY ./Cargo.* ./<br>RUN mkdir src &amp;&amp; \<br> echo &quot;fn main() {}&quot; &gt; src/main.rs &amp;&amp; \<br> RUSTFLAGS=&quot;-C codegen-units=1&quot; cargo build - release</pre><p>Important note here. You need that RUSTFLAGS=&quot;-C codegen-units=1&quot; flag because it ensures that C++ and Rust play along.</p><p>2. Then build your actual application:</p><pre>COPY ./src src<br>RUN cargo build - release</pre><p>This way, Docker caches the dependency build, and you only rebuild your app code when it changes. Much faster!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*lpToUEGdoUQ7zF_IC116hA.png" /></figure><h4>A Critical Warning About Memory</h4><p>This is important: during the C++ build steps, you’ll need machine with 20+ GB of memory (I used 32Gb). And here’s the part that cost me almost a day of debugging — if you don’t have enough memory, you won’t get a clear error message (or any to be honest). Instead, your build will mysteriously timeout, leaving you wondering what went wrong. I learned this one the hard way!</p><h3>Wrapping It Up</h3><p>Now we have a working Rust environment with Catboost that can handle all the good stuff: categories, text data, embeddings. Getting here wasn’t exactly easy.</p><p>Next time we’ll cover:<br>- Building an Axum web service<br>- Smart model loading patterns<br>- Real-world performance tricks I learned along the way</p><p>So we’ll turn this foundation into something that can actually serve models in production! I ran into some interesting problems while figuring this out, like accidentally loading the same model multiple times for each handler call.</p><p>And if you’ve tried this out and hit any weird issues, let me know. It’s always interesting to hear what problems other people run into.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82f15447c4deb4cde39527948d60c645/href">https://medium.com/media/82f15447c4deb4cde39527948d60c645/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60a7303ded8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Write your own neural network from scratch (Part 1)]]></title>
            <link>https://medium.com/@ilekseev/write-your-own-neural-network-from-scratch-part-1-15931de8579b?source=rss-27cd59fddc81------2</link>
            <guid isPermaLink="false">https://medium.com/p/15931de8579b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Andrey Alekseev]]></dc:creator>
            <pubDate>Fri, 17 Feb 2023 15:59:34 GMT</pubDate>
            <atom:updated>2023-02-17T15:59:34.578Z</atom:updated>
            <content:encoded><![CDATA[<p>Seeing what stable diffusion models and ChatGPT can achieve would make one think that deep learning must be very hard, but as in everything else, once you unravel the basics it is much easier to tackle complex problems right away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*waN3De-nFjirckB9" /><figcaption>Photo by <a href="https://unsplash.com/@kierinsight?utm_source=medium&amp;utm_medium=referral">Kier... in Sight</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Today I want to guide you through the basics of neural networks. We will need only the knowledge of linear algebra, derivatives, and some experience with Python and NumPy.</p><h4>The task</h4><p>Our task will be to classify the good old MNIST dataset. Each element of this dataset is a picture of a handwritten digit the size of 28x28 pixels. This means that for each picture there are 784 values. Overall MNIST dataset consists of 10 000 pictures. For our task that will be enough!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*-KrFuhtoT9Rr-o525mgL3w.png" /><figcaption>MNIST dataset (<a href="https://www.researchgate.net/publication/306056875_An_analysis_of_image_storage_systems_for_scalable_training_of_deep_neural_networks">source</a>)</figcaption></figure><p>Great! Let’s reshape each picture from 28x28 to 784x1 values. We can easily do so using NumPy. Now each picture is a vector of 784 dimensions. In our case, we want to have some function that could say from this vector what number is displayed in it. This function will be the neural network. Mathematically speaking we want to solve this task:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*HMbudeP7KyWp_t0cthhUrg.png" /></figure><p>The F function, our neural network, consists of simpler functions that are applied sequentially, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fEgllGNCWCsi24dDlf4RWA.png" /></figure><p>Let’s look closely at our “simple” functions. These functions are called layers. Of course, there are many types of layers, and I will show the most basic ones.</p><h4>Fully connected layer</h4><p>A fully connected layer (FC) is a perfect example. FC layer is able to transform our initial vector to a new size. This way our neural network can extract information. For example, FC can transform the initial picture of size 784x1 to 100x1. How does it do that? If you remember linear algebra you will easily recognize matrix multiplication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*_yuBm31J8AXePcw1jSKDkA.png" /></figure><p>Don’t worry I’ll explain it in a second. W is just a big table of values the size of 100x784 (matrix) and x is just a vector (the number next to it just shows its size). But how exactly do we calculate new x?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aamc4JHdH6uVmlxWuMY6QQ.png" /><figcaption>Numbers next to the W and x correspond to position. W 1,1 means first row, first column and so on.</figcaption></figure><p>First of all the new x is also a vector that consists of 100 values. In order to get its first element all we have to do is multiply the first value of the initial x with the value of matrix W that is placed in <em>the first row</em> and the <em>first column</em> then adds the second value of x multiplied by the value of W in <em>the first row </em>and <em>second column</em>, and so on until we reached the final 784th value of x. To calculate the second element of our new x we have to repeat the same algorithm but only with <em>the second row</em> of W.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*_EBoRRSKS3sT_zSEF4JKeA.png" /><figcaption>Each element of the row of W multiply with each element of the initial x and summarize the result</figcaption></figure><h4>Activation function</h4><p>Then activation function is applied to the new vector. The activation function should be non-linear otherwise the neural network would not be able to extract complex patterns from our data. Then we repeat the process until we have our answer: the last layer of the NN should yield the answer. In our case answer is the number shown in the picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*ihEe-C2Ft2sFW_fY-eeQXA.png" /><figcaption>Overall architecture of the simple NN</figcaption></figure><h4>Error function and backpropagation</h4><p>But wait! How do we know exactly the values of the matrix W? Those values are called weights and the neural network “learns” them during the training process. For that we have to know the real answer for at least the subset of the pictures, so we would be able to compare the result of the NN to the real label. At first FC weights are initiated randomly or according to some distribution. Then NN calculates the result for the picture and we compare it to the real label using the error function. It shows how close the NN answer is to the real one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*pCgtTQgL25mWMF6771jc3g.png" /></figure><p>And I also want to have these restrictions on the error function. The closer NN is to the real answer the smaller the result of the function. It should be non-negative and the derivative of this function should exist. Additionally, I want that every layer of the NN has a derivative as well.</p><p>Finally, we have all the requirements in place to train the NN. At first, I do forward propagation and calculate the error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qm-43O6XfMORtDwY3CSEng.png" /><figcaption>Forward pass</figcaption></figure><p>Then I can calculate the partial derivative for each weight in the NN and use it to minimize the overall error function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*Jkw3zZ8kcDBvPykzm7gxmg.png" /><figcaption>Partial derivative</figcaption></figure><p>This partial derivative indicates the direction in which ω needs to be changed to increase L. So making change in the opposite way will decrease L. This is called the backpropagation algorithm.</p><h4>Conclusion</h4><p>In this post, I tried to describe how the neural network works in general terms, what parts it consists of, and what conditions these parts must satisfy in order for the neural network to work.</p><p>In the next article, I want to delve into mathematics and show exactly how a neural network works from the inside.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15931de8579b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time series forecasting with ETNA: first steps]]></title>
            <link>https://medium.com/its-tinkoff/time-series-forecasting-with-etna-first-steps-dfaf90c5b919?source=rss-27cd59fddc81------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfaf90c5b919</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Andrey Alekseev]]></dc:creator>
            <pubDate>Fri, 08 Apr 2022 10:54:55 GMT</pubDate>
            <atom:updated>2022-04-08T10:54:55.384Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*gWm3LBeGqa73wJioadWuKw@2x.png" /></figure><p>Hi! My name is Andrey, I’m a development lead of the <a href="https://l.tinkoff.ru/etna_medium_github">ETNA</a>, time series forecasting package. We already showed how to start forecasting time series with ETNA in our last article here at Medium.</p><p><a href="https://medium.com/its-tinkoff/forecasting-with-etna-fast-and-furious-1b58e1453809">Forecasting with ETNA: Fast and Furious</a></p><p>In this tutorial, I want to show how to use ETNA for simple time series analysis and introduce several feature engineering techniques already built-in in ETNA. I will be using the <a href="https://www.kaggle.com/c/tabular-playground-series-jan-2022">Tabular Playground Series with the Jan 2022</a> dataset from Kaggle as an example.</p><p>In this Kaggle competition, researchers should predict merchandise sales — mugs, hats, and stickers — at two imaginary store chains across three Scandinavian countries. The forecast is expected to be a year in advance and the quality of predictions is measured by the SMAPE metric.</p><h3>Load the data</h3><p>First of all, let’s load the data and take a look at it. I will use Pandas.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fa71ec33cab0d90b4607cee4b5f2d40/href">https://medium.com/media/7fa71ec33cab0d90b4607cee4b5f2d40/href</a></iframe><p>There are several columns: date, num_sold, country, store, and product. The description of the competition said that we need to forecast the country-store-item combination. We should find out how many combinations we have. Also, we need to make our dataset suitable for the ETNA package. ETNA expects the dataset to have a special column that specifies different time series. For that I will create a new column called “segment” — this is the name ETNA expects. The segment is a combination of country, store, and product columns. This is exactly what we need to forecast. And finally, let’s check how many combinations we got. Turns out 18: there are 3 products in 2 store chains, each of which is located in 3 countries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/daac9690805aac6824321394952e0376/href">https://medium.com/media/daac9690805aac6824321394952e0376/href</a></iframe><p>Let us look at what we got:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdc4dea77002f026591bbf2d01b9998f/href">https://medium.com/media/bdc4dea77002f026591bbf2d01b9998f/href</a></iframe><p>Looks a lot better, but the dataset is still a few steps away from the ETNA format. I’m going to fix that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8eaeb0b064a1ceb7535cdfba52a7d0b5/href">https://medium.com/media/8eaeb0b064a1ceb7535cdfba52a7d0b5/href</a></iframe><p>This is the format that is required for the ETNA package. Target — is a reserved name for the column we want to forecast. Timestamp — is a column where we store our timestamp. We don’t call it “date” because ETNA knows how to work with hourly and weekly data as well.</p><p>To sum up, timestamp, segment, and target are compulsory columns for ETNA. However, ETNA can also work with exogenous data and we will show how to use it correctly in one of the next tutorials.</p><p>We are almost ready to analyze and forecast the data. To do that we need to put it inside a TSDataset class. TSDataset expects data in a multi-index format. But we can easily reformat the data with the method of TSDataset: <em>TSDataset.to_dataset()</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ac36fbb55b348fd44a6278fe9afc644/href">https://medium.com/media/6ac36fbb55b348fd44a6278fe9afc644/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kroNHrRkd725WPdDw3Xx5A.png" /><figcaption>In our practice, it is easier to store data in the following format.</figcaption></figure><p>Finally, we create a new TSDataset instance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef8bad25868e8bdf774dc54abffc0db4/href">https://medium.com/media/ef8bad25868e8bdf774dc54abffc0db4/href</a></iframe><p>Apt to ask why we need to do so much work to convert our data from one format to the other? TSDataset allows convenient indexing, validates data, works with other parts of the package, has built-in analysis methods, and generates necessary features for future prediction. I will show how TSDataset can be used in time series analysis and forecasting.</p><p>And here is the first TSDataset’s essential feature — indexing. I can easily index by timestamp, segment, and individual column name.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/526c950f0eab83179f6382ca1c2308a4/href">https://medium.com/media/526c950f0eab83179f6382ca1c2308a4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*wKny1-4cuFbV5J_itz9W8A.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3236887a08e778645b06601c4f8156eb/href">https://medium.com/media/3236887a08e778645b06601c4f8156eb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*l3If6tjGT8q2iKeXgjFQjg.png" /></figure><h3>Time series analysis</h3><p>After I’ve put my data in a TSDataset, I’d like to look at it. First of all, I will run a describe method. It will show me basic info about my dataset: how many time series I have, start date, end date, series length, and a number of missing values.</p><p>As we can see this dataset is perfect. No missing values and all series end at the same date.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd06c4ce280212c5cf241db05d3cae86/href">https://medium.com/media/cd06c4ce280212c5cf241db05d3cae86/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xFidmK8q6WyYrF3km-EQTg.png" /></figure><p>Also we can visualize the data, by using a plot method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd1f06ad071fee06c37048adbf05a885/href">https://medium.com/media/dd1f06ad071fee06c37048adbf05a885/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIQ_jGcqoPakPM3WIijw5g.png" /><figcaption>The series has yearly seasonality. Peaks are arranged not in a random fashion. It could be holidays of some sort.</figcaption></figure><p>Let’s zoom in to see if the series has monthly or weekly seasonality.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08b865a9302d509e05ddfb163750e173/href">https://medium.com/media/08b865a9302d509e05ddfb163750e173/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j_zi3vz3lxEsCcDMxa_3ZQ.png" /><figcaption>By analyzing four segments, it is safe to say that weekly seasonality is present.</figcaption></figure><h3>Feature generation</h3><p>In this section I’d like to focus on features that could be applied to time series.</p><h3>Lags</h3><p>Lag is some previous value of the time series. For example, the first lag is yesterday’s value, and the fifth lag is the value five days ago. Lags are essential for regression models, like linear regression or boosting, because they allow these models to grasp information about the past.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lfY6TSh9iS8RO96LLNrBQ.png" /></figure><p>The model tries to predict the next value by looking at the lags — the previous values of the series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pXEfrKVopuDXGdXlD2X3Yw.gif" /></figure><p>Let’s try to use ETNA’s LagTransform for our data. First of all, we import it from the ETNA package. Then I will try to generate the first lag by setting parameter <em>lags=[1]</em>. As we can see, there is a new column with the lag we wanted. LagTransform is applied for all time series in our TSDataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/59be86b968cfbe816d38da2b84c6cd74/href">https://medium.com/media/59be86b968cfbe816d38da2b84c6cd74/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*qbxBywjiQlkGxz66fk_DBQ.png" /></figure><p>If we need to generate several lags it can be easily done with ETNA. All we need is to specify all the necessary lags in a list.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a16dee86627ec365074e7f1e3bf7ebe/href">https://medium.com/media/4a16dee86627ec365074e7f1e3bf7ebe/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*IPFSPbV1-14KwUyPPJKpMg.png" /></figure><p>And for more advanced users, lags parameters could be set by using range function or list comprehension.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/39813245596d754a1001bae626a09d98/href">https://medium.com/media/39813245596d754a1001bae626a09d98/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*XJYIMv7SVr1AFEXiaW7EmA.png" /></figure><h3>Statistics</h3><p>Statistics are another essential feature. It is also useful for regression models as it allows them to look at the information about the past but in different ways than lags. I’m talking about the mean, median, standard deviation, minimum and maximum on the interval.</p><p>I will show how all this works using MeanTransform. MeanTransform is also easy to use. To make it up and running we just need to specify one parameter — window. Let’s do that!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6699aa2e4f79e31f03dc4cf8f89e6a7a/href">https://medium.com/media/6699aa2e4f79e31f03dc4cf8f89e6a7a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*vaJ62rFtkdQTpaQFObmA_g.png" /></figure><p>But what does it mean? The window specifies how many previous values we want to average. Let’s understand it step by step.</p><p>Step 1. MeanTransform wants to average 5 values previous to value 18, including it. But we have no values before it. So the average value is 18.</p><p>Step 2. Before 26, including itself, we have only values 26 and 18. So MeanTransform averages only them. And so on. Up until step 5.</p><p>Step 5. At this step MeanTransform averages all five values.</p><p>For the next steps, MeanTransform averages only five previous values including the current one. Why five? Because this is the window that we chose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhsbyVd9ZsoqkwSpYxA97w.gif" /></figure><p>This feature is useful when we want to pass the info about mean value during the last week or month. But we also may want to use the information about the average value of every seventh day for example. It can be useful if we expect weekly seasonality in our series. Fortunately, we can easily do it with ETNA. Here is the example for every second day:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecf750df439acaa0fa1f5bac60f8428c/href">https://medium.com/media/ecf750df439acaa0fa1f5bac60f8428c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*lZfqT27UFhDQ7vjnBpu3xA.png" /></figure><p>Here we indicated that we want to average 2 points that go in increments of 2. Let me make that clear with another gif:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*loLHycPWMGEOQn70SvWHFw.gif" /></figure><h3>Dates</h3><p>The time series also has the timestamp column that we have not used yet. But date number in a week and in a month, as well as week number in year or weekend flag can be really useful for the machine learning model. And ETNA allows us to extract all this information with DateFlagTransform. Let’s apply it to our dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b955d076a63cd2aca8d231f06d9ad2d4/href">https://medium.com/media/b955d076a63cd2aca8d231f06d9ad2d4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEECSpDWweNJjQp6PRacKw.png" /></figure><p>We do not need to set the column name we want to apply this transform to. Setting up flags that we are interested in will be enough. There are 9 available flags, I chose only 4 of them.</p><h3>Holidays</h3><p>HolidaysTransform also works with the timestamp column. It uses a holiday package that has knowledge of the holidays for most of the countries. Users need to specify only the ISO-code of a country and HolidaysTransform is good to go.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed8f1005640bb98159d2d83342a1093a/href">https://medium.com/media/ed8f1005640bb98159d2d83342a1093a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*T7BLZYruGk9SIMiBcGoVdw.png" /></figure><h3>Logarithm</h3><p>I’ve already told you about transforms that use target columns and about transforms that use timestamp columns to create new features columns. However, there is another type of transform that alters the column itself. We call it “inplace transform”. The easiest is LogTransform. It logarithms values in a column.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aec8d8a97a0d666027929c32b42ff23/href">https://medium.com/media/1aec8d8a97a0d666027929c32b42ff23/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*PKnLJalkcFZjf7zcCdIjlA.png" /></figure><p>We can use the inverse_transform method to unapply this transform.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/091bc3f8cd63aec54d845cee3c72ee6f/href">https://medium.com/media/091bc3f8cd63aec54d845cee3c72ee6f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*MYCQtXWAKXpCDrnviMDiUw.png" /></figure><p>And if we want to get the logarithm of our values and still keep the original column we can set the parameter <em>inplace=False</em> for the transform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*uPXVBWFYppd5x0zKudlQHA.png" /></figure><h3>Forecasting</h3><p>We’ve learned a lot about transforms and how to use them. Let’s use this knowledge to predict sales of the merchandise from Kaggle Tabular Playground Series — Jan 2022.</p><p>I will use a linear regression model. And that’s why I need to generate features. The difference between autoregression and regression models we will cover in next tutorials.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1ecd60b4cdc01093457d04cb6ea8f4f/href">https://medium.com/media/d1ecd60b4cdc01093457d04cb6ea8f4f/href</a></iframe><p>As you know we need to forecast sales year in advance. So the horizon of our prediction will be 365. And, by doing prior analysis we know that the series has weekly seasonality, I will use lags from 365 to 371. But why can’t we just use lags from 1 to 7? Because the model won’t be able to forecast anything due to the fact that it won’t get feature values for all horizons more than 7. Conclusion: the horizon defines the smallest lag that we can use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlQAnxtfnSl4l6HObHB1vQ.png" /></figure><p>Let’s visualize it for a smaller horizon. Imagine that the horizon is 3. Model trains on lags and tries to predict the series value. If we use lag 1 and 2, we get our series shifted by 1 and 2. So at horizon 2 we won’t get values for lag 1, and at horizon 3 we won’t get values for lag 1 and lag 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wghUiuUs4Vd198ne63hM9w.png" /></figure><p>So the smallest usable lag is 3. Same logic applies to statistics, so I apply MeanTransform to lag 365. And I want to consider weekly seasonality in MeanTransform, that’s why I set seasonality to 7 and window to 104. It means that I want to average every weekday for the last 2 years. For mondays it is average of 104 prior mondays, for tuesdays it is average of 104 prior tuesdays and so on.</p><p>Let’s look at the holidays. Competition says that stores are located in three countries. So I set 3 holiday features for these countries. However I expect sales to be different after the holidays, so I apply lags to holiday features as well.</p><p>And finally, let’s move on to training.</p><p>LinearPerSegmentModel is a model I use. PerSegment means that for every segment will be created and fitted separate linear model.</p><p>Pipeline is supposed to make transforms and models work together. It also allows running backtest for the series. It makes the researcher’s life easier.</p><p>SMAPE is a metric. We could interpret it as a percentage error of our model prediction.</p><p>I run a <a href="https://l.tinkoff.ru/etna_medium_backtest_example">backtest</a> to check the model’s performance, calculate metrics and plot the forecasts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ff23434fe558d468e61fed6f10621b6/href">https://medium.com/media/0ff23434fe558d468e61fed6f10621b6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpbJtnCIP5MFr3NeFAbsNg.png" /></figure><p>I use the plot_backtest function. This is one of ETNA’s plotting functions. It allows us to compare real and predicted values. Seems like our model is okay. Let’s fit on the whole dataset and forecast the future! For that I use an already configured pipeline.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4532d1f9618ba6e5b9245bf657253464/href">https://medium.com/media/4532d1f9618ba6e5b9245bf657253464/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30WKtYwfw6QoN2ggpgKUtg.png" /></figure><p>The final step is to convert the forecast to Kaggle submission format.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29a99eaafe3150ca1f0550de1fd16817/href">https://medium.com/media/29a99eaafe3150ca1f0550de1fd16817/href</a></iframe><h3>Instead of conclusion</h3><p>We also created a robust <a href="https://l.tinkoff.ru/etna_medium_kaggle_tps-jan-2022">Kaggle-notebook for the Tabular Playground Series— Jan 2022 competition</a>.</p><p>In this tutorial we have learned:</p><ul><li>how to use TSDataset</li><li>how to use lags, statistics, date flags, holidays flags and logarithm</li><li>how to use models and pipelines</li><li>how to use backtest and plots</li><li>how to forecast a series</li></ul><p>In next tutorials we will talk about more advanced transforms and time series analysis tools. Stay tuned :-)</p><p>If you‘d like to propose a new feature or ask a question, <a href="https://l.tinkoff.ru/etna_medium_github">welcome to our GitHub page</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfaf90c5b919" width="1" height="1" alt=""><hr><p><a href="https://medium.com/its-tinkoff/time-series-forecasting-with-etna-first-steps-dfaf90c5b919">Time series forecasting with ETNA: first steps</a> was originally published in <a href="https://medium.com/its-tinkoff">IT-компании Тинькофф</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>