<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by 1StepFromMagic on Medium]]></title>
        <description><![CDATA[Stories by 1StepFromMagic on Medium]]></description>
        <link>https://medium.com/@1StepFromMagic?source=rss-6fb54d99528a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by 1StepFromMagic on Medium</title>
            <link>https://medium.com/@1StepFromMagic?source=rss-6fb54d99528a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:10:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@1StepFromMagic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A/B-testing checklist]]></title>
            <link>https://medium.com/@1StepFromMagic/a-b-testing-checklist-f683935776e1?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f683935776e1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[checklist]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[experiment]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 15:41:43 GMT</pubDate>
            <atom:updated>2023-11-06T15:41:43.935Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is a checklist that highlights important technical aspects of conducting A/B-tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CNlVpKHlq4mhXHMN" /><figcaption>Photo by <a href="https://unsplash.com/@jdent?utm_source=medium&amp;utm_medium=referral">Jason Dent</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’m not going to describe why A/B-tests are crucial. The only thing I want to mention as a motivation is that A/B-test is usually the best among available methods for establishing causal relationships between some events.</p><p>So, let’s jump straightly to the technical aspects.</p><h4>Correct interpretation of statistical results</h4><ul><li>It’s not correct to jump to conclusions based on statistic significance only. It’s important to consider statistical power as well.</li><li>It’s important to remember what the “p-value” is in order to interprete it correctly. P-value is the probability to get the same or even greater result (in comparison with an observed one) assuming the null hypothesis is correct.</li><li>You shouldn’t stop an experiment at the moment when you like its current result.</li><li>You should make adjustments when testing multiple hypotheses. Use stricter significance threshold while doing multiple simultaneous comparisons</li></ul><h4>Using confidence intervals</h4><ul><li>It’s incorrectly to consider confidence intervals separately for the control and test groups and to assume that if they overlap, the change has no statistically significant effect.</li><li>However, if they do not overlap, there is a statistically significant difference with the specified p-value threshold.</li></ul><h4>Independence of experimental objects</h4><ul><li>Objects are often not independent. For example, Internet store customers may be acquainted and share information about products or customer experience with each other. This might not be a problem, but you can be certain only after analyzing historical data.</li><li>Object dependencies can be reduced by isolating related objects, for example, during randomization.</li></ul><h4>Ability to generalize/divide results by segments</h4><ul><li>It’s not advisable to generalize conclusions obtained in one segment to another. It’s better to conduct a new experiment. This is also applicable to conducting a similar experiment at different time intervals.</li><li>Segmenting the sample and analyzing them separately can lead to insights (ideas for new experiments). So you can do this.</li></ul><h4>Proper randomization</h4><ul><li>Users should always see the same variant.</li><li>The combination of a user and a variant should not reveal information about the combination of another user and their variant.</li><li>You can run experiments sequentially, independently dividing objects into groups using a hash function.</li><li>You can run experiments simultaneously, but ensure that experiments do not interfere with each other. Users can participate in all experiments simultaneously, but be careful with combimations of experiments’ variants because they may lead to a poor user experience. The combinations can be isolated by separating “layers” of non-interfering experiments or coloring the experiments’ graph.</li></ul><h4>Sample size sufficiency</h4><ul><li>A practical rule for the minimum necessary number of observations (N):</li></ul><blockquote>N = 355 * s², where s is the coefficient of skewness of the sample distribution of the variable Y</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*qn6ws2bHoFD5i4Qk1dggXA.jpeg" /></figure><ul><li>However, if absolute value of s is equal to or less then 1, this approach doesn’t work, and you should use the following formula:</li></ul><blockquote>N &gt;= 16*sigma² / delta², where sigma² is the sample variance, and delta is the minimum effect you want to detect</blockquote><ul><li>To reduce the sample size, in the first case, you should reduce skewness, for example, by replacing Y with Y-Y’ (where Y’ is a predicted value), and in the second case, reduce variance or increase the minimum detectable effect.</li></ul><h4>Choosing the level of statistical significance</h4><ul><li>It’s a good idea to categorize all controlled effects into three groups (those the experiment should impact, those it might indirectly impact, and those it’s unlikely to impact) and set a significance level for each group separately.</li><li>The more you believe in the truth of the null hypothesis, the lower the significance level you should use.</li></ul><h4>Reducing variance</h4><ul><li>Removing outliers is important.</li><li>Replace the indicator with one with lower deviation (e.g., replacing the sum of purchases with the fact of purchase).</li><li>Use stratification.</li><li>Randomize objects at a more detailed level.</li><li>Conduct a paired experiment showing both the test and control variants to the same user.</li><li>Combine control groups from different experiments into one large group.</li></ul><h4>Choosing the experiment duration</h4><ul><li>The duration depends on seasonality, the effects of primacy and novelty that can have place at the beginning of your experiment.</li><li>In most cases, 1–2 weeks are sufficient to capture short-term effects.</li><li>Long-term effects (&gt;3 months) may not necessarily be significant. Short-term volatility tends to suppress long-term trends.</li></ul><h4>Monitoring the progress of the experiment</h4><ul><li>Before conducting an A/B-test, it’s advisable to run an A/A-test to check that groups are homogeneous.</li><li>Running A/A-tests in parallel with A/B-tests is useful for detecting unaccounted effects and deal with them as soon as possible.</li><li>Controlling Sample Ratio Mismatch (SRM). Even a 0.99 or 1.01 ratio with a large number of objects can indicate a serious problem. Consider stopping the experiment, identifying the error, and starting over.</li></ul><blockquote>SRM is the ratio of the number of objects between variants during the experiment</blockquote><p>While conducting your experiments, remember Twyman’s law. The more unusual or interesting the data, the greater the likelihood that it was obtained due to an error.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f683935776e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3D World Representation]]></title>
            <link>https://pub.aimind.so/3d-world-representation-5c6ab065f220?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c6ab065f220</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[3d-modeling]]></category>
            <category><![CDATA[3d]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 19:47:36 GMT</pubDate>
            <atom:updated>2023-08-25T15:39:42.357Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article there are described popular ways of 3D objects’ representation methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SKAH8KOtUcAZ5bkD" /><figcaption>Photo by <a href="https://unsplash.com/@shijub?utm_source=medium&amp;utm_medium=referral">Shiju B</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We live in a three-dimensional world, but we frequently store information about it in a 2D form (using different projections, for example, in the form of photos or drawings). From this point of view the task of transition between 2D and 3D worlds naturally arises. And if everything is more or less clear with dimensionality reduction, then restoring a 3D image based on 2D information is a non-trivial task.</p><p>However, both tasks start with an understanding of how a 3D world can be represented in a computer’s memory. The most precise and universal methods are “Point Cloud”, “Level Set” and “Mesh”. These approaches do not make additional assumptions and do not impose additional restrictions on the model of the world. Let’s understand what each of them means:</p><ul><li><strong>Point Cloud</strong> is (surprisingly) a set of points. For exapmle, it’s what a Lidar returns, when it sends lazer rays arout itself, and the rays return after facing with an obstacle. Thus we have a bunch of uniformly distributed rays in the space, and for each ray we have a value that represents the distance between the sensor and the obstacle. In other words we have coordinates of surfaces around in the senser’s coordinate system. And the coordinates can be easily translated into another coordinate system. On the one hand it gives us detailed information, but on the other hand, there are a lot of points, and their storage requires a large amount of memory.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X8-i6gYT8fda-DLFMe8tvw.jpeg" /><figcaption>By Wikipedia Commons</figcaption></figure><ul><li><strong>Level Set</strong> is rather similar to the idea of geographical maps or Point Cloud, where the points are not uniformly distributed, but located at a fixed distance from each other along one of the dimensions forming layers. For example, if you want to depict a landscape, you can get rid of one of the dimensions (height) by imagining that you are looking at it from a bird’s eye view. Doing so, you do not lose information about the height, you can make reasonable assumptions about it based on density of the level lines. Thus, using this approach you will just save several contours of a complex shape, where each line corresponds to a certain height, for example, 1000 m, 2000 m, etc.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ1gcoXD7HXvKWeF-KKq5g.jpeg" /><figcaption>By Wikipedia Commons</figcaption></figure><ul><li><strong>Mesh</strong> is also a kind of Point Cloud, but the points here are located in such a way that they could be combined into triangles that would describe the surface with a minimum loss of information about its shape. This allows you not to abandon one of the dimensions (as in the Level Set), but the accuracy of the data is also reduced. The less points exist in the mesh, the less precise the data is.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*1o0nqDn_Q_BU0d6x_YjOkA.png" /><figcaption>By Wikipedia Commons</figcaption></figure><p>But there are such objects when none of these three ways of displaying is the best choice for. For example, for a cube image, storing in memory information about millions of points from the Point Cloud is wasteful. In fact, such detailization is not needed. It is enough to use a simpler data representation. I think it’s obvious that there are different levels of simplification. Here are a few more approaches at different levels of abstraction:</p><ul><li><strong>Planes</strong>. In general, a surface of any complexity can be defined by a set of arbitrary planes (more precisely, by combining their parts). If the surface is extremely complex, this representation degenerates into a Point Cloud or a Mesh. If the shape is simple (for example, it can be a cube), then it is enough to set only a few (in case of the cube, it’s six) planes in order to keep full information about an object.</li><li><strong>State Center World (SCW)</strong>. This representation is very similar to the Planes, but if there are several planes directed (or slanted) in the same way, they will be assigned to a single entity. The entity here is their common direction. In the case of the cube, three directions will be found.</li><li><strong>Manhattan World (MW)</strong>. Perhaps, this is the maximum simplification. This approach assumes that the scene has a Cartesian x, y, z coordinate system and the image statistics will be determined by the alignment of the viewer with respect to this system. In the cube example, this representation of the world guarantees that those three directions, that we’ve counted in the previous approach description, are orthogonal, while SCW did not impose such a requirement. But despite the extra terms, the Manhattan world assumption is plausible for indoor and outdoor city scenes. In addition, people got used to imagine the world in this way (a number of optical illusions are associated with this). Therefore, this model of the world is so popular in the difficult Computer Vision tasks.</li></ul><p>But a “world” rarely consists of one object. Moreover, these objects can be located at different angles of rotation or in different planes. A great example is a curved street of the city, where the houses are turned following the bend in the road. Therefore, it is important to talk about intermediate variations between MW and SCW:</p><ul><li><strong>Atlanta World (AW)</strong> allows objects to be rotated in the same plane</li><li>Whereas <strong>Mixture of Manhattan Frames (MMF) </strong>allows both rotations and shifts in more than one plane</li></ul><p>In many scientific articles related to 3D Computer Vision tasks, you can see an explicit mentioning that the authors use Manhattan World Assumption (or refuse it). Now you understand what it means.</p><h4>A Message from AI Mind</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*5Wm7sOfTpe5DEbhg.gif" /></figure><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://pub.aimind.so/">AI Mind Publication</a></li><li>🧠 Improve your <a href="https://www.aimind.so/prompt-generator?utm_source=pub&amp;utm_medium=message">AI prompts effortlessly and FREE</a></li><li><strong>🧰 Discover </strong><a href="https://www.aimind.so/?utm_source=pub&amp;utm_medium=message"><strong>Intuitive AI Tools</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c6ab065f220" width="1" height="1" alt=""><hr><p><a href="https://pub.aimind.so/3d-world-representation-5c6ab065f220">3D World Representation</a> was originally published in <a href="https://pub.aimind.so/">AI Mind</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizating areas of given elevation]]></title>
            <link>https://medium.com/@1StepFromMagic/visualizating-areas-of-given-elevation-1074ec427969?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1074ec427969</guid>
            <category><![CDATA[srtm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[nasa]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Sun, 21 May 2023 14:54:18 GMT</pubDate>
            <atom:updated>2023-05-21T14:54:18.574Z</atom:updated>
            <content:encoded><![CDATA[<h4>with help of Folium and SRTM</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kzgptsml_qz3hvDO" /><figcaption>Photo by <a href="https://unsplash.com/@therawhunter?utm_source=medium&amp;utm_medium=referral">Massimiliano Morosinotto</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The story starts with a man and his wish to do some physical exercises. Different training overload can be achieved on a different elevation level above the sea. So he was curious where is the closest appropriate place of particular elevation could be found.</p><p>And here we are, creating a map for him. At the same time the goal of the article is to show how easily such a task can be managed. First of all, I’ll talk about available data, and move to visualization geodata with help of Folium after that.</p><h3>Getting data</h3><p>If you’re looking for a geographical data related to the Earth surface, the first source of information that I would recommend you to check is an open data provided by NASA. You can explore the available datasets using this <a href="https://www.earthdata.nasa.gov/">link</a>.</p><p>Among them there is such an amazing dataset as SRTM which is an abbreviation for “Shuttle Radar Topography Mission”. It was an international research project that created a digital elevation model of our planet. The model covers land surface between 56°S and 60°N.</p><p>The elevation model is split into “squares”, each covering one degree of latitude and one degree of longitude. Each square is named according to its south western corner. For example, “N05W075.hgt”. The resolution of the data can be either 1 arcsecond (30 m along the equator) or 3 arcsecond (90 m along the equator). The versions are called SRTM1 and SRTM3 respectively. The square is related to the SRTM1 has 3,601 rows, each consisting of 3,601 points. The SRTM3’s square includes 1,201*1,201 points.</p><p>The dataset can be downloaded after a short signing-up on the same web page. It’s also important that you can download the needed data (several squares) just selecting them and pushing the only button. Easy.</p><p>The web page you’re interested in should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jb7v_Lfy_8Zxh6fzQhoMEw.jpeg" /><figcaption>The most important things are highlighted in yellow.</figcaption></figure><p>It’s just amazing that such the data is available to anyone nowdays.</p><h3>Preprocessing data</h3><p>When you’ve got the data, you just need to read it and visualize.</p><h4>Reading file</h4><p>The files have an unusual extension “.hgt” that is specific for SRTM data and implies on the fact that such a file includes elevations (heights).</p><p>You can read the file using the following function in Python:</p><pre>import os<br>import numpy as np<br>import zipfile<br><br>SRTM_DICT = {&#39;SRTM1&#39;: 3601, &#39;SRTM3&#39;: 1201}<br><br>def get_raw_elevation_data(path, filename, srtm_type=&#39;SRTM1&#39;):<br><br>    SAMPLES = SRTM_DICT[srtm_type]<br><br>    if filename[-4:] == &#39;.hgt&#39;:<br>        with open(os.path.join(path, f&#39;{filename[:7]}.hgt&#39;), &#39;rb&#39;) as f:<br>            all_elevation_data = np.fromfile(<br>                f,  # binary data<br>                np.dtype(&#39;&gt;i2&#39;),  # data type<br>                SAMPLES * SAMPLES  # length<br>            ).reshape((SAMPLES, SAMPLES))<br>    elif filename[-4:] == &#39;.zip&#39;:<br>        with zipfile.ZipFile(os.path.join(path, filename), &#39;r&#39;) as zf:<br>            zf.extractall(path)<br>        with open(os.path.join(path, f&#39;{filename[:7]}.hgt&#39;), &#39;rb&#39;) as f:<br>            all_elevation_data = np.fromfile(<br>                f,  # binary data<br>                np.dtype(&#39;&gt;i2&#39;),  # data type<br>                SAMPLES * SAMPLES  # length<br>            ).reshape((SAMPLES, SAMPLES))<br>    else:<br>        raise ValueError(&quot;File extension should by either .hgt or .zip&quot;)<br><br>    return all_elevation_data</pre><p>Pay attention that it’s important to choose a correct dataset resolution, because otherwise you’ll get a wrong matrix with elevations. <br>Let’s demonstrate it. I consider a file “N05W075.hgt”. Initially the file is from the SRTM1.</p><p>In this case running the following code I’ll get the following image:</p><pre>from matplotlib import pyplot as plt<br><br>path = &#39;./data&#39;<br>filename = &#39;N05W075.hgt&#39;<br>elevations = get_raw_elevation_data(path, filename, srtm_type=&#39;SRTM1&#39;)<br><br>plt.imshow(elevations, cmap=&#39;hot&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*rMWj_kjoKG-JypRVyIxZ3w.jpeg" /></figure><p>Let’s compare the image with the NASA web page mentioned above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*uKeyDN5O4Is1kB4l15mtSQ.jpeg" /></figure><p>Seems pretty similar. It means that we do everything correctly. But when I replace srtm_type=“SRTM1” on srtm_type=“SRTM3”, the result is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*NsJn2jqqOqFKm3CILHh1Tg.jpeg" /></figure><p>This one is far from the ground truth. Thus, be attentive!</p><h4>Filtering data</h4><p>The previous function returns a matrix of numbers which equal to elevation in meters above the sea level. Filtering such a matrix is an extremely easy task which can be managed with help of the following function:</p><pre>def collect_dots_with_appropriate_elevation(elevation_matrix, lat_list, lon_list, min_elevation=None, max_elevation=None):<br>    if min_elevation is not None:<br>        if max_elevation is not None:<br>            bool_ele = (elevation_matrix &gt;= min_elevation) &amp; (elevation_matrix &lt;= max_elevation)<br>        else:<br>            bool_ele = elevation_matrix &gt;= min_elevation<br>    else:<br>        if max_elevation is not None:<br>            bool_ele = elevation_matrix &lt;= max_elevation<br>        else:<br>            bool_ele = [[True]*len(elevation_matrix)]*len(elevation_matrix)<br><br>    return [<br>        (round(lat_list[lat_ind], 4), round(lon_list[lon_ind], 4))<br>        for lat_ind, bool_row in enumerate(bool_ele)<br>        for lon_ind, bool_elem in enumerate(bool_row)<br>        if bool_elem<br>    ]</pre><p>The function works well in case of both one-sided and two-sided limitations. For example, if you’re interested in points above the particular elevetion or between some minimum and maximum values.</p><p>But it requires to make one step ahead, run the function that makes a net of latitude and longitude coordinates related to the elevation matrix. It works with a standard SRTM filename’s format.</p><pre>def get_coords_net_for_file(filename, shape=3601):<br>    ns, lat, ew, lon = filename[0], int(filename[1:3]), filename[3], int(filename[4:7])<br>    if ns == &#39;S&#39;:<br>        lat = -lat<br>    if ew == &#39;W&#39;:<br>        lon = -lon<br><br>    return np.linspace(lat, lat+1, shape), np.linspace(lon, lon-1, shape), (lat+0.5, lon-0.5) </pre><h4>Visualization</h4><p>And finally, let’s show the result on the geographical map. For this purpose I’ll use heatmap from Folium library. The following function creates the heatmap and save it as a HTML-file on your computer:</p><pre>import folium<br>from folium.plugins import HeatMap<br>import webbrowser<br><br>def draw_heatmap(clusters, center, html_page=’./outputs/default.html’):<br>  m = folium.Map(location=center, zoom_start=10)<br>  for cluster in clusters:<br>    HeatMap(cluster, min_opacity=1e-5).add_to(m)<br>  m.save(html_page)</pre><p>But it’s not necessary to find and open the file manually, you can ask python to open it after the saving.</p><p>Then the overall pipeline looks like this:</p><pre>filename = &#39;N05W075.hgt&#39;<br>path = &#39;./data&#39;<br><br>lat_list, lon_list, center = get_coords_net_for_file(filename, shape=3601)<br>elevations = get_raw_elevation_data(path, filename, srtm_type=&#39;SRTM1&#39;)<br>points = collect_dots_with_appropriate_elevation(<br>  elevations, <br>  lat_list, lon_list, <br>  min_elevation=1500<br>)<br><br>html_page = r&#39;.\outputs\heatmap.html&#39;<br>draw_heatmap([points], center, html_page=html_page)<br>webbrowser.open(html_page)</pre><p>And it allows us to see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gV-rsQXSJjnsnywpD_Mgtg.png" /></figure><p>Pay attention that you can change the map’s scale, but it will influence on the shape of highlighted areas, because they become more or less detailed.</p><p>If you have a particular problem or question, you’d like to read about, feel free to send me a message with your request.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1074ec427969" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Common sampling mistakes and how to avoid them]]></title>
            <link>https://medium.com/@1StepFromMagic/common-sampling-mistakes-and-how-to-avoid-them-ec0142627d1b?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec0142627d1b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[sampling]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Fri, 22 Jul 2022 18:17:33 GMT</pubDate>
            <atom:updated>2022-07-22T18:17:33.172Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WpEuZYNgShJtA-Oa" /><figcaption>Photo by <a href="https://unsplash.com/@holatgr?utm_source=medium&amp;utm_medium=referral">Tomas Gonzalez de Rosenzweig</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Modeling, forecasting, analytics are important, but they won’t work if they are fed non-representative data. As they say, garbage in, garbage out.</p><blockquote><strong>Representativeness </strong>is a property of a sample selected from a general population which means that this sample has the same properties as the general population. In other words, the sample can represent the population.</blockquote><p>How to make the sample representative? The first idea is to use data of the entire population. The entire population definitely can represent itself. But if the population is more or less large (which is usually assumed in Data Science tasks), and the data, that characterize the population, is even larger. Taking it into account, you can imaging that it can be hardly impossible to collect information on the entire population as well as rather expensive to store and to process. So, it’s better to collect just a representative sample.</p><p>In the article I’ll describe what mistakes are the most common while collecting a sample. I’ve already <a href="https://medium.com/@1StepFromMagic/how-to-deal-with-outliers-in-time-series-c6d5e03298f">written</a> about data quality related to time series, but in the article is about sampling independent observations.</p><h4>Non-random sampling</h4><p>The simplest mistake is related to non-random (bias) selection of observations. Actually, random choice does not obligatory return an unbiased sample, but the probability of such an event is low.</p><p>We all make this mistake in everyday life when we generalize our experience or the experience of our friends to some conclusion about a whole population. For example, “the election results are definitely rigged, because none of my acquaintances voted for the chosen candidate.” Usually, your acquaintances have the same characteristics as you are (not all not some of them), and represent a particular social group, whereas there are other group, you’re not in touch with. Thus, your sample is biased.</p><p>At the same time there can be such sircumstances which don’t allow to reach the whole population and pick observations randomly. A classic example tells us about the World War II aircrafts. Researchers tried to determine what parts of the aircrafts should be additionally protected based on those planes which were available for observation (which returned to flight base after some fightings). And not immediately and not everyone realized that the sample is biased and it was necessary to strengthen those parts where there were no bullet holes, because if the plane returned, the damage was not critical.</p><h4>Choosing a specific time intervals</h4><p>This is a very logical kind of bias that is often overlooked. It’s often situation when the events which you observes (and collect data about) are seasonal. The seasonality can be annual, quarterly, monthly, weekly, daily, whatever. For example, your project is related to a restorant, and you try to improve a client experience. Those people who come to the restaurant at lunchtime and in the evening are completely different categories of customers. At lunchtime, clerks from neighboring offices are the most common visitors. They are very limited in terms of spare time, that’s why they are interested in getting their not too expensive business lunchs quickly and return to work. And in the evening it’s more probable to meet there couples, groups of friends who want to relax and as a result value calm environment. Do you feel the difference?So, if you need to conduct a survey of visitors, then you can not draw conclusions taking into account dinner visitors or evening visitors only. Both groups must be represented in a sample.</p><h4>Cherry-picking</h4><p>And again my exaple will be related to a customer survey because this is a very intuitively understandable way to collect information. Imagine that you need to collect information about potential customers of your offline pharmacy. You realize that the potential customers are people who live in the nearby buildings. So, collecting data based of catching passers-by seems like a good idea. You try to stop people on the street, but not everyone stops (some people are in a hurry, some others refuse to take the survey on principle). As a result, by the end of the day you have the required number of questionnaires in your hands, but all of them are from pensioners and lunatics who are lonely and therefore were happy to talk to you. Is it correct to stop at this stage? Will this sample be representative? Of course not, because not only pensioners live in the district, although they go to the pharmacy quite often. Be persistent and look for ways to get information from sources that are harder to reach in order to make your sample more balanced.</p><h4>Endless search (or Who is looking for something, will find something)</h4><p>Data scientists, analysts and so on believe that the more data they have the deeper and less expected conclutions they can make. And if they (we) thoroughly dig into the data, analyze it, twist it in this and that way, build “explaining” models in order to isolate new knowledge from the data, then we will definitely benefit the business. The good news is that, indeed, in the large datasets, you will certainly find something interesting. But there is a bad piece of news as well. You can’t be sure that the result you’ve found is systematic, and not a random outlier. So, the finding needs to be critically evaluated and verified. To order to do that, you need to select a control sample for comparison. For example, you can collect a comparable fresh dataset (the only changed factor is time when the data collected) and check whether the conclusion is still true there. Or you can do a permutation test (target shuffling, which assumes that you mix the target variable and check if the “ signal” in the data is not there after the operation).</p><h4>Stop experiment when its result looks promising</h4><p>At it was mentioned above, you can be interested in collecting data in order to cheching your hypothesis. It is very important to determine the sample size before the start of the data collection, otherwise there are temptations to end the process either too early (“well, I like my hypothesis, and I see 5 out of 5 respondents’ data supports the hypothesis, then let’s stop here, before someone contradicts it”) or too late. The last option can be very seductive in A/B testing (whatever statistical significance you define as a measure of expected results’ confidence, there will always can be collected such a huge sample that will provide you with this very degree of confidence, it just need to continue to collect the data).</p><p>So there is only one way out. You must determine parameters of data collection before the experiment. In case of A/B testing, you can decide that A is better than B if you see a statistically significant difference in conversion at the level of N according to the X method of evaluation and based on survey of K customers. Everything that is needed for the experiment is fixed here: the compared options, the sample size, the comparison method and the confidence criterion.</p><p>That’s it. Forewarned is forearmed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec0142627d1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to ensure data type correctness at Python]]></title>
            <link>https://faun.pub/how-to-ensure-data-type-correctness-at-python-906abdc49963?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/906abdc49963</guid>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-type]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Sat, 11 Jun 2022 19:11:48 GMT</pubDate>
            <atom:updated>2022-06-14T06:38:05.613Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GsMoPOhKFjzQJYqn" /><figcaption>Photo by <a href="https://unsplash.com/@santesson89?utm_source=medium&amp;utm_medium=referral">Andrea De Santis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Python is a programming language which is famous for its polymorphism. It means that the same operation will work differently with objects of different data types. It’s very useful but in some cases can lead to unexpected behaviour of your code. Let’s imagine, that you’ve sent a request and have got a response. You expected to get two numbers a=1 and b=2, and you were going to calculate their sum (a+b=3). But you’ve received the same numbers in form of strings (a=”1&quot; and b=”2&quot;). Then a+b=”12&quot;. In this case your code will work, but the answer is incorrect. What can be done in this case? You should assure that the variables have expected data types, and if they do not, change them.</p><p>Let’s consider the situation using the described case, but in form of function which gets two arguments and allows to calculate their sum:</p><pre>def summation(a, b):<br>    return a+b</pre><p>First of all, you can announce what data types are expected:</p><pre>def summation(a: int, b: int) -&gt; int:<br>    return a+b</pre><p>The syntax won’t lead to an error occurrence in case of using wrong data types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5upgNg-1GKD9x9lFEo6Jw.png" /></figure><p>But it is a good practice which makes it easier for you and for anybody else, who is going to reuse your code, to understand what the function should take and return.</p><p>Now, we can move on to converting data types. The simplest solution will be the following:</p><pre>def summation(a: int, b: int) -&gt; int:<br>    return int(a)+int(b)</pre><p>On the one hand, if a and b have data types which can be converted into integers without any problems (e.g., strings from the example above), it will work. But if at least one variable can’t be converted to integer, you’ll get a ValueError:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHRXBB2hXromO6DXBpb-lA.png" /></figure><p>On the other hand, if variables are represented by floats, then the function won’t return any error, but the resulting sum will be wrong because applying “int” rounds a float down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*KPNVRW9_kL0p8hrLZ56M4A.png" /></figure><p>So, it’s not a perfect solution. First of all, let’s try to make error message clearer.</p><pre>def summation(a: int, b: int) -&gt; int:<br>    if type(a) == int and type(b) == int:<br>        return a+b<br>    raise ValueError(f«a is {type(a)} and b is {type(b)} while a is int and b is int are expected»)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MYIXAAqAuA4LNcuKkpAk8Q.png" /></figure><p>Thus, you can see that any deviation from required data types will return a ValueError with a provided text which allows to understand what variable became a source of the problem.</p><p>At the same time we’ve lost an opportunity to make the calculations in case when a and b are strings which can be easily converted to integers.</p><p>Moreover, even for such a simple function we had to add some functionality which distracts from its main functionality and makes the code less transparent. In order to save clearness of the code, and add data types verification at the same time, we can use pydantic library. It can be easily installed using “pip install pydantic” and used for data types validation as well as converting them to required ones when it’s possible.</p><p>Initially, we need to create a class which declares what data types are expected, and use the class in our function.</p><pre>from pydantic import BaseModel, conint</pre><pre>class MyModel(BaseModel):<br>    a: conint(strict=True)<br>    b: conint(strict=True)</pre><pre>def summation(a: int, b: int) -&gt; int:<br>    model = MyModel(a=a, b=b)<br>    return model.a + model.b</pre><p>Pay attention that instead of “int” I use “conint(strict=True)”. This is because “int” or “conint(strict=False)” will suffer from rounding down even in this library:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*giclI4CfwFk8KlaH8jUT3A.png" /></figure><p>As you can see in the last example on the screenshot, it returns an error which enumerates all variables which became a reason for the error occurence. It helps easily solve any problem related to data types.</p><p>But again, we had to add additional code for the validation. Pydantic can do better. It provides us a decorator which interpretes types described at the first line of a function declaration, and automatically transforms this into a model.</p><pre>from pydantic import validate_arguments</pre><pre>@validate_arguments<br>def summation(a: int, b: int) -&gt; int:<br>    return a+b</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqVvgdsyguiK8nmtkH9nzQ.png" /></figure><p>For getting an error in case of float inputs, you’d better declare types as following:</p><pre>@validate_arguments<br>def summation(a: conint(strict=True), b: conint(strict=True)) -&gt; int:<br>    return a+b</pre><p>And finally let me say couple more words about the pydantic library.</p><p>This library allows to create whole systems of nested classes as well as supports recurrent validation of such data types as List[int]. Moreover, developers declare that it works much faster than its competitors. Detailed documentation can be found by the link: <a href="https://pydantic-docs.helpmanual.io/">https://pydantic-docs.helpmanual.io/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*X3CcF31D3PmCtPA8.png" /></figure><h4>If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀Developers: Learn and grow by keeping up with what matters, <a href="https://faun.to/8zxxd">JOIN FAUN.</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=906abdc49963" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/how-to-ensure-data-type-correctness-at-python-906abdc49963">How to ensure data type correctness at Python</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wrapping your code into a Docker container]]></title>
            <link>https://blog.devops.dev/wrapping-your-code-into-a-docker-container-fd8095668f93?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd8095668f93</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Tue, 07 Jun 2022 20:30:28 GMT</pubDate>
            <atom:updated>2022-06-09T11:33:21.084Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iF9S22oB-sO3gIMT" /><figcaption>Photo by <a href="https://unsplash.com/@carrier_lost?utm_source=medium&amp;utm_medium=referral">Ian Taylor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Having a machine learning model is not enough for any business. The model should be deployed in order to use it easily when it’s needed. The very first step in this direction is saving the model into a file.</p><pre># it&#39;s just a one among plenty possible ways to save your model<br>import pickle</pre><pre>with open(&#39;model.pkl&#39;, &#39;wb&#39;) as f:<br>    pickle.dump(model, f)</pre><p>And the second step is wrapping the model in such a way which will allow you to move the model to any server and use it there without any technical issues. Docker container is a very popular wrapper (actually, it’s a industry standard).</p><p>In the article, I’ll focus on the second step.</p><h3>What is a Docker container</h3><p>According to <a href="https://www.docker.com/">the official web-site</a>, “a container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another”.</p><p>Every Docker container is initially a lightweight image with instuctions which describe everything needed to run an application from required system libraries to your own code. Container images become containers when they run (the provided instructions executed one by one). Before that each Docker image is just a read-only entity. You can easily combine several images in order to create your own custom image.</p><p>Except the light weight, there is one more significant advantage of the containers. They isolate software from its environment and ensure that they works uniformly in different environments.</p><h3>How to install</h3><p>A Docker container usage requires Docker installment.</p><p>In order to do this, you need to follow <a href="https://www.docker.com/get-started/">the link</a>, download the file appropriate for your OS (Docker is compatible with Linux, Windows and iOS, and it’s free for personal usage.). Then you should launch the downloaded .exe file. Installation will take some time, so you’ll have an opportunity to spend this time on sign-up to Docker Hub in order to reuse Docker images which were created and shared by other Docker users. It will make the process of creating your containers quicker, because you’ll be able to use shared containers as a base, and add your code as a level above it.</p><blockquote>I’ve faced with error related to the installation process. I have to follow the provided link at the error description and install additional Linux component.</blockquote><p>When Docker’s installed, let’s check that it works. In order to do this, open a terminal and type “docker ps”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkwIa5dKz8l9LEuLcUOHpg.png" /></figure><p>If you’ve got a comparable output (an empty table with a particular set of columns), everything works.</p><h3>How to use</h3><p>Initially, an installed Docker includes a deamon (a server for containers which warrant container’s isolation), a client interface that can be used in a terminal (instructions should start with «docker &lt;command&gt;», e.g. «docker run» or «docker pause») and API for managing containers.</p><p>There are two ways of interactions with the API. Firstly, you can type and execute commands one by one in a terminal. For example, «docker build» allows to create a new image, «docker commit» and «docker push» (similarly to Git) allows to upload your image into DockerHub. Whereas «docker pull» allows to get an existing image from DockerHub.</p><p>But, believe me, you don’t want to type all the commands in the terminal every time when you need to launch a new container. That’s why there is the second option. You can type all commands once in a Dockerfile and execute it with the only line of code in the terminal. Dockerfile is so useful and powerful way to deal with Docker containers. You can just type all commands there once and launch it each time when you need to launch the container.</p><p>A file with « Dockerfile» name should be created at the same directory with files which should be added into container. In sake of clear coding, remove everything which is not necessary during the container usage (i.g., data or Jupyter notebooks where you tested hypothesis while building you model).</p><p>Then you should add instructions to this file, using a bit different syntax in comparison with the terminal commands described above. Main rules are following:</p><ul><li>one line, one command</li><li>the file must include and start with FROM command</li><li>comments can be added using hash symbol before them(#)</li><li>each command (even FROM one) can follow ARG command which specify parameters used at the following command. For example:</li></ul><pre>ARG OS_VERSION=14.04<br>FROM Ubuntu:${OS_VERSION}</pre><ul><li>commands execute from the top to the bottom</li><li>CMD command can be used just once, it launches after the container’s running only. It should look like this:</li></ul><pre>CMD [&quot;executable&quot;,&quot;param1&quot;,&quot;param2&quot;]</pre><pre># for example:<br>CMD [&quot;echo&quot;, &quot;$HOME&quot;]</pre><p>Here you can find other widely used command and their meanings:</p><ul><li>ADD copies files from the mentioned repository at local OS to the particular repository at the container</li><li>ENTRYPOINT defines a file which will be executed automatically after container’s launch. The file must include defined either ENTRYPOINT or CMD command</li><li>EXPOSE defines port number which will be listened by the launch container</li><li>RUN creates a new layer. There can be added as many layers as you wish</li><li>WORKDIR defines working directory for RUN, CMD, ENTRYPOINT, COPY and ADD commands</li></ul><h3>Consider an example</h3><p>Let consider basic operations using a toy example.</p><h4>Creating a container via a terminal</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdiA-LhrcG1bU-Q4fLC8nA.png" /></figure><p>I’ve ran a container with ubuntu inside. It’s image was automatically downloaded from the DockerHub. The command which I used includes the following arguments:</p><ul><li>“docker” is standard attribute of all commands related to Docker</li><li>“run” tells us that we’re launching a new container</li><li>“-it” specifies that Docker should open an interactive Node terminal (as you can see at the last line on the screenshot</li><li>“ubuntu” is an image that will be used while the container’s creation</li><li>“/bin/bash” is a command which should be executed after the container had launched</li></ul><blockquote>Pay attention that if you want to use a Git-Bash as a replacement for the terminal, the provided command won’t work.</blockquote><p>You can open a new terminal and check that the container is working:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YhMKqYgxameC07r0iLC1ew.png" /></figure><h4>Stopping the container</h4><p>You can see the “CONTAINER ID” field in the table above. Let’s use it in order to stop a particular container (terminate the container constantly):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*lo7K8Kfrs-Q4qiALx03oQA.png" /></figure><p>And you can check that it works as it was expected looking into the terminal where you’ve run the container. It should exit from the interactive mode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*K-NM99AtpzwDs2xTw_GBWg.png" /></figure><p>At the same time you can check a list of running containers. It shouldn’t contain the container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*NqwEpUO_5qPIWzeZgcGQ5A.png" /></figure><h4>Creating an image from a Dockerfile</h4><p>It’s easier to keep an image instead of a full set of required files and used images. So, you may want to create your own image that will contain everything you need in your container.</p><p>First of all, you can create a new image avoiding Dockerfile and using a running container. You can do this using the command:</p><pre>docker commit -m “&lt;comment&gt;” &lt;running_container_name&gt; &lt;name_of_the_new_image&gt;</pre><p>But, again, it’s much easy to work with the Dockerfile. In this case your actions should be a bit different:</p><pre>docker build -t &lt;image_name&gt;:&lt;tag&gt; &lt;directory with the Dockerfile&gt;</pre><ul><li>“-t” means that there is a directory mentioned after the tag</li><li>“tag” usually means a version (e.g., “latest” or “v1”)</li></ul><p>Let’s follow the instruction:</p><ol><li>Create a Dockerfile that contains</li></ol><pre>FROM ubuntu<br>ENTRYPOINT /bin/bash</pre><p>2. Build a new image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ecK1vfsy6-2VRNPlOO0Ww.png" /></figure><p>3. Check that the new image exists</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*1D907OhVaK7nNgFgvHyPjA.png" /></figure><p>The command returns all images that you have locally on your computer.</p><h4>Running the image</h4><p>The new image can be transfered into a container using a simple instruction</p><pre>docker run &lt;image name&gt;</pre><p>Again, let’s do this and check that the command works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*v9MOLWE5OykMut5fZW2V0Q.png" /><figcaption>Terminal 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2hIOr4UI1IJJjMW6AD85rg.png" /><figcaption>Terminal 2 (while Terminal 1 is still working)</figcaption></figure><h4>Removing the image</h4><p>Let’s try to remove the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXXMDYQ6YEClJw0SCDkSPA.png" /><figcaption>Terminal 2 (while Terminal 1 is still working)</figcaption></figure><p>The text of the error means that we can’t remove an image while there are containers which use the image. The removing command won’t work at the situation when the working container is paused as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*KeUvTl5Ld0lg5uNabljQkg.png" /><figcaption>Terminal 2 (while Terminal 1 is still working)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXXMDYQ6YEClJw0SCDkSPA.png" /><figcaption>Terminal 2 (while Terminal 1 is still working)</figcaption></figure><blockquote>“docker pause &lt;container ID&gt;” temporary stops working container untill getting “docker unpause &lt;container ID&gt;” instruction</blockquote><p>But if you stop the container and remove all containers which used the image, you’ll be able to remove the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aMlLOc6mCeY41oNkhIQyyQ.png" /><figcaption>Check the full list of containers (including stopped ones)</figcaption></figure><p>You need to remove all containers which have “myapp:dev” mentioned at the “IMAGE” column. We have two of them. Thus, we need to repeat the following command (using different container IDs) twice:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*756IHW3d3r-S1eXEYGwcsA.png" /></figure><p>When it’s done, remove the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4CeRs2VyWS6lbYV9NkSLA.png" /></figure><p>Let’s check that it’s removed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i6tzHWnQIaPpUjCeuqH46A.png" /></figure><h4>Share your image</h4><p>And finally, you can share your unique image with other community members. The operation is really close to working with Git repositories. You can push the new image into a public DockerHub repository and everyone will be able for find it and use.</p><p>That’s it for today. Thank you for your attention. If you like this article, <a href="https://medium.com/@1StepFromMagic">follow me</a> to support my work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd8095668f93" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/wrapping-your-code-into-a-docker-container-fd8095668f93">Wrapping your code into a Docker container</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring of CPU and RAM usage from a Jupyter Notebook]]></title>
            <link>https://medium.com/@1StepFromMagic/monitoring-of-cpu-and-ram-usage-from-a-jupyter-notebook-f5da263ddbf6?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5da263ddbf6</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Sun, 22 May 2022 20:54:53 GMT</pubDate>
            <atom:updated>2022-05-22T20:54:53.257Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CbbPfLPF2PH-bPH2" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There are several linux tools which allow to monitor a current load of RAM, CPU and other metrics related to the running processes. <em>htop</em>, <em>top</em>, <em>nmon</em> are among them, and they are more than enough for many practical tasks.</p><p>But I’ve faced with a situation when I needed to monitor a recent history of these metrics’ changes instead of just their current state.</p><p>For this purpose I’ve written a small script that, initially, monitored the load of RAM and CPU for a particular process ID (pid) in form of graphs which refreshed with a chosen frequency (enriched by a new portion of data). But after testing it, I’ve realized that pids can change during a program’s processing. That’s why the code, that I share, depends on a username instead of the pid. The idea is rather simple — if there are some other (less inportant) processes working in the background, they won’t influence significantly on the graph, and you’ll understand the overall picture anyway.</p><p>And here is the script:</p><ol><li>Import required libraries</li></ol><pre><strong>import</strong> psutil<br><strong>import</strong> numpy <strong>as</strong> np<br><strong>import</strong> matplotlib.pyplot <strong>as</strong> plt<br><strong>%matplotlib</strong> notebook</pre><p>Pay attention to the first and the last lines of code. Importing “psutil” allows to get information about the current states of RAM and CPU usage. Using matplotlib in “notebook” mode allows to refresh plots inplace (refresh the plots which have already been showed).</p><p>2. Initialize variables</p><pre>x <strong>=</strong> [] # iterations<br>y <strong>=</strong> [] # current CPU load<br>z <strong>=</strong> [] # current RAM load<br>i <strong>=</strong> 0</pre><p>3. Set parameters</p><pre>user <strong>=</strong> &#39;&lt;write your username here&gt;&#39; <br>refresh_freq <strong>=</strong> 1 <em># seconds (frequency of the graph refreshing)</em></pre><p>You can understand what is your username by launching htop and finding your working process there (even the htop launch will be shown there as a separate process). The following script will aggregate information about all working processed associated with the chosen username.</p><p>4. Initialize graphs</p><pre>fig <strong>=</strong> plt<strong>.</strong>figure(figsize<strong>=</strong>(12, 7))<br>ax1 <strong>=</strong> fig<strong>.</strong>add_subplot(211)<br>ax2 <strong>=</strong> fig<strong>.</strong>add_subplot(212)<br>graph_cpu, <strong>=</strong> ax1<strong>.</strong>plot(x, y, &#39;g-&#39;) <em># CPU plot</em><br>graph_ram, <strong>=</strong> ax2<strong>.</strong>plot(x, z, &#39;b-&#39;) <em># RAM plot</em></pre><p>5. Run a while-loop that will refresh the graphs</p><pre>zeros <strong>=</strong> 0<br><strong>while</strong> zeros <strong>&lt;</strong> 10:<br>    x<strong>.</strong>append(i)<br>    vals_cpu <strong>=</strong> []<br>    vals_ram <strong>=</strong> []<br>    vals_cpu<strong>.</strong>append(sum([p<strong>.</strong>cpu_percent() <strong>for</strong> p <strong>in</strong> psutil<strong>.</strong>process_iter() <strong>if</strong> p<strong>.</strong>username() <strong>==</strong> user]))<br>    vals_ram<strong>.</strong>append(sum([p<strong>.</strong>memory_info()[1]<strong>/</strong>(2.<strong>**</strong>30) <strong>for</strong> p <strong>in</strong> psutil<strong>.</strong>process_iter() <strong>if</strong> p<strong>.</strong>username() <strong>==</strong> user]))<br>    y<strong>.</strong>append(np<strong>.</strong>mean(vals_cpu))<br>    z<strong>.</strong>append(np<strong>.</strong>mean(vals_ram))<br>   <br>    <em># deal with while-loop stopping criterion</em><br>    <strong>if</strong> len(vals_cpu) <strong>&lt;</strong> 2 <strong>or</strong> vals_cpu[<strong>-</strong>1] <strong>!=</strong> vals_cpu[<strong>-</strong>2]:<br>        zeros <strong>=</strong> 0<br>    <strong>else</strong>:<br>        zeros <strong>+=</strong> 1<br>   <br>    graph_cpu<strong>.</strong>set_xdata(x)<br>    graph_cpu<strong>.</strong>set_ydata(y)<br>    graph_ram<strong>.</strong>set_xdata(x)<br>    graph_ram<strong>.</strong>set_ydata(z)<br> <br>    ax1<strong>.</strong>set_xlim((<strong>-</strong>1, i<strong>+</strong>1))<br>    ax2<strong>.</strong>set_xlim((<strong>-</strong>1, i<strong>+</strong>1))<br>    ax1<strong>.</strong>set_ylim((min(y)<strong>-</strong>0.1, max(y)<strong>+</strong>0.1))<br>    ax2<strong>.</strong>set_ylim((min(z)<strong>-</strong>0.1, max(z)<strong>+</strong>0.1))<br>    ax1<strong>.</strong>set_title(&#39;CPU usage, %&#39;)<br>    ax2<strong>.</strong>set_title(&#39;RAM usage, Gb&#39;)<br> <br>    fig<strong>.</strong>canvas<strong>.</strong>draw()<br>    plt<strong>.</strong>pause(refresh_freq)<br>    i <strong>+=</strong> 1</pre><p>You may choose your own criterion for the script stopping. I prefer to stop the refreshment when CPU usage doesn’t change during last 10 iterations, while it can be a bad choice in case of several working processes in the background. If you have any ideas about better criteria, please, let me know via comments.</p><p>You can also easily add a code that will save the information on the disc instead of just drawing it.</p><p>The code is published here, so you can download and use the notebook:</p><p><a href="https://github.com/dariaip/monitoring/blob/main/monitoring.ipynb">monitoring/monitoring.ipynb at main · dariaip/monitoring</a></p><p>Thank you for your attention. If you like this article, <a href="https://medium.com/@1StepFromMagic">follow me</a> to support my work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5da263ddbf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to deal with outliers in time series]]></title>
            <link>https://medium.com/@1StepFromMagic/how-to-deal-with-outliers-in-time-series-c6d5e03298f?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6d5e03298f</guid>
            <category><![CDATA[outliers]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Mon, 02 May 2022 15:57:47 GMT</pubDate>
            <atom:updated>2022-05-02T15:57:47.922Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous <a href="https://medium.com/@1StepFromMagic/time-series-forcasting-methods-347f92710a3f">article</a> about time series analysis, I’ve described methods which allow to predict what will happen next. But all of them rely on one important prerequisite that I haven’t mentioned. Your data should be correct, otherwise your models won’t be robust and precise. But we’re living in the real world which is full of mistakes and extraordinary situations. The COVID-19 pandemic is an extreme example that first comes to mind. This article explains how to work with such noisy time series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VFLB_8oUCyOUzkHZ" /><figcaption>Photo by <a href="https://unsplash.com/@moneyphotos?utm_source=medium&amp;utm_medium=referral">olieman.eth</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s imagine, that you have some suspecious points in your time series. Let’s call them outliers. First of all, you should understand about each of these datapoints whether it’s an error or a real value. It’s important because there should be different approachs applied to these two types of outliers.</p><blockquote>You may think that they can be ignored because you have just a few of them. Partly, you’re right if your model is not crucial for the business and your source of data is so stable that its representatives can vow that their data won’t include outliers. But partly this decision can lead to wrong forcast which, in its turn, can mean the additional costs for the business and spoiling your reputation as a data scientist.</blockquote><h3>Errors</h3><p>Errors usually occur because of a human factor, sensor malfunction, miscalculation, etc. You definitely don’t want to consider this data while making a prediction.</p><p>Your actions will depend on what source of data you have. If you can influence on the source (for example, asking to fix the mistake), it’s amazing, and you can get correct data without applying any more tricky techniques.</p><p>But if the error can’t be fixed easily, you need to remove it from data. It can be made by:</p><ul><li><strong>Explicit deletion</strong> the datapoint and treating it as a missing value</li><li><strong>Replacing</strong> it by a predicted value. It can be the value predicted by your time series forcasting model (based on the previous observations), or regression model (based on other information that correlates with your time series), or even extrapolation (what value could be observed between previous and following correct points).</li></ul><p>In case you’re not sure that the observation is a mistake, it’s better to use the logic described below.</p><h3>Real but Rare situations</h3><p>Outliers can occur because of extremely rare unexpected situations. The key point here is that, with tiny probability, the same situation can happen somewhen in the future.</p><p>The data, you deal with, is actually correct, and it’s almost impossible that the similar situation will occur in the future. Then you don’t want to take the outliers into consideration as is. But in case the situation will happen, it’s better to keep information about previous experience somehow.</p><p>What methods can be applied here in order to treat the outliers?</p><blockquote>They should not be deleted. Otherwise, you assume that nothing like this will happen in the future.</blockquote><ul><li>The first approach is exactly what was described above. You replace the value itself by a predicted one which is far from the initial outlier, but take the initial value of the datapoint into account while calculation variance and the related confidence interval.</li><li>On the oposite side, you can slightly change the initial value using windsorisation. And use this adjusted value while calculating variance, because it will still lead to getting wider confidence interval for your forecast.</li></ul><blockquote><strong>Winsorisation </strong>replaces extreme data values with less extreme values making tails of a distribution heavier. It can be done either by applying provided boundaries (all values which are bigger than the provided upper boundary should be equal to it, and all values which are lower than the provided lower bound should be equal to it), or replacing outliers by more probable values that can be taken from the tails of the same distribution. If you consider the first way of winsorisation, you will need to think carefully about the boundaries. You can also apply only one boundary (either high, or low).</blockquote><ul><li>It’s also possible to use the initial values of the outliers in the model with lower weights in comparison with other datapoints.</li></ul><h3>Real but Unexpected situations</h3><p>This category is last but not the least important. It’s even the most frequent situation, when you catch outliers which can be repetitive and not anomaly at all. They usually mean that your model is too simple and it needs ajustments.</p><p>Of course, it’s up to you to decide what will be more costy — either improving the model or dealing with additional false positive outliers. But if you choose to adjust your model, then your need to explore the context and try to find a reason, why you hasn’t expected the correct value. The analysis conclusions should lead to adding new predictors to your model which allow you to account for the outliers in the future forcasts.</p><p>Thank you for your attention. If you liked this article, <a href="https://medium.com/@1StepFromMagic">follow me</a> to support my work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6d5e03298f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to find the shortest path in a graph]]></title>
            <link>https://medium.com/@1StepFromMagic/looking-for-an-optimal-path-ebe4be578777?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebe4be578777</guid>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[shortest-path]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[dijkstra]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Thu, 21 Apr 2022 19:25:07 GMT</pubDate>
            <atom:updated>2022-05-03T18:43:26.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wHnYSl53HEAzwB8z" /><figcaption>Photo by <a href="https://unsplash.com/@gcalebjones?utm_source=medium&amp;utm_medium=referral">Caleb Jones</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The task is an old one, but there still doesn’t exist such an algorithm that would allow to find precise solution quickly. In this article I’ll describe some algorithms that can solve the task precisely, despite the fact that you would prefer them to work quicker.</p><p>But let’s start from several introductory statements:</p><ul><li>Surely, an optimal path can be looking for in different tasks, but the most transparent example is related to the road net. That’s why I’ll talk about this case.</li><li>A road network can be represented as a graph, where crossroads are associated with vertices and roads refer to edges. That’s why all provided algorithms deal with graph structures. The graphs can be both directed and undirected.</li><li>Technically, the task is about finding the shortest path between particular vertices A and B in a provided graph. “Shortest” means that either number of edges between A and B is as small as possible, or sum of edges’ weights (which are located on the path) is minimal.</li><li>Graphs can keep in memory in different forms (using different data types) which influence on time and space comlexity of the algorithms.</li><li>All described algorithms guarantee returning of a properly optimal path.</li></ul><p>And when the short introduction is finished. Algorithms are coming.</p><h3>Bellman Ford’s Algorithm</h3><p>The algorithm is looking for the shortest path from a particular vertex A to all other vertices in a provided graph. Thus, if you’re interested in an optimal path from A to B, then you just need to find the vertex B and the related distance in the results returning by the algorithm.</p><p>Dynamic programming approach uses for solving the task. At the very beginning all distances are initialized as infinitely huge. Then the algorithm proceeds each and every edge trying to include it into the shortest paths and therefore to minimize the distances from the starting point A to all other vertices.</p><p>If there are negative weights in a graph, negative cycles can occur which lead to stucking into the cycle instead of finding an optimal path. At the same time dealing with dynamic programming approach allows easily reconstruct the optimal path instead of just getting a number describing the length of the path.</p><h3>Floyd Warshall’s Algorithm</h3><p>The algorithm also returns minimal distances from starting point A to all other vertices. It considers all triples of vertices, looking for shortcuts in these “triangles”.</p><p>It means that if vertices X, Y and Z are included in the considered triangle, and the sum of distances from X to Y and from Y to Z is smaller than the distance from X to Z directly, then path X -&gt; Y -&gt; Z is a part of an optimal path with higher probability.</p><p>Unfortunately, for considering all available triangles, V³ operations is needed (where V is a number of vertices in the graph), which is a huge number.</p><h3>Dijkstra’s Algorithm</h3><p>It seems that the algorithm is the most well known and rather interesting because it became a basis for several advanced algorithms.</p><p>It tries to find a path between A and B vertices only. Doing this, the algorithm goes from one vertex X (at the very beginning it’s a starting point A) to other vertices which are connected to X by edges, and checks whether adding the edge to the previously constructed path can make the path shorter and, thus, more optimal. All new variations of the path (previous path + a new edge) should be added into queue to process them further by adding even more edges until the optimal path will be found.</p><p>Main limitation of the algorithm is a fact that it works with non-negatively weighted graphs only. But there is one more disadvantage. In this basic configuration, the algorithm checks all vertices in the increasing radius around the starting point which can mean processing huge number of possible paths until the optimal one will be found. This disadvantage leads to occurence of new and better algorithms described below.</p><h3>Bidirectional Search</h3><p>Bidirectional Search is rather similar to the Dijkstra’s algorithm. It works with non-negatively weighted graphs only, but deals with the graph quicker because Dijkstra’s algorithm is looking for an oprimal path in surrounding of starting point A with a radius R (where R is the minimal distance between A and B). But we can start the search from A and from B simultaneously, make the search in surroundings with radiuses R/2 around both of the vertices and allow the parts of the optimal path meet somewhere inbetween the vertices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGFn9p1rdj4BNwNlv8Mpxg.png" /></figure><p>Such an approach allows to decrease number of operations significantly. But the algorithm does not always work well with roads, because they are not symmetric (Optimal paths from A to B and from B to A can be different).</p><h3>A* Algorithm</h3><p>At the same time dealing with roads can have their advantages as well. For example, it’s possible to measure approximately, whether we become closer to the destination B using geographical coordinates. It’s exactly what A-star algorithm does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6JA9tN0fCzyNecOY9NyhVg.png" /></figure><p>Here we use Dijkstra’s algorithm, but adding new vertices in a queue we order them by the estimation of potential distance. The smaller the distance, the higher probability that the point is on the shortest path and the quicker the path variants related to the vertex will process.</p><p>The algorithm can be bidirectional.</p><h3>Contraction Hirerarchies</h3><p>And here is one more algorithm related to road networks. The idea is rather natural. There is a hierarchy of roads: local, medium-size, etc., and finally highways. The higher a road in the hierarchy, the more crucial it is in terms of trafic. It can even mean that these highways can be unaviodable if you want to move from point A to point B.</p><p>The hierarchy reflects properties of the graph, and it is universal for any selection of points A and B. So, creating of the hierarchy is a part of data preprocessing, and can be done once. Optimal path’s search follows afterwards, and include movements along the hierarchy (from small roads to more significant and vise versa).</p><p>Of course, there are formal description of the algorithms, and their implementations in code, but using them here would make the article too long. I’m sure that the details can be googled easily when you know what algorithms exist.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebe4be578777" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time series forcasting methods]]></title>
            <link>https://medium.com/@1StepFromMagic/time-series-forcasting-methods-347f92710a3f?source=rss-6fb54d99528a------2</link>
            <guid isPermaLink="false">https://medium.com/p/347f92710a3f</guid>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[time]]></category>
            <dc:creator><![CDATA[1StepFromMagic]]></dc:creator>
            <pubDate>Mon, 11 Apr 2022 18:13:41 GMT</pubDate>
            <atom:updated>2022-05-28T16:20:38.149Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_GDfEf-LJsasfgrr" /><figcaption>Photo by <a href="https://unsplash.com/@alx2bgx?utm_source=medium&amp;utm_medium=referral">Alex Guillaume</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Time series exist as long as the time itself. That’s why there are lots of methods that allow not only to analyze the data but also predict what will happen next. In the article you can find short overview of the methods, I’ve faced in practice.</p><h3>1. Quick and dirty</h3><p>First of all, there are rather simple methods based on the idea that next point is strongly connected with particular observations in the past. Moving Average and Autoregressive models are among them. These methods can work as a stand-along method and as a trend extraction method before applying something else to residuals.</p><h4><strong>Moving Average</strong> (MA)</h4><p>MA calculates the predicted value as a weighted average of previous N observations. In the simplest version it’s an arithmetic average (when all weights are the same, and they sum up to 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*tZbh2m3LXuv3gtiWq9E4Gg.jpeg" /></figure><h4><strong>Autoregressive model</strong> (AR)</h4><p>AR represents values of time series as a linear combination of N previous observations of the same time series. As any regression it also includes coefficients related to each observation and a bias.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*WXpmB8jc9eCcl__eBvsOAw.jpeg" /></figure><h3>2. Classics</h3><p>AR and MA together led to appearance of such well-known methods as Autoregressive Moving Average, Autoregressive Integrated Moving Average and Seasonal Autoregressive Integrated Moving Average. All of them deal with stationary time series.</p><blockquote>Shortly speaking, <strong>stationarity<em> </em></strong>means that parameters of the observations’ distribution (such as mean and variance) don’t change over time.</blockquote><p>And because this requirement is too strict there are different methods for different types of data preprocessing that allow to make data closer to stationarity.</p><h4><strong>Autoregressive Moving Average</strong> (ARMA)</h4><p>ARMA is the simplest version of the approach that just joins two previous methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6hbJXu4M_nRfEhWet_JasA.jpeg" /><figcaption>Epsilon here stands for the “error” mentioned in the previous formula.</figcaption></figure><p>One can see that it includes the AR model as is, and additionally extracts significant fluctuations from the errors which AR couldn’t catch but MA could.</p><p>Thus, it is an example of <strong><em>additive decompositional models</em></strong>.</p><blockquote>Decompositional models are widely used in time series analysis. They represent a time series as a combination (either additive or multiplicative) of the following factors: trend (or main tendency), cyclicity, seasonality and unpredictable noise.</blockquote><h4><strong>Autoregressive Integrated Moving Average</strong> (ARIMA)</h4><p>This version of ARMA works with non-stationary time series which can be made stationary by using not the time series themselves but using differences between their observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DAQe6Kl5xz-FMiuYqQVrzg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*9hasaeELUvueoAP8jYFSTQ.jpeg" /></figure><h4><strong>Seasonal Autoregressive Integrated Moving Average</strong></h4><p>SARIMA or Seasonal ARIMA includes one more improvement. It can work with seasonal data because of including special summands into the previous formula. They allow to extract seasonal information related to a particular fixed duration of a season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-CFrAAR2LgW3AJZHw2iFQA.jpeg" /><figcaption><strong>s</strong> stands for the number of time steps that a single seasonal period includes</figcaption></figure><h4>etc.</h4><p>There are even more advanced methods such as <strong>TBATS </strong>(Trigonometric, Box-Cox transform, ARMA errors, Trend and Seasonal components) which allows to deal with changing seasonality. But I won’t go deeper in this direction.</p><h3><strong>3. Machine learning</strong></h3><p>When the Big Data Era began the models observed above became too slow. Fortunatelly, machine learning methods could solve the problem. Actually, any time series forcasting task can be treated as a standard regression task. Time series just needs an additional data preprocessing.</p><p>The dataset can be formed in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*RA0UD4L1kY4cD8Z2dYWgYA.gif" /></figure><p>It can also include additional generated features that help the model identify seasonality and other specific characteristics of the time series.</p><blockquote>Be careful using the method, because rows in the dataset are not independent. That’s why preventing leakage is needed while splitting data into train and test sets.</blockquote><h3>4. Deep learning</h3><p>And when deep learning became extremely popular there occured new methods which apply deep learning to the task of time series forcasting.</p><p>The most natural choice here is LSTM networks which allow to keep some information about previous observations in so called “memory”.</p><p>But at the same there occured other models such as:</p><ul><li><a href="https://www.sciencedirect.com/science/article/pii/S0169207019301888">DeepAR</a> (a methodology for producing accurate probabilistic forecasts, based on training an autoregressive recurrent neural network model on a large number of related time series)</li><li><a href="https://arxiv.org/pdf/1905.10437.pdf">N-BEATS</a> which uses a multi-layer fully connected network as a basic building block</li><li>Temporal Fusion Transformer (<a href="https://arxiv.org/pdf/1912.09363.pdf">TFT</a>)</li></ul><p>Their architectures are rather complicated, so it would deserve a separate article. So, I’ve just added links to the sources of the additional information.</p><h3>Conclusion</h3><p>And last but not the least, before using all the described methods it’s strongly recommended to understand whether the considered time series is predictable or it’s just a random sequence of values. The “garbage in, garbage out” principle is extremely important in any task related to data.</p><p>If you liked this article, <a href="https://medium.com/@1StepFromMagic">follow me</a> to support my work, and check out my new article “<a href="https://medium.com/@1StepFromMagic/how-to-deal-with-outliers-in-time-series-c6d5e03298f">How to deal with outliers in time series</a>” related to the same topic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=347f92710a3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>