<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marco Marchioro on Medium]]></title>
        <description><![CDATA[Stories by Marco Marchioro on Medium]]></description>
        <link>https://medium.com/@quantisland?source=rss-507818b3b756------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WoGUtfhj8m8GkiLHWn0NdQ.jpeg</url>
            <title>Stories by Marco Marchioro on Medium</title>
            <link>https://medium.com/@quantisland?source=rss-507818b3b756------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:29:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@quantisland/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop Loosing Money by Using Artificial Intelligence to Understand Market Directions]]></title>
            <link>https://medium.com/@quantisland/marco-marchioro-multi-period-market-trends-203eaedd2d2b?source=rss-507818b3b756------2</link>
            <guid isPermaLink="false">https://medium.com/p/203eaedd2d2b</guid>
            <category><![CDATA[cryptocurrency-investment]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[market-trends]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <dc:creator><![CDATA[Marco Marchioro]]></dc:creator>
            <pubDate>Fri, 13 May 2022 07:18:58 GMT</pubDate>
            <atom:updated>2022-05-13T07:18:58.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A frustrated bear wondering why Marco Marchioro wrote this post about multi-period market trends" src="https://cdn-images-1.medium.com/max/1024/1*IVPgyrjIzuuESLteso-HxQ.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/raincarnation40-747390/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1383980">Rain Carnation</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1383980">Pixabay</a></figcaption></figure><h3>Stop Losing Money by Using Artificial Intelligence to Understand Market Directions</h3><p>How to compute the <a href="https://dev.cryptoaidashboard.com/charts/bitcoin"><em>Multi-Period Market Trends</em></a>, using a mix of Fourier analysis and machine-learning techniques.</p><h3>The HODL strategy</h3><p>Remarkably, one of the most profitable strategies in playing the crypto-currency markets is the so-called HODL strategy. According to the HODL strategy, one should just buy as much as possible cryptos and hold them for as long as possible. Astonishingly, in the long run, at least that’s what happened in the last ten years (but I cannot guarantee for the future) hodling resulted in superior returns, even compared to much more sophisticated strategies.</p><p>To understand more about hodling you can read the old post, <a href="https://www.coindesk.com/markets/2018/04/16/hodl-on-in-defense-of-bitcoins-best-strategy/">HODL On: In Defense of Bitcoin’s Best Strategy</a>, which is still relevant at the time of writing. The hodl strategy is so popular that entire internet websites are built around this idea, for example <a href="https://dailyhodl.com/">The Daily Hodl</a>. Note that this type of strategy is usually just called <em>buy and hold</em> in other markets, such as the equity market<em>.</em></p><p>Given the volatility of the crypto markets, the problem with hodling is that one might suffer <em>significant</em> losses when the market direction turns south. For example, at the end of 2017, the Bitcoin price was just shy of 20,000 USD, while just a year later was down below 4,000 USD. The people who understood the market direction during that period and acted on it by selling their cryptos saved a considerable amount of money (possibly to be used later to buy at lower prices).</p><h4>Is there a strategy that performs better than hodling?</h4><p>Inspired by the considerations above, we can think of the following strategy: <em>we wait for an upward direction to buy the cryptos of our choice, just to sell them when the market trends downward.</em></p><p>This plan of action is known as a <em>trend-following strategy</em>. At least in principle, this tactic does better than just buying and holding as proposed by the hodlers. There is only one problem: we need to have a way to measure the market direction, i.e. the <em>current trend</em>.</p><h3>Is the trend your friend?</h3><p>One popular saying on the internet among traders is: “the trend is your friend.” The problem with this statement is that in reality, there isn’t just one single trend, but there are many depending on the investment time horizon (does it mean that we have many friends?-). Consider for example the path of the Bitcoin price, in USD, at the end of 2021. We split this path by sampling the price using periods of 7 and 30 days, i.e. one week and one month, as shown in Figure 1.</p><figure><img alt="A chart of the bitcoin price at the end of 2021, together with samples taken for weekly and monthly periods" src="https://cdn-images-1.medium.com/max/1024/1*UetGl9UPTJJ-KzVDtwD8Ag.png" /><figcaption>Figure 1: The bitcoin price at the end of 2021, together with weekly and monthly samples.</figcaption></figure><p>We then compute the relative returns on these sampled periods and, for the time being, forget about the bitcoin price in between, as shown in Figure 2.</p><figure><img alt="A chart of the weekly and monthly samples of the bitcoin price at the end of 2021, together the computed relative returns" src="https://cdn-images-1.medium.com/max/1024/1*0gXei573egACxVWrTR7ExQ.png" /><figcaption>Figure 2: the weekly and monthly returns computed for each period of the data as shown in Figure 1.</figcaption></figure><p>Note that the return values computed using the weekly periods are different from those computed using the monthly ones.</p><p>To compute the daily trends, we simply divide the returns by the number of days present in each period. The results are shown in Figure 3.</p><figure><img alt="The sampled bitcoin chart at the end of 2021, together with the computed daily trends for each period" src="https://cdn-images-1.medium.com/max/1024/1*VRdcbp8fH5QjpfLSd1N5bg.png" /><figcaption>Figure 3: the daily trends computed by dividing the returns by the number of days in the corresponding period.</figcaption></figure><p>Note how higher trends mean higher returns, only if they are observed for an extended period of time. This means that a high trend, that doesn’t last a long time, may not give rise to a higher return than that provided by a smaller trend, that lasts much longer.</p><p>In summary, both the returns and the trends depend on the sample period. This is why there isn’t just a single trend but there are many, depending on the time horizon we are considering. For example, the market might be currently trending up considering a horizon of a few days, have a negative trend if we consider a weekly period, and be bullish on the monthly horizon, all at the same time.</p><h3>Triangles and trading patterns</h3><p>To gain more insights into the trend measurement, we split the series of returns into their geometrical components. Hence, starting from the results shown in Figure 2, we take each return and shift it to always start on the <em>time</em>-axis (see Figure 4).</p><figure><img alt="The chart of returns shifted so that their starting point is always on the time-axis" src="https://cdn-images-1.medium.com/max/1024/1*SwQ3EqbMbqMSa0pTr8aJXw.png" /><figcaption>Figure 4: the returns as computed in Figure 2, however, shifted to always start at the time<em>-axis. The price-axis indicates the returns and the text labels in the chart indicate the trends.</em></figcaption></figure><p>We notice that the trend functions, as seen in Figure 4, can be computed as the superposition of sawtooth-shaped triangular waves. Similarly to<a href="https://en.wikipedia.org/wiki/Fourier_series"> a Fourier series</a>, that can approximate the shape of any graph using the <em>sine</em> and <em>cosine</em> functions, it is possible to come as close as needed to any path by using triangular waves. In other words, if we were to compute the trends on smaller and smaller scales, e.g. monthly, weekly, daily, hourly, and so on; we could perfectly reproduce the whole price chart of Figure 1, just using a superposition of triangles.</p><p>Indeed this is not a new idea, as a several technical analysis methods split the price chart into triangles at different scales. For example, <a href="https://fbs.com/analytics/guidebooks/what-is-a-triangle-273">this interesting post</a> classifies the trading patterns into five different kinds of triangular shapes: <em>Horizontal</em>, <em>Barrier</em>, <em>Expanding</em>, <em>Running</em> and <em>Skewed</em>.</p><figure><img alt="An example of barrier and expanding triangular patterns" src="https://cdn-images-1.medium.com/max/1024/1*2-bNKWKwNL2doJu358swtQ.png" /><figcaption>Figure 5, barrier and expanding triangular patterns, as described in <a href="https://fbs.com/analytics/guidebooks/what-is-a-triangle-273">https://fbs.com/analytics/guidebooks/what-is-a-triangle-273</a></figcaption></figure><p>The trend method described in this post is actually more complete than most technical-analysis techniques, because keeps into consideration a virtually-infinite number of triangular patterns. Also, as we are about to find out in the following paragraphs, the method presented here does not rely on the precise price observations.</p><h3>Optimizations using machine-learning techniques</h3><p>So far our definition of trends, as described in the previous sections, is based on the price path sampled at regular intervals. This definition is less than ideal for a number of reasons.</p><ul><li>The segments chosen to compute the returns might not be <em>representative</em> of the underlying price graphs. As seen in Figure 1, the segments we obtain by sampling do not adhere to the price path.</li><li>The resulting trends are not resilient to market-data errors. If there are minor measurement errors, as it sometimes happens, the resulting trend figures may significantly change. (This same problem is seen in most technical analysis techniques.)</li><li>All trading periods are treated equally. However in the markets, there are sometimes periods of high volatility, in which the trend varies often. Also, there are other periods when there isn’t as much trading, where the trend is more stable and lasts longer.</li></ul><p>For these reasons, using machine-learning techniques (which somebody may call artificial intelligence), we modify the regular triangular waves defined above and stretch them so that the resulting patterns are closer to the initial price graph. Mathematically this approach is similar to what happens in a <a href="https://en.wikipedia.org/wiki/Wavelet">wavelet transform</a> where the original wave shape, in our case the triangles, is modified and adapted to the underlying signal. The wavelet technique is often used in image compression. In our case, we modify the original triangular patterns so that we need a smaller number of trend lines to approximate the same price chart.</p><p>In practical terms, we use a machine-learning technique, for each chosen time frame, to stretch the periods and compute new return values, so that the resulting triangular waves better match the price-graph patterns. Note that the stretch is performed in both directions: the time axis and the price axis. For example, for the period of one month, using the same data as Figure 1, this method yields the results illustrated in Figure 6.</p><figure><img alt="The chart of the bitcoin price together with the trend line as computed both by naive sampling and by the stretching technique described in this post" src="https://cdn-images-1.medium.com/max/1024/1*mFmVy82N7EtCsxYywkPE8w.png" /><figcaption>Figure 6: s<em>tretching the monthly returns of Figure 2 using the technique described in the text. The text labels denote the new trend values.</em></figcaption></figure><p>In the specific example of Figure 6, we notice how we obtain a longer central upward trend, while the first and the last periods are slightly shrunk. In other market conditions the effects of the stretching might be different.</p><p>Note that the shifted nodes do not necessarily match the price graph but might be above it, possibly indicating a future price increase, or below it signaling a plausible market correction. Later we use this property to compute the <em>model returns</em>.</p><h3>Multi-period market trends</h3><p>The method described in this post acquires more significance when multiple periods are computed simultaneously. In this context we call the results of this technique the <em>Multi-Period Market Trends</em>.</p><p>On any given time scale, once we measure the multi-period market trends, there are some interesting quantities that we can compute.</p><ul><li>The <em>Intrinsic Return</em>, i.e. the return that we would obtain if the market would revert instantly to the trend line.</li><li>The <em>Trading Score</em>, i.e. the last value of <a href="https://nl.wikipedia.org/wiki/Z-score">the z-score</a> of the difference between the price and the current trend line. Usually the trading score is between <em>-2</em> and <em>2</em>. However a score above <em>2</em>, or below <em>-2</em>, may signal an imminent trend change or a buy/sell opportunity.</li><li>The latest <em>Trend</em>, i.e. the value obtained for the daily trend as computed in the last segment.</li><li>The <em>Trend Days</em>, i.e. the length in days, of the latest segment</li></ul><p>For example on <em>2022–05–13</em> at <em>11:13, </em><a href="https://dev.cryptoaidashboard.com/charts/bitcoin"><em>using the companion web app</em></a>,<em> </em>for the bitcoin chart we computed the quantities as shown in Table 1.</p><figure><img alt="A table with a sample of the multi-period trend parameters for the Bitcoin chart, as observed on 2022–05–11 at 11:13" src="https://cdn-images-1.medium.com/max/1024/1*-KArwvUtoVAQ9Gn5rPMbiw.jpeg" /><figcaption>Table 1: a sample of the multi-period trend parameters for Bitcoin, as <a href="https://dev.cryptoaidashboard.com/charts/bitcoin">observed on <em>2022–05–13</em> at <em>11:13</em></a>.</figcaption></figure><p>Note that, in Table 1, the <em>Very short</em> numbers are computed using 4 days, the <em>Short </em>results using 7 days, the <em>Medium</em> results using 15 days, while the <em>Medium/long</em> numbers with 30 days, the <em>Long</em> figures using 60 days, and finally the <em>Very long</em> trends with 90 days.</p><p>Also in the specific case of the results of Table 1, we notice that we have a strong upward trend for the <em>very short</em> period, a null trend for the <em>short</em> period, very negative trends from the <em>medium</em> to the <em>long</em> periods, and, finally, a milder negative trend for the <em>very long</em> period. Once again all the trends are indicated in daily percentages.</p><h3>Summary and conclusions</h3><p>In this post, we argue that there isn’t just one trend, or market direction, but many according to the investment horizon. Therefore, to understand the current market conditions, we need to measure many trends simultaneously at many different time scales. We show how to compute the <em>Multi-Period Market Trends</em> using a mix of <a href="https://en.wikipedia.org/wiki/Fourier_analysis">Fourier analysis</a> and machine-learning techniques<em>. </em>Finally we gave the results for the trends at the time of writing.</p><p>For a group of cryptos, the latest trends will be made available on <a href="https://dev.cryptoaidashboard.com/">the companion web app</a>.(Note that this app is still in development.)</p><p><strong>Important legal disclaimer:</strong> <em>the material on this post has no regard for the specific investment objectives, financial situation, or particular needs of any user. This post is presented solely for informational and entertainment purposes and is not to be construed as a recommendation, solicitation, or an offer to buy or sell / long or short any securities, commodities, cryptocurrencies, or any related financial instruments. Nor should any of its content be taken as investment advice.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=203eaedd2d2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ethereum: how to save even more on gas price with a weekly plan]]></title>
            <link>https://medium.com/dextf/ethereum-how-to-save-even-more-on-gas-price-with-a-weekly-plan-c6689ac09fe6?source=rss-507818b3b756------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6689ac09fe6</guid>
            <category><![CDATA[ethereum-gas-price]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[dextf]]></category>
            <dc:creator><![CDATA[Marco Marchioro]]></dc:creator>
            <pubDate>Sun, 02 Aug 2020 05:06:36 GMT</pubDate>
            <atom:updated>2020-08-02T06:40:29.324Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Image of a weekly planner together with a pen and a phone" src="https://cdn-images-1.medium.com/max/1024/1*MhoT8csdFjtmVZxmmByWTg.jpeg" /><figcaption>Image courtesy of <a href="http://www.ktcupoftea.com/">http://www.ktcupoftea.com</a></figcaption></figure><p>An analysis of the weekly periodicity of the Ethereum blockchain reveals that the best time to submit a transaction is during the weekend.</p><p>As I have shown in<a href="https://medium.com/dextf/when-is-the-best-time-to-send-ethereum-transactions-80d53ae8e8d1"> a companion post</a>¹, Ethereum transactions are observed to be more frequent during the Asian daytime which means that one can <strong>save on gas prices</strong> if they are willing to wait for the <strong>Asian wee hours</strong>. In this post I will share how to save <em>even more </em>on gas prices by waiting until weekend to submit the most costly transactions. I also show the hourly dependence of the gas price and the throughput for each single day of the week.</p><p>Recently the Ethereum blockchain has been performing close to capacity in terms of average transactions per second (TPS). As a consequence miners can be <strong>pickers</strong> and choose only transactions with <strong>higher gas prices</strong>. In turn, this logical behaviour has increased the average gas price paid to submit transactions to the Ethereum blockchain. The success story of many <a href="https://defiwiki.org/wiki/Main_Page">DeFi</a> solutions has caused many headaches because of the rising gas costs and the long queues in submitting transactions.</p><p>As part of <a href="https://dextf.com/">DEXTF</a> Protocol project, I often have to submit transactions to the blockchain and interact with many different smart contracts. While some of these transactions have an immediate urgency and need to be performed as soon as possible, many others <strong>can wait for a few days</strong>.</p><p><em>To minimize this problem, you can create a weekly plan to submit non-urgent transactions in days when both gas prices and the throughput are low.</em></p><h3>Weekly periodicity of transactions</h3><p>We do not need to run a full Ethereum node to obtain transaction data since¹, fortunately for us, there is a mirror of the blockchain accessible as a <a href="https://cloud.google.com/bigquery/public-data/">BigQuery public dataset</a>. Using the same dataset as the companion post¹ we investigate here what is the best day in a week to submit transactions to the blockchain in order to save on gas prices. Since the first few years of the blockchain history are somewhat unique¹, we only analyze transactions data for recent years, i.e. since January 2018.</p><p>In the following chart (Fig.1), we show the <strong>hourly-average throughput</strong>, measured in transactions per second (TPS), for the Ethereum blockchain from Jan-2018 onwards:</p><figure><img alt="Chart of the hourly-average throughput for the Ethereum blockchain for recent years" src="https://cdn-images-1.medium.com/max/1024/1*Z6UmyF4IgkdElJRxf2UX3A.png" /><figcaption>Fig.1: hourly-average throughput for the Ethereum blockchain for recent years, i.e. since January 2018</figcaption></figure><p>In the above chart, the<strong> thick red graph</strong> plots the <strong>weekly average throughput</strong>. We can immediately recognize that the above process is not <em>stationary</em> since even the weekly average is not constant. A <a href="https://en.wikipedia.org/wiki/Stationary_process">stationary process</a> is a stochastic variable that may change with time, for which the statistical properties, such as the mean and the standard deviation, are constant. When performing a statistical analysis it is easier to work with <a href="https://en.wikipedia.org/wiki/Stationary_process">stationary processes</a>. Hence, in order to make our variables stationary we <strong>compute the surplus</strong>, i.e. the percentage increase, of the hourly-average with respect to the weekly average. For the throughput we obtain the following result:</p><figure><img alt="Chart of the surplus of the hourly throughput measured as a percentage of its week-average" src="https://cdn-images-1.medium.com/max/1024/1*ysqME9VqrID5ZJAuuhZUsg.png" /><figcaption>Fig. 2: surplus of the hourly throughput measured as a percentage of its week-average</figcaption></figure><p>Similarly for the gas price we have:</p><figure><img alt="Chart of the surplus of the hourly gas price measured as a percentage of its week-average" src="https://cdn-images-1.medium.com/max/1024/1*ODaGeaz71A_jeFVsxKKv2A.png" /><figcaption>Fig. 3: surplus of the hourly gas price measured as a percentage of its weekly average. (For visual convenience the percentages higher than 300% are not shown.)</figcaption></figure><p>The charts shown in Fig. 2 and 3 are remarkably different since the <em>throughput surplus</em> is <strong>mostly limited to 100%</strong>, while the <em>gas-price surplus</em> is <strong>often above 100%</strong>. While the statistical analysis of these two variables is beyond the scope of this work, even a casual glance at the two charts reveals that the two distributions are noticeably different.</p><p>Since the surplus variables are stationary, or rather almost (-quasi) stationary, we <strong>can compare their values at different dates</strong>. In the companion post¹ we indicate the hourly dependence of these variables, here we compute the <strong>average surplus conditionally to the day of the week</strong>:</p><figure><img alt="Chart of the average surplus of gas prices and throughput for each day of the week" src="https://cdn-images-1.medium.com/max/1024/1*e_paP5ZsAi7gIwV1fVTreg.png" /><figcaption>Fig. 4: average surplus of gas prices and throughput for each day of the week</figcaption></figure><p>As someone might have suspected by the subtitle of this post, for both variables we observe a positive average surplus during the weekdays and a negative one on weekends. This means that in order to save on gas, on average, it is better to wait for the weekend to submit transactions to the Ethereum blockchain.</p><p>Remarkably the average surplus during each day is quantitatively similar for both variables. This is surprising since, as noted earlier, the two variables have markedly different statistical properties. Indeed, there must be a close relationship between these two variables as they influence each other. This relationship will be explored in a future work.</p><h3>Weekend hourly periodicity results</h3><p>In the companion paper¹ the conclusion was that it is better to wait for the Asian nighttime in order to submit transactions with lower gas prices. In the previous section we observed lower throughput and gas prices, on average, during the weekends. Suppose that we want to follow both these advices, a natural question arises:</p><p><em>What is the best time, on a weekend to submit transactions?</em></p><p>To answer this question we compute the <strong>average of the variable surpluses</strong> <em>conditional</em> to both the <strong>hour </strong>and the <strong>day</strong> concurrently. Note that the hours are always measured using the <a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">Coordinated Universal Time</a> (aka UTC). The results for Saturdays are shown in the following chart:</p><figure><img alt="Chart of the average surplus of gas prices and throughput for Saturdays" src="https://cdn-images-1.medium.com/max/1024/1*JviB7ARmaDMlIFQOg7ADjw.png" /><figcaption>Fig. 5: Average surplus of gas prices and throughput for Saturdays</figcaption></figure><p>Similarly, for Sundays we have</p><figure><img alt="Chart of the average surplus of gas prices and throughput for Sundays" src="https://cdn-images-1.medium.com/max/1024/1*_8yhfRvBF3eI8XOpuCrcsw.png" /><figcaption>Fig. 6: Average surplus of gas prices and throughput for Sundays</figcaption></figure><p>The result for both days are very similar and there is no obvious choice of which day would give more benefits.</p><p>We also notice that the gas-price surplus <strong>stays negative for most of the weekend</strong>, with <strong>small positive peaks </strong>at 2 pm and 3 pm UTC on both Saturdays and Sundays.</p><p>On the other hand, the<strong> throughput is more irregular</strong> and has some noticeable positive peeks at 7 am and 8 am UTC time. Again the best saving on gas prices are during the Asian after hours, i.e. <strong>between 6 pm UTC </strong>(2 am in Singapore) and <strong>midnight UTC </strong>(8 am in Singapore).</p><h3>Combined day and hourly periodicity results</h3><p>We have established that the best time to save on gas prices, on average, is to submit transactions between 2 am and 8 am Singapore time. This is consistent with Singapore being one of the most vibrant places for companies in the DeFi space. However, one might have the <strong>urgency</strong> to submit transaction on any given day of the week. Hence we may ask ourselves the following question:</p><p><em>On any given day, what is the hourly distribution of gas prices and throughput?</em></p><p>To answer this question we plot, separately for each day, the hourly average of the two variables under consideration. For the gas price we have:</p><figure><img alt="The 7 charts of the average surplus of gas price for each single day of the week" src="https://cdn-images-1.medium.com/max/1024/1*rWHFlBueVHCaYGAbV_ZFjg.png" /><figcaption>Fig. 7: average surplus of gas price for each single day of the week</figcaption></figure><p>While for the throughput we have:</p><figure><img alt="The 7 charts of the average surplus of throughput for each single day of the week" src="https://cdn-images-1.medium.com/max/1024/1*0egSiCHLi_563y6iMya6Ew.png" /><figcaption>Fig .8: average surplus of throughput for each single day of the week</figcaption></figure><p>Again, qualitatively on any given day the shape of the plot is similar for both variables, as noticed earlier, this is an indication of the <strong>co-dependence of these two variables</strong>. Also, in both cases the shape of the charts for the weekends are different from those in the weekdays. Looking a bit more carefully at the chart for the weekdays, we notice more uniformity in the charts for the throughput and less in the charts for the gas price. Specifically the shape of the hourly gas-price surplus on <strong>Thursday</strong> is different from the other days and has an <strong>higher peak of about +30% around noon UTC</strong>.</p><p>It is sometimes useful to look at the same data from different point of views. Very often we can plot the same data using different chart types to obtain more insights from the data. In this case, it is useful to look at the percentage surplus variables by plotting them on a <strong>two-dimensional heatmap</strong>. The gas-price heatmap is shown in the following chart:</p><figure><img alt="Chart of the surplus gas-price heatmap for each day of the week (the vertical coordinate) and hour (the horiz. coordinate)" src="https://cdn-images-1.medium.com/max/1024/1*db_SbUBinf44h19ow-Ex-g.png" /><figcaption>Fig. 9: the percentage surplus gas-price heatmap for each day (the vertical coordinate) and hour (the horizontal coordinate) of the week</figcaption></figure><p>Similarly, the throughput heatmap is shown in the following chart:</p><figure><img alt="Chart of the surplus throughput heatmap for each day of the week (the vertical coordinate) and hour (the horiz. coordinate)" src="https://cdn-images-1.medium.com/max/1024/1*a3wo2RQ_eyiDVc2ALhEzFg.png" /><figcaption>Fig. 10. the percentage surplus throughput heatmap for each day of the week (the vertical coordinate) and hour (the horizontal coordinate)</figcaption></figure><p>Both heatmaps are characterized by a dark-red (higher surplus) zone during the Asian working day on weekdays and by a dark blue zone (lower surplus) during the Asian weekend nights. In the throughput heatmap the vertical lines are more apparent while the gas-price heatmap looks more jagged.</p><p>This means that hourly throughput <strong>does not depend</strong> much on the exact day it is observed, either on weekdays or on weekends. Also we note that the <strong>worst time</strong> during the week for gas prices is observed on <strong>Thursdays at UTC noon</strong>, or 8 pm in Singapore. On the other hand the throughput seems to be consistently highest on all weekdays at 8 am UTC time, or 4 pm in Singapore.</p><h3>Conclusions</h3><p>In this post we show the dependence of the average surplus in gas prices and throughput for each day of the week and each hour.</p><p>As shown by the heatmap charts above, it turns out that both variables show a strong dependency on the hour and the day <strong>regardless of weekday or weekend considerations</strong>.</p><p>We also establish that, in order to save on gas prices, the <strong>best time</strong> is to submit our transaction is the <strong>early weekend morning </strong>Singapore time<strong> </strong>and the <strong>worst time</strong> is on <strong>Thursdays at around 8pm</strong>, Singapore time.</p><p>Since the Ethereum blockchain does not sleep and does not take weekend breaks, the fact that there is indeed a time periodicity means that transactions are submitted to the blockchain by humans, according to their own schedule. With the <strong>growing presence</strong> of sleepless and unfaltering <strong>bots</strong>, and the fast evolution of artificial intelligence, we expect to observe in the years to come <strong>less and less hourly or weekly periodicity</strong> on transactions in all blockchains.</p><p><strong>Footnotes</strong></p><ol><li>Refer to <a href="https://medium.com/dextf/when-is-the-best-time-to-send-ethereum-transactions-80d53ae8e8d1">this earlier post</a> in which we discusses the average hourly gas price and throughput</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6689ac09fe6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dextf/ethereum-how-to-save-even-more-on-gas-price-with-a-weekly-plan-c6689ac09fe6">Ethereum: how to save even more on gas price with a weekly plan</a> was originally published in <a href="https://medium.com/dextf">DEXTF Protocol</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When is the best time to send Ethereum transactions?]]></title>
            <link>https://medium.com/dextf/when-is-the-best-time-to-send-ethereum-transactions-80d53ae8e8d1?source=rss-507818b3b756------2</link>
            <guid isPermaLink="false">https://medium.com/p/80d53ae8e8d1</guid>
            <category><![CDATA[dextf]]></category>
            <category><![CDATA[etherum-gas-price]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Marco Marchioro]]></dc:creator>
            <pubDate>Sun, 26 Jul 2020 00:43:19 GMT</pubDate>
            <atom:updated>2020-07-30T01:39:41.484Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9F2OIvsNWZ_LFh10H-EHYQ.jpeg" /><figcaption>(Title picture taken from <a href="https://www.hdwallpaper.nu/">https://www.hdwallpaper.nu</a>)</figcaption></figure><p>An analysis of the hourly periodicity of the Ethereum blockchain shows that the transactions are busiest from the start of the Asian business day through to the end of the Russian one</p><p>When I started to use the Ethereum blockchain a few years ago, I was rather happy that I was able to <strong>transfer</strong> tokens <strong>relatively fast</strong> compared to the infrequent transactions of Bitcoin. As I became a more sophisticated user, I started to interact with smart contracts and hence I paid more attention to the <strong>gas price</strong>. Indeed, while sending ETH, or any ERC-20 token, may consume little gas, contract interactions can work out to a hefty sum. In recent months this problem has re-emerged as the rate of utilization of the Ethereum blockchain rose exponentially with new and exciting DeFi projects gobbling up bandwidth which has very clearly inflated gas prices these days.</p><p>Recently, while working on the <a href="https://dextf.com/">DEXTF</a> project, I had to deploy a number of smart contracts on the Ethereum blockchain and had to practically experience how this task was becoming harder and harder. And this is because:</p><ol><li>More and more transactions were sent to the blockchain</li><li>Gas prices have risen close to an all-time high</li></ol><p>However I also noticed that I could somewhat mitigate these problems by sending the transactions at specific times during the day. Since I did not find much information on the internet on the hourly periodicity of transactions on the Ethereum blockchain, I decided to investigate a bit further by myself.</p><h3>Obtaining the Ethereum transaction data</h3><p>The main feature of any <em>public blockchain</em> is in its name: all the data is public. However running an Ethereum node, so that we have access to the whole blockchain in real time, is not an easy task. Fortunately there is a mirror of the Ethereum blockchain accessible as a <a href="https://cloud.google.com/bigquery/public-data/">BigQuery public dataset</a>. Anybody can access these datasets by logging in as a registered user (some restrictions may apply, please read and consult the T&amp;Cs before using it). Once we reach the public-dataset page, we can easily find Ethereum among the blockchains that are available.</p><p>Since the gas price used is stored in the transaction itself we can write a query that aggregates transactions by date and hour.</p><p>In the next figure I show an example of a query that can be run on the <a href="https://cloud.google.com/">Google Cloud Platform</a> to obtain aggregate transaction dates for each hour since the Ethereum genesis block.</p><figure><img alt="Screenshot of the Google cloud platform web interface for Ethereum public BigQuery dataset" src="https://cdn-images-1.medium.com/max/1024/1*ZQ8S78U0eLo5R33lBaygGA.png" /><figcaption>Screenshot of BigQuery results for querying Ethereum transaction by date and hour</figcaption></figure><p>Using the BigQuery web interface we can click the <strong>Run</strong> button, wait for the query job to finish, and save the data either locally (only for a limited number of rows) or on the Google Drive associated to the account.</p><p><em>Note that in this study we use ETH as the main currency and we do not compute the gas price for any other currency.</em></p><h3>Ages of the Ethereum blockchain</h3><p>At this point we assume that we have all the data we need to perform the analysis on the hourly periodicity of transactions. There are many tools that can be used to process the data. My favorite method is to import everything into a J<a href="https://jupyter.org/documentation">upyter Notebook</a> and use the P<a href="https://pandas.pydata.org/">ython/Pandas tools</a> to analyze the data. In any data analysis it is always very important to clean the data in the so-called pre-processing phase:</p><ol><li>Eliminate the first couple of weeks of data, since at the beginning of the Ethereum blockchain there was a very high gas price and very low number of transactions per second (TPS).</li><li>Eliminate the very large outliers in the gas price as they might substantially alter the results of averages.</li><li>Round the fields <em>min_block_timestamp</em> and <em>max_block_timestamp</em> to the nearest lower/higher hour</li><li>Use the <em>pd.Grouper(freq=’W-MON&#39;)</em> to compute the week-average gas price and throughput (i.e. transaction per seconds, or TPS)</li><li>Use the python package <em>seaborn</em> to obtain charts that are easier on the eyes than the standard <em>matplotlib</em> charts</li></ol><p>I will not bore the reader with the details of the Python code used to process the data. Next, looking at the historical gas price, which it is expressed in <a href="https://www.investopedia.com/terms/g/gwei-ethereum.asp"><strong>Gwei</strong></a>, we notice this variable fluctuates during the blockchain life.</p><figure><img alt="Historical chart of Ethereum gas prices and their week average" src="https://cdn-images-1.medium.com/max/1024/1*HYm_NW6j3rmab6OcEkTAnw.png" /><figcaption>Historical gas prices (in GWei) and their week average. Prices are truncated to 200 GWei for visualization</figcaption></figure><p>In this charts we notice at least four different “<strong>ages</strong>”:</p><ol><li>A first period in 2015 where gas prices were consistently <strong>higher than 50 Gwei</strong></li><li>A second period, roughly in years 2016 and 2017, where the gas price <strong>rarely exceeded 50 Gwei</strong></li><li>A third period, beginning roughly at the end of 2017, where gas prices had a higher volatility but stayed <strong>below 75 Gwei</strong></li><li>The last period, starting sometime in 2020, with gas prices <strong>often</strong> being <strong>above 50 Gwei</strong></li></ol><p>We can compare the historical gas prices with the corresponding TPS, obtained from the hourly data.</p><figure><img alt="Historical throughput, i.e. transactions per seconds, observed for each hour and its week average" src="https://cdn-images-1.medium.com/max/1024/1*UPabIr0qWELE73Up6h3GBg.png" /><figcaption>Historical throughput, i.e. transactions per seconds, observed for each hour and its week average</figcaption></figure><p>Looking at the chart for the historical throughput we can roughly see <strong>four ages</strong> that do not exactly match the periods that we observed for the gas prices:</p><ol><li>A first period of adoption, from the beginning of the blockchain to the end of 2016</li><li>A period of exponential growth of throughput, ending at the start of 2018. This period coincided with the ICO mania.</li><li>A period of stabilisation, roughly coinciding with the years 2018 and 2019</li><li>A new period of growth roughly starting with 2020</li></ol><h3>Early hourly periodicity of throughput</h3><p>As seen in the previous charts both variables, the gas prices and the throughput (TPS), vary widely in different periods of their history. Hence it is not appropriate to compare directly the variables at different times. In order to be able to carry out the comparison we need to make the variable<strong> time series</strong> <em>quasi stationary</em>, following these steps:</p><ol><li>We split the whole transaction history into weeks, starting from Monday</li><li>We take the week average for each variable</li><li>For each variable, we compute the percentage increase, namely the surplus, with respect to the week average</li></ol><p>When computing the variable increase, we choose to use the week average rather than the day average so that at a <em>later stage</em> we can also look at the <strong>effects of the</strong> <strong>weekday periodicity</strong> (the weekday-periodicity results will be published in a future post).</p><p>We first look at the chart for the hourly average of the throughput variable:</p><figure><img alt="Chart of the hourly throughput, measured in transaction per seconds (TPS), using the whole blockchain history" src="https://cdn-images-1.medium.com/max/1024/1*WkmjXaZpKTNkZODGO0YLbg.png" /><figcaption>Hourly throughput, measured in transaction per seconds (TPS), using the whole blockchain history</figcaption></figure><p>Note that the hours are always measured using the <a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">Coordinated Universal Time</a> , or shortened as UTC. <strong>At first sight</strong>, the above chart doesn’t seem to show any daily periodicity and this contradicts our experience that some hours are better than others. Also, we note that there are four relative maxima at 2 am, 9 am, 2 pm, and 9 pm. This seems to be too much of a coincidence.</p><p><em>It looks like somebody has been using the blockchain specifically at these four hours during the day.</em></p><p>To further our understanding of the the data, we proceed to split the weeks by years. Then we plot the hourly excess throughput for the years 2015 and 2016:</p><figure><img alt="Chart of the hourly throughput, measured in transaction per seconds (TPS), for the years 2015 and 2016" src="https://cdn-images-1.medium.com/max/1024/1*MbIezKF2IoBDqPQOCluTGQ.png" /><figcaption>Hourly throughput, measured in transaction per seconds (TPS), for the years 2015 and 2016</figcaption></figure><p>It turns out that in the year 2015 transactions were concentrated at 9 am and 9pm, while in the year 2016 transactions were mostly completed at 2 am, 9 am, 2 pm, and 9 pm.</p><p><em>This means that we should not use the earlier years of the Ethereum blockchain to compute the hourly averages.</em></p><p>I am sure that there is a good explanation for the reason why, in Ethereum’s first few years, transactions were concentrated in specific hours. (Contact me, please, if you know the reason as I could not find it on the internet.)</p><h3>Recent hourly periodicity of throughput and gas prices</h3><p>Since the early years seem to have peculiarities, I have decided to discard the data before 2018 in order to have a clean view of the hourly periodicity of the two variables under observation: the gas price and the throughput.</p><p>Therefore, let’s look at the charts for our variables for <strong>each of the last three years</strong>. For the throughput, measured in transactions per seconds, we have:</p><figure><img alt="Chart of the average throughput increase/decrease for each hour (UTC) separately for the year 2018, 2019, and 2020" src="https://cdn-images-1.medium.com/max/1024/1*fbwCTwORL0kyDOdyoIOX5Q.png" /><figcaption>Average throughput increase/decrease for each hour (UTC) separately for the year 2018, 2019, and 2020</figcaption></figure><p>While for the gas price we have</p><figure><img alt="Chart of the average gas-price increase/decrease for each hour (UTC) separately for the year 2018, 2019, and 2020" src="https://cdn-images-1.medium.com/max/1024/1*fbwCTwORL0kyDOdyoIOX5Q.png" /><figcaption>Average gas-price increase/decrease for each hour (UTC) separately for the year 2018, 2019, and 2020</figcaption></figure><p>Both charts have very similar features. Starting from the year 2018 we seem to have been developing a growing dependence of the gas price and the throughput from the time of the day. In the year 2018 the relative increase/decrease, does not jump often above an absolute value of 10%. In the following years, for the gas price we had a more accentuated increase in the hourly periodicity, while for the throughput this periodicity was higher in 2019 than it is in 2020.</p><p>It is also interesting to compare the hourly dependence for both variables on the same chart:</p><figure><img alt="Chart of the verage hourly increase/decrease for each hour (UTC) for the gas price and the throughput" src="https://cdn-images-1.medium.com/max/1024/1*Y5Yn1tcyx1snuPgn9WLpjw.png" /><figcaption>Average hourly increase/decrease for each hour (UTC) for both variables, the gas price and the throughput, together for all recent dates (2018 and after)</figcaption></figure><p>This chart clearly shows that <strong>both observed variables</strong> have a very <strong>similar periodicity</strong> during a single day. We note that the increase in the gas price and the throughput start both at around 2 am UTC and end around 4 pm UTC.</p><p><em>Interestingly this more or less coincides with the start of the Asian business day and ends approximately with the end of the Russian business day.</em></p><p>Therefore looking at this chart one is tempted to say that the transactions on the Ethereum blockchain are preponderantly submitted during the<strong> Asian</strong> and <strong>Russian</strong> daytime.</p><h3>Conclusion</h3><p>In conclusion we found the answer to our original question: <em>when is the best time to send Ethereum transactions?</em></p><p>The best time to send a transaction in order to <strong>minimise</strong> the <strong>gas price</strong> and to increase its chance of being picked up, i.e. in periods of low throughput, is between<strong> 8 pm and 11 pm UTC</strong>.</p><p>Note that one important variable, which for the purpose of this study was not considered, is the weekday periodicity, i.e. how the choice of the day during the week is also a very important variable to consider.</p><p>The results that includes this type of periodicity will be published in a separate study.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80d53ae8e8d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dextf/when-is-the-best-time-to-send-ethereum-transactions-80d53ae8e8d1">When is the best time to send Ethereum transactions?</a> was originally published in <a href="https://medium.com/dextf">DEXTF Protocol</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are we past the Coronavirus peak?]]></title>
            <link>https://medium.com/@quantisland/are-we-past-the-coronavirus-peak-3f8562dde283?source=rss-507818b3b756------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f8562dde283</guid>
            <category><![CDATA[covid-19-crisis]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[mathematical-modeling]]></category>
            <category><![CDATA[quant]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[Marco Marchioro]]></dc:creator>
            <pubDate>Wed, 08 Apr 2020 13:09:05 GMT</pubDate>
            <atom:updated>2020-04-09T09:11:14.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTbwMmixbjd6WqljWIYzug.jpeg" /></figure><h3>Are we past the Coronavirus first peak?</h3><h4><em>The computation of almost-smooth trends for public data related to the Coronavirus pandemic give us a new hope</em></h4><p>In my life as a <em>quant</em>, otherwise boringly referred to as <em>financial quantitative analyst</em>, I deal every day with temporal sequences of numbers that I need to analyze and from which I need to extract as much information as possible. Since the outbreak of the novel Coronavirus we have been bombarded by the news fraught with a large number of data about the pandemic evolution such as: how many cases/deaths/recoveries are present in each region, state, or country and so on and so forth. I am afraid that most people do not have enough statistical background to comprehend the pandemic data or worse, they might be misled to read numbers that they do not understand. In an attempt to mitigate this issue, I have devised a quantitative method to extract almost-smooth trend figures from the published daily numbers, and believe that this new method can improve our understanding of the on-going Coronavirus pandemic.</p><h3>Results of quantitative trends computations</h3><h4>Trend of cases in Singapore</h4><p>As an example let’s take the following panel published on the Wikipedia page for the official cases observed in<a href="https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_Singapore"> Singapore</a> as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*zIE9QesA6u4OpMG9N5PLDA.png" /><figcaption>Number of cases and deaths in Singapore together with their percentage increase (source: <a href="https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_Singapore">wikipedia</a>)</figcaption></figure><p>Let us disregard the statistics related to deaths in Singapore and consider the number of registered cases and its previous-day percentage increase. In just the first seven days of April 2020, we have a percentage increase that goes as low as 4.9% and as high as 10%. I am of the opinion that both the public and policy-makers would have trouble in interpreting these numbers as they are. With the help of some statistical methods, which are also traditionally used in quantitative finance, I was able to compute a much smoother figure for the trend of cases, as shown in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mGuptsOVjkcCh08N8vJoPw.png" /><figcaption>Percentage increases of cases vs model computation of trend</figcaption></figure><p>In the above chart the blue line represents the percentage increase computed from the observed data (the number published by Wikipedia) and the red line represents the trend computed by the quantitative model. While the interpretation of the blue line is somewhat problematic, the red line clearly shows a decrease of the trend until the end of March 2020 and a plateau thereafter, with a possible small increase in the trend in the first week of April.</p><p>To further understand how Singapore reached the trends shown in the chart above, let’s extend the plot back to the whole month of March 2020, as shown in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UHNsklOSCxDr4JpOwqRY8g.png" /></figure><p>By looking only at the blue line, i.e. the daily percentage increase of the cases, we are not sure what happens. However a bit more of insight can be gained by observing the computed quant trend, shown as the red line. Indeed the red line shows that there was an increase in the trend of new cases in the first two weeks of March, followed by a flat trend in the third week, and a decrease at the end of the month. What we see here is <strong>evidence of a peak</strong> for the trend of Coronavirus cases happening around the third week of March 2020.</p><p>As far as I understand most people don’t care about number (which is a pity!). People only want to know when the pandemic will end or at least when they will be able to return to the freedom that was taken for granted only a few months ago. I believe that policy makers could also benefit from a timely and more precise measurement of the effectiveness of the quarantines measures they have implemented. The quantitative trends previously computed can be used to measure what is the current state of Coronavirus figures with a greater accuracy than possible otherwise.</p><h4>World trends: have we reached the first peak?</h4><p>I believe I have demonstrated how the computation of a quantitative trend can turn into a useful tool in understanding the current state of the Coronavirus pandemic. Let us turn our attention to the trend figures for both the global number of cases and deaths:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U36TsgUfEChvEj2wHbGJnQ.png" /><figcaption>Computed trends for worldwide confirmed cases and deaths</figcaption></figure><p>In this chart we can notice that the trends for confirmed cases and deaths run parallel to each other (unfortunately the death rate is on top) and that a peak for both variables was reached also in the second half of March 2020. Returning to the chart above, we notice that we cannot really talk about a peak but rather we have a plateau that extends for many days (about ten days).</p><p>Please note that the fact that we had a local maxima for the world trend of cases doesn’t necessarily mean that this trend will be descending forever, that really depends on so many other variables. The computation of trends is only able to tell us what is the most recent figure, and it should not be understood as a multi-day forecast.</p><h4>Trends in virtuous European countries</h4><p>We should also look at the computed trends for the confirmed cases in three virtuous European countries. In my opinion these countries had a reasonably good response to the pandemic, and thus were all able to lower their trends in a reasonable amount of time. The trends for the confirmed cases of Austria, Italy, and Switzerland are shown in the following chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TpnvZlCPjGJ9l0aFlVdZkA.png" /><figcaption>Computed trends of confirmed cases in three virtuous European countries</figcaption></figure><p>Notice how the trends of Austria and Switzerland were able to drop to the lower final rates of Italy at a quicker pace. In order to quantify how much faster Austria and Switzerland are with respect to Italy, we compute the smoothed day-to-day trend variation as in the following chart:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJGBQ0QZon0SByQSq7AnjQ.png" /><figcaption>Computed smoothed day-to-day variations of the observed COVID-19 cases trend for the three virtuous European countries</figcaption></figure><p>In this chart we notice that while Italy had a day-to-day trend variation mostly between -1% and 0, Austria and Switzerland boasted day-to-day trend variations as low as -3.5%. (The trend variation is therefore a measure of how quickly the trend is changing. Negative values for this number are associated with a decreasing trend, while positive numbers signal an increasing trend.)</p><h3>Quantitative computation of trends</h3><p>This is not a quantitative paper and therefore I don’t want to bother the reader with the full details of the trend computation. However I want to describe here what is the approach that I used and what are the methods that I didn’t use, and why.</p><p>First of all I want to remark that much of the data collected by the different agencies around the world are not perfect and they might be off from the actual count for many different reasons. In addition, since the numbers in any epidemic tend to grow exponentially, all computations were performed on the logarithm of the raw data and then they were adjusted to reflect the actual daily rates. Using the logarithm of the Coronavirus number has the advantage of cancelling the effect of any constant multiplier. Suppose, for example, that one country decides to only report one third of the official cases (who would do that?). By applying the logarithm to the raw data allows us to compute the actual trends without knowing the value of the constant multiplier.</p><h4>How the trend was not computed</h4><p>Since the percentage increase is a <em>noisy</em> variable, one is tempted to take the running averages of the data over a certain number of days (usually five or ten days). These averages are also known as <a href="https://en.wikipedia.org/wiki/Moving_average">moving averages</a>. This method would produce smoother results, however it has at least two major shortcomings. Firstly the trends thus obtained suffer a lag, typically about half of the number of days used in the running average, so that we are never up to date with the latest state available. Secondly, all days considered in the average have the same weights and an old observation falling out of the average can affect the result as much as a new incoming observation; thus resulting in unexplained changes in the trend.</p><p>To overcome the shortcomings of the previous method, one could think of using an <a href="https://school.stockcharts.com/doku.php?id=technical_indicators:moving_averages">exponentially decreasing moving average</a>. However while this is definitely an improvement over the fixed-window time average, it is not clear what should be the exponent that dictates the rate of decrease of older weights.</p><p>Another approach that I have seen adopted by newscasters is polynomial fitting of the whole trend curve. This method provides very smooth trend curves that stretch from the first published data to the last. The problem with this method is that in order to fit the first part of the curve, which is typically exponential, monomials of high degrees have non-negligible coefficients. These large coefficients are then affecting the end part of the curve, typically giving artificially high convexities as opposed to the more natural almost-linear nature of the trend after the peak</p><h4>How the trend was computed</h4><p>The approach that I used instead isbased on the <a href="https://en.wikipedia.org/wiki/Filtering_problem_(stochastic_processes)">filtering problem of stochastic processes</a>. More precisely I compute a local log-linear trend using a modification of the <a href="https://en.wikipedia.org/wiki/Kalman_filter">Kalman filter</a>. This filter works in two steps: a prediction step and an update step. In the <strong>prediction step</strong> the model tries to <em>predict</em> the next value of the trend and its error bar. While the<strong> update step</strong> the new observation is compared with the prediction and the prediction method is progressively updated. If the new observation is consistent with the prediction within the error bar then the model doesn’t need any major update and the error bar is reduced. On the other hand, when the new observation is far from the prediction it means that the prediction and the error bar not correct and they are updated accordingly. In this way we obtain at least three advantages compared to the other methods. Firstly new observations do not change the trend significantly, unless they are very far from the previous trend. Secondly the effect of old observations gradually fades away. Thirdly the rate at which old observation become irrelevant are adaptive and are computed from the data themselves. I have looked at the results produced by the models for many days before publishing them and I believe that this method provides reasonable numbers for the computed trends. However feedback from other observers are always welcome.</p><h4>Computation of day-to-day trend variations</h4><p>The computation of trends is similar to the computation of a first derivative, however for a stochastic process. Hence the computation of the day-to-day trend variation should be similar to the computation of a second derivative. However since a smooth trend is not a pure stochastic process we cannot apply the same method to compute it in order to determine its second derivative. Instead we can use a smoothed (three-point) finite difference method that gives proportionally higher weights to latest data.</p><h4>Which data has been used to compute the trend</h4><p>It turns out that the raw data I am using for the computation of trends is taken from the same place where I was studying for my Ph.D. many years ago: the <a href="https://www.jhu.edu/">Johns Hopkins University</a> in Baltimore, Maryland. Indeed the <a href="https://systems.jhu.edu/">Center for Systems Science and Engineering (CSSE)</a> has published, and has ever since been kept updated, a BigQuery database that is public thanks to the support of Google.</p><h3>Final remarks</h3><p>God willing, I will publish daily data of trends and charts either on <a href="https://twitter.com/marchioro_marco">my Twitter account</a>, <a href="https://medium.com/@quantisland">here on Medium</a> , or on <a href="http://sg.linkedin.com/in/marchioro">my LinkedIn account</a>. Contact me on <a href="http://sg.linkedin.com/in/marchioro">LinkedIn</a> if you want to know more details about this model.</p><p>Finally I would like to thank everybody at <a href="https://dextf.com/">DEXTF</a> for helping me in revising the last draft of this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f8562dde283" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>