<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jesús Corrius on Medium]]></title>
        <description><![CDATA[Stories by Jesús Corrius on Medium]]></description>
        <link>https://medium.com/@jesus.corrius?source=rss-d9466f7f5c3d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*nQ6ygCjtRa1tJ2Uh.</url>
            <title>Stories by Jesús Corrius on Medium</title>
            <link>https://medium.com/@jesus.corrius?source=rss-d9466f7f5c3d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:29:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jesus.corrius/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Packaging a .NET Core Service for Ubuntu]]></title>
            <link>https://medium.com/bluekiri/packaging-a-net-core-service-for-ubuntu-4f8e9202d1e5?source=rss-d9466f7f5c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f8e9202d1e5</guid>
            <category><![CDATA[packaging]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[net-core]]></category>
            <dc:creator><![CDATA[Jesús Corrius]]></dc:creator>
            <pubDate>Thu, 15 Nov 2018 10:38:49 GMT</pubDate>
            <atom:updated>2018-11-15T10:38:49.748Z</atom:updated>
            <content:encoded><![CDATA[<h3>Packaging a .NET Core Service for Ubuntu</h3><p>This is a step-by-step guide on how to package a <a href="https://www.microsoft.com/net/download/dotnet-core/2.1">.NET Core 2.1</a> service for <a href="http://releases.ubuntu.com/18.04/">Ubuntu 18.04 LTS</a>, but most of the concepts explored here should work for other versions of .NET Core and/or any Debian-based Linux distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*swZLW_updEGFnWLb" /><figcaption>Photo by <a href="https://unsplash.com/@kadh?utm_source=medium&amp;utm_medium=referral">Kira auf der Heide</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Building a demo API on Linux</h3><p>We will start the tutorial by building a very simple API on Linux and the first step is to install the .NET Core SDK on our working environment following the <a href="https://www.microsoft.com/net/download/linux-package-manager/ubuntu18-04/sdk-current">instructions</a> from the official website:</p><pre>$ wget -q <a href="https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb">https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb</a><br>$ sudo dpkg -i packages-microsoft-prod.deb<br>$ sudo apt-get install apt-transport-https<br>$ sudo apt-get update<br>$ sudo apt-get install dotnet-sdk-2.1</pre><p>Next, we will create a very simple application using the provided webapi template:</p><pre>$ dotnet new webapi -o demoapi<br>$ cd demoapi<br>$ dotnet run</pre><p>That was easy! Now the sample application should be up and running on your system.</p><p>Before we complete this task, we are going to make a couple of modifications to the source code to make the packaging process less complicated. First, we are going to make the application <a href="https://docs.microsoft.com/en-us/dotnet/core/deploying/">self-contained</a> so all the dependencies of our service are included in the package and are isolated from other applications installed in the system. This saves us from the trouble of distributing the .NET Core libraries and the .NET Core runtime.</p><p>For this purpose we create a <em>&lt;RuntimeIdentifiers&gt;</em> tag in the <em>&lt;PropertyGroup&gt;</em> section of your <strong>demoapi.csproj</strong> file that defines the <a href="https://docs.microsoft.com/en-us/dotnet/core/rid-catalog">runtime identifier</a> of our service targets, that is: <em>ubuntu.18.04-x64:</em></p><pre>&lt;PropertyGroup&gt;<br>    &lt;RuntimeIdentifiers&gt;ubuntu.18.04-x64&lt;/RuntimeIdentifiers&gt;<br>&lt;/PropertyGroup&gt;</pre><p>Also, to further reduce the size of the deployment we are going to the use <a href="https://github.com/dotnet/corefx/blob/master/Documentation/architecture/globalization-invariant-mode.md">globalization invariant mode</a> for NET Core, which also has the advantage to remove the dependency from <a href="https://packages.ubuntu.com/bionic/libicu60">libicu</a>. To activate it, we also add the following section to the csproj:</p><pre>&lt;ItemGroup&gt;<br>    &lt;RuntimeHostConfigurationOption Include=&quot;System.Globalization.Invariant&quot; Value=&quot;true&quot; /&gt;<br>&lt;/ItemGroup&gt;</pre><p>The final version of the <strong>demoapi.csproj </strong>should look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3eea4e81012b022594c978078aa6817c/href">https://medium.com/media/3eea4e81012b022594c978078aa6817c/href</a></iframe><p>Now we can run the following command to build the self-contained service:</p><pre>$ dotnet publish -c Release --self-contained -r ubuntu.18.04-x64</pre><p>Once the project is built, the generated binaries are located in: <em>bin/Release/netcoreapp2.1/ubuntu.18.04-x64/publish/</em></p><p>The PDBs for the service code are already generated in the publish directory mentioned above, but we might want to debug the service in production or analyse performance issues, in this case is good have the symbols for the .NET Framework and the native runtime DLLs (typically <em>libcoreclr.so</em>).</p><p>Fortunately, .NET Core 2.1 comes with an excellent symbols download tool named <a href="https://www.nuget.org/packages/dotnet-symbol/">dotnet-symbols</a> that makes our lives very easy (<strong>note:</strong> you may need to restart the shell after the first command):</p><pre>$ dotnet tool install -g dotnet-symbol<br>$ dotnet-symbol --symbols bin/Release/netcoreapp2.1/ubuntu.18.04-x64/publish/</pre><h3>Packaging the demo API</h3><p>The first step to package our application will be to install and configure the Ubuntu packaging tools, additionally we will set two environment variables that the tools will use later on to customise it:</p><pre>$ sudo apt install gnupg pbuilder ubuntu-dev-tools apt-file dh-make bzr-builddeb<br>$ export DEBEMAIL=&quot;service@bluekiri.com&quot;<br>$ export DEBFULLNAME=&quot;Bluekiri Demo&quot;</pre><p>Now we have to tidy up the source code in order to create a clean source package:</p><pre>$ cd demoapi<br>$ dotnet clean<br>$ rm -rf bin obj<br>$ cd ..</pre><p>For the tools to work, you should make sure the source directory is in the format <em>&lt;name&gt;-&lt;version&gt;</em>. This is only necessary the first time we run the packaging tools and we can revert to the original name afterwards (yes, it’s possible to override the requirement, but we’ll follow the best practises here). Then we can create the compressed package with the source code of the application:</p><pre>$ mv demoapi demoapi-1.0<br>$ tar cvzf demoapi-1.0.tar.gz demoapi-1.0</pre><p>At this point, we have everything ready to run the helper to prepare a Debian package for our original source archive. The <a href="http://manpages.ubuntu.com/manpages/trusty/en/man8/dh_make.8.html">dh_make</a> tool has many options, here we are going to specify that we have already created our own source file, we want to create a single binary native package and that we are going to use the <a href="https://opensource.org/licenses/MIT">MIT license</a> for it:</p><pre>$ cd demoapi-1.0<br>$ dh_make -f ../demoapi-1.0.tar.gz -s -c mit -n</pre><p>The main things to notice is that the command has generated a new folder named <em>debian </em>on the top level directory of your application. We are going to spend the rest of this section customising its contents.</p><p>We’ll start by deleting some of the optional example files and documentation that we are not going to use in our project:</p><pre>$ cd debian<br>$ rm *ex *EX<br>$ rm README README.Debian README.source demoapi-docs.docs</pre><p>The remaining files are all important.</p><p>Our first task will be to modify the <strong>changelog</strong> file. We will edit it to change the version number to an Ubuntu version: <em>1.0–0ubuntu1</em> (upstream version 1.0, Debian version 0, Ubuntu version 1). Also change unstable to the current development Ubuntu release to<em> bionic</em>. The final result should look very similar to this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cf32b0bd9d220192da2baffa1f15ed4/href">https://medium.com/media/7cf32b0bd9d220192da2baffa1f15ed4/href</a></iframe><p>One of your future tasks is to mantain this <strong>changelog</strong> file up-to-date for any new version of the package. We can edit the file manually or use <a href="http://manpages.ubuntu.com/manpages/trusty/man1/debchange.1.html">debchange</a> this purpose.</p><p>The very important <strong>control</strong> file contains all the metadata of the package. The first paragraph describes the source package. The second and following paragraphs describe the binary packages to be built.</p><p>In our version of this file we are going to define the metadata for our main package and the debug package. Please note that the debug package depends on the main package and that the main package has all the <a href="https://docs.microsoft.com/en-us/dotnet/core/linux-prerequisites?tabs=netcore21">binary dependencies</a> needed to run .NET Core in Linux, as the runtime is embedded in your own application. Also notice that one of the build dependencies of the source package is <em>dh-systemd</em>, a helper to generate a <a href="https://www.freedesktop.org/wiki/Software/systemd/">systemd</a> service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27a179623c8c24131a9498d200cd7555/href">https://medium.com/media/27a179623c8c24131a9498d200cd7555/href</a></iframe><p>The last file, <strong>rules</strong>, is the most complex of all of them. This one is basically a Makefile which compiles the code and turns it into a binary package.</p><p>In our version, we override some of the build helpers processes to build and install the application. We have seen the build process in the previous section, so in this file we just launch the same command.</p><p>To install the the application, we copy both the application files and the symbols to the <em>/opt/bluekiri/demoapi </em><a href="https://wiki.debian.org/FakeRoot">fakeroot</a> of both packages.</p><p>The remaining overrides are for tools that are intended to run for other programming languages and are not compatible with our .NET Core application.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f200b73fd43394c49b22558e4811904b/href">https://medium.com/media/f200b73fd43394c49b22558e4811904b/href</a></iframe><p>Finally, we have added a very simple <strong>demoapi.service</strong> file that systemd will use to control our application. You will need a better version for a production ready service (specially from a <a href="http://0pointer.net/public/systemd-nluug-2014.pdf">security</a> point of view), but this one will do the trick.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/940a93891d6c65e10cca8b802be89a6a/href">https://medium.com/media/940a93891d6c65e10cca8b802be89a6a/href</a></iframe><p>With all these files and modifications in place, we can now build the package with the <a href="http://manpages.ubuntu.com/manpages/xenial/man1/dpkg-buildpackage.1.html">dpkg-buildpackage</a> command. Use use the options to generate only the binary files and skip the signing of the package:</p><pre>$ dpkg-buildpackage -b --no-sign</pre><p>The result packages will be generated in the parent directory of the current location. You can install the generated package in the following way:</p><pre>$ cd ..<br>$ sudo dpkg -i demoapi_1.0-0ubuntu1_amd64.deb</pre><p>You can find the full source code of this demo api in the following location: <a href="https://github.com/bluekiri/demoapi">https://github.com/bluekiri/demoapi</a></p><h3>Creating a private Debian package repository</h3><p>If you followed all the steps in the previous section, you should now have a debian package with your application. Now we are going to take a look on how to distribute the application using a private repository. In this tutorial, we are going to first create a local repository and then upload it to the well know <a href="https://github.com/">github</a> service. The later is just for convenience.</p><p>First we are going to install the prerequisties and then generate a private key using <a href="https://www.gnupg.org/">GnuPG</a> that users will later use to verify the integrity of the packages.</p><pre>$ sudo apt-get install gnupg rng-tools<br>$ gpg --gen-key</pre><p>For this tutorial we are just going to use default settings, so basically you only have to type your name and email. Also we are going to use a blank password so it doesn’t ask for it each time.</p><p>Next we will install and configure <a href="http://manpages.ubuntu.com/manpages/xenial/man1/reprepro.1.html">reprepro</a>, a very useful tool produce and manage local repositories:</p><pre>$ sudo apt-get install reprepro<br>$ mkdir repo <strong>&amp;&amp;</strong> cd repo<br>$ mkdir conf <strong>&amp;&amp;</strong> cd conf<br>$ touch distributions</pre><p>Now we need to create a new <strong>distributions</strong> file, which is where we store all the required configuration of the repository. The <em>SignWith</em> field should match the key your created in the previous step. This is a sample configuration, with the minimum required information:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3801d15a54f78d7527967e749342369/href">https://medium.com/media/c3801d15a54f78d7527967e749342369/href</a></iframe><p>Now we can import the generated packages to the local repository and check that are actually there:</p><pre>$ cd ~<br>$ reprepro --basedir ~/repo includedeb bionic demoapi*.deb<br>$ reprepro --basedir ~/repo list bionic</pre><p>In this point, the local repository is ready to go. It’s a good idea to add the local repository to <em>/etc/apt/sources.list </em>to test it before we upload the repo.</p><p>Let’s upload the repository to github, so anyone can test. For that purpose we will first export the public key to the repository directory root:</p><pre>$ gpg --output PUBLIC.KEY --armor --export service@bluekiri.com<br>$ mv PUBLIC.KEY ~/repo</pre><p>Then we upload everything to a previously created github repo:</p><pre>$ cd ~/repo<br>$ git init<br>$ git add --all<br>$ git commit -m &quot;Package repository&quot;<br>$ git remote set-url origin git@github.com:bluekiri/demorepo.git<br>$ git push origin master</pre><p>And done! At this point we have a public private repo in github that contains our application. You can download and install the service in this tutorial following these instructions:</p><pre>$ wget -qO - <a href="https://raw.githubusercontent.com/bluekiri/demorepo/master/PUBLIC.KEY">https://raw.githubusercontent.com/bluekiri/demorepo/master/PUBLIC.KEY</a> | sudo apt-key add -<br>$ echo &quot;deb <a href="https://raw.githubusercontent.com/bluekiri/demorepo/master/">https://raw.githubusercontent.com/bluekiri/demorepo/master/</a> bionic main&quot; | sudo tee /etc/apt/sources.list.d/bluekiridemo.list<br>$ sudo apt-get update<br>$sudo apt-get install demoapi</pre><p>Thank you very much for you attention, I hope you enjoyed this article and you learnt something new. Stay tuned for more content related to .NET Core on Linux.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f8e9202d1e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluekiri/packaging-a-net-core-service-for-ubuntu-4f8e9202d1e5">Packaging a .NET Core Service for Ubuntu</a> was originally published in <a href="https://medium.com/bluekiri">bluekiri</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cointegration tests on time series]]></title>
            <link>https://medium.com/bluekiri/cointegration-tests-on-time-series-88702ea9c492?source=rss-d9466f7f5c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/88702ea9c492</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[timeseries]]></category>
            <dc:creator><![CDATA[Jesús Corrius]]></dc:creator>
            <pubDate>Tue, 09 Oct 2018 06:32:16 GMT</pubDate>
            <atom:updated>2018-10-09T06:32:16.145Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nR2Yzwjcppk1yRJt" /><figcaption>“person using black and gray laptop computer” by <a href="https://unsplash.com/@rawpixel?utm_source=medium&amp;utm_medium=referral">rawpixel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In a <a href="https://medium.com/bluekiri/simple-stationarity-tests-on-time-series-ad227e2e6d48">previous post</a>, we examined the fundamental tools to test for <a href="https://en.wikipedia.org/wiki/Stationary_process">stationarity</a> on <a href="https://en.wikipedia.org/wiki/Time_series">time series</a> using <a href="https://www.python.org/">Python</a>, one of my favorite programming languages. If we use the tools described in the article, we will very soon realise that most time series are neither stationary nor <a href="https://en.wikipedia.org/wiki/Regression_toward_the_mean">mean revertin</a>g. In this new article we are going to examine how we can test two (or more) non-stationary time series to check whether the combined value is stationary.</p><p>This is where we should introduce the notion of <a href="https://en.wikipedia.org/wiki/Cointegration">cointegration</a> .</p><blockquote>If we are able to find a stationary linear combination of several time series that are not themselves stationary, then these are called <em>cointegrated</em>.</blockquote><p>We are going to see two different tests: the simpler Cointegrated <a href="https://en.wikipedia.org/wiki/Augmented_Dickey%E2%80%93Fuller_test">Augmented Dickey-Fuller test</a> (CADF) will be useful for pairs only, but we can apply the <a href="https://en.wikipedia.org/wiki/Johansen_test">Johansen test</a> to any number of time series.</p><h3>Cointegrated Augmented Dickey-Fuller Test</h3><p>In the previous post, we saw how the <a href="https://en.wikipedia.org/wiki/Augmented_Dickey%E2%80%93Fuller_test">ADF</a> and <a href="https://www.mbaskool.com/business-concepts/statistics/7363-variance-ratio.html">Variance Ratio</a> can test a given time series for mean reversion and stationarity, but we don’t know the number of units o percentage we should use to combine them into the stationary basket of elements we are looking for.</p><blockquote>We have to be aware that just because a set of time series is cointegrating doesn’t mean that any random linear combination of the series will form a stationary basket of elements</blockquote><p>To easily create the test we can use the procedure by <a href="https://en.wikipedia.org/wiki/Cointegration#Engle%E2%80%93Granger_two-step_method">Engle and Grange<em>r</em></a>, which can be defined as the following steps:</p><ol><li>Determine the optimal hedge ratio by running a lineal regression fit between the two series.</li><li>Use the hedge computed in <em>step 1 </em>to form a portfolio.</li><li>Run a stationarity test on the portfolio created in <em>step 2.</em></li></ol><p>Taking advantage of the code already written in the <a href="https://medium.com/bluekiri/simple-stationarity-tests-on-time-series-ad227e2e6d48">previous article</a>, we can write the test easily with the help of the <a href="http://www.numpy.org/">numpy</a> and <a href="https://www.statsmodels.org/stable/index.html">statsmodels</a> libraries in the following way:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b15051c841baa42e4b3f047be191e1ba/href">https://medium.com/media/b15051c841baa42e4b3f047be191e1ba/href</a></iframe><h3>Johansen test</h3><p>In order to test for cointegration of more than two variables, we have to use the <a href="https://en.wikipedia.org/wiki/Johansen_test">Johansen test</a>. If we start with the linear model we already described in the previous article:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*4BmM03l6sX3uC2IvlzCXKQ.png" /></figure><p>We can generalize it to the case where the variable <em>y(t)</em> are vectors representing multiple series, and the coefficients <em>λ</em> and <em>α</em> are actually matrices (we are also going to assume <em>βt=0</em> for simplicity) and we can rewrite the equation in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*NiAbS0trQq5_JdkOqeda8w.png" /></figure><p>Just like in the previous case with just one variable, if <em>λ = 0</em> we don’t have cointegration. Let’s assume the rank of <em>λ</em> is <em>r</em> and the number of time series is <em>n. </em>The number of independent baskets that can be formed by different linear combinations of the cointegrating series is equal to <em>r</em>. And the Johansen test will calculate that number for us in two different ways, both of them based on the eigenvector decomposition of <em>λ</em>: the first test produces the trace statistic, and the second one produces the eigen statistic.</p><p>Here you can find a complete implementation of the Johansen Test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d322b8f82c8f77bb8af24bedd13498a/href">https://medium.com/media/3d322b8f82c8f77bb8af24bedd13498a/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88702ea9c492" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluekiri/cointegration-tests-on-time-series-88702ea9c492">Cointegration tests on time series</a> was originally published in <a href="https://medium.com/bluekiri">bluekiri</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple stationarity tests on time series]]></title>
            <link>https://medium.com/bluekiri/simple-stationarity-tests-on-time-series-ad227e2e6d48?source=rss-d9466f7f5c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad227e2e6d48</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jesús Corrius]]></dc:creator>
            <pubDate>Wed, 06 Jun 2018 13:06:29 GMT</pubDate>
            <atom:updated>2018-10-09T09:16:17.904Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBYBJ5wlbcR7WyaIW-mv7w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/ulTMKa88rEM?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tristan Gassert</a> on <a href="https://unsplash.com/search/photos/time-series?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this first post we are going to look for ways to test for mean reversion on time series using the Python programming language, which will give us the basic toolbox to deal with <a href="https://en.wikipedia.org/wiki/Cointegration">cointegration</a> in future posts.</p><p><strong>Testing for mean reversion</strong></p><p>Mathematically, a continuous mean-reverting time series can be represented by an <a href="https://en.wikipedia.org/wiki/Ornstein%E2%80%93Uhlenbeck_process"><em>Ornstein-Uhlenbeck</em></a> stochastic differential equation in the following form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/0*TcoJB47hEIGd7kx7" /></figure><p>Where <em>θ</em> is the rate of reversion to the mean, <em>μ</em> is the mean value of the process, σ is the variance of the process and, finally, <em>Wt </em>is a <a href="https://en.wikipedia.org/wiki/Wiener_process"><em>Wiener process</em></a>.<br>The given equation implies that the change of the time series in the next period is proportional to the difference between the mean and the current value, with the addition of <a href="https://en.wikipedia.org/wiki/Gaussian_noise"><em>Gaussian noise</em></a>.</p><p><strong>Augmented Dickey-Fuller Test</strong></p><p>The <a href="https://en.wikipedia.org/wiki/Augmented_Dickey%E2%80%93Fuller_test">Augmented Dickey-Fuller tes</a>t provides a quick check and confirmatory evidence that your time series is stationary or non-stationary. We should take into account that is a statistical test, and as such it can only be used to inform the degree to which a null hypothesis can be accepted or rejected. So the given result must be interpreted to be meaningful.</p><p>The ADF test is based on the simple observation that if the value level is higher than the mean, the next move will be downward while if the value is lower than the mean, the next move will be upward.<br>We can describe the value changes with the following linear model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*beS2g7ISAa_r_rDm" /></figure><p>where <em>Δy(t) ≡ y(t) — y(t-1)</em>, <em>Δy(t-1) ≡ y(t-1)-y(t-2)</em> and so on.</p><p>The function of the ADF is to test if <em>λ=0</em>. If the hypothesis <em>λ=0</em> can be rejected, that means the next move <em>Δy(t)</em> depends on the current level <em>y(t-1)</em> and so it is not a random walk. The test statistic is the regression coeficient λ divided by the standard error of the regression fit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/145/0*DO-6XgI2-cs-n7Rb" /></figure><p>Dickey and Fuller have <a href="http://www.econ.uiuc.edu/~econ508/DFtable.pdf">already calculated</a> the distribution of this test statistic, which allows us to determine the rejection of the hypothesis for any chosen percentage critical value. Since we expect mean regresion to be negative and it has to be more negative than the critical value for the hypothesis to be rejected.</p><p>In this exercise we will simply interpret the result using the<em> p-value</em> from the test. A <em>p-value</em> below a specified threshold (we are going to use 5%) suggests we reject the null hypothesis (stationary), otherwise a <em>p-value</em> above the threshold suggests we accept the null hypothesis (non-stationary).</p><p>A naive implementation of the ADF test in Python is shown here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d330ebececba93d86f593ae02a189fc5/href">https://medium.com/media/d330ebececba93d86f593ae02a189fc5/href</a></iframe><p>For the verification of our implementation we can compare the output of our implementation with the result of the function <a href="http://www.statsmodels.org/dev/generated/statsmodels.tsa.stattools.adfuller.html"><em>adfuller</em></a><em>, </em>included in the Python module <a href="http://www.statsmodels.org/stable/index.html"><em>statsmodels</em></a><em> </em>from the SciPy ecosystem, when used with the same input.</p><p><strong>Hurst Exponent</strong></p><p>It is also possible to determine the stationary nature of the time series by measuring its speed of diffusion, because a stationary time series should diffuse from its initial value more slowly than a geometric random walk.</p><p>We can model the speed by the variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*0jUXlnYc2Z8_pKv1" /></figure><p>where <em>z</em> is the log values, <em>τ</em> is an arbitrary time lag, and <em>&lt;…&gt;</em> is an average over all <em>t</em>’s. For a geometric random walk, we know that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/0*QhicG5G1cTCXYHFL" /></figure><p>This relationship turns into an equality with some proportionally constant for a large value of <em>τ</em>, but it may deviate for a small <em>τ</em>. So if the time series is mean reverting this won’t hold.</p><p>Now we have to introduce the <a href="https://en.wikipedia.org/wiki/Hurst_exponent">Hurst exponent</a> <em>H</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/0*aEg6DU_2aE0TqN8E" /></figure><p>For a time series with a geometric random walk behaviour, <em>H=0.5</em>, for a mean reverting series, <em>H&lt;0.5</em>, and, finally, for a trending series <em>H&gt;0.5</em>. <em>H</em> also is an indicator for the degree of mean reversion or trendiness: as <em>H</em> decreases towards 0, the series is more mean reverting and as it increases towards 1, it is more trending.</p><p>This is a simple implementation I made of the Hurst Exponent described above. Unfortunately, I couldn’t find any suitable implementation in any SciPy or standard library to compare it to:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1131a8dc06a848326773888dcfdecfa/href">https://medium.com/media/e1131a8dc06a848326773888dcfdecfa/href</a></iframe><p>Because of the finite sample size, we need to know the statistical significance of an estimate value of <em>H</em> to be sure we can reject the null hypothesis that <em>H</em> is 0.5. To verify this, we are going to use the <a href="https://www.mbaskool.com/business-concepts/statistics/7363-variance-ratio.html">Variance Ratio Test</a> which can be expressed as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/0*RnttyBdk96sXf1_U" /></figure><p>We can express this ratio in the following Python code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1609b5ede700dbfda57e7cdce49d3e0/href">https://medium.com/media/d1609b5ede700dbfda57e7cdce49d3e0/href</a></iframe><p><strong>Half-Life of Mean Reversion</strong></p><p>Calculating the <a href="https://en.wikipedia.org/wiki/Half-life">half-life</a> of a mean reversion time series is very interesting because it gives us the measure of how long it takes to mean revert.</p><p>This measure is a way to interpret the λ coefficient in the equation we have already seen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*beS2g7ISAa_r_rDm" /></figure><p>For this new interpretation we have to transform this discrete time series into a differential form while ignoring the drift <em>βt</em> and the lagged differences <em>Δy(t-1) … Δy(t-k)</em> and we get the following <em>Ornstein-Uhlenbeck</em> formula for the mean reverting process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/0*jH9af4UWqOvXTIak" /></figure><p>This form allows us to get an analytic solution for the expected value of <em>y(t)</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/0*58501WUJfi6HpvtB" /></figure><p>If λ is negative for a mean-reverting process, the expected value of the value decays exponentially to the value <em>-μ/λ</em> with the half-life of decay being <em>log(2)/λ</em>.</p><p>Finally, to calculate the half-life mean reversion test we can use this simple implementation in Python:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c841a097adf9196ef45153fd632be9a/href">https://medium.com/media/1c841a097adf9196ef45153fd632be9a/href</a></iframe><p>All this theory and practical examples in Python will help us to understand and master the concept of cointegration, which will be thoroughly discussed in a <a href="https://medium.com/bluekiri/cointegration-tests-on-time-series-88702ea9c492">following post</a>. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad227e2e6d48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluekiri/simple-stationarity-tests-on-time-series-ad227e2e6d48">Simple stationarity tests on time series</a> was originally published in <a href="https://medium.com/bluekiri">bluekiri</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Bootcamp 2018]]></title>
            <link>https://medium.com/bluekiri/azure-bootcamp-2018-b1d9a4b59f49?source=rss-d9466f7f5c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1d9a4b59f49</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cosmosdb]]></category>
            <category><![CDATA[api-management]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Jesús Corrius]]></dc:creator>
            <pubDate>Thu, 26 Apr 2018 09:56:33 GMT</pubDate>
            <atom:updated>2018-04-26T09:56:33.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ipM4k1yfTh41U9YriRUMg.jpeg" /></figure><p>Bluekiri loves <a href="https://azure.microsoft.com/en-gb/">Azure</a>, and our clients love Azure too.<br>Just a few days ago, <a href="https://medium.com/bluekiri/hello-botkiri-f652fd9a756d">we announced</a> a cutting edge bot application, in partnership with Microsoft, to deliver a world class conversational experience to our customers. This solution is hosted, developed, tested, deployed and run on Azure.<br>So it was only natural to us to sponsor and collaborate on the <a href="https://global.azurebootcamp.net/">Global Azure Bootcamp 2018</a>. The <a href="https://www.meetup.com/Netsaimada/events/247126728/">local event</a> was held last weekend in <a href="http://www.parcbit.es/wparcbitfront/">Parc Bit</a> in Palma de Mallorca, organized by the local .NET group <a href="https://www.meetup.com/Netsaimada/">Netsaimada</a>, and it was made possible with the help of many friends and volunteers, including our partners, <a href="https://www.xmltravelgate.com/en">XML Travelgate</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZuGzUTE_0lrT8HyzTcjpA.jpeg" /><figcaption>Bluekiri and XML Travelgate teams at Global Azure Bootcamp 2018</figcaption></figure><p>The event kicked off with a warm welcome and an introduction by <a href="https://www.kenneth-truyers.net/">Kenneth Truyers</a>. Then Carlos Tur, from XML Travelgate, explained how his company migrated from Cloud Services to ARM.</p><p>My talk, and practical demo, was about the <a href="https://azure.microsoft.com/en-us/services/api-management/">API Management</a> service, which is a very powerful tool to, among other things, publish securely existing legacy back-end API on the internet in just a few minutes.</p><p>Finally, <a href="https://twitter.com/leomicheloni?lang=en">Leonardo Micheloni </a>gave us a comprehensive but, at the same time, newbie friendly talk about <a href="https://azure.microsoft.com/en-us/services/cosmos-db/">Cosmos DB</a>.</p><p>It was a fun event and full of great people. I am already looking forward for Azure Bootcamp 2019!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1d9a4b59f49" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluekiri/azure-bootcamp-2018-b1d9a4b59f49">Azure Bootcamp 2018</a> was originally published in <a href="https://medium.com/bluekiri">bluekiri</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas documentation sprint]]></title>
            <link>https://medium.com/bluekiri/pandas-documentation-sprint-90f5a76c0e24?source=rss-d9466f7f5c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/90f5a76c0e24</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[community]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[pydata]]></category>
            <dc:creator><![CDATA[Jesús Corrius]]></dc:creator>
            <pubDate>Mon, 12 Mar 2018 08:43:48 GMT</pubDate>
            <atom:updated>2018-03-12T08:43:48.434Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k0r4IX4bqIlYLhdm0xWY7A.jpeg" /><figcaption>Volunteers improving Pandas in Parc Bit, Palma de Mallorca</figcaption></figure><p>Last Saturday <a href="https://bluekiri.com/">Bluekiri</a> sponsored a <a href="https://python-sprints.github.io/pandas/">Pandas documentation sprint</a> in <a href="http://www.parcbit.es/wparcbitfront/">Parc Bit</a> (Palma de Mallorca), organized by <a href="https://www.meetup.com/PyData-Mallorca/">PyData Mallorca</a> and <a href="https://www.meetup.com/Python-Sprints/">London Python Sprints</a>. It was part of a very sucessful worldwide event which connected us with more than 500 developers located in 30 different cities. There were many people involved in the organization of the event, but one person in particular, <a href="https://twitter.com/datapythonista">Marc Garcia</a>, deserves a standing ovation for making it possible. We absolutely love events where people learn new skills and contribute back to the community and this was a very fine example of all this.</p><p><a href="https://pandas.pydata.org/">Pandas</a> is an amazing open source Python library for data manipulation and analysis which has become in the last few years the de-facto standard for data science and data engineering. A huge number of individuals and companies use this library nowadays, but the number of core developers remains the same. Our <a href="https://bluekiri.com/team.html#big_data">Big Data team</a> makes an extensive use of this library in almost all their projects, so it is only fair to contribute back to fix some outstanding issues with the documentation and help it become an even better tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dORM3_RGinVC_bTx6ops6w.jpeg" /><figcaption>Bluekiri proudly sponsored the Pandas sprint</figcaption></figure><p><strong>Doing open source the right way</strong></p><p>Companies use a wide variety of open source tools today, and many of them are key to maintaining their ongoing operations. And while it’s not uncommon for them to pay a lot of money in licenses for tools that are way less critical for their business, they usually take open source software for granted. And they never give anything back to the community. Some projects are backed by big companies indeed, but it isn’t hard to find a project that is critical for your own business and run exclusively by volunteers.</p><p>Generally, open source projects don’t want your money. Ok, let’s admit that money is better than nothing. But this is like buying very expensive toys for<strong> </strong>your children and never spending any quality time with them. You can argue that toys are better than nothing, yes, but let me tell you that <em>you don’t get it</em>. What open source projects want is quality time. They want you to write good bug reports, create tests, contribute code, clean up documentation, write tutorials, etc. You can’t buy love with money, you buy love with effort, commitment and dedication.</p><p>Be a doer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90f5a76c0e24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluekiri/pandas-documentation-sprint-90f5a76c0e24">Pandas documentation sprint</a> was originally published in <a href="https://medium.com/bluekiri">bluekiri</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What does Spectra and Meltdown mean for developers?]]></title>
            <link>https://medium.com/bluekiri/what-does-the-intelbug-mean-for-developers-4764a8ab9830?source=rss-d9466f7f5c3d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4764a8ab9830</guid>
            <category><![CDATA[intel]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[bugs]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Jesús Corrius]]></dc:creator>
            <pubDate>Fri, 05 Jan 2018 07:58:15 GMT</pubDate>
            <atom:updated>2018-10-16T06:37:39.398Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5EgrZ9p2jTZwoCEn_FJGA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/1K9T5YiZ2WU?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tim Gouw</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>One of the first steps executed when booting a computer is patching the processor with a firmware update. Virtually all CPUs of all vendors released in the last 40 years contain bugs. Most of them are trivial, many mild, some are severe, a few critical, and then you have Meltdown. Of the two recent security bugs found on Intel processors, Meltdown could arguably be the worst security CPU bug ever found.</p><p>Meltdown affects most Intel processors manufactured since 1995 and allows hackers to bypass the hardware barrier between applications run by users and the computer’s core memory. The problem is especially severe in a virtualized environment where a hacker with access to one of the VMs could bypass security and read the memory contents of all the machines hosted physical server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*3rd91T42kECN3pp-zC6zVg.png" /><figcaption><a href="https://danielmiessler.com/blog/simple-explanation-difference-meltdown-spectre/">https://danielmiessler.com/blog/simple-explanation-difference-meltdown-spectre/</a></figcaption></figure><p>The fix keeps kernel and userland memory more carefully apart so that side-effects from speculative execution tricks are mitigated. All this is good, but this change is not free. As a side effect, it tampers with caching. As CPU caching is one of the main tricks for a CPU to boost performance, anything that reduces the effectiveness of the cache is likely to produce a performance hit, as is the case.</p><blockquote>You can compare this fix to the HTTPS protocol, which adds a layer of security on top of HTTP but makes it computationally more expensive.</blockquote><p>The low level OS fix required to secure our computers is probably the most radical change in OS design we have seen in years, as it basically changes how your operating system manages memory for the running processes. The implications are profound.</p><p>Arguably the most important of these implications is that the toolchain we use to build our applications has to be adapted to produce optimal code to target this new design.</p><p>For us, it has two main consequences:</p><ol><li>Update our build tools (compilers, etc.) to the latest version.</li><li>Make sure all the source code from our applications, including third-party modules or dependencies, is properly recompiled.</li></ol><p>As usual, legacy code that can’t be updated, obsolete dependencies or third-party unmaintained libraries will be our main adversaries. I boldly predict that many applications will be rewritten from scratch to solve the performance issues described in this article.</p><p>Applications running on Linux may have an advantage here over the Windows counterparts and their migrations should be smoother. Big Linux vendors recompile all the packages in their repositories with every new version to take advantage of the new compiler, kernel and glibc optimizations and developers rarely install dependencies outside the supported channels.</p><p>Unfortunately this toolchain update and recompiling discipline is rarely known in the Windows world where backwards compatibility is king. We all know application that can only be compiled with a particular (and obsolete) version of Visual Studio or, even worse, abandoned nuget packages that are critical for a particular application (or that DLL with a fancy name whose source code is lost in time)</p><h3>But it’s not all bad news.</h3><p>A big breaking change is a chance too. For example, a chance to activate security features that are now disabled by default for compatibility reasons, get rid of obsolete features or entire subsystems, rethink optimizations, etc. It’s a chance to modernize and improve the very same foundations on top of which we build our applications. In short: to evolve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*QDa8nrODzbjfaRqfnHbgrg.png" /></figure><p>If you are already a <a href="https://bluekiri.cloud/">Bluekiri customer</a>, you can rest assured our infrastructure is being updated with the latest security patches as soon as they become available.</p><p>The majority of Bluekiri customers should not see a noticeable performance impact with this update. We’ve worked to optimize the CPU and disk I/O path and are not seeing noticeable performance impact after the fix has been applied.</p><p>Additionally, if you are impacted by <a href="https://www.phoronix.com/scan.php?page=article&amp;item=linux-415-x86pti&amp;num=2">performance degradation</a> or want to start planning an upgrade to your legacy code, feel free to <a href="https://bluekiri.com/">start a conversation with us</a> :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4764a8ab9830" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bluekiri/what-does-the-intelbug-mean-for-developers-4764a8ab9830">What does Spectra and Meltdown mean for developers?</a> was originally published in <a href="https://medium.com/bluekiri">bluekiri</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>