<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sébastien Cararo on Medium]]></title>
        <description><![CDATA[Stories by Sébastien Cararo on Medium]]></description>
        <link>https://medium.com/@sebcararo?source=rss-a0b1d27359d5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1emajB6No9i7C8AgxE-Otw.jpeg</url>
            <title>Stories by Sébastien Cararo on Medium</title>
            <link>https://medium.com/@sebcararo?source=rss-a0b1d27359d5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:35:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sebcararo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Modelizing a tennis match with Markov Chains]]></title>
            <link>https://medium.com/analytics-vidhya/modelizing-a-tennis-match-with-markov-chains-b59ca2b5f5bf?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b59ca2b5f5bf</guid>
            <category><![CDATA[model]]></category>
            <category><![CDATA[markov-chains]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[sports]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Mon, 04 Jan 2021 23:06:15 GMT</pubDate>
            <atom:updated>2023-08-08T21:51:15.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>Modeling a tennis match with Markov Chains</h3><blockquote>#HHMM.</blockquote><p>This article focuses on modelling a tennis match with mathematical objects called <strong><em>Markov chains </em></strong>[1]<strong><em>.</em></strong> It is quite illustrated, and designed to be intelligible towards everybody — no prerequisites are required.</p><p>The scoring in tennis is quite particular when compared to other sports. If you’re not familiar with it, please refer to [2] for a simple introduction. Nonetheless, this article might also be a very clear explanation of this scoring system. We’ll illustrate the sequential organization of the scoring through elegant state models known as <em>Markov Chains</em>, and eventually compute winning probabilities :</p><figure><img alt="Figure 1: Models’ nesting for a tennis match" src="https://cdn-images-1.medium.com/max/481/1*uSo-3Z_euy0mSLHkgHsE3Q.png" /><figcaption>Figure 1: Models’ nesting for a tennis match</figcaption></figure><p>The whole code to generate this modelization is available in a dedicated GitHub repository [3].</p><p>I — Modelize a game</p><p>II — Modelize a tie-break</p><p>III — Modelize a set</p><p>IV — Modelize a match</p><p>V — Assembly all models in a single model</p><p>VI — Conclusion</p><p>The only two parameters we’ll to modelize a match are the elementary winning probabilities :</p><ul><li><em>ppoint_srv1</em>: the probability that the player 1 wins a service point</li><li><em>ppoint_srv2</em>: the probability that the player 2 wins a service point</li></ul><p><strong>I — Modelize a game</strong></p><p>In this part, we seek to modelize a game based on the probability that the server wins a point <em>ppoint_srv1. </em>The principle of Markov chains is simply to define <strong><em>states </em></strong>(in our example the states are simply the scores), and the <strong><em>probabilities </em></strong>to get from a given state to another one. If you are familiar about tennis scoring, then the following matrix (Figure 2) is a no-brainer. It simply illustrates the transition probabilities between scores for <em>ppoint_srv1 = 0.7.</em></p><p><strong>To know the states values after 1 iteration, we simply multiply the transition matrix with the initial states matrix</strong>. When we start from ‘0–0’, the states matrix contains 1 in the case ‘0–0’ and 0 in every other state. To know the states values after <em>n </em>iterations, we simply multiply the transition matrix with itself <em>n </em>times, and finally multiply this with the initial states matrix.</p><p>We can observe the stationary states (HOLD &amp; BREAK), as when we reach this score, we cannot <em>move </em>to another state as the game is over :</p><figure><img alt="Figure 2: Markov matrix for a game (ppoint_server = 0.7)" src="https://cdn-images-1.medium.com/max/1024/1*8S5mdoUk05ZeYBTg6I7zwQ.png" /><figcaption>Figure 2: Markov matrix for a game (ppoint_server = 0.7)</figcaption></figure><p>We can also plot this matrix for illustration purpose :</p><figure><img alt="Figure 3: Game model representation with a Markov Chain" src="https://cdn-images-1.medium.com/max/1024/1*wqjdeKEwXYx5JMHIX-BK5A.png" /><figcaption>Figure 3: Game model representation with a Markov Chain</figcaption></figure><p>Finally, we can build in R a function<em> resGAME() </em>that takes the parameters <em>ppoint_srv1 </em>and the <em>score </em>at the beginning as input, and <strong>computes the outcomes probabilities — that are either an hold or a break</strong>. With <em>ppoint_srv1 = 0.7</em> and <em>score = ‘0–0’</em> the output is the following :</p><figure><img alt="Figure 4: Computing game outcome probabilities with Markov chains" src="https://cdn-images-1.medium.com/max/665/1*yDZrIxO8QHd5AhZ-rf4aXA.png" /><figcaption>Figure 4: Computing game outcome probabilities with Markov chains</figcaption></figure><p>Now, if we want to compute probabilities from<em> ‘0–15’</em> we just need to set the starting score to <em>‘0–15’</em> by changing the states :</p><figure><img alt="Figure 5: Computing game outcome probabilities from a given score in the game" src="https://cdn-images-1.medium.com/max/662/1*mxnCKFEOrtB2mLX_mzYfjw.png" /><figcaption>Figure 5: Computing game outcome probabilities from a given score in the game</figcaption></figure><p><strong>II — Modelize a tie-break</strong></p><p>Now that we have explained the principle used for modelizing a game, we can apply the same principle to <strong>modelize a tie-break</strong>. One should pay attention to the sequential progress of a tie-break : first P1 serves, then P2 serves twice, then P1 serves twice, etc…To do this, we just pay attention to setting the right transition probabilities in the construction of the matrix. The input parameters for this block are <em>ppoint_srv1 </em>and <em>ppoint_srv2.</em></p><p>The Markov chain looks like this with <em>ppoint_srv1 = 0.7</em> and <em>ppoint_srv2= 0.6 </em>:</p><figure><img alt="Figure 6: Tie-break model representation with a Markov Chain" src="https://cdn-images-1.medium.com/max/1024/1*fKx0QimS1iif0japzifz6A.png" /><figcaption>Figure 6: Tie-break model representation with a Markov Chain</figcaption></figure><p>As during the game modelization, we can build in <em>R </em>a function<em> resTIE() </em>that takes the parameters <em>ppoint_srv1, ppoint_srv2 </em>and the <em>score </em>at the beginning as input, and <strong>computes the outcomes probabilities </strong>— that are either P1 or P2 winning the set. With <em>ppoint_srv1 = 0.7, ppoint_srv2 = 0.6,</em> and <em>score = ‘0–0’</em> the output is the following :</p><figure><img alt="Figure 7: Computing tie-break’s outcome probabilities" src="https://cdn-images-1.medium.com/max/695/1*FyZ5b7lbKTgh0dCV6cVmTQ.png" /><figcaption>Figure 7: Computing tie-break’s outcome probabilities</figcaption></figure><p>We can eventually compute the outcome’s probabilities from any score :</p><figure><img alt="Figure 8: Computing tie-break’s outcome probabilities from 3–4" src="https://cdn-images-1.medium.com/max/707/1*PijTxJuMabeFtbt-Uzqprw.png" /><figcaption>Figure 8: Computing tie-break’s outcome probabilities from 3–4</figcaption></figure><p><strong>III — Modelize a set</strong></p><p>Following the very same technique, we can <strong>modelize a set</strong>. The input parameters are the <em>holding probabilities</em>, and the <em>probability that P1 wins the tie-break.</em> The Markov chain looks like the following one :</p><figure><img alt="Figure 9: Set model representation with a Markov Chain" src="https://cdn-images-1.medium.com/max/1024/1*YJ8bTaRnKE0EBEWxrZgF7w.png" /><figcaption>Figure 9: Set model representation with a Markov Chain</figcaption></figure><p>As we have previously built the <em>game </em>and <em>tie-break </em>modelizations, we can compute the holding and tie-break probabilities from the point winning probabilites, such that the inputs can be only the point winning probabilities :</p><figure><img alt="Figure 10: Compute set winning probabilities from the point winning probabilities" src="https://cdn-images-1.medium.com/max/823/1*RBGSH5flQizeSw4QPKA7qg.png" /><figcaption>Figure 10: Compute set winning probabilities from the point winning probabilities</figcaption></figure><p>We can also compute the probabilities from a given score :</p><figure><img alt="Figure 11: Figure 10: Compute set winning probabilities from the point winning probabilities from a given score" src="https://cdn-images-1.medium.com/max/821/1*Mm0nkrZ2GPW3tz90WvD1kA.png" /><figcaption>Figure 11: Figure 10: Compute set winning probabilities from the point winning probabilities from a given score</figcaption></figure><p><strong>IV — Modelize a match</strong></p><p>Following the very same principle, we can <strong>build a model for the match in its whole</strong>. We can then compute the probabilities of victory, given the set winning probabilities :</p><figure><img alt="Figure 12: Match model representation with a Markov Chain" src="https://cdn-images-1.medium.com/max/1024/1*8UyvuKIRUrrpcQXdYaMlKw.png" /><figcaption>Figure 12: Match model representation with a Markov Chain</figcaption></figure><p>The input parameters are the set winning probabilities, however as presented earlier the set winning probabilities can be inferred from the point winning probabilities, such that we can build this model with just the two elementary parameters inputted (step V):</p><figure><img alt="Figure 13: Computing exact score probabilities" src="https://cdn-images-1.medium.com/max/525/1*GiAt04oaVmD-qZqkAyLE6w.png" /><figcaption>Figure 13: Computing exact score probabilities</figcaption></figure><p>We can also compute these probabilities from a given score :</p><figure><img alt="Figure 14: Computing exact score probabilities from a given score" src="https://cdn-images-1.medium.com/max/547/1*F4-kv-3Jve5QNQxYSMp_Vw.png" /><figcaption>Figure 14: Computing exact score probabilities from a given score</figcaption></figure><p><strong>V — Assembly all models in a single model</strong></p><p>As presented during the introduction, once we have correctly defined the <em>game</em>, <em>tie-break</em>, <em>set </em>and <em>match </em>Markov chain models, we can<strong> imbricate these models to set up our final modelization</strong>. This is done by simply linking the outputs to the inputs (<em>e.g.</em> the output of the set model will be used as input for the match model parameters). After having done this step, <strong>we can compute the winning probabilities from any state of the match, with the only two basic parameters </strong>:</p><figure><img alt="Figure 15: Computing winning probabilities from any score, with only the point winning probabilities inputed" src="https://cdn-images-1.medium.com/max/787/1*hwXthQzv27fqTDg4gOM3zg.png" /><figcaption>Figure 15: Computing winning probabilities from any score, with only the point winning probabilities inputted</figcaption></figure><p><strong>VI — Conclusion</strong></p><p>This type of model is generally used in the sports betting industry, as <strong>the model computes the estimated winning probabilities from any state of the match</strong>. For example, we could infer <em>ante </em>point winning probabilities on serve from historical statistics, for each player. This estimation would depend on the conditions, the surface, the form of the players, etc…Once we would have estimated these two parameters, the package we just presented would allow us to estimate the chances of victory, but also intermediary event probabilities (holding a game, winning a set, winning with a specific score, etc…)</p><p>Also, I can only imagine that the bookmakers take several input parameters in order to make models as precise as possible. The functions we presented are an overview of some of the models used, and can be a good starting point to build more powerful models.</p><p>Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionnaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Sources</strong></p><p>[1] Markov chains introduction</p><p><a href="https://setosa.io/ev/markov-chains/">Markov Chains explained visually (setosa.io)</a></p><p>[2] Tennis scoring system</p><p><a href="https://www.sportingnews.com/us/tennis/news/tennis-scoring-explained-rules-system-points-terms/7uzp2evdhbd11obdd59p3p1cx">Tennis scoring, explained: A guide to understanding the rules, terms &amp; points system at the US Open | Sporting News</a></p><p>[3] GitHub repository</p><p><a href="https://github.com/Seb943/Markov4Tennis">Seb943/Markov4Tennis: R script to modelize a tennis match with Markov chains (games, tie-breaks, sets, match) (github.com)</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b59ca2b5f5bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/modelizing-a-tennis-match-with-markov-chains-b59ca2b5f5bf">Modelizing a tennis match with Markov Chains</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A complete  guide to fitting Machine Learning models in R]]></title>
            <link>https://medium.com/analytics-vidhya/a-complete-guide-to-fitting-machine-learning-models-in-r-24a3dea2ee82?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/24a3dea2ee82</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Mon, 04 Jan 2021 23:05:19 GMT</pubDate>
            <atom:updated>2023-08-08T21:52:32.227Z</atom:updated>
            <content:encoded><![CDATA[<h3>A complete guide to fit Machine Learning models in R</h3><blockquote>It is more simple than you would think</blockquote><p>This article describes how one can train and make predictions with various Machine Learning models in <em>R</em>. <em>R </em>is a powerful programming language, that is mainly oriented towards statistical studies, even though it has a wider range of applications. We will expose the <strong>syntactic implementation</strong>, plus we will describe the usual <strong>fitting options</strong> and <strong>hyperparameters </strong>for each model, how to use <strong>cross-validation</strong>, and eventually how to <strong>make predictions</strong>. We will also show how to <strong>save the models</strong>.</p><p>The data we’ll be using can be found at [1]. It consists of 12 054 <em>NBA </em>games, along with the date, the final result and a few predictive variables that are commonly used in sports analytics (Win/Loss record, offensive and defensive performance). This is just an example dataset, and obviously the codes I’m going to show are not related to this specific dataset. The dataset looks like this :</p><figure><img alt="Figure 1: Overview of the dataset (predictive variables in green, outcome in yellow, miscellaneous info in grey)" src="https://cdn-images-1.medium.com/max/1024/1*c_BzTX1vlnBX8WD88ueNkA.png" /><figcaption>Figure 1: Overview of the dataset (predictive variables in green, outcome in yellow, miscellaneous info in grey)</figcaption></figure><p>0 — Load libraries and data</p><p>I — Random Forests</p><p>II — LDA, QDA, etc.</p><p>III — Support Vector Machines</p><p>IV— averaged Neural Networks (avNNet)</p><p>V — Make predictions</p><p><strong>0 — Load libraries and data</strong></p><p>First we import the libraries, load the dataset and pick only the predictive variables <em>X </em>and the independent variable <em>Y </em>(<em>Winner </em>in the case of our dataset) to create our training and testing sets. The following models will be trained to predict a categorical variable — as <em>Y </em>is either the home team (<em>H</em>) or the away team (<em>A</em>) — but the code to predict quantitative outputs is strictly the same.</p><p><em>NB : Whole code is available at [2]</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/466094b2487818a874ee16ccf0bda940/href">https://medium.com/media/466094b2487818a874ee16ccf0bda940/href</a></iframe><p>We eventually define the <em>trainControl </em>object to proceed to cross-validation, this tool is optional but it reduces the variance in the training process and will be used in the training process:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc32017567f7f6a64506b53e1355b960/href">https://medium.com/media/cc32017567f7f6a64506b53e1355b960/href</a></iframe><p><strong>I — Random Forests</strong></p><p>We’ll use the <em>randomForest </em>package to build and train a random forests model. The usual key parameters are the number of trees that we average (<em>ntree</em>) and the metric used for the construction (<em>metric</em>). Further details can be found at [3].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e649fe6b590191792557b5aebfdee80c/href">https://medium.com/media/e649fe6b590191792557b5aebfdee80c/href</a></iframe><p><strong>II — LDA, QDA, etc.</strong></p><p>We’ll use there the <em>train()</em> function contained in the <em>MASS </em>package [4]. It is very useful because the argument ‘<em>method’ </em>allows us to train any sort of ML model among a variety of already-included models. An exhaustive list of the methods and further documentation can be found at [5]. The code to build and fit a QDA model is presented below :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44ef903fd7ffdcdfec1f455069f5319e/href">https://medium.com/media/44ef903fd7ffdcdfec1f455069f5319e/href</a></iframe><p><strong>III — Support Vector Machines</strong></p><p>Even though it is possible to train <em>SVM </em>models with the <em>train()</em> function, we’ll use the <em>e1071</em> package here. This package contains advanced methods for the <em>SVM</em> models, in such way that there are multiple options such as the type of classification, the type of kernel (<em>linear</em>, <em>radial</em>, <em>polynomial</em>, custom function), the metric, and the possibility to train the model to predict class probabilities aswell. Further documentation can be found at [6].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49279f8d5e102b670c78da9b2497b062/href">https://medium.com/media/49279f8d5e102b670c78da9b2497b062/href</a></iframe><p><strong>IV — averaged Neural Networks (avNNet)</strong></p><p>During this last part, we’ll overview the <em>avNNet </em>package. It is an exceptional package, that allows to build and train averaged Neural Networks : concretely, the function fits a certain number of neural networks with different starting seeds, averages them and builds a unique model from them. The process allows to reduce variance in the training process and hence to obtain more robust models. The key parameters are the number of NN built (<em>repeats</em>) and the number of hidden layers for each model (<em>size</em>). Further documentation can be found at [7].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f89c21d60122aee26e22e460b899a67/href">https://medium.com/media/8f89c21d60122aee26e22e460b899a67/href</a></iframe><p><strong>V — Make predictions</strong></p><p>Once we have trained and saved our models, it could be interesting to make predictions on new data. We simply need to load the model, then to make predictions thanks to the <em>predict()</em> function. The key parameter here is the <em>type </em>argument. Further documentation can be found at [8].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04e4236c063fea1c68cbd4054ae5ab7f/href">https://medium.com/media/04e4236c063fea1c68cbd4054ae5ab7f/href</a></iframe><p>Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionnaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Sources</strong></p><p>[1] Dataset location</p><p><a href="https://raw.githubusercontent.com/Seb943/PCA_and_ML_R/main/Data/Example_dataset.csv">https://raw.githubusercontent.com/Seb943/PCA_and_ML_R/main/Data/Example_dataset.csv</a></p><p>[2] Whole code</p><p><a href="https://github.com/Seb943/PCA_and_ML_R/blob/main/Codes/FittingMLmodelsInR.R">PCA_and_ML_R/FittingMLmodelsInR.R at main · Seb943/PCA_and_ML_R (github.com)</a></p><p>[3] Random forest documentation</p><p><a href="https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest">randomForest function | R Documentation</a></p><p>[4] MASS package documentation</p><p><a href="https://cran.r-project.org/web/packages/MASS/MASS.pdf">MASS.pdf (r-project.org)</a></p><p>[5] <em>train </em>function in the <em>MASS </em>package</p><p><a href="https://www.rdocumentation.org/packages/caret/versions/4.47/topics/train">train function | R Documentation</a></p><p>[6] e1071 package documentation (<em>SVM</em>)</p><p><a href="https://cran.r-project.org/web/packages/e1071/e1071.pdf">e1071.pdf (r-project.org)</a></p><p>[7] avNNet package documentation</p><p><a href="https://www.rdocumentation.org/packages/caret/versions/6.0-86/topics/avNNet">avNNet function | R Documentation</a></p><p>[8] predict function documentation</p><p><a href="https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/predict">predict function | R Documentation</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*kyWjqiYZKUvss8tP.jpg" /><figcaption>Figure 9 : Cover picture</figcaption></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24a3dea2ee82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-complete-guide-to-fitting-machine-learning-models-in-r-24a3dea2ee82">A complete  guide to fitting Machine Learning models in R</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying PCA in R]]></title>
            <link>https://medium.com/analytics-vidhya/applying-pca-in-r-e41468c94f15?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e41468c94f15</guid>
            <category><![CDATA[correlation]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Mon, 04 Jan 2021 23:04:41 GMT</pubDate>
            <atom:updated>2023-08-08T21:52:48.728Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>And eventually exporting the model</blockquote><p>PCA is a powerful Machine Learning technique which can be useful for multiple tasks : data visualization, data analysis and exploration, reducing variance in datasets, increase the Signal-To-Noise ratio, or even creating datasets that are suitable for Machine Learning and Deep Learning training.</p><p>However, for many individuals the underlying concept and the implementation may seem out of reach, sometimes because this technique is not well introduced. Concerning the theoretical part, I would suggest reading some academic works such as the <em>ISLR </em>[1]. The <em>ISLR </em>is an exceptional textbook for whom may wish to learn statistical learning. This article will cover the implementation in <em>R</em>, the syntax and the possibilities of application through a simple example.</p><p>0 — The data we’ll use</p><p>I — Fitting the PCA and saving the components</p><p>II — Example of when we apply it later</p><p>III — Conclusion</p><p><strong>0 — The data we’ll use</strong></p><p>The data we’ll be using can be found at [2]. It consists of 12,054 <em>NBA </em>games, along with the date, the final result and several predictive variables that are commonly used in sports analytics (Win/Loss record, offensive and defensive performance). This is just an example dataset, and obviously the codes I’m going to show are not related to this specific dataset. The dataset looks like this :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuA3WTX51ARu2SeSGFjJUA.png" /><figcaption>Figure 1: Overview of the dataset (predictive variables in green, outcome in yellow, miscellaneous info in grey)</figcaption></figure><p>The correlation matrix looks like the figure below, where we can observe some variables are quite correlated. The same information is present multiple times in the dataset :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AP-6dk4moFhj-_O1hQrE9A.png" /><figcaption>Figure 2 : Correlation matrix of the predictive variables in the training set</figcaption></figure><p>First we import the libraries, load the dataset and pick only the variables of interest, that are the predictive variables. We also divide the dataset into train set and test set (which is more realistic concerning real-life applications) to display how we can apply the PCA to new data (the test set):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fb4955ce3cfe1671c754d5d20df8aa3/href">https://medium.com/media/7fb4955ce3cfe1671c754d5d20df8aa3/href</a></iframe><p><em>NB : Whole code is available at [3]</em></p><p><strong>I — Fitting the PCA and saving the components</strong></p><p>The code presented below allow us to build the PCA object. The parameter ‘<em>scale.’</em> is a boolean corresponding to whether we wish to scale the variables before fitting the PCA model. It is commonly admitted that it is a good idea when coping with variables that are not in the same unit. We can then plot the Proportion of Variance Explained (<em>PVE</em>) in each PC that we built, and eventually the cumulative PVE. We eventually save the PCA components as .<em>csv</em> objects. These objects are basically matrices, and are useful when we want to apply the PCA without having to fit it each time (see next section for predictions on new data).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZU5DWPBKRZGcw6RnsmPLg.png" /><figcaption>Figure 4: Proportion of Variance Explained in each Principal Component created</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wm6LWXBJkhQALEm7whdY0A.png" /><figcaption>Figure 5: Cumulative PVE graph</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b3b5393db29e0c8db08dada48d4d0f0/href">https://medium.com/media/3b3b5393db29e0c8db08dada48d4d0f0/href</a></iframe><p><strong>II — Example of when we apply it later</strong></p><p>Once we have built our PCA model and saved its components, we can use them to apply the PCA on new data. We simply use the <em>scale()</em> function with the saved components, and we finish the PCA application with a matrix multiplication. The last two lines are there to assert we have correctly done the computation :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06713e1957abf7fa655fcf3cabea3301/href">https://medium.com/media/06713e1957abf7fa655fcf3cabea3301/href</a></iframe><p><strong>III — Conclusion</strong></p><p>We have displayed how one can build, train and export a PCA object with the <em>R </em>programming language. Moreover, we also displayed how to plot the proportion of variance explained (PVE) in each constructed component over the train set.</p><p>Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionnaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Sources</strong></p><p>[1] <em>ISLR</em></p><p><a href="https://www.academia.edu/38159350/An_Introduction_To_Statistical_Learning_with_Applications_in_R_ISLR_Sixth_Printing_">(PDF) An Introduction To Statistical Learning with Applications in R (ISLR Sixth Printing) | ym xue — Academia.edu</a></p><p>[2] Dataset location</p><p><a href="https://raw.githubusercontent.com/Seb943/PCA_and_ML_R/main/Data/Example_dataset.csv">https://raw.githubusercontent.com/Seb943/PCA_and_ML_R/main/Data/Example_dataset.csv</a></p><p>[3] Code location</p><p><a href="https://github.com/Seb943/PCA_and_ML_R/blob/main/Codes/PCA_in_R.R">PCA_and_ML_R/PCA_in_R.R at main · Seb943/PCA_and_ML_R (github.com)</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e41468c94f15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/applying-pca-in-r-e41468c94f15">Applying PCA in R</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring NBA injuries with Python]]></title>
            <link>https://medium.com/analytics-vidhya/monitoring-nba-injuries-with-python-bf05ea2aec68?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf05ea2aec68</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[html]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[nba]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Mon, 28 Dec 2020 23:16:03 GMT</pubDate>
            <atom:updated>2023-08-08T21:53:10.232Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>In 30 seconds : collect updates on the web, store them in a csv, send via e-mail.</blockquote><p>As a lifelong NBA fan, I have always been interested into knowing about which players would play tonight. Living in France, I would always fear waking up at 3 AM just to see that the expected <em>game of the night</em> turned into a <em>blowout </em>game because some superstar didn’t play that night.</p><p>To avoid this, I used to check the <em>Rotoworld </em>website [1] in order to get the very last updates regarding players’ injuries. But, <strong>why bother spending time looking at the same page when a Python code can do it for you and send you the updates via an elegant e-mail ?</strong></p><figure><img alt="Figure 1: Rotoworld page for NBA injuries" src="https://cdn-images-1.medium.com/max/849/1*bTS_c5R0ZnC_ASG3MQF9SQ.png" /><figcaption>Figure 1: Rotoworld page for NBA injuries</figcaption></figure><p>0 — Importing libraries and setting driver location</p><p>I — Defining scraping functions</p><p>II — Scraping the page</p><p>III — Saving the <em>.csv</em> and sending it via e-mail</p><p><strong>0 — Importing librairies and setting driver location</strong></p><p>During this first step we first import the required packages, and we <strong>define the location of our Chrome driver on line 10. </strong>We also import a custom library called <em>Send_mail</em> which we will expose in section III. Finally, we open the Chrome driver and go to the <em>Rotoworld</em> NBA injuries’ page.</p><p>The initialization code is written below :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f6a46e1571a3c41aa15e5d10e3ee069/href">https://medium.com/media/2f6a46e1571a3c41aa15e5d10e3ee069/href</a></iframe><p><strong>I — Defining scraping functions</strong></p><p>We then define a function named <em>ffi(a)</em> to browse easily over the page : its purpose is to collect a text associated to an <em>xpath</em>, without yielding any error if it doesn’t exist. In short, <strong>this function handles errors</strong> by shutting them down it there is nothing written somewhere.</p><p>We then define a function named <em>collect_info_line()</em>, which takes the <strong><em>team </em></strong>number (from 1 to 30 as there are 30 teams) and the row <strong><em>i </em></strong>as inputs, and <strong>collects the associated info</strong> on the <em>Rotoworld </em>page :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c7bf791522d9c0f72c76af6e04f6984/href">https://medium.com/media/5c7bf791522d9c0f72c76af6e04f6984/href</a></iframe><p><strong>II — Scraping the page</strong></p><p>Once we have defined the function to collect the info at each row for each team of the page, we simply need to browse over all teams and all rows, in order to collect all data. The code to do this is given just below :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdf150f01d6f7597bed5edba3d840e3b/href">https://medium.com/media/bdf150f01d6f7597bed5edba3d840e3b/href</a></iframe><p><strong>III — Saving the <em>.csv</em> and sending it via e-mail</strong></p><p>Once we have scraped the raw data during the last step, we may now want to save it in a table and send it via e-mail.</p><p>First we create a script to send a <em>.csv</em> file via e-mail (sending attachments via email on Python — [2]), one should pay attention to <strong>modifying the receiving email address on line 16</strong> :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd9fdfc10060a5c5e072e5e1d420b4d6/href">https://medium.com/media/fd9fdfc10060a5c5e072e5e1d420b4d6/href</a></iframe><p>Finally, we transform the <em>scraped</em> data into a table, save the <em>.csv</em> and send it via e-mail :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b82ef3b0dd60e2ac645092752b63e630/href">https://medium.com/media/b82ef3b0dd60e2ac645092752b63e630/href</a></iframe><p><strong>IV — Conclusion</strong></p><p>After running the code, the final <em>scraped</em> table which is sent via e-mail looks like this :</p><figure><img alt="Figure 7: Final table overview" src="https://cdn-images-1.medium.com/max/757/1*xfX-kfjiKXQjYPNXQZFvgA.png" /><figcaption>Figure 7: Final table overview</figcaption></figure><p>We have now built our code for scraping the injuries on the <em>Rotoworld </em>website. The code can be scheduled to run at specific times, so that the user can receive daily updates on his mailbox. Also, <strong>one could think about adapting the code to fit his own needs</strong> : for example, if you’re a <em>Clippers </em>fan you may want the program to send you a specific e-mail if Kawhi Leonard does play tonight. This could be done by simply checking if the <em>Status </em>value for Kawhi Leonard is set to <em>Active</em>. You could also adapt the code for various purposes : sports betting forecasting, statistics, monitoring injuries, etc…</p><p>Finally, we can also recall that the entire code is available in a dedicated GitHub repository [3]. You can pick it, run it and adapt it freely. Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Sources</strong></p><p>[1] Rotoworld NBA injuries page</p><p><a href="https://www.rotoworld.com/basketball/nba/injury-report">Rotoworld fantasy sports news and analysis for NFL, MLB, NBA, NHL, CFB, Golf, EPL and NASCAR</a></p><p>[2] Sending attachments via email in Python</p><p><a href="https://stackoverflow.com/questions/3362600/how-to-send-email-attachments">python — How to send email attachments? — Stack Overflow</a></p><p>[3] GitHub repository</p><p><a href="https://github.com/Seb943/NBA_injuries_report">Seb943/NBA_injuries_report</a></p><figure><img alt="Figure 8 : Cover picture (Kawhi Leonard injury vs Denver Nuggets)" src="https://cdn-images-1.medium.com/max/638/1*HVQiw853rzILf0c8O-2wZA.jpeg" /><figcaption>Figure 8 : Cover picture (Kawhi Leonard injury vs Denver Nuggets — <a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fsports.yahoo.com%2Fclippers-lose-kawhi-leonard-to-bloody-injury-late-in-win-over-nuggets-061344718.html&amp;psig=AOvVaw3exwWT4I-UX5ibe5hx-vQk&amp;ust=1609266451161000&amp;source=images&amp;cd=vfe&amp;ved=0CAIQjRxqFwoTCKCr3Oum8e0CFQAAAAAdAAAAABAE">https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fsports.yahoo.com%2Fclippers-lose-kawhi-leonard-to-bloody-injury-late-in-win-over-nuggets-061344718.html&amp;psig=AOvVaw3exwWT4I-UX5ibe5hx-vQk&amp;ust=1609266451161000&amp;source=images&amp;cd=vfe&amp;ved=0CAIQjRxqFwoTCKCr3Oum8e0CFQAAAAAdAAAAABAE</a> )</figcaption></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf05ea2aec68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/monitoring-nba-injuries-with-python-bf05ea2aec68">Monitoring NBA injuries with Python</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IV. Deploy a Poloniex trading bot]]></title>
            <link>https://medium.com/analytics-vidhya/iv-deploy-a-poloniex-trading-bot-c3af87eba48c?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3af87eba48c</guid>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Mon, 28 Dec 2020 23:13:13 GMT</pubDate>
            <atom:updated>2023-08-08T21:53:30.110Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Using the Poloniex API to create an autonomous trading bot</blockquote><p>This article is part of a series of articles that describe the step-by-step process to create, backtest, and deploy a trading bot. The first article focused on <strong>computing predictive variables</strong>, the second on <strong>building the Deep Learning models</strong>, the third on <strong>identifying profitable trading strategies</strong> with the models, and this final one on the real-life implementation in a program (<strong>deployment</strong>). Links are given at the end of this article.</p><p>Once we have now built our models and identified our trading strategy, time is now to implement it in a Python script so that it can generate profit by itself. The objective of this final article is to produce an <strong>autonomous Python code which will take trading decisions in-real time</strong>, according to the profitable <em>bullish</em> strategies that we identified and assessed in the previous article (article III — [1]). The following code takes only one trade at a time (<strong>1TAAT</strong>), but some direct adaptation can make it suitable for multiple simultaneous trades monitoring. As this is a bit more sophisticated, I only expose the <em>1TAAT </em>principle and let the choice to readers to adapt it to their own needs.</p><p>0 — Importing libraries and loading data</p><p>I — Initializing Poloniex API connection</p><p>II — Defining buy and sell functions, computing variables</p><p>III — One trade at a time (1TAAT) trading bot</p><p>IV — Conclusion</p><p><strong>0 — Importing libraries and loading data</strong></p><p>The very first step is to import the necessary libraries, and (re-)define the constants which we used along the construction of this trading bot. This includes the <strong><em>stoploss</em></strong>, the <strong><em>takeprofit</em></strong>, the <strong><em>period</em></strong><em> </em>of candles, but also the <strong>pair </strong>which we trade — as referred to on the Poloniex API and website (<em>e.g.</em> <em>USDT_BTC </em>for Bitcoin trading) and the <strong>amount </strong>in dollar which we want to invest for each trade. Furthermore, we also set our threshold probability <strong><em>min_threshold</em></strong> from which we may follow our model during <em>bullish</em> predictions. By default, it is set to 0.5 (<em>i.e.</em> we follow the model every time it is <em>bullish</em>) but we can set a higher threshold to generate a more selective trading strategy (see article III — [1]). One should also pay attention to modifying the path on line 14 according to his directories structure.</p><p>We also import a custom library called <em>Deployment_functions </em>that will be described in section II of this article.</p><p>The initialization code is given below :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82e499d4dffc925d6b9f9308fe651ccc/href">https://medium.com/media/82e499d4dffc925d6b9f9308fe651ccc/href</a></iframe><p><strong>I — Initializing Poloniex API connection</strong></p><p>Once we have imported the libraries, we can initialize the connection to the Poloniex API by inputting our API keys and secret. These hexadecimal codes are unique to each account, and allow the user to take trades directly from Python. More information regarding the API and how to get your keys can be found at [2]. With our program, you just need to <strong>set yours at lines 62 &amp; 63 in <em>Deployment_functions.py </em></strong>(next paragraph).</p><p>Furthermore, the code to initialize the connection is as simple as this :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/111b78f9cc1db2d6c87e9138f591e6b2/href">https://medium.com/media/111b78f9cc1db2d6c87e9138f591e6b2/href</a></iframe><p><strong>II — Defining buy and sell functions, computing variables</strong></p><p>As presented earlier, to lighten the trading script I decided to store all functions inside a custom library called <em>Deployment_functions.py</em>. This file contains previously introduced functions such as the computation of predictive variables (see article I — [3]). It also contains two very important functions, that are the <em>buy_asset()</em> and <em>sell_asset()</em> functions. These functions enable us to <strong>buy/sell a given amount at the best possible price </strong>available in the market, and are achieved only if the entire amount is bought/sold. These functions are important to define well and clearly, in order to avoid pending orders which would deteriorate the real-life trading. The underlying idea is to make the code as performant as possible, in order to obtain real-life results that are as close as possible to the backtested results (see article III — [1]).</p><p>The code for these functions is given below, where as specified before you should change lines 62 &amp; 63 :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04ee564c8ae75e302c76b2d65830ad4c/href">https://medium.com/media/04ee564c8ae75e302c76b2d65830ad4c/href</a></iframe><p><strong>III — One trade at a time (1TAAT) trading bot</strong></p><p>Now that we have built and defined all of our tools, time is now to create the trading loop. In our case (<em>1TAAT</em>), we simply repeat a sequential process over and over. We wait for the candle’s data to be delivered, then we apply these 3 tasks : <strong>Request the recent data</strong>, <strong>compute predictions</strong> by computing the variables and applying the model, then finally if our bullish criterion is reached (see article II [4] for how we define our min and max) we <strong>enter the trade and track it</strong>. When we enter into the trade, first we buy the predefined amount of the pair, then we wait until the price reaches either the <em>stoploss </em>or the <em>takeprofit</em>. When it finally reaches a limit, we sell our asset. And we repeat this process over and over.</p><p>The code was made to display some monitoring information in the console (<em>Figure 4</em>) and to store the characteristics of each transaction made (<em>Figures 5 &amp; 6</em>). <em>NB : The Poloniex website stores all trades’ information anyhow.</em></p><p>The console looks like this when a trade in progress, <em>i.e.</em> when we are waiting for the price to reach either the stoploss or the takeprofit:</p><figure><img alt="Figure 4: Console when a trade is being monitored" src="https://cdn-images-1.medium.com/max/866/1*OZjKSf0hZXX7HzgzDYizAg.png" /><figcaption>Figure 4: Console when a trade is being monitored</figcaption></figure><figure><img alt="Figure 5: Data stored during a Buy action (entry point of the trade)" src="https://cdn-images-1.medium.com/max/718/1*luv2Q4NuyDTV8WoeN59P3A.png" /><figcaption>Figure 5: Data stored during a Buy action (entry point of the trade)</figcaption></figure><figure><img alt="Figure 6: Data stored during a Sell action (exit point of the trade)" src="https://cdn-images-1.medium.com/max/869/1*jxhnw2sBgrkR1i4ZbE6F0g.png" /><figcaption>Figure 6: Data stored during a Sell action (exit point of the trade)</figcaption></figure><p>The trading loop is quite refined, and it fits in 26 lines which makes it clear to understand :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6425e64ecf8c13dfba80416307837be/href">https://medium.com/media/a6425e64ecf8c13dfba80416307837be/href</a></iframe><p><strong>IV — Conclusion</strong></p><p>First we can recall that the entire project, along with the code, is available in a dedicated GitHub repository [5].</p><p>This project is a great accomplishment, as we managed to display how to build a profitable trading bot from scratch. The pipeline is easily adaptable and can fit various financial markets. Also, the required input data is limited, so that the access to historical data should not be an issue.</p><p>We showed that the pipeline was yielding <strong>robust and profitable trading strategies on backtested simulations</strong> (article III) and we exposed in this very last article how to implement the principle of a trading bot in a simple Python script.</p><p>We may also recall that the purpose of this series of articles was to make this notion accessible to everyone, and that obviously one can enhance the proposed pipeline by multiple means. In particular, one could think about <strong>adding some more predictive variables, increasing the number of layers in the Deep Learning architecture, or even ameliorate the implementation loop</strong>. It is also quite feasible to adapt the trading loop in order to take and monitor multiple trades (possibly over multiple markets) at the same time.</p><p>To give an overview of some possibilities, I can invite the interested reader to look at academic papers, especially MSc and PhD thesis. At another level, I wrote a report concerning my construction of such advanced trading bot, available at [6].</p><p>Don’t hesitate to leave any feedback/questions/<em>clap </em>or to contact me for more information.</p><p><strong>04/01/2021 : </strong>There are a couple of bugs in the buy/sell functions. Looking forward to correcting them asap.</p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Other articles of the series :</strong></p><ul><li><a href="https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7">I. A turnkey Python code for creating a stock forecasting dataset | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db">II. Forecasting crypto tendencies with Deep Learning in Python | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/iii-creating-profitable-trading-strategies-1858a2fc8b85">III. Creating profitable trading strategies | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li></ul><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning) </em><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p>Sources :</p><p>[1] Article III</p><p><a href="https://medium.com/analytics-vidhya/iii-creating-profitable-trading-strategies-1858a2fc8b85">III. Creating profitable trading strategies | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p>[2] Poloniex API</p><p><a href="https://docs.poloniex.com/#create-an-api-key">Reference</a></p><p>[3] Article I</p><p><a href="https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7">I. A turnkey Python code for creating a stock forecasting dataset | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p>[4] Article II</p><p><a href="https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db">II. Forecasting crypto tendencies with Deep Learning in Python | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p>[5] GitHub repository</p><p><a href="https://github.com/Seb943/TBpolo">Seb943/TBpolo: A profitable trading bot using Python and Poloniex API (github.com)</a></p><p>[6] Full paper</p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//seb943.github.io/Data/Paper_CreatingATradingBot.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce81dc17c81a9aeb3eac33b9b3cbb393/href">https://medium.com/media/ce81dc17c81a9aeb3eac33b9b3cbb393/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*Jq8ndUzBrLl-ynPtKhppaA.png" /><figcaption>Figure 8: Cover picture</figcaption></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3af87eba48c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/iv-deploy-a-poloniex-trading-bot-c3af87eba48c">IV. Deploy a Poloniex trading bot</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[III. Creating profitable trading strategies]]></title>
            <link>https://medium.com/analytics-vidhya/iii-creating-profitable-trading-strategies-1858a2fc8b85?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1858a2fc8b85</guid>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Mon, 28 Dec 2020 23:10:37 GMT</pubDate>
            <atom:updated>2023-08-08T21:53:49.703Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Two ways to generate profits thanks to Deep Learning models.</blockquote><p>This article is part of a series of articles that describe the step-by-step process to create, backtest, and deploy a trading bot. The first article focused on <strong>computing predictive variables</strong>, while the second focused on <strong>building the Deep Learning models</strong>, this third focuses on <strong>identifying profitable trading strategies</strong> with the models, and the last one on the real-life implementation in a program (<strong>deployment</strong>). Links are given at the end of this article.</p><p>Now that we have built our models with the training set and compared them on the validation set (Article II — [1]), we will <strong>backtest 2 strategies</strong> in front of totally new data (the test set). The <strong>first approach</strong> is simply to pick the best-performing model over the validation set, and follow its advice when it predicts a bullish trend (<em>preds = 1</em>). The <strong>second approach</strong> is more evolved : we look for the threshold <em>l </em>such that we can generate the highest ROI when selecting timesteps where the best model predicts a probability higher than <em>l, i.e. </em>we seek for <em>l</em> such that limiting our trades to the cases where <em>proba_best_model &gt; l </em>yields the best profit.</p><p><strong>0 — Loading libraries and datasets</strong></p><p><strong>I — Defining functions for backtesting</strong></p><p><strong>II — A simple strategy : pick our best model and follow it when bullish</strong></p><p><strong>III — A more evolved strategy</strong></p><p><strong>IV — Conclusion</strong></p><h3>0 — Loading libraries and datasets</h3><p>As during the two previous articles, we first need to import the libraries, the data, and define the constants (see other articles for further details). Especially, we define the <strong><em>stoploss</em></strong>, <strong><em>takeprofit</em></strong>, the list of <strong><em>nPCs</em></strong><em> </em>that we compared during the construction of the models. For the backtesting, we also need to set the value of the <strong><em>fees </em></strong>for each transaction : in the case of the Poloniex [2] exchange these are about 0.125%, but you can adapt this value to yours. We also define the <strong><em>minimal number of trades required</em></strong><em> </em>to consider a strategy during the second approach (50 trades is a good threshold). The initialization code looks like this :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b4306997f60797ac306df9d6f81b270/href">https://medium.com/media/3b4306997f60797ac306df9d6f81b270/href</a></iframe><h3>I — Defining functions for backtesting</h3><p>Once we have imported the data, time is now to create two backtesting functions. The first one is a function that computes the earnings and loss, given a specific set of <em>stoploss</em>, <em>takeprofit </em>and the exchange <em>fees</em>. This step is important so that our backtesting is coherent with real-life trading conditions.</p><p>The second function really is the backtesting function. It takes the testing set as input, the model which we want to backtest, and our constants. <strong>It then computes the predictions over the test set, and computes the backtested earnings in a column named <em>‘EarningsBullish’</em></strong>. As we seek to build a <em>bullish </em>trading bot, we filter out the <em>bearish </em>predictions (<em>preds = 0</em>). We can have a look at what this column looks like after computation, with our set of stoploss and takeprofit. We recall that ‘<em>result</em>’ is the ground truth result and ‘<em>preds</em>’ is the model’s prediction :</p><figure><img alt="Figure 2 : Overview of the EarningsBullish column" src="https://cdn-images-1.medium.com/max/497/1*LxdkMt3B54helavFy6NrFg.png" /><figcaption>Figure 2 : Overview of the EarningsBullish column</figcaption></figure><p>The code to generate the backtesting is given below :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3290f971b98cfe487540d138304887ad/href">https://medium.com/media/3290f971b98cfe487540d138304887ad/href</a></iframe><h3>II — A simple strategy : pick our best model and follow it when bullish</h3><p>Once we have defined our backtesting functions, we can try to look at the first approach. <strong>This approach is simply to follow the best model’s bullish predictions</strong>. We then display the <strong><em>PnL </em></strong>(Profits And Loss) profile, the <strong>entry points</strong>, and the <strong><em>ROI </em></strong>(Return On Investment). Finally, we may wish to evaluate to which degree the performance is great. To do this, we compare the approach’s performance to the average return over the testing period (with <em>stoploss </em>and <em>takeprofit</em>) : that is, we look at the expected returns if we’d always bet <em>bullish </em>at each timestep over the test set. With this technique, we can really assess the performance of our method, independently of the market’s conditions. We will obtain a metric called <em>avg_return_benchmark</em>, which we will compare to our performance. To do this, we test the statistical significance of our performance by comparing it to the average benchmark through a <em>t-test </em>[3].</p><p>The performance is superior to the benchmark, and<strong> it is even significative when we perform a t-test (p-value &lt;&lt; 0.05)</strong>.</p><p>What we obtain in the case of our example is like this :</p><figure><img alt="Figure 4: PnL evolution of our best model over the test set" src="https://cdn-images-1.medium.com/max/1024/1*TKyKd4cDui13QWenQigiDg.png" /><figcaption>Figure 4: PnL evolution of our best model over the test set</figcaption></figure><figure><img alt="Figure 5: Best model’s entry points" src="https://cdn-images-1.medium.com/max/1024/1*TqUxs0mTQjF9dr5Afsk1Lg.png" /><figcaption>Figure 5: Best model’s entry points</figcaption></figure><figure><img alt="Figure 6: Evaluating the significance of the performance" src="https://cdn-images-1.medium.com/max/472/1*zNow2Au2-OY4I2agEeWPNw.png" /><figcaption>Figure 6: Evaluating the significance of the performance</figcaption></figure><p>We could even zoom-in at the first 3 months of 2020 to better capture the relevancy of the trading bot. We can see that the trading bot made profitable bets at the right times, and stopped being <em>bullish</em> when the market was collapsing. If we used it there, it would have allowed us to win money in the first place, and to avoid losing some in the second place :</p><figure><img alt="Figure 7: Approach 1 zoomed-in during the period January-March 2020" src="https://cdn-images-1.medium.com/max/1024/1*QH1mjvOFhDuOq-rIKivvMA.png" /><figcaption>Figure 7: Approach 1 zoomed-in during the period January-March 2020</figcaption></figure><p>For information purpose, this approach yielded a <strong>ROI per trade of 3.3698%</strong> over the test set, with <strong>897 trades</strong> being taken and 516 ending up reaching the takeprofit (<strong>57.52% accuracy</strong>).</p><p>The code to generate and backtest this first strategy is the following :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8440603664c2b7e59dfa16f34a1cf35c/href">https://medium.com/media/8440603664c2b7e59dfa16f34a1cf35c/href</a></iframe><h3>III — A more evolved strategy</h3><p>As presented earlier, the principle of this second approach can be summarized as following : <strong>we pick the best model, and look for the threshold <em>l </em>such that limiting the trades to cases where <em>proba_bullish &gt; l</em> yields the most profitable strategy</strong>. To do this, we need to evaluate the best model on the <strong><em>validation_set</em></strong>, otherwise the approach would not be generalizable. The very first step is to create a function that creates a recapitulative table such like this one (<em>Figure 8</em>), which we can display for illustration purpose (<em>Figure 9</em>) :</p><figure><img alt="Figure 8: The recapitulative table of each section for the best model on the validation set" src="https://cdn-images-1.medium.com/max/641/1*uhxZ7WITqT4Q0pattZufJQ.png" /><figcaption>Figure 8: The recapitulative table of each section for the best model on the validation set</figcaption></figure><figure><img alt="Figure 9: Plotting of the precedent table : maximum is achieved for l = 0.90" src="https://cdn-images-1.medium.com/max/1024/1*WAFOhHirD9wg4CI-mIGtqw.png" /><figcaption>Figure 9: Plotting of the precedent table : the maximum ROI is achieved for l = 0.95</figcaption></figure><p>Once we have computed and saved this table, we simply pick the threshold which yields the best ROI. In our example we can see that the best threshold is <em>l =0.95</em>. We can then evaluate this strategy (keeping predictions where <em>proba_best_model &gt; 0.95</em>) on the test set so that we can see how this approach performs on totally new data. We observe that <strong>the strategy is once again yielding great profits, with a ROI that is even higher than during the first approach (3.41% vs 2.90%)</strong>. All of these observations tend to confirm the robustness and the validity of our method.</p><p>Also,<strong> the performance is also significative (p-value &lt;&lt; 0.05)</strong>, which is a great point :</p><figure><img alt="Figure 10: Second approach’s PnL over the test set" src="https://cdn-images-1.medium.com/max/1024/1*9P7OKZRyc1avAyBBWGWfNg.png" /><figcaption>Figure 10: Second approach’s PnL over the test set</figcaption></figure><figure><img alt="Figure 11: Second approach’s entry points" src="https://cdn-images-1.medium.com/max/1024/1*XLPHLXB3dWMBSRBIZ1QHnA.png" /><figcaption>Figure 11: Second approach’s entry points</figcaption></figure><figure><img alt="Figure 12: Evaluating the significance of the performance" src="https://cdn-images-1.medium.com/max/470/1*eCbuMab04bxnxEe_N97iIQ.png" /><figcaption>Figure 12: Evaluating the significance of the performance</figcaption></figure><p>As during the first approach, we can zoom-in during the first months to see the relevancy of such trading bot :</p><figure><img alt="Figure 13: Approach 2 zoomed-in during the period January-March 2020" src="https://cdn-images-1.medium.com/max/1024/1*_rd3lURjWx5V7xx_2jU5CQ.png" /><figcaption>Figure 13: Approach 2 zoomed-in during the period January-March 2020</figcaption></figure><p>When we look at the statistics of this approach, we can notice that it is quite <strong>more selective than the first approach</strong> we presented earlier. Over the same testing period it took <strong>less trades</strong>, and had a<strong> better success rate</strong>.</p><p>This approach yielded a <strong>ROI per trade of 4.1668%</strong> over the test set, with <strong>498 trades</strong> being taken and 313 ending up reaching the takeprofit (<strong>62.85% accuracy</strong>).</p><p>The code to generate the second approach is written down there :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b43829c6bcb11beb5856fb9d965f105/href">https://medium.com/media/8b43829c6bcb11beb5856fb9d965f105/href</a></iframe><h3>IV — Conclusion</h3><p>First we may recall that the full code is available in the repository at [4]. I even made a script called <strong><em>Pipeline_I_to_III.py</em></strong><em> </em>[5] : you only need to set the <em>stoploss</em>, <em>takeprofit</em>, <em>splitting dates</em> and the <em>dataset location</em> and the pipeline does all steps automatically (variables computation, PCA, building models, assessing, identifying profitable strategies) ! By taking the very same dataset that I took over the course of this series of article (available in the repository aswell), you will eventually find out the exact same results.</p><p>During this article, we have created two profitable trading strategies. These strategies perform better than the benchmark — <em>i.e.</em> they are accurate and yield sufficient profit — and are even yielding significant results. <strong>We checked that the strategies were yielding profit on totally new data, which lets indicate that their real-life implementation will also generate profits in the long run.</strong></p><p>It is worth noting down that we can observe the model is not predicting any more <em>bullish </em>trends since the BTC price has exploded over 15k $ : this might be due to the fact our dataset is not <em>stationary</em>. The model thinks the price will go down to its usual values over the train set. <strong>We could avoid this issue by making it stationary in the first place, changing the stoploss/takeprofit for shorter-term fluctuations or — more simply — by adding some normalized control variables such as MACD.</strong> However, if we apply the pipeline to an asset that’s been in a limited range for years the problem might solve itself. Also, as the purpose of a trading bot is to bet on <em>short-term</em> tendencies rather than long-term investment, this is not a huge issue. We can also observe that the recent times where the model predicts a bullish trend are highly accurate (<strong>high <em>selectivity</em></strong>).</p><p>Now that we have identified and backtested our strategies, we may wish to implement them for real-life trading. That is exactly the purpose of the next article (article IV), which will expose how one can deploy such trading bot in Python. The trading bot will be developed with the Poloniex API, which is a cryptocurrencies exchange platform. However, the framework can easily be adapted to any other API or coding system.</p><p>Also, the full paper that I wrote concerning an advanced trading bot is available at [6]. Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Other articles of the series :</strong></p><ul><li><a href="https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7">I. A turnkey Python code for creating a stock forecasting dataset | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db">II. Forecasting crypto tendencies with Deep Learning in Python | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/iv-deploy-a-poloniex-trading-bot-c3af87eba48c">IV. Deploy a Poloniex trading bot | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li></ul><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionnaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Sources :</strong></p><p>[1] Second article</p><p><a href="https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db">II. Forecasting crypto tendencies with Deep Learning in Python | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p>[2] Poloniex exchange</p><p><a href="https://poloniex.com/exchange/USDT_BTC">Poloniex - Crypto Asset Exchange</a></p><p>[3] T-test</p><p><a href="https://www.scribbr.com/statistics/t-test/">An Introduction to t Tests | Definitions, Formula and Examples</a></p><p>[4] Full code</p><p><a href="https://github.com/Seb943/TBpolo/blob/master/Codes/III.AssessAccuracyAndBacktest.py">Seb943/TBpolo</a></p><p>[5] <em>Pipeline_I_to_III.py</em> (still inside repositoy)</p><p><a href="https://github.com/Seb943/TBpolo/blob/master/Codes/Pipeline_I_to_III.py">TBpolo/Pipeline_I_to_III.py at master · Seb943/TBpolo (github.com)</a></p><p>[6] Full paper</p><p><a href="https://seb943.github.io/Data/Paper_CreatingATradingBot.pdf">https://seb943.github.io/Data/Paper_CreatingATradingBot.pdf</a></p><p><strong>Software :</strong> Python 3.8.5 on Pyzo IDE</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*Jq8ndUzBrLl-ynPtKhppaA.png" /><figcaption>Figure 15 : Cover picture</figcaption></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1858a2fc8b85" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/iii-creating-profitable-trading-strategies-1858a2fc8b85">III. Creating profitable trading strategies</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[II. Forecasting crypto tendencies with Deep Learning in Python]]></title>
            <link>https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5948909617db</guid>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Tue, 22 Dec 2020 21:54:59 GMT</pubDate>
            <atom:updated>2023-08-08T21:54:09.445Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>In less than 65 lines.</blockquote><p>This article is part of a series of articles that describe the step-by-step process to create, backtest, and deploy a trading bot. The first article focused on <strong>computing predictive variables</strong>, while this second focuses on <strong>building the Deep Learning models</strong>, the third on <strong>identifying profitable trading strategies</strong> with the models, and the last one on the real-life implementation in a program (<strong>deployment</strong>). Links are given at the end of this article.</p><p>Now that we have created our predictive variables <em>X </em>and our output <em>Y </em>(article I — [1]), it is time to build and train several Deep Learning models to predict <em>Y </em>from <em>X</em>. To do this, I will expose how one can easily build such a model in Python, train it, and finally assess its accuracy on a validation set. The code can also be easily adapted to fit any other input format.</p><p>The process is summarized in this article in order to display the very key information. The article is decomposed as followed :</p><p><strong>0 — Import libraries and define constants</strong></p><p><strong>I — Train Deep Neural Networks</strong></p><p><strong>II — Compare models’ performance on the validation set</strong></p><p><strong>III — Conclusion</strong></p><h3><strong>0 — Import libraries and define constants</strong></h3><p>The very first step is to import the libraries, (re-)define our <em>stoploss</em>/<em>takeprofit</em> and also define the sets of Principal Components that we want to fit our models on. For instance, in the code furnished below, we will build and fit Deep Learning models with various amount of PCs between 10 and 40. The objective is then to compare their accuracies over the validation set, and eventually to pick the best one in order to create a trading strategy (see article n°III).</p><p>We recall that during the previous steps (article I [1]), we saved our datasets but also the associated datasets after applying scaling and PCA. These latters are called <em>xxxx_</em><strong><em>final</em></strong><em>_xxx.csv, </em>and <strong>these sets will be used for predictions</strong>.</p><p>The code to load data and libraries is written down there :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f3298940349db2f1d501b7e764ce64b/href">https://medium.com/media/6f3298940349db2f1d501b7e764ce64b/href</a></iframe><h3><strong>I — Train Deep Neural Networks</strong></h3><p>Once we have loaded all the data, the next step is now to build and train several Deep Learning models. I decided to build DL models because of their impressing performances, and the fact that it is fairly easy to build them with <em>Keras</em>.</p><p>The architecture that I developed is fairly simple, and composed of <strong>4 hidden layers</strong>. The input layer has a dimension of <em>nPCs</em>, where <em>nPCs </em>if the number of Principal Components that we use for the specific regression. The output layer has the <strong>sigmoid function</strong> [2] as activation function, and the model was trained in <em>keras </em>using the <strong>Adam optimizer</strong> [3], the <strong>binary cross-entropy loss function</strong> [4] and <strong>accuracy</strong> as assessment metric.</p><p>We recall that the PCs are ordered by rank of importance (<em>i.e.</em> the amount of variance contained in them), which means that the first PCs are supposed to be the ones carrying the most relevant information. That is not always the case, but it usually is.</p><p>The architecture of my model is displayed down there, with <em>nPCs = 10</em> in this specific case :</p><figure><img alt="Figure 2: Deep Learning network architecture (software used for illustration [5])" src="https://cdn-images-1.medium.com/max/682/1*W_lNlsKQRHQklUdydW99Cg.png" /><figcaption>Figure 2: Deep Learning network architecture (software used for illustration [5])</figcaption></figure><p>The code for building and training these several DL models is written down there :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbf72da5e5cdc56eb719beb41332d546/href">https://medium.com/media/fbf72da5e5cdc56eb719beb41332d546/href</a></iframe><h3><strong>II — Compare models’ performance on the validation set</strong></h3><p>Once we have built our models, time is now to compare them over the validation set. This step is important because there we rank our models, and this will eventually determine which model we will use for potential real-life trading.</p><p>To do this, we simply load every model that we previously trained, compute predictions onto the validation set, and store all results in a <em>.csv</em> file.</p><p><strong><em>NB :</em></strong><em> As we seek to bet only on </em><strong><em>bullish </em></strong><em>trends, we evaluate accuracies</em><strong><em> only on bullish predictions</em></strong><em> (i.e. when the model predicts 1)</em><strong><em>. </em></strong><em>This means that an accuracy of 46.63 % for nPCs = 10 should be interpreted as : when the model predicted ‘1’, 46.63 % of the time it happened.</em></p><p>In our example with have stoploss = 5% and takeprofit = 10%, so <em>a priori</em> we would expect an accuracy of around 33%. In reality, there was the following proportion of bullish results (‘<em>result’ = 1</em>) over the validation set, as the period was quite bullish overall :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*CNITo6U1n1gLotyQ9wPOBw.png" /><figcaption>Figure 4 : Validation set average bullish result</figcaption></figure><p>Obtaining <strong>46.98% can then be considered a good performance (see Figure 5)</strong>. However, the accuracy is only useful for ranking the models. Also, all of our models did better than the average repartition on the validation set : that is a good point, we perform better than trading blindly (!).</p><p>We can then have a look at the recapitulative table and plot it. We can observe that the best performance is achieved with <em>nPCs = 20</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*WRFJJLQHhXrFKW_D7NxqNg.png" /><figcaption>Figure 5: Comparative of accuracies over the validation set</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jrLdedkWVRrmj8cfuW9Usg.png" /><figcaption>Figure 6: Plotting of the previous table</figcaption></figure><p>The code for assessing the models on the <em>validation set</em> is furnished there down :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9db1eb273a787008ed34f1cca0e6b837/href">https://medium.com/media/9db1eb273a787008ed34f1cca0e6b837/href</a></iframe><h3><strong>III — Conclusion</strong></h3><p>After this step, we have now completed building, training and assessing various Deep Learning models on our datasets. We can be quite confident about our results, and the ranking of our models will allow us to create profitable strategies during our next step (article III —<em> coming soon</em>).</p><p>We can also recall that the entire code is available at [6]. The full paper I wrote concerning the construction of an advanced trading bot is at [7].</p><p>Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Other articles of the series :</strong></p><ul><li><a href="https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7">I. A turnkey Python code for creating a stock forecasting dataset | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/iii-creating-profitable-trading-strategies-1858a2fc8b85">III. Creating profitable trading strategies | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/iv-deploy-a-poloniex-trading-bot-c3af87eba48c">IV. Deploy a Poloniex trading bot | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li></ul><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionnaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Sources</strong></p><p>[1] First article</p><p>https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7</p><p>[2] The sigmoid function</p><p>https://www.youtube.com/watch?v=WsFasV46KgQ</p><p>[3] The Adam optimizer</p><p>https://keras.io/api/optimizers/</p><p>[4] The binary cross-entropy loss function</p><p>https://gombru.github.io/2018/05/23/cross_entropy_loss/#:~:text=Binary%20Cross-Entropy%20Loss%20Also%20called%20Sigmoid%20Cross-Entropy%20loss.,component%20is%20not%20affected%20by%20other%20component%20values.</p><p>[5] Software used for the illustration</p><p><a href="http://alexlenail.me/NN-SVG/index.html">NN SVG</a></p><p>[6] GitHub repository</p><p>https://github.com/Seb943/TBpolo</p><p>[7] Full paper</p><p>https://seb943.github.io/Data/Paper_CreatingATradingBot.pdf</p><p><strong>Software used :</strong> Python 3.8.5 in the Pyzo IDE</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5948909617db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db">II. Forecasting crypto tendencies with Deep Learning in Python</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I. A turnkey python code for creating a stock forecasting dataset]]></title>
            <link>https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7?source=rss-a0b1d27359d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad25f673f3b7</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Sébastien Cararo]]></dc:creator>
            <pubDate>Tue, 22 Dec 2020 15:19:38 GMT</pubDate>
            <atom:updated>2023-08-08T21:54:31.212Z</atom:updated>
            <content:encoded><![CDATA[<h3>I. A turnkey Python code for creating a stock forecasting dataset</h3><blockquote>Creating a ready-to-train dataset from basic historical information.</blockquote><p>This article is part of a series of articles that describe the step-by-step process to create, backtest, and deploy a trading bot. This first article focuses on <strong>computing predictive variables</strong>, while the second focuses on <strong>building the Deep Learning models</strong>, the third on <strong>identifying profitable trading strategies</strong> with the models, and the last one on the real-life implementation in a program (<strong>deployment</strong>). Links are given at the end of this article.</p><p>The pipeline presented down there can be applied to any sort of financial market (stock market, crypto, electricity, etc…). The trading bot that we build only concerns <strong>bullish </strong>moves, <em>i.e.</em> we only bet on increasing trends. I decided not to study <em>bearish </em>trading because of the multiple differences over the exchanges.</p><p>What you’ll only need to <strong>input</strong>:</p><ul><li>An historical dataset containing the following variables within candles: ‘<em>close</em>’, ‘<em>date</em>’, ‘<em>high</em>’, ‘<em>low</em>’, ‘<em>open</em>’, ‘<em>volume</em>’</li><li>Set a <em>stoploss </em>and a <em>takeprofit</em></li><li>Split your dataset into three distinct parts : you’ll just need to set the two splitting dates</li></ul><p><strong>What you’ll have</strong> after applying the pipeline :</p><ul><li>A set of Deep Learning models, along with their performance over the validation set</li><li>A simple trading method : follow the best model’s instructions</li><li>A more evolved trading method : follow the best model in the most profitable cases only</li><li>An operating trading bot code for Poloniex (can be adapted to any platform)</li></ul><p>I tried to make this series as intelligible as possible towards non-financial readers : apart from certain technical points, the articles are quite illustrated and the underlying notions are usually easy to understand. This first article has the following outline :</p><p><strong>0 — Import libraries and constants</strong></p><p><strong>I — Define standard predictive variables</strong></p><p><strong>II — Build the function to compute all variables</strong></p><p><strong>III — Compute the output</strong></p><p><strong>IV — Apply PCA and save results</strong></p><p><strong>V — Conclusion</strong></p><h3><strong>0-Import libraries and constants</strong></h3><p>The very first step consists of initializing and inputting the required data. After importing the libraries, the user should provide an historical dataset containing the date and 5 basic informations within each candle, that are the <strong>opening </strong>and <strong>closing</strong> prices, the <strong>min </strong>and <strong>max </strong>values within the candle, and the <strong>volume </strong>exchanged. The whole pipeline will be constructed upon these features.</p><p>Also, the user just needs to set the values of the <strong>stoploss </strong>and the <strong>takeprofit</strong>, plus the dates at which we should split the dataset into <em>training set</em>, <em>validation set</em> and <em>testing set</em>. The <em>training set </em>is used for training the Deep Learning models, while the <em>validation set</em> is used to evaluate the accuracy of the different models. Finally, the <em>test set</em> is used to check that our final trading strategy is effectively working on totally new data.</p><p>In my example, I used a cryptocurrency dataset that I downloaded from the Poloniex API : it consists of candles of 7200 secs (2 hours) for the pair <em>BTC_USDT</em>, from 20/02/2015 to 21/12/2020. The dataset I used, along with the whole code, is available in a dedicated GitHub repository [2].</p><figure><img alt="Figure 1: Overview of my historical dataset (relevant variables in green)" src="https://cdn-images-1.medium.com/max/900/1*NMJ-lbebkXT4MuN7gKpB9w.png" /><figcaption>Figure 1: Overview of my historical dataset (relevant variables in green)</figcaption></figure><p>During the example, I did set a training period of 45 months, a validation set of 12 months and a testing set of 12 months (roughly <em>65%/17.5%/17.5%</em>). I also decided to choose <em>stoploss = 5%</em> and<em> takeprofit = 10%</em>, and my splitting yields the following sets :</p><figure><img alt="Figure 2 : Splitting between train set, validation set and test set." src="https://cdn-images-1.medium.com/max/1024/1*8-pLjGxofEjaknwz3qb4zA.png" /><figcaption>Figure 2 : Splitting between train set, validation set and test set.</figcaption></figure><p>The initialization code is written below, the user should just change lines 10 &amp; 11 aswell in order to fit with his own file paths:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c62d04724abb7d462be8c032857e0855/href">https://medium.com/media/c62d04724abb7d462be8c032857e0855/href</a></iframe><h3><strong>I — Define standard predictive variables</strong></h3><p>The next step is to <strong>define the variables which we want to compute</strong> for forecasting the tendencies. I chose to compute only a couple of these variables in my example, however we will see that this is sufficient to build accurate models and generate profits. In the next step, these variables will be computed over various window sizes in order to account for both short-term and long-term movements.</p><p>The code for computing Simple Moving Averages [3] and RSI [4] is given down there :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85823c52843efd851200072c2a0139c7/href">https://medium.com/media/85823c52843efd851200072c2a0139c7/href</a></iframe><h3><strong>II — Build the function to compute all variables</strong></h3><p>Once we have defined our basic predictive variables, time is now to <strong>compute these variables in the dataset</strong>. Moreover, we will also compute some usual variables such as the min/max on rolling windows, the moving average of the volume, <em>modulos</em>, etc… You can choose which variables you may want to add/remove in your own trading bot. Usual forecasting variables include metrics such as Bollinger Bands [5], MACD [6] or even support/resistance trend lines [7].</p><p>The code for computing the variables on a given dataset is given down there:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/314a8bbaaf52e567e877fae9339d5b64/href">https://medium.com/media/314a8bbaaf52e567e877fae9339d5b64/href</a></iframe><h3><strong>III — Compute the output</strong></h3><p>Now that we have defined our predictive variables <em>X,</em> it is time to <strong>define the output of our dataset</strong>, that is the result <em>Y </em>that we seek to predict. As trading bots operate in a binary way — that is either we buy/sell the asset — I decided to encode my output in a binary way. Concretely, given my <em>stoploss </em>and my <em>takeprofit</em>, I defined <em>Y </em>as following :</p><figure><img alt="Figure 6 : Computing Y" src="https://cdn-images-1.medium.com/max/965/1*3w6mt2EytWeHhqKtaKbHoA.png" /><figcaption>Figure 6 : Computing Y</figcaption></figure><p>The observations where <em>Y = -1</em> are then removed (these are only the most recent timesteps), because we wouldn’t be able to exploit them anyway.</p><p>To do this, for each timestep we browse over the dataset from this point until the price reaches either the <em>stoploss </em>or the <em>takeprofit</em>. As we only have a limited set of information (close, open, min, max, volume), I consider that within a candle it is not possible to reach both the <em>stoploss </em>and the <em>takeprofit</em>. This assumption is realistic given the values of the candles’ length and the values of <em>stoploss</em>/<em>takeprofit</em>, however it is worth noting it down.</p><p>The code for computing the output <em>Y </em>is the following one :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2234897690c31f2deb1ca9b07f4615dd/href">https://medium.com/media/2234897690c31f2deb1ca9b07f4615dd/href</a></iframe><h3><strong>IV — Apply PCA and save results</strong></h3><p>Once we have computed <em>X </em>and <em>Y</em>, the final step in building a ready-to-train financial dataset is to <strong>apply Principal Components Analysis</strong> (PCA — [8]). This technique is very useful to reduce the variance in the dataset before training DL models, especially in the case of highly correlated predictive variables.</p><p>Actually, many of the variables that we computed are highly correlated. That is mainly because they come from the same signal. Such correlation can be a problem for Deep Learning forecasting, as apart from increasing the training times it also increases the variance in the dataset, which often results in poor forecasting performances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4FXnK_c-M648-ROvZUdaHg.png" /><figcaption>Figure 8 : Correlation matrix of the training set (Pearson)</figcaption></figure><p>To deal with this, we apply PCA to create a new set of orthogonal components, that contain exactly the same information overall. This will be useful in the training process, because we will be able to compare models with different set of Principal Components (PCs) via cross-validation, in order to pick the very best model.</p><p>We proceed to scaling the variables both before and after applying the PCA process. This permits a gain in both training and forecasting times — that is useful when you need the trading bot to take decisions as quick as possible.</p><p><strong><em>NB :</em></strong><em> We pay attention to fitting the scalers and the PCA on the </em><strong><em>trainset only</em></strong><em>, otherwise it would generate causality issues in the next sections (i.e. we would be predicting past observations when having future information).</em></p><p>We pay attention to removing the output (‘<em>result</em>’) and ‘<em>date’ </em>variables before fitting the scalers and the PCA. For illustration purpose, we can display what the datasets look like before and after applying this section (<em>scaler n° 1 + PCA + scaler n° 2</em>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*LS0Phcq4vPFNwz7dUdUB2g.png" /><figcaption>Figure 9 : Overview of the training set variables before applying this block</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*_hpgRkNgvHL5kWjMP9EfuA.png" /><figcaption>Figure 10 : Overview of the trainset_final variables (we can observe that the variables are well scaled to have mean 0 and std = 1)</figcaption></figure><p>The code to scale, apply PCA, and scale the variables is written down there :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78ed0ffb3a680688f00442d0bcb17ef5/href">https://medium.com/media/78ed0ffb3a680688f00442d0bcb17ef5/href</a></iframe><p><strong>V — Conclusion</strong></p><p>First we can recall that the full code is available at [2]. As a conclusion, we can be satisfied for having transformed our basic historical dataset into a dataset that is ready-to-train on. To apply this pipeline, you just need to copy/paste the code and modify the input accordingly to your needs.</p><p>What we have created over the course of this article :</p><ul><li>Train set, validation set, test set with <em>X </em>and <em>Y</em></li><li>2 scalers and a PCA function</li><li>The 3 datasets after applying the scalers and the PCA : these datasets are <em>ready-to-train</em> (<em>ready-to-predict</em>) datasets</li></ul><p>The next article will explain how we can use these datasets to build and train several Deep Learning models. The final goal will be to compare the models, pick the best one and create profitable and robust trading strategies (article III). We will conclude this series by deploying the bot in real-life conditions (article IV).</p><p>Don’t hesitate to leave any feedback/questions/<em>claps</em> or to contact me for more information.</p><p><strong>Other articles of the series :</strong></p><ul><li><a href="https://medium.com/analytics-vidhya/ii-forecasting-crypto-tendencies-with-deep-learning-in-python-5948909617db">II. Forecasting crypto tendencies with Deep Learning in Python | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/iii-creating-profitable-trading-strategies-1858a2fc8b85">III. Creating profitable trading strategies | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li><li><a href="https://medium.com/analytics-vidhya/iv-deploy-a-poloniex-trading-bot-c3af87eba48c">IV. Deploy a Poloniex trading bot | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></li></ul><p><strong>Contact :</strong> sebcararo@hotmail.fr</p><p><strong>Another article </strong><em>(building a sports-betting algorithm with Machine Learning)</em><strong> </strong><a href="https://medium.com/analytics-vidhya/how-covid-19-prevented-me-from-being-a-millionnaire-in-2020-5b2144e8bdef">How Covid-19 prevented me from being a millionnaire in 2020 | by Sébastien Cararo | Analytics Vidhya | Dec, 2020 | Medium</a></p><p><strong>Sources :</strong></p><p>[1] Full paper</p><p>https://seb943.github.io/Data/Paper_CreatingATradingBot.pdf</p><p>[2] GitHub repository</p><p>https://github.com/Seb943/TBpolo</p><p>[3] Simple Moving Averages</p><p>https://www.wallstreetmojo.com/moving-average-formula/</p><p>[4] RSI (Relative Strength Index)</p><p>https://www.investopedia.com/terms/r/rsi.asp</p><p>[5] Bollinger bands</p><p>https://www.bollingerbands.com/bollinger-bands</p><p>[6] MACD (Moving Average Convergence Divergence)</p><p>https://www.investopedia.com/terms/m/macd.asp</p><p>[7] Support/resistance trendlines</p><p>https://trading.info/support-resistance</p><p>[8] PCA (Principal Components Analysis — EigenVectors decomposition)</p><p>https://blog.clairvoyantsoft.com/eigen-decomposition-and-pca-c50f4ca15501</p><p><strong>Software :</strong> Python 3.8.5 on Pyzo IDE</p><figure><img alt="Figure 12 : Cover picture" src="https://cdn-images-1.medium.com/max/916/1*Jq8ndUzBrLl-ynPtKhppaA.png" /><figcaption>Figure 12 : Cover picture</figcaption></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Ever interested in high-quality tools for professionals and DIY enthusiasts ? Check out this awesome online store ! <a href="https://toolsyland.com/">Home — ToolsyLand</a></p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad25f673f3b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/i-a-turnkey-python-code-for-creating-a-stock-forecasting-dataset-ad25f673f3b7">I. A turnkey python code for creating a stock forecasting dataset</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>