<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jason Zivkovic on Medium]]></title>
        <description><![CDATA[Stories by Jason Zivkovic on Medium]]></description>
        <link>https://medium.com/@jaseziv83?source=rss-b3835330254b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*t5GVu75IrKtDYFZ20Lg9KQ.jpeg</url>
            <title>Stories by Jason Zivkovic on Medium</title>
            <link>https://medium.com/@jaseziv83?source=rss-b3835330254b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:22:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jaseziv83/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[EXTRACT FOOTBALL (SOCCER) DATA IN R WITH ZERO R EXPERIENCE]]></title>
            <link>https://medium.com/analytics-vidhya/extract-football-soccer-data-in-r-with-zero-r-experience-bee36287ad37?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bee36287ad37</guid>
            <category><![CDATA[tidyverse]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[soccer-analytics]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Mon, 14 Jun 2021 17:37:24 GMT</pubDate>
            <atom:updated>2021-06-14T17:37:24.292Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6A13FNomKUYz5B5Zoz7wNQ.png" /><figcaption>Combining R, worldfootballR and RStudio to analyse football data</figcaption></figure><h3>Introduction</h3><p>Do you find your love of football and curious mind converging to the point where you want to dig a little into the data to confirm if what your eyes are seeing is in fact what’s happening, but you just don’t know how to get started, or where to get data from? Or maybe you have already started doing some analysis in visualisation software, including PowerBI, Tableau, Excel, etc, or know enough about statistics to want to perform some statistical modelling, but you just need to know how to get access to more data, quicker and easier? Then this post will be for you.</p><p>This post will hopefully teach you how to use worldfootballR — an R package built to aid in the extraction of world football data from a number of popular websites (with their consent).</p><p>The post will demonstrate how to use the package to extract your data programmatically and save it in a suitable file format that your report ingests, saving you the manual steps of extracting data from popular websites listed below.</p><p>Additionally, I will also aim to demonstrate the visualisation tools available in R, if you so choose to invest the time into learning R.</p><p>A final aim of the post is to highlight what’s possible in R for those new to R programming in the hope that it stokes enough interest to get you started on your R learning journey.</p><p>What this post won’t be is an in depth how-to-code-in-R post — there are plenty of online resources, including MOOCs, posts, screencasts, etc to do that. This post will focus on getting you up and running with data in your hands… or on your screens.</p><h4>What data can I get?</h4><p>The package as at version 0.3.2 provides access to data from the following data sites:</p><ul><li>FBref.com (a whole host of data to analyse, including results, match stats, season long stats, player and team stats, etc);</li><li>Transfermarkt.com (player market values, team transfer history, player transfer history); and,</li><li>Understat.com (shot locations data for matches played in the major leagues)</li></ul><p>These three sites are regularly used by analysts the world over, however the package is constantly evolving and may include data from additional sites in the future.</p><p>To my knowledge, only FBref provide the ability to export data to a file — the other two don’t, so you’d need to find some other way to get your data (painfully slow copy and paste), and that’s where worldfootballR come in.</p><h4>Prerequisites</h4><p>The only prerequisites to this post are that you have a computer, internet connection and the desire to analyse world football data. This post is designed to take any aspiring analyst with absolutely no R coding experience to being able to extract data programmatically using the worldfootballR R library.</p><h4>What is R / Why would I use R?</h4><p>From R’s official site (<a href="https://www.r-project.org/about.html">https://www.r-project.org/about.html</a>);</p><blockquote><em>R is a language and environment for statistical computing and graphics.</em></blockquote><p>You can find out a lot more about R as you get more experienced with the language on the home page, but all you need to know for now is that R will be another tool in your toolkit to perform the analyses you want to do. R is an amazing programming framework that allows you to do a number of things, including data cleaning, performing statistical analysis and modelling and building fully customisable visualisations using the R programming language.</p><p>Importantly, anything programmed means it’s repeatable. Set up your code upfront, then simply rerun it to get consistent results.</p><p>The world is your oyster if you have the drive to learn.</p><h3>Installing R and RStudio</h3><p>This first section will detail the steps required to install R and RStudio, before you can get started with pulling data using worldfootballR.</p><h4>Getting started</h4><p>Ok so a few things before we dive right in… this is in layman’s terms (because I myself am a layman). R is the engine, the coding language driving everything you want to do, and RStudio is the pretty graphical user interface (GUI pronounced “gooey”) that gives you additional functionality when writing code in it. I am going to propose installing both R and RStudio in this post, but if you don’t want RStudio (I advise you to go with RStudio), you could still run the code to get the data you need (I will assume here everything is done in RStudio).</p><h4>Install R</h4><p>You remember how I said I’m a layman? It’ll extend itself here. We’re going to install R and we’re going to use all of the default options (just keep clicking “<em>accept</em>” or “<em>agree</em>”, or “<em>continue</em>”, and nothing else until R has been successfully installed. You can customise the installation when you’re better versed in R, but for now, just show me the data!</p><p>So how to find R to install R you ask? Well, you have to type into Google some really sophisticated stuff… “<em>download r windows</em>” or on Mac OS… yep, you guessed it, “<em>download r mac</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*cKtsxuooGJ2zSJyU.jpg" /></figure><p>If you can’t be bothered doing that, here are the links:</p><ul><li>For Windows OS: <a href="https://cran.r-project.org/bin/windows/base/">https://cran.r-project.org/bin/windows/base/</a></li></ul><p>Select “Download R X.X.X for Windows”, with “X.X.X” being for whatever is the most recent version of R:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*zqvBUg6QX7lNXkxR.jpg" /></figure><ul><li>For Mac OS: <a href="https://cran.r-project.org/bin/macosx/">https://cran.r-project.org/bin/macosx/</a></li></ul><p>Depending on what version of Mac OS you’re running will determine which version to install, but generally speaking it’ll only be one of the highlighted options below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*wg3FniV6JppxNJFJ.jpg" /></figure><p>Once you have selected either of the Windows or Mac OS versions, install as you would any software, remembering my one special rule at the start of this section… default options all the way.</p><h4>Now for RStudio</h4><p>Once R has been installed, we can now go ahead and install RStudio, remember, the GUI that will allow you to code in and provide additional functionality.</p><p>Back to our intricate Google search query… “<em>download rstudio</em>”, and select the first result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*mLgeoIf5RXti2NMh.jpg" /></figure><p>Or for the URL: <a href="https://www.rstudio.com/products/rstudio/download/">https://www.rstudio.com/products/rstudio/download/</a></p><p>Once we’ve opened the page, we’re presented with a few pricing options, but if you’re like almost all amateur analysts (let’s be honest, even professional analysts), you’re probably only going to need the free version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*OvHDJ6W2pef7rFi4.jpg" /></figure><h4>We’re (almost) ready to start coding!</h4><p>Well, we’re almost ready to start coding, but first we’ll do a bit of a tour of RStudio (again, for those that have forgotten, RStudio is the software where we type and execute our R code).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*7kGyNeeRT7TBFOJ3.jpg" /></figure><p>The console pane is where we will be able to see our code running (or run our code in here if we’re not using the scripts), whether there are any “errors” or “warnings” and printed outputs.</p><p>The environment is the sections where we will see the objects that we’ve imported, or created, including data frames (think of something similar to an Excel table).</p><p>In the viewers pane, there are a few tabs, including being able to navigate through the file system (‘Files’ tab), any visualisations we’ve created (‘Plots’ tab), the libraries/packages we have loaded on our machines and have available to us “locally” (‘Packages’ tab), a help section, where we can get additional details on what we need to provide the functions to work and a ‘Viewer’ tab to see HTML outputs.</p><h4>Now we’re ready to start coding</h4><p>Open up an R script (there are a number of different options to code in, but a .R script is the best place to start.</p><p>To open up a script, you can either press control+shift+n on a Windows machine, or command+shift+n on a Mac, or you can select a new script with the drop down in the top-left corner:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/0*VzA6BCf086FHZGhd.jpg" /></figure><p>Changing the layout as such:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*yzHcfuM0GIfDxqUc.jpg" /></figure><p>We will start writing code in the top-left quadrant.</p><h3>worldfootballR</h3><p>So now we’re here, ready to start coding right? Well we’ve just got one more thing to cover off. What is <a href="https://jaseziv.github.io/worldfootballR/">worldfootballR</a>?</p><p>worldfootballR is a package made up of functions that aid in the extraction of data from popular football sites.</p><p>I know what you’re thinking — what are functions??</p><p>From <a href="https://www.datacamp.com/community/tutorials/functions-in-r-a-tutorial">DataCamp</a></p><blockquote><em>In programming, you use functions to incorporate sets of instructions that you want to use repeatedly or that, because of their complexity, are better self-contained in a sub program and called when needed. A function is a piece of code written to carry out a specified task; it can or can not accept arguments or parameters and it can or can not return one or more values.</em></blockquote><p>Crystal clear right? Well all you need to know is that worldfootballR has taken lines and lines of code to extract data and essentially condensed it down to a line (and more importantly for you, no web scraping knowledge by you).</p><p>Say for instance, you wanted to get match summary information for a particular match(es) from FBref.com… rather than learning about R programming and web scraping to be able to write this code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*NPROxUV4IBS16Yf-.jpg" /></figure><p>All you needed to do was run this line of code:</p><pre><strong>get_match_summary</strong>(match_url)</pre><p>Ok, so now I think we’re ready to start you writing code to extract and save data in a form you need.</p><h4>Writing code</h4><p>Before I fully get in to the nuts and bolts of the code, I just want to point you to some resources.</p><p>A website has been created for the worldfootballR package that will give you a full run down of all the functions available and the arguments you need to pass to those functions to get the data you want. The website can be found <a href="https://jaseziv.github.io/worldfootballR/">here</a>.</p><p>Now we can really start writing code.</p><p>As we’ve already established, packages contain functions (code to do something). To use these packages they need to be installed (downloaded onto our laptops) first.</p><p>Packages will typically (but not limited to) either be housed on CRAN (a central repository for all packages) or on GitHub. worldfootballR is housed on GitHub, so the instructions for installing this package are slightly different to the packages housed on CRAN. Before we can install worldfootballR, we need to install the devtools package to enable the installation of worldfootballR. We will also install another package, xlsx that will enable us to save files in ‘.xlsx’ format.</p><p>We type (just copy and paste) these lines of code into our R script the first time we use R <strong>OR</strong> when we update our version of R <strong>OR</strong> the package updates its functionality:</p><pre><em># to allow us to install packages from github:</em><br>install.packages(&quot;devtools&quot;)</pre><pre><em># a meta package that contains packages for data cleaning, analysis and visualisation</em><br>install.packages(&quot;tidyverse&quot;)</pre><pre><em># to save xlsx files</em><br>install.packages(&quot;xlsx&quot;)</pre><p>You’ll know that they have successfully installed if there are no error messages and you can see the you see the “&gt;” symbol and a blinking cursor in the console section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*UQarBxrC12YMgino.jpg" /></figure><p>Then we will install worldfootballR (it’s what we’re all here for) and importantly load the library - it’s not enough to just install the package, we need to load it to make the functions easily available in our current session.</p><p><strong>Note:</strong> We only need to install the package the first time we install R <strong>OR</strong> if we update our version of R, <strong>OR</strong> if there is a update to a newer version of the package.</p><pre>devtools::install_github(&quot;JaseZiv/worldfootballR&quot;, <strong>ref</strong> = &quot;main&quot;)</pre><p><strong>Note 2:</strong> While we only need to install it very infrequently, we need to load the package <strong>every time</strong> we want to use the package.</p><pre><strong>library</strong>(worldfootballR)</pre><h4>The world is your oyster</h4><p>There are a number of functions in worldfootballR to enable you to get all the data you need.</p><p>To find these functions, you can visit the reference page on the package website <a href="https://jaseziv.github.io/worldfootballR/reference/index.html">here</a>. The reference page will give you a brief summary of what the function does — for more detailed info, click on the function and it’ll take you to the function page.</p><p>Additionally, there are instructions and example code on the website for each of the sites data is collected from:</p><ul><li>Click <a href="https://jaseziv.github.io/worldfootballR/articles/extract-fbref-data.html">here</a> for <strong>FBref</strong> data</li><li>Click <a href="https://jaseziv.github.io/worldfootballR/articles/extract-transfermarkt-data.html">here</a> for <strong>Transfermarkt</strong> data</li><li>Click <a href="https://jaseziv.github.io/worldfootballR/articles/extract-understat-data.html">here</a> for <strong>Understat</strong> data</li></ul><h4>Let’s get some data</h4><p>Now the fun part — getting data!</p><p>Before we get started, in the code below, when we see a “#” at the beginning of text, that is a comment, not executable code, and it’s how R programmers write notes to themselves, or others who use the code to give an overview of what is being done.</p><p>At the start of every script, it is best practise to include all the packages you’re going to be using/needing in your analysis. Installing packages only needs to be done the first time you start using R, when installing new versions or R, or when newer versions of the package become available and you want the updates that have been created.</p><p>The code we would run is below. A few things to remember:</p><ol><li>To ‘run’ code we’ve typed out, there are two main ways;</li></ol><p>a. Highlight the code you want to run and press control+enter or command+enter on MacOS</p><p>b. Have the cursor on the line you want to run and press control+enter or command+enter on MacOS</p><p>2. A “#” in a script signifies comments, and I will use them in the code to tell you what we’re doing</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93985957abfa55b471909e36d34e8157/href">https://medium.com/media/93985957abfa55b471909e36d34e8157/href</a></iframe><h4>I want shooting data!</h4><p>Let’s say we want to get the shooting data from FBref for teams in the 2020–2021 EPL season, to be able to analyse how many goals a team scored, or how they performed against expectation (xG), or from what average distance they shot from:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/0*rmhtujL63Fr2XjIU.jpg" /><figcaption>Source: FBref <a href="https://fbref.com/en/comps/9/Premier-League-Stats">https://fbref.com/en/comps/9/Premier-League-Stats</a></figcaption></figure><p>Then we run the specific code to extract shooting data from FBref.com:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abe3f4d9e6cc3461358e35b183f4c1c8/href">https://medium.com/media/abe3f4d9e6cc3461358e35b183f4c1c8/href</a></iframe><p>And this is what the data looks like in R now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*FlUgZH94rmOPtCQh.jpg" /></figure><h4>I just want the data and nothing else</h4><p>If all you want is a csv or Excel output (to take to some other software/framework), then the below lines of code will address that need:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de9643c7ed71c061e11ea1f9d818320f/href">https://medium.com/media/de9643c7ed71c061e11ea1f9d818320f/href</a></iframe><p>And this is how our output looks, and this output can be fed in to any visualisation software we choose to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*8NNEz5edhlyVi5Nd.jpg" /></figure><h4>But R has more to offer right?</h4><p>R is a whole lot more than a tool for extracting and saving data.</p><p>We can create fully customisable visualisations with code to analyse our data. This flexibility means your creative juices would only be constrained by your desire to practise and learn and practise and learn R.</p><p>Below, we have plotted a team’s performance against expected goals for the season. Teams that finished above the line scored more goals that xG would have had them scored, while teams below the dashed line scored less.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a9b9194be595cc481fda727a4bc434a/href">https://medium.com/media/6a9b9194be595cc481fda727a4bc434a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rSb1dECHPYVp84Bf.png" /></figure><p>We could also calculate how closely correlated xG is to expected goals using the cor() function in R, which we see printed out to be 0.8805275 - highly correlated indicating teams score pretty close to expected:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88529eb146e0b2f792c44f004614b2cb/href">https://medium.com/media/88529eb146e0b2f792c44f004614b2cb/href</a></iframe><h4>More data please</h4><p>FBref.com is a great site that has given analysts unprecedented access to their data, including the ability to extract to csv’s manually. This is great when there is only one extract you need, but when there are multiple extracts, then this becomes infinitely more time consuming! Functions have been created for these, and the official <a href="https://jaseziv.github.io/worldfootballR/">worldfootballR site</a> will get you going with them.</p><p>Others, like Transfermarkt.com and Understat.com don’t give you the ability to export to files. worldfootballR is here to help there too!</p><p>First we’ll get the data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5786c8f31aa25c23853c72487bd2ce3a/href">https://medium.com/media/5786c8f31aa25c23853c72487bd2ce3a/href</a></iframe><p>Then we’ll save it to file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54e8169b76ce227a2f8065a6905682e4/href">https://medium.com/media/54e8169b76ce227a2f8065a6905682e4/href</a></iframe><p>Finally, we will also create a visualisation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eae7f13020b248d7bbeaa4ad52ef8024/href">https://medium.com/media/eae7f13020b248d7bbeaa4ad52ef8024/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1sI1i__iy0U4Czlo.png" /></figure><h3>Conclusion</h3><p>Hopefully you’ve made it this far!</p><p>We’ve gone through an overview of the worldfootballR package, designed to help you extract football data from popular soccer data websites, we’ve installed R and RStudio, we’ve started pulling data using the worldfootballR, we’ve saved it in either .xlsx or .csv, we’ve seen some brief statistical analysis, AND we’ve also created some visualisations using a mixture of dplyr (to get our data ready) and ggplot.</p><p>Hopefully this has stoked your interests enough into the possibilities available to you using R and worldfootballR!</p><p>If you have any questions or feedback, feel free to comment, or get in touch on twitter <a href="https://twitter.com/jaseziv">jaseziv</a>.</p><p>This post originally appeared on the Don’t Blame the Data blog and can be found <a href="https://www.dontblamethedata.com/blog/extract-data-using-worldfootballr/">here</a>.</p><h4>Resources</h4><ul><li><a href="https://jaseziv.github.io/worldfootballR/">worldfootballR</a></li><li><a href="https://fbref.com/en/">FBref</a></li><li><a href="https://understat.com/">Understat</a></li><li><a href="https://www.transfermarkt.com/">Transfermarkt</a></li><li><a href="https://www.tidyverse.org/">tidyverse</a> — a collection of very valuable R packages for manipulating, analysing and visualising data</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bee36287ad37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/extract-football-soccer-data-in-r-with-zero-r-experience-bee36287ad37">EXTRACT FOOTBALL (SOCCER) DATA IN R WITH ZERO R EXPERIENCE</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Historically bad Melbourne Victory as told by Elo]]></title>
            <link>https://medium.com/@jaseziv83/historically-bad-melbourne-victory-as-told-by-elo-5bd4fd01ce28?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bd4fd01ce28</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[elo]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[soccer-analytics]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 20:32:13 GMT</pubDate>
            <atom:updated>2021-04-19T20:32:13.886Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/862/0*XV8ZX0mBc6jjxwaP" /><figcaption>Source: <a href="https://www.abc.net.au/news/2021-03-07/melbourne-victory-woes-laid-bare-after-record-derby-defeat/13224654">abc.net.au</a></figcaption></figure><p>To say the A-League’s Melbourne Victory in Australia are having a bad season is a gross understatement. Football Twitter, whatever publication wishes to cover football (soccer) in Australia (funnily enough more coverage when things are bad), almost anyone with an opinion, all are in agreement that this is a horror show this season — from the board all the way down. This culminated in the recent 7–0 drubbing at the hands of cross-town (and City Group cashed up) rival Melbourne City and subsequent sacking of coach and former player Grant Brebner. Add to this a 6–0 loss to City just a few weeks back and it didn’t take a genius to know Brebner was in trouble.</p><p>Just how bad is this season? Well that’s somewhat of a subjective statement - there are many ways to define measures of success (or failure in this case), including ladder position, points earned, wins, etc, but these don’t give us the whole picture.</p><h3>Hello Elo</h3><p>The Elo rating system should not be foreign to football fans worldwide — FIFA have used Elo as a way of ranking the Women’s national teams since their introduction in 2003 and have used it as the basis of calculating Men’s rankings since 2018.</p><p>Elo was created by Arpad Elo, a Hungarian-American physics professor who wanted to design a rating system for chess players. The beauty of the method lies in its zero-sum notion, meaning that one team’s gains come at the expense of their opponent. Draws are also handled for — the lower ranked team/player would typically gain ranking points from the higher ranked opponent. Importantly for the purposes of this piece, Elo is calculated after every match played, meaning we can continue to get updated rankings. The real value in these ranking comes the ability to calculate the predicted outcome of a contest. General implementations of the model would suggest that an opponent with a rating 100 points greater than their opponent would lead to a predicted win probability of 64%.</p><h4>Applying Elo</h4><p>Now for the casual fan, the following may be a tad dreary, so feel free to skip along to the analysis, but for those wishing to understand the application of Elo here, read ahead.</p><p>Teams typically start the season with an Elo rating of 1500 and given the zero-sum condition of the calculation, it can be said that an “average” team has an Elo of 1500. So while the team starts the season (or any period of time you want to have the “starting” point for Elo) on 1500, the ranking updates after every game. If you win, you are rewarded, and even more handsomely for big wins and/or wins against higher ranked opponents.</p><p>The very popular website <a href="https://fivethirtyeight.com/">FiveThirtyEight</a> website uses Elo ratings as the basis for a number of their sports prediction models, with a lot of these models using a running Elo ranking that began its calculation for teams a long time ago, making tweaks to handle starting ratings of teams in new seasons vs carrying forward the previous season’s ranking. <em>Note, this analysis calculates a team’s Elo rating for each single season analysed, not carrying over the team’s previous season rating, nor carrying them forward to the next season.</em></p><p>There are a number of approaches to calculating Elo, however there are two main areas where decisions need to be made.</p><p>The first is deciding whether to include a “home advantage”, meaning should the home team be expected to win more often than not on average… The answer to this is yes in my opinion. What this rating should be is oft debated by analysts all over the globe, but I have settled for a 40 point advantage for the home team.</p><p>The other factor for consideration is how much weight to place on the margin of victory. It has been argued that once a team has a big win, say by over a three goal margin, then the additional rating points you earn shouldn’t have a perfectly linear relationship with ranking points earned. Smoothing has been applied to the goal index with the following formula for wins of a margin of three goals or more:</p><pre>G = (11+N)/8<br>* Where N is the goal difference</pre><h3>Worst A-League Season Elo Rankings</h3><p>Ok, enough of the very high level math and theory. Let’s get in to seeing just how bad things are looking for the Victory…</p><p>Data has been extracted from fbref.com using the <a href="http://jaseziv.github.io/worldfootballr/">worldfootballR</a> R package. Data contains all A-League regular season matches from the 2013/14 season up to and including matches up to the 18th April, 2021.</p><p>It can be seen below that when it comes to bad seasons, Central Coast have a monopoly, owning the three worst Elo seasons, with the three-win 2015/16 side the worst of the bunch (they also only won three matches in the 18/19 season).</p><p>The Victory’s position on this table could be seen through either a half-full, or half-empty glass. Currently, they’re on track for the eighth worst season, however with eleven games yet to play, anything could happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_yPaFzDjx6a8ETm5kRIHg.png" /><figcaption>Melbourne Victory could yet be the worst ever team… Data captured using worldfootballR from fbref.com.</figcaption></figure><h3>Look Away Victory Fans</h3><p>If we take the glass half-empty view of the current season, the Victory have the second worst record after 16 games played with only the 2018/19 Mariners worse at this stage of the season. More worryingly, that season’s Mariners were able to pull themselves together to “recover” somewhat. If we take them out of the equation, the Victory have the worst record after 16 games and are 10 points lower than the worst every 2015/16 Mariners.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQojePFfSJDKawQoebIhwg.png" /><figcaption>This could spell trouble for the Victory… Data captured using worldfootballR from fbref.com.</figcaption></figure><h3>Fingers Crossed</h3><p>Hopefully for the Victory fans interim Head Coach Steve Kean can turn this woefully dreadful season around and salvage some pride over the remainder of the season. If not, history beckons.</p><p>This full post and code originally appeared on Don’t Blame The Data <a href="https://www.dontblamethedata.com/blog/historically-bad-melbourne-victory/">here</a>. Check it out if you want to see the code to reproduce the analysis. And remember to check out <a href="https://github.com/JaseZiv/worldfootballR">worldfootballR</a> if you want easy access to football(soccer) data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bd4fd01ce28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysing Passing in the Top Five European Domestic Leagues]]></title>
            <link>https://medium.com/data-science/analysing-passing-in-the-top-five-european-domestic-leagues-907f18231a20?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/907f18231a20</guid>
            <category><![CDATA[soccer-analytics]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[r-programming]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Sun, 31 Jan 2021 04:16:54 GMT</pubDate>
            <atom:updated>2021-01-31T04:16:54.458Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using the worldfootballR R package to analyse different passing behaviour in the top five domestic European Soccer leagues</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1U1fg9pObJbhJzuolRDZBQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@echaparro?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Edgar Chaparro</a> on <a href="https://unsplash.com/s/photos/soccer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Football (soccer) as often referred to as “the beautiful game”, is made all the more beautiful because of the intricacies of playing styles between different nations.</p><p>This piece aims to begin exploring some of these differences in the playing style of the top five European domestic leagues. For reference, when we refer to the top five, we’re referring to the following domestic leagues:</p><ul><li>Fußball-Bundesliga (Germany)</li><li>La Liga (Spain)</li><li>Ligue 1 (France)</li><li>Premier League (England)</li><li>Serie A (Italy)</li></ul><p>Specifically, this piece will focus on passing and analyse whether there are any differences in how teams in these leagues move the ball around the pitch.</p><p>Data for this piece comes from <a href="https://fbref.com/en/">fbref</a>, from the worldfootballR R package. This package can currently be installed from GitHub, with the repository <a href="https://github.com/JaseZiv/worldfootballR">here</a>.</p><h3>Attempting and Completing Passes</h3><p>First, we turn our attention to the leagues that attempt the most number of passes per 90 minutes of football.</p><p>Other than when Serie A had the highest passing attempts in the 2017–18 season, the German Bundesliga has reigned supreme when it comes to the number of passes attempted, with the current season the highest number of passes attempted (514).</p><p>Teams in France’s League 1 have continued to increase their passing attempts, overtaking The EPL and Serie A for the first time this season. Interestingly, in the 2017–18 season, League 1 and La Liga were equally the lowest passing leagues, and in three short years, the respective leagues have diverged in their passing attempts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VxrN-H42Y0y3uT7K.png" /></figure><p>Drilling down further and analysing individual teams within league seasons, it can be seen below that while the Bundesliga and EPL have some of the highest passing rates, they also have some of the highest variability, with the discrepancy being greatest between the highest and lowest passing teams within each league (although La Liga has the greatest variability of all).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fuFlGtAxveApJ3M8.png" /></figure><h3>Completion Rates</h3><p>Attempting passes is no doubt important to the game, but one could argue that completing those passes is more vital.</p><p>The boxplot below shows that generally speaking, teams in each of the leagues have increased their passing completion rates over the last four years (although the Bundesliga has somewhat plateaued).</p><p>Serie A teams have consistently had higher passing accuracy than teams in the other leagues, while La Liga has had the greatest completion rate variability of all the leagues — no doubt driven by the vast differences in quality between The Madrid teams and Barca, and the rest of the league.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rlVCPtGye2cfrAPX.png" /></figure><h3>Why does this matter?</h3><p>It can be seen below that teams that attempt more passes during a season tend to accumulate more league points, with a fairly strong positive correlation between the variables. The below data does not include the current season.</p><p>This holds more true for the EPL correlation of (0.829), Bundesliga (0.805) and Serie A (0.796), while the relationship is not as strong for La Liga and Ligue 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ipU7hQ24D3_9JjGo.png" /></figure><h3>Which Direction are the Passes Heading?</h3><p>So, we have seen that there are some differences between the leagues with regards to how often, and how successfully they move the ball to teammates. What about the direction that the ball travels in when their making passes?</p><p>For this, we can look at the progressive distances (in yards) of completed passes and compare this to the total yardage of all passes completed. Progressive yards are defined by fbref as:</p><blockquote>Total distance, in yards, that completed passes have travelled towards the opponent’s goal. Note: Passes away from opponent’s goal are counted as zero progressive yards</blockquote><p>The resulting figure indicates how many <em>progressive</em> yards a ball has travelled per yard of completed passes. A higher number indicates a team moves the ball more directly, while a lower figure indicates a greater share of a teams completed passing yards are not as direct to the opponent’s goal.</p><h3>Leagues Passing Less Directly?</h3><p>Leagues have progressively been moving away from more direct passing routes, with the EPL leading the way (0.319 progressive yards per total yards completed), while La Liga are the least direct with 0.339 progressive yards per total yards completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*22OjW1__3TM6JBKC.png" /></figure><p>Like with the correlation between league points and passes attempted, there also appears to be a relationship between <em>directness</em> and league points, however the relationship is negative, meaning that teams that are less direct with their passing also tend to earn more league points at the end of a season.</p><p>Additionally, this relationship isn’t as strong for all leagues. The relationship is fairly strong for the EPL (correlation of -0.711) and Serie A (-0.706), while the weakest in La Liga (-0.402).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0CYkCcHFiP7Jp8Lu.png" /></figure><p>We can also look at the individual team seasons that have been the most and least direct passing seasons.</p><p>The 20 least direct teams over the last four seasons are listed below, with the top of the list dominated by EPL clubs, specifically Man City, which hold four of the top six places.</p><p>Interesting to note, it appears that teams on this list have also finished nearer the top of the table RK that the bottom of the table, reinforcing our correlation above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_97RS5kij6ND-IlGvgY0Ow.png" /></figure><p>Similarly, we can also look at the 20 most direct passing teams of the last four seasons.</p><p>The finishing positions of these clubs looks to be somewhat closer to the bottom of their league tables (Getafe in La Liga 2018–19 and RP Leipzig in Bundesliga 2018–19 the notable exceptions).</p><p>Cardiff City’s lone Premier League season was the most direct passing season, with 0.464 progressive yards per passing yards completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-M0n1MMUW6gA7qkEU8pz_g.png" /></figure><p>While not a considerable difference in the passing activities between the top five European domestic leagues, there are still some differences that will be interesting to see how their leagues evolve over the coming years.</p><p>As always, any feedback or opinions would be valued in the comments sections, or feel free to get in touch on Twitter [@jaseziv](<a href="https://twitter.com/jaseziv">https://twitter.com/jaseziv</a>).</p><p>This full post originally appeared on dontblamethedata.com. The code for the article can be found <a href="https://github.com/JaseZiv/DontBlameTheData/blob/master/content/post/top_five_euro_league_passing.Rmd">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=907f18231a20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analysing-passing-in-the-top-five-european-domestic-leagues-907f18231a20">Analysing Passing in the Top Five European Domestic Leagues</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Expected Goals and Liverpool — An Intro to worldfootballR]]></title>
            <link>https://medium.com/analytics-vidhya/expected-goals-and-liverpool-an-intro-to-worldfootballr-da1f02c17622?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/da1f02c17622</guid>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[premier-league]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[worldfootballr]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Fri, 22 Jan 2021 15:42:25 GMT</pubDate>
            <atom:updated>2021-01-22T15:42:25.922Z</atom:updated>
            <content:encoded><![CDATA[<h3>Expected Goals and Liverpool — An Intro to worldfootballR</h3><h4>This post continues a series of posts that aims to showcase the new <a href="https://github.com/JaseZiv/worldfootballR">worldfootballR</a> R package for extracting world football (soccer) data from popular data site fbref.com.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*vlTLp4tqdIRyI0WeoW117g.png" /><figcaption>worldfootballR R package</figcaption></figure><p>This post will aim to analyse expected goals and actual goals, primarily focusing on Liverpool.</p><p>This season sees Liverpool in a struggle with a few teams in the hope of going back-to-back. I want to see if there is a difference in how the Reds are scoring this season as opposed to the last few seasons.</p><h3>Extract Data Using worldfootballR</h3><p>As always, we first start by installing the package if you haven’t already done so and loading any libraries necessary for the analysis.</p><pre>devtools::install_github(&quot;JaseZiv/worldfootballR&quot;)<br><strong>library</strong>(tidyverse)<br><strong>library</strong>(worldfootballR)</pre><p>To get match results, including goals and xG data, we can use the function get_match_results(). Because we want it for the Men’s Premier League, we pass the following values to the function’s arguments:</p><pre>xg_data &lt;- get_match_results(country = &quot;ENG&quot;, gender = &quot;M&quot;, season_end_year = c(2018:2021))</pre><p>We can also see what the data looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KFCgEfm50VeDt-Ki1_QSGw.png" /><figcaption>Output of get_match_results()</figcaption></figure><p>To get season league tables (which contains goals scored for and against, and expected goals), we can use the get_season_team_stats() functions, and pass the following values to the function arguments:</p><pre>end_season_summary &lt;- get_season_team_stats(country = &quot;ENG&quot;, gender = &quot;M&quot;, season_end_year = c(2018:2021), stat_type = &quot;league_table&quot;)</pre><p>And inspect the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkISv8FKT0-hA9YiftrY-g.png" /><figcaption>Output of get_season_team_stats()</figcaption></figure><h3>The Analysis</h3><p>So what are expected goals? From <a href="https://www.optasports.com/services/analytics/advanced-metrics/">Opta Sports</a>;</p><blockquote>EXPECTED GOALS (XG) MEASURES THE QUALITY OF A SHOT BASED ON SEVERAL VARIABLES SUCH AS ASSIST TYPE, SHOT ANGLE AND DISTANCE FROM GOAL, WHETHER IT WAS A HEADED SHOT AND WHETHER IT WAS DEFINED AS A BIG CHANCE.</blockquote><blockquote>ADDING UP A PLAYER OR TEAM’S EXPECTED GOALS CAN GIVE US AN INDICATION OF HOW MANY GOALS A PLAYER OR TEAM SHOULD HAVE SCORED ON AVERAGE, GIVEN THE SHOTS THEY HAVE TAKEN.</blockquote><p>Why are we focusing on expected goals in this post?</p><p>Well it can be seen below that for the 2017/18 to 2019/20 seasons, the more teams scored above their expected goals tally, the more points they tended to finish the season on, indicating a fairly strong positive relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2biolhArgct6qYrp.png" /></figure><h3>How are the Reds tracking?</h3><p>Which leads us to Liverpool. Below we can chart Liverpool’s cumulative progress after each match played for the last four seasons and can see that in each of the three seasons before the current season, the Reds outperformed their expected goals tally, with last season’s championship team exceeding their expected goals total for the whole season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qiDOibWbjfU7Fkjk.png" /></figure><p>This current season, other than the 7–0 drubbing (xG of 2.2) against Crystal Palace at Selhurst Park, Liverpool’s goals tally has closely tracked the expected goals. The Reds marksmen just aren’t able to convert those guilt edge chances this season, which might explain the struggle at the top.</p><h3>Wrap Up</h3><p>This was the second post in a series of analyses that will make use of various data extraction functions in the worldfootballR package.</p><p>Here we saw that Liverpool appeared to have regressed to the mean somewhat and are struggling to score much higher than they’re expected, based on their shots taken. This is in contrast to the previous three seasons, where the goals scored outpaced the expected goals.</p><p>As always, any questions/comments about the piece or the R package, feel free to reach out through the regular channels.</p><p>The full version of this post appears <a href="https://www.dontblamethedata.com/blog/liverpool-expected-goals-worldfootballr/">here</a> on dontblamethedata.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da1f02c17622" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/expected-goals-and-liverpool-an-intro-to-worldfootballr-da1f02c17622">Expected Goals and Liverpool — An Intro to worldfootballR</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to the worldfootballR R Package]]></title>
            <link>https://medium.com/data-science/an-introduction-to-the-worldfootballr-r-package-8932b192065d?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8932b192065d</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[soccer]]></category>
            <category><![CDATA[football]]></category>
            <category><![CDATA[rstats]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 13:11:36 GMT</pubDate>
            <atom:updated>2021-01-13T13:11:36.984Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using the worldfootballR R package to extract Premier League substitution data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXwuL9WEo_UcAwW91039zQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@viennachanges?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Vienna Reyes</a> on <a href="https://unsplash.com/s/photos/soccer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>With the creation of the <a href="https://github.com/JaseZiv/worldfootballR">worldfootballR</a> R package; a new R package to aid in the extraction of Football (Soccer) data from fbref.com, I will be trying to highlight ways the package can be used.</p><p>The package released is currently a development version and only available on GitHub, but will be made available on CRAN once user testing has been completed.</p><p>In this piece, I will analyse Liverpool’s use of substitutions in the English Premier League (EPL) over the last three seasons to understand whether there has been a change in behaviour in the current season, and whether there was a change around the COVID outbreak and subsequent pause taken during the 2019–2020 season.</p><p>Data was extracted for the last two and a half seasons, up to the end of Matchweek 17 in the 2020/21 EPL season.</p><h3>Extract Data Using worldfootballR</h3><p>The package has a function called get_match_summary which extracts match summary (goals, subs, red/yellow cards) data for a match URLs (along with other metadata regarding the matches.</p><p>To get the match URLs, another function — get_match_urls - can be used, which accepts a country code (“ENG” for England), gender and the year the season(s) ended, and returns a vector of match URLs.</p><pre>devtools::install_github(&quot;JaseZiv/worldfootballR&quot;)<br>library(worldfootballR)</pre><pre>match_urls &lt;- get_match_urls(country = &quot;ENG&quot;, gender = &quot;M&quot;, season_end_year = c(2019:2021))</pre><pre>match_summaries &lt;- get_match_summary(match_url = match_urls)</pre><p>This is what the output of the function looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GnBcO2Rk3KncppmskYgcVA.png" /><figcaption>Lots of event data to analyse. Source: worldfootballR</figcaption></figure><h3>Teams making their first sub</h3><p>Now that the package instructions are out of the way, we turn our attention to the analysis of substitution data in the Premier League, and specifically ask the question, has there been a change in Liverpool’s substitution behaviour.</p><p>The analysis was loosely inspired by a piece done by FiveThirtyEight on <a href="https://fivethirtyeight.com/features/bundesliga-teams-could-be-using-more-substitutions-but-they-arent/">Bundesliga team susbstitutions before and after the league restart</a>.</p><h3>Using the data to explain Liverpool’s substitutions</h3><p>In all seasons analysed, Liverpool make slightly more subs in a season compared to the league average, with the average number of subs in a season peaking after the restart last season to 4.67 subs per game. The current season has seen this revert to pre-COVID levels.</p><p>Interestingly, while Liverpool’s first sub typically comes slightly later than the league average (56 minutes compared to 52 minutes), the Reds have stuck close to their first sub time after the season restart, while the league average has reverted back to pre-COVID levels, reversing the league-wide trend.</p><h3>More in the first hour now?</h3><p>The Reds made 12% of their substitutions in the first hour during the combined 2019–20 season (double that of their 2018–19 season), and this season has seen this percentage again double through the first 16 games. The league however has experienced a direct reversal of this trend, spiking to 24% during the restarted part of the 19/20 season, and settling back down at the league average for the previous seasons.</p><p>It will be interesting to see how this trend continues throughout the latter parts of this season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7jhi5FlH46wDVdF7RHcdg.png" /><figcaption>Data sourced using worldfootballR. Visualisation made using gt</figcaption></figure><h3>Wrap Up</h3><p>This was the first in a series of analyses that will make use of various data extraction functions in the worldfootballR package.</p><p>Here we saw that Liverpool are making their first substitution earlier than their previous trends, and are deviating from the league averages. I will follow this analysis to see if this trend has continued.</p><p>As always, any questions/comments about the piece or the R package, feel free to reach out through the regular channels.</p><p>GO REDS!!</p><p>The full version of this post appears <a href="https://www.dontblamethedata.com/blog/liverpools-early-substitutions-worldfootballr/">here</a> on dontblamethedata.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8932b192065d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-introduction-to-the-worldfootballr-r-package-8932b192065d">An Introduction to the worldfootballR R Package</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Agony and the Ecstasy of my First Open Source Contribution]]></title>
            <link>https://medium.com/data-science/my-first-open-source-contribution-7698950895a9?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7698950895a9</guid>
            <category><![CDATA[data-journalism]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Sat, 21 Dec 2019 14:32:23 GMT</pubDate>
            <atom:updated>2019-12-21T14:32:23.243Z</atom:updated>
            <content:encoded><![CDATA[<h4>My first open source contribution wasn’t all smooth sailing, but it sure was thoroughly rewarding</h4><p>For the last year or so, I’ve had this desire to contribute to an open source R package, but like a lot of people, I found the thought of tackling the task frightening.</p><p>While I work in a really dynamic and close team every day, and in the world of remote repositories (Git), I’ve had really limited exposure to collaborative working in these remote repositories… We tend more to work on projects largely on our own, so the concepts of pull requests (PRs), merging, forking… well, it was all a bit daunting.</p><p>The following is a glimpse at the journey, and will be explained in this post:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JFGzIujPWhxT2zYY.png" /><figcaption>The journey I took</figcaption></figure><p>What this post won’t be is an exhaustive step-by-step guide of every touch point, rather a medium-high level summary.</p><blockquote>Life is not meant to be easy, my child; but take courage: it can be delightful. — George Bernard Shaw</blockquote><p>With George’s words in mind, I thought time to push myself to jump in.</p><p>Scrolling through Twitter (as one does when nursing a newborn), I came across a tweet about a package I’ve used in a few analyses on <em>Don’t Blame the Data</em> that said that the package was now live on CRAN (a great achievement).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/0*npU5I6OzmPgKxzyc.png" /><figcaption>The tweet that started everything</figcaption></figure><p>This naturally led me to the repository on <a href="https://github.com/jimmyday12/fitzRoy">Github</a>, at which point I noticed there were open “Issues”, and one of these being for a function to create a ladder for any round.</p><h4>THE FITZROY PACKAGE</h4><p>The fitzRoy package, created by <a href="https://github.com/jimmyday12">James Day</a>, is a package designed to help R users extract and analyse Australian Football League (AFL) data for both the men’s and women’s competitions:</p><blockquote>The goal of fitzRoy is to provide a set of functions that allows for users to easily get access to AFL data from sources such as afltables.com and footywire.com. There are also tools for processing and cleaning that data.</blockquote><p>While I certainly haven’t done any extensive analysis on this point, I would guess that a large proportion of all AFL data analytics projects are completed with the help of this package.</p><h4>JUMPING RIGHT IN</h4><p>So rather than think about how good it would be to contribute, why not just get in touch with James and offer to address the open issue…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*as-hRfaUVJba_DSZ.png" /></figure><p>James was super easy to deal with, and boy was he helpful (and patient with this bumbling fool).</p><p>Then came the time to write the function. Well, sort of write the function. Fortunately, I had already written this function for a linear regression model I built for predicting the attendance of AFL home and away games <a href="https://www.dontblamethedata.com/blog/building-a-linear-regression-model-in-r-to-predict-afl-crowds/">here</a>. The function was aptly named return_ladder()… I’m a Data Scientist, not a poet.</p><p>The function was modified somewhat though to take advantage of the get_match_results() function in the package to return the starting data frame for return_ladder When writing the function, I wanted to address the requirement that the ladder is returned for any round, and for it to be returned for even earlier than the 2011 season, which another API already offered.</p><p>With that in mind, the function written takes in three arguments, all of which have the option of being blank, as well as specified:</p><ul><li>match_results_df - A data frame extracted using get_match_results(),</li><li>season_round - The round of the season the user wants the ladder for,</li><li>season - The season the ladder is required for.</li></ul><p>If these are all left blank, the function will return the ladder for every round of every season since the 1897 season.</p><p>Having the function written was one thing, it also required roxygen notes, that are returned to the user in the help docs of the function. Hadley’s <a href="http://r-pkgs.had.co.nz/man.html">R Packages</a> book does a good job explaining these.</p><h4>I’M READY TO BE A CONTRIBUTOR</h4><p>I’ve written the function, the help docs, and have checked the package using devtools::check() to make sure I haven’t made any mistakes that would cause the package to fail its build… Nothing looks alarming (well there are some warnings about <em>No visible binding for global variable</em> or something but I’m sure there’s nothing to worry about), but all looked good to me.</p><p>My local changes were committed and a PR was made, I’m ready to be a contributor, and then bam! <strong>Failed codecov</strong>!! What is that?! An email to James and I’m told it’s because there were no tests written. Ok cool, I’ll write some tests… WHAT ARE TESTS?! HOW DO I WRITE THESE TESTS?! I found this <a href="https://katherinemwood.github.io/post/testthat/">post</a> to be really helpful, as well as Hadley’s tests in the R Packages book.</p><p>Once these tests were written, I commit my changes, I’m ready to be a contributor, and then bam! Changes have been made to the master that I haven’t got in my PR… ok so I need to merge the master in my PR — easy (for some maybe, I have no idea). A bit of googling, seems pretty easy, but after typing git merge origin/master, I get this editor pop up in terminal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*2A53kEqnVGTz-1C0.png" /></figure><p>My initial thoughts? What the is this?!</p><p>Bit of googling, ok, it’s a VIM editor. Easy. Write a commit message and then all should be good… WAIT?! How do I get out of this screen?! Bit more googling and after typing :WQ, we’re ready to rock.</p><h4>WHAT. AM. I. DOING?!!</h4><p>Ok, so things were looking good. I’d committed my changes, all checks passed, happy days.</p><p>You know that line I had earlier about <em>well there are some warnings about ‘No visible binding for global variable’ or something but I’m sure there’s nothing to worry about</em>?? Well, that was nagging away at me, because as James had advised, these would cause issues when trying to include the update on CRAN. So I fixed those, and also updated the Men’s vignette. It’s at this point that I’m a bit hazy on what I did, but all I know is is that I must have spun myself into a Git web…</p><p>The Master of my forked repo was two commits behind my branch Ladder, which was five commits ahead of Origin/Master. What. Am. I. Doing?! Trial and error, error and trial.</p><h4>Success!</h4><p>After much heartache (I can’t stress enough how much heartache), eventually, I got myself all sorted, created another PR and… SUCCESS!!!</p><p>Finally, I can say I have successfully made my first contribution to an open source project. I hope that users of this package find the function useful and as with everything, can find improvements to make it even better.</p><p><strong><em>This post was originally written and posted on the Don’t Blame the Data blog </em></strong><a href="https://www.dontblamethedata.com/"><strong><em>https://www.dontblamethedata.com</em></strong></a><strong><em>. Go there to see some examples of the new function in use.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7698950895a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/my-first-open-source-contribution-7698950895a9">The Agony and the Ecstasy of my First Open Source Contribution</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When the statistic used changes the AFL age narrative for 2020]]></title>
            <link>https://medium.com/data-science/when-the-statistic-used-changes-the-afl-age-narrative-for-2020-954e938076ef?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/954e938076ef</guid>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[afl]]></category>
            <category><![CDATA[data-journalism]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Mon, 09 Dec 2019 04:51:17 GMT</pubDate>
            <atom:updated>2019-12-09T04:51:17.735Z</atom:updated>
            <content:encoded><![CDATA[<h4>Should we use the mean or median?</h4><p>After seeing my beloved Hawthorn Hawks tweet out an <a href="https://www.hawthornfc.com.au/news/2019-12-04/hawks-no-longer-the-oldest-list">article</a> on their website regarding player ages for each team, it got me riled up that the media love to cite Champion Data’s “average age” as their measure.</p><p>As can be seen with the age distribution of players as at the start of the 2020 AFL Premiership Season, the figure is going to be skewed by the older players on lists, especially with guys like Shaun Burgoyne, Kade Simpson, Gary Ablett Junior…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-sJ1SAGfHvOx4doNR1DrQ.png" /><figcaption>Would the mean or median be a better measure with this distribution?</figcaption></figure><h3>MEAN, MEDIAN…WHAT OF IT?</h3><p>The <em>mean</em>, from <a href="https://en.wikipedia.org/wiki/Mean">Wikipedia</a>;</p><blockquote>For a data set, the arithmetic mean, also called the mathematical expectation or average, is the central value of a discrete set of numbers: specifically, the sum of the values divided by the number of values.</blockquote><p>The <em>median</em>, also <a href="https://en.wikipedia.org/wiki/Median">Wikipedia</a>;</p><blockquote>The median is the value separating the higher half from the lower half of a data sample (a population or a probability distribution). For a data set, it may be thought of as the “middle” value. For example, in the data set {1, 3, 3, 6, 7, 8, 9}, the median is 6, the fourth largest, and also the fourth smallest, number in the sample. For a continuous probability distribution, the median is the value such that a number is equally likely to fall above or below it.</blockquote><p>When data is not normally distributed around the mean (as is the case here), using it to describe the measure of centre is misleading. In cases like this, the median provides a more representative statistic. With a positively skewed distribution like the one above, statistics 101 tells us that the median will be less than the average, while the opposite would hold true where the distribution was negatively skewed.</p><p>The average age of all players on team lists for the 2020 AFL premiership season is 24.17 years, while the median age is 23.6 years.</p><h3>SO WHY DO THEY STILL REPORT AVERAGES?</h3><p>I suspect media outlets report the <em>average</em> age for two reasons;</p><ol><li>Who can be bothered explaining the nuances between the mean and the median; or more likely,</li><li>The average is more prone to pull this number higher for some teams, fuelling the narrative they want to run with.</li></ol><h3>IMPLICATIONS OF THE DIFFERENT MEASURES</h3><p>When we plot both the average and median ages of each of the teams, we can see that we get some very different outcomes.</p><p>We are led to believe that the Geelong Cats have the fourth oldest list but when using the median as the statistic, they are the 11th oldest playing list, with the Kangaroos, GWS, Saints, Demons, Bombers, Bulldogs and Tigers all having older lists than them.</p><p>Some of the other implications:</p><ul><li>The Brisbane Lions have the youngest list for the 2020 season, not the Gold Coast</li><li>The Hawks are actually the fourth oldest list, with North Melbourne being third</li><li>The Demons are the 7th oldest list, not the 12th oldest</li><li>The Power are the equal third (with the Swans) youngest list, not the 10th oldest</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k3MXPuvTOQ03qe7J_H1nlQ.png" /><figcaption>Orange points represent the mean, while blue points the median</figcaption></figure><h3>WHY DOES ANY OF THIS MATTER?</h3><p>How this figure is reported probably doesn’t matter ultimately; the age of a playing list likely isn’t all that important when it comes to the ultimate glory of premiership success, as can be seen in the different age distributions of recent premiership teams in this great little analysis at <a href="https://thearcfooty.com/2016/04/24/how-old-is-the-typical-premiership-team/">TheArc</a>.</p><p>That aside, it would be nice to see this figure reported a little more accurately.</p><p>Rant. Over.</p><p><em>The data for this post and the code used to scrape it can be found </em><a href="https://github.com/JaseZiv/Dont-Blame-the-Data-Data-Files/tree/master/analysing_afl_team_age"><em>here</em></a><em>.</em></p><p><strong><em>This post was originally written and posted on the Don’t Blame the Data blog </em></strong><a href="https://www.dontblamethedata.com/"><strong><em>https://www.dontblamethedata.com</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=954e938076ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/when-the-statistic-used-changes-the-afl-age-narrative-for-2020-954e938076ef">When the statistic used changes the AFL age narrative for 2020</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Static Plots with Animations]]></title>
            <link>https://medium.com/data-science/enhancing-static-plots-with-animations-ca987801a6b0?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca987801a6b0</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Fri, 30 Aug 2019 23:41:26 GMT</pubDate>
            <atom:updated>2019-08-30T23:41:26.398Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using gganimate to spice up ggplot2 visualisations</h4><p>This post aims to introduce you to animating ggplot2 visualisations in r using the gganimate package by Thomas Lin Pedersen.</p><p>The post will visualise the theoretical winnings I would’ve had, had I followed the simple model to predict (or tip as it’s known in Australia) winners in the AFL that I explained in this <a href="https://www.dontblamethedata.com/blog/simplifying-afl-tipping/">post</a>. The data used in the analysis was collected from the AFL Tables website as part of a larger series I wrote about on AFL crowds. The wider project can be found <a href="https://github.com/JaseZiv/AFL-Crowd-Analytics">here</a>.</p><pre>library(tidyverse)<br>library(lubridate)<br>library(scales)<br>library(gganimate)</pre><pre># set themes<br>theme_set(theme_minimal() +<br> theme(axis.title.x = element_text(size = 16, hjust = 1), <br> axis.title.y = element_text(size = 16), <br> axis.text = element_text(size = 13),<br> plot.title = element_text(size = 19)))</pre><pre># create a colour pallette<br>colours &lt;- c(“#5EB296”, “#4E9EBA”, “#F29239”, “#C2CE46”, “#FF7A7F”, “#4D4D4D”)</pre><pre># — — — Read in data — — -#<br>afl &lt;- read.csv(&quot;<a href="https://raw.githubusercontent.com/JaseZiv/AFL-Crowd-Analytics/master/data/cleaned_data/afl_model_data.csv">https://raw.githubusercontent.com/JaseZiv/AFL-Crowd-Analytics/master/data/cleaned_data/afl_model_data.csv</a>&quot;, stringsAsFactors = F)</pre><pre># Data pre-processing — — — — — — — — — — — — — — — — — — — — — — — — — — -</pre><pre># make all variables character type to make splitting and string manipulation easier<br>afl &lt;- afl %&gt;% <br> mutate_if(is.factor, as.character) %&gt;% <br> mutate(team1_score = as.numeric(team1_score),<br> team2_score = as.numeric(team2_score)) %&gt;% <br> mutate(fav_team = ifelse(AwayLineClose &lt; 0, team2, team1)) %&gt;% <br> mutate(winning_team = ifelse(winner == “Home”, team1, ifelse(winner == “Away”, team2, “Draw”))) %&gt;% <br> mutate(fav_win = ifelse(fav_team == winning_team, “Yes”, “No”)) %&gt;% <br> filter(season &gt;= 2014,<br> !str_detect(round, “F”)) %&gt;%<br> mutate(tip = ifelse(abs(AwayLineClose) &lt; 3, team1, fav_team))</pre><pre># function to calculate odds<br>calculate_odds_available &lt;- function(tip, winning_team, team1, team2, HomeOddsClose, AwayOddsClose) {<br> if(tip == winning_team) {<br> odds_available &lt;- ifelse(tip == team1, HomeOddsClose, AwayOddsClose)<br> } else {<br> odds_available &lt;- 0<br> }<br>}</pre><pre># apply function and calculate returns<br>afl &lt;- afl %&gt;% <br> mutate(odds_available = mapply(calculate_odds_available, tip, winning_team, team1, team2, HomeOddsClose, AwayOddsClose),<br> game_return = odds_available * 10,<br> game_profit = game_return — 10)</pre><pre># create a df that calculates total winnings per round<br>money_per_round &lt;- afl %&gt;% <br> group_by(season, round) %&gt;% <br> summarise(total_profit = sum(game_profit)) %&gt;% ungroup()</pre><pre># add a round 0, where all seasons start at $0<br>zeros &lt;- data.frame(season = (2014:2019), round = 0, total_profit = 0)</pre><pre># join zeros df on to money_per_round<br>money_per_round &lt;- money_per_round %&gt;% rbind(zeros)</pre><pre># create a df that sums up winnings cumulatively<br>total_money &lt;- money_per_round %&gt;% <br> arrange(season, round) %&gt;% <br> group_by(season) %&gt;% <br> mutate(cumulating_winnings = cumsum(total_profit)) %&gt;% ungroup()</pre><h4>Let’s start</h4><p>Ok, so the first step I took when writing the original post was to create a ggplot2 visual to plot the winnings (or losses) I would’ve made using my simple strategy.</p><p>This was the result:</p><pre>total_money %&gt;%<br> ggplot(aes(x= round, y= cumulating_winnings, <br> group = season, colour = as.character(season))) +<br> geom_line(size = 1) +<br> geom_point(size = 2, colour = “black”) +<br> labs(x= “Round”, y= “Cumulative Wins/Losses”, colour = “Season”) +<br> scale_x_continuous(limits = c(0, 27), <br> labels = c(0, 3, 6, 9, 12, 15, 18, 21, 24), <br> breaks = c(0, 3, 6, 9, 12, 15, 18, 21, 24)) +<br> scale_colour_manual(values = colours) +<br> ggtitle(“2017 WOULD’VE BEEN A BAD YEAR”) +<br> theme(legend.position = “bottom”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dUFs66b0zGyMMZzg.png" /><figcaption>The legend is hard to follow here</figcaption></figure><p>Not bad, but certainly could be improved. To my mind, with six seasons being plotted, the legend is hard to map to the line itself. Also, other than the 2017 season, which was particularly bad, the other seasons’ variation between rounds was hard to see.</p><p>Additionally, plotting the data this way made it hard to realise without expending far too much energy focusing on where I would’ve made money, and where I would’ve lost it.</p><h4>LABELS AND ANNOTATIONS</h4><p>Yuck — you can’t just simply add the season as a label — you can’t read anything!</p><pre>total_money %&gt;%   <br>ggplot(aes(x= round, y= cumulating_winnings, <strong>group</strong> = season, colour = <strong>as</strong>.character(season))) +   <br>geom_line(size = 1) +   <br>geom_point(size = 2, colour = &quot;black&quot;) +   <br>geom_hline(yintercept = 0, linetype = 2, size = 2) + <strong># added in horizontal line at $0</strong>   <br>geom_text(aes(label = season), hjust = -1, size = 6) + <strong># season labels added</strong>   <br>scale_colour_manual(values = colours) +   <br>labs(x= &quot;Round&quot;, y= &quot;Cumulative Wins/Losses&quot;) +   scale_x_continuous(limits = c(0, 27), labels = c(1, 3, 6, 9, 12, 15, 18, 21, 24), breaks = c(1, 3, 6, 9, 12, 15, 18, 21, 24)) +   scale_y_continuous(labels = dollar) + <strong># y-axis formatted to dollar format using scales</strong>   <br>annotate(&quot;text&quot;, x= 26, y= 6, label = &quot;Break Even $&quot;, size = 6) + <strong># added text to break even line</strong>   <br>ggtitle(&quot;2017 WOULD&#39;VE BEEN A BAD YEAR&quot;) +   <br>theme(legend.position = &quot;none&quot;) <strong># turned legend off</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k6ZGvdu8UDfnD9eR.png" /><figcaption>That didn’t work!</figcaption></figure><p>Instead, only one season label was applied, and applied at the end of each line’s run. This looks much better.</p><p>As we can see, further elements were added to our static chart, including:</p><ul><li>Adding the break-even line;</li><li>Formatting the y-axis to a dollar format; and</li><li>Adding labels and removing the legend</li></ul><p>This has greatly improved the readability of the plot.</p><pre>total_money %&gt;% <br>ggplot(aes(x= round, y= cumulating_winnings, <strong>group</strong> = season, colour = <strong>as</strong>.character(season))) + <br>geom_line(size = 1) + <br>geom_point(size = 2, colour = “black”) + <br>geom_hline(yintercept = 0, linetype = 2, size = 2) + <strong># added in horizontal line at $0</strong> <br>geom_text(data = total_money %&gt;% filter(round == max(round)), aes(label = season), hjust = -0.3, size = 6) + <strong># season labels added, but only one label per season</strong> <br>scale_colour_manual(values = colours) + <br>labs(x= “Round”, y= “Cumulative Wins/Losses”) + scale_x_continuous(limits = c(0, 27), labels = c(1, 3, 6, 9, 12, 15, 18, 21, 24), breaks = c(1, 3, 6, 9, 12, 15, 18, 21, 24)) + scale_y_continuous(labels = dollar) + <strong># y-axis formatted to dollar format using scales</strong> <br>annotate(“text”, x= 26, y= 6, label = “Break Even $”, size = 6) + <strong># added text to break even line</strong> <br>ggtitle(“2017 WOULD’VE BEEN A BAD YEAR”) + <br>theme(legend.position = “none”) <strong># turned legend off</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*636bPX5mfhElW-ww.png" /><figcaption>Looking better…</figcaption></figure><h4>HELLO ANIMATIONS!</h4><p>While the above chart looks a lot better, there are no theatrics!</p><p>Enter animations from gganimate!</p><p>Using an animated plot allows us to remove even more elements. With the right mix of labelling and animation, the y-axis no longer is necessary — with each round, we can follow the winnings or losses as we go, while the break-even line gives us a reference point.</p><p>The other things that were done here include the slowing down of frames using fps (frames per second) and adjusting the range in transition_reveal() to be longer than the rounds it’s transitioning over (ie adjusting the range to c(0,25)). This allows the animation to pause after it has finished its cycle.</p><p>Finally, to increase the size of the output, adjust the width and height arguments to your liking.</p><pre>total_money_anim &lt;-  total_money %&gt;%   <br>ggplot(aes(x= round, y= cumulating_winnings, <strong>group</strong> = season, colour = <strong>as</strong>.character(season))) +   <br>geom_line(size = 2) +   <br>geom_point(size = 3, colour = &quot;black&quot;) +   <br>geom_hline(yintercept = 0, linetype = 2, size = 2) +   geom_text(aes(label = paste0(season, &quot;: &quot;, dollar(cumulating_winnings))), hjust = -0.3, size = 6) +   scale_colour_manual(values = colours) +   <br>labs(x= &quot;Round&quot;, y= &quot;Cumulative Wins/Losses&quot;) +   scale_x_continuous(limits = c(0, 27), labels = c(1, 3, 6, 9, 12, 15, 18, 21, 24), breaks = c(1, 3, 6, 9, 12, 15, 18, 21, 24)) +   scale_y_continuous(labels = dollar) +   <br>annotate(&quot;text&quot;, x= 26, y= 6, label = &quot;Break Even $&quot;, size = 6) +   ggtitle(&quot;2017 WOULD&#39;VE BEEN A BAD YEAR&quot;) +   <br>theme(legend.position = &quot;none&quot;, axis.text.y = element_blank(),          axis.title.y = element_blank(), panel.grid.major.y = element_blank(), panel.grid.minor.y = element_blank()) +   transition_reveal(round, range = c(0, 25)) <br></pre><pre>animate(total_money_anim, width = 900, height = 900, fps = 5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*BQ2T1a1cUTk3j4XKtKAbtQ.gif" /><figcaption>Animations make things so much better</figcaption></figure><p>Hope this has given you some inspiration to go out and start producing some animated visualisations of your own.</p><p>Let me know in the comments if you have any questions or suggestions.</p><p><strong><em>This post was originally written and posted on the Don’t Blame the Data blog </em></strong><a href="https://www.dontblamethedata.com/"><strong><em>https://www.dontblamethedata.com</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca987801a6b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/enhancing-static-plots-with-animations-ca987801a6b0">Enhancing Static Plots with Animations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysing Team USA vs Australian Basketball Tweets]]></title>
            <link>https://medium.com/data-science/analysing-team-usa-vs-australian-basketball-tweets-using-rtweet-in-r-465b62a29a5d?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/465b62a29a5d</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[twitter-analytics]]></category>
            <category><![CDATA[basketball]]></category>
            <category><![CDATA[text-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Thu, 29 Aug 2019 11:52:19 GMT</pubDate>
            <atom:updated>2019-08-29T12:12:21.858Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using rtweet and the tidyverse in R</h4><p>On the 24th of August, 2019, the Australian Men’s Basketball team, the Boomers, created history when they were able to enact their own David and Goliath moment, taking down Team USA for the first time ever. More impressively was the fact that the US hadn’t lost a game in 13 years. It was a marvellous moment in Australian basketball history, and while the US fielded what is widely considered one of their weaker teams, it was a moment for us to all savour. The fact that we were able to bring the best country in the world to play some exhibition games should have been celebrated, regardless of the result.</p><p>That was not to be the case…</p><p>After the historic first men’s basketball game at Marvel Stadium, the narrative from the media appeared to be full of negativity, predominantly focusing on the expensive seating with poor views for fans.</p><p>I suspect that this happens everywhere, but we Aussies seem to let our opinions swing more than most — we’re more than happy to rag on individuals or teams when things aren’t tracking so well, but flip pretty quickly when the are (Nick Kyrgios says hi).</p><p>This analysis will look at the Twitter activity around the time of both games in an attempt to prove or disprove the narrative change.</p><p>To collect tweet data, the rtweet and ROAuth packages were used. Tweets between the 22nd August and 2pm on the 25th of August were collected and analysed in the blow analysis. Tweets using the official hashtag - #BoomersUSA are referencing @BasketballAus were scraped.</p><p>The full code base and data for this project can be found <a href="https://github.com/JaseZiv/Boomers-vs-USA-Tweets">here</a></p><p>If the tweet was recorded after 2019-08-24 04:00:00 UTC (tip-off time for game 2), it was classed as a tweet in Game2 Starts. This allows us to compare tweets leading in to the historic second game with those that occurred during and after the magical night.</p><h3>TWEET ANALYSIS</h3><p>Looking at all tweets since game 1, we can see that game 2 had more tweets per hour. No doubt the shock result played a massive part in this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1lweaXx-QhpmJccs.png" /><figcaption>Tweets per hour peaked during the historic game 2</figcaption></figure><p>With 4019 ‘favourites’, the following tweet from NBATV was the most favoured tweet:</p><blockquote>It feels awesome. … I hope we can all build on this.” @Patty_Mills describes his emotions after leading @BasketballAus to its first ever win over the U.S.</blockquote><p>The most retweeted tweet from the period analysed came from NBL and had 833 retweets. The tweet was:</p><blockquote>Patty Mills in the 4th quarter is a piece of art 🔥 #BoomersUSA @SBSVICELAND@SBSOnDemand</blockquote><h3>TWEET WORDS USED</h3><p>Before we can measure the sentiments of tweets, the tweet strings need to be split into ‘tokens’ (or individual words).</p><p>Once these tokens have been un-nested (split out), we can plot the most frequently used words. Importantly, stop-words and other words we don’t want in our analysis have been omitted. Stop-words include “and”, “the”, “a”, etc — words that don’t add a lot to a sentiment analysis. Additionally, “BoomersUSA” was removed, as this was the hashtag for the game and was mentioned in almost all tweets.</p><p>The 20 most frequently used words for tweets that occurred before and after game 2 are plotted below.</p><p>As expected, “seats”, “plastic”, “seating” were words that frequently appeared in tweets before Game 2, where only “seats” appeared in tweets after Game 2 started. For tweets after game 2 started, “history”, “Patty Mills”, “awesome” and “love” all appeared in tweets frequently — very soft and mushy hey?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YHhXOz_CTUWrRlfa.png" /><figcaption>Not so many seating issues for tweets after game 2 started (left panel)</figcaption></figure><p>The plot below allows us to get an even better look at the differences between the words used before and after Game 2. Words to the bottom right of the diagonal line running through the plot indicate words more frequently used in tweets before game two, while words above the line were more frequently used during and after Game 2.</p><p>Looks like we were a happy bunch finally…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LcfiwwiUuSlxg8jW.png" /></figure><h3>TWEET SENTIMENT ANALYSIS</h3><p>Once the tokens have been separated, a sentiment score can be calculated.</p><p>The method that will be used is the common lexicon for sentiment analysis created by Finn Årup Nielsen (<a href="http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010">http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010</a>), called the <em>AFINN</em> lexicon. Words that are more positive (say, “awesome” for example) are given positive score further from zero than more negative words (like “devastated”), which are given socres further below zero.</p><p>To get a feel for the power of sentiment analysis, the following tweet was the most positive tweet, with a positivity index of 17:</p><blockquote>Totally thrilled with the Boomers win today, and Patty Mills was just brilliant when it counted. Really wonderful team effort. Bring on the World Cup! #BoomersUSA</blockquote><p>At the (complete) opposite end of the spectrum, the following tweet was the most negative, with a score of -18 (sorry for the profanities, I’ve done my best to clean them out):</p><blockquote>Who the f!&amp;% hired these people to organise this event, what a f!&amp;%ing bs stich up. $ chairs from Bunnings does that come with a forking snag c!&amp;%. Like WTF surely the company who funding to set up the stadium is not that broke.#BoomersUSA</blockquote><p>No surprises, the most positive tweet was after we won, the most negative after Game 1.</p><p>Plotting the distribution of sentiment scores for each tweet, we can clearly see that the tweets after game 2 started became considerably more positive — the median positivity score (the ratio of positive to negative words) for these tweets was 0.95, over double the 0.43 median for tweets prior to game 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TGlF3QHBSzVNbYDb.png" /></figure><p>As I suspected, we became much happier after our historic win… almost to the point where we’d forgotten about the seating “debacle”, even though the second game was played at the exact venue, with the exact seating arrangement… very strange.</p><p>I might put this theory of us Aussies to the ultimate test one day and see how we respond to Kyrgios’ success… if he ever tastes the ultimate success!</p><p><strong><em>This post was originally written and posted on the Don’t Blame the Data blog </em></strong><a href="https://www.dontblamethedata.com/"><strong><em>https://www.dontblamethedata.com</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=465b62a29a5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analysing-team-usa-vs-australian-basketball-tweets-using-rtweet-in-r-465b62a29a5d">Analysing Team USA vs Australian Basketball Tweets</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Linear Regression Model in R to Predict AFL Crowds]]></title>
            <link>https://medium.com/data-science/building-an-linear-regression-model-in-r-to-predict-afl-crowds-735b16a1f7c6?source=rss-b3835330254b------2</link>
            <guid isPermaLink="false">https://medium.com/p/735b16a1f7c6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <dc:creator><![CDATA[Jason Zivkovic]]></dc:creator>
            <pubDate>Mon, 29 Jul 2019 09:45:52 GMT</pubDate>
            <atom:updated>2019-08-29T11:03:20.319Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>There is a lot of talk about crowd behaviour and crowd issues with the modern day AFL. I personally feel the crowd issues are a case of <em>business-as-usual</em>; I’ve been going to AFL games on-and-off for close to three decades and the crowd behaviour is no different today as it was 20 years ago… just now everyone with a phone and Twitter account is a journo.</p><p>Anyway, that’s my two cents worth. This analysis is not on crowd behaviour. This post is primarily concerned with building a model to predict AFL crowds during the season proper. While data was collected since the beginning of the 2000 season, the model will be built on training data from the beginning of the 2013 season to the end of the 2018 season (Finals series excluded), with the trained model then used to predict the attendance figures of AFL games during the 2019 season up until round 18 which concluded on Sunday 21st July, 2019. The linear model will be built using R (version 3.5.3) in <a href="http://www.rstudio.com/">RStudio</a>.</p><p>This post is a follow up to my introductory analysis done on bandwagon fans. It can be found <a href="https://medium.com/@jaseziv83/using-data-to-determine-which-afl-fans-jump-on-the-bandwagon-8adbcc4160b0?sk=b4c0788b1f54aca1cfd50e2766eadd5c">here</a>.</p><p>The article was inspired by the amazing work Tony Corke does, specifically his post on <a href="http://www.matterofstats.com/mafl-stats-journal/2016/3/16/the-predic">The Predictability of AFL Crowds</a>.</p><p>All code for this project can be found on GitHub <a href="https://github.com/JaseZiv/AFL-Crowd-Analytics">here</a></p><h3>The Data</h3><p>Three sources of data were used in the project:</p><ol><li><a href="https://afltables.com/">AFL Tables</a> was used as the primary source of information. This is no doubt the go to for anyone looking at doing AFL analysis</li><li><a href="http://www.aussportsbetting.com/">Australian Sports Betting</a> was used to get betting data for matches from the 2013 season</li><li><a href="http://www.bom.gov.au/climate/data/">Australian Bureau of Meteorology</a> was used for climate data (rain and temperature). Part of the scraper for BOM data was taken from James Day’s <a href="https://github.com/jimmyday12/fitzRoy">fitzRoy</a> R package. Big thanks to him. The rain and temperature data had some missing observations. Missing rain data was filled by averaging out the next taken reading over the missed data points before it. Missing temperature was filled taking the average temperature for that month.</li></ol><h3>Feature Engineering</h3><p>The data contained at AFL Tables is a rich source of information, but some steps were needed to glean some additional information from the data.</p><p>The original three data sources combined contained approximately 35 variables for potential use. Feature engineering (the process of creating additional explanatory variables to use in a predictive model) was then conducted and resulted in ~95 features in total to see if they would ultimately be added to the final model to be used.</p><h3>Features Created</h3><p>Based on some feedback from the Bandwagon article and other things that I wanted to explore, some new features were created to see if they had any predictive power.</p><p>Take the game start time as an example — having the time is great, but a regression model may find it better to categorise the time to a period of the day to reduce some of the noise. So, with that in mind, a feature (time_period) was created - if the game started by 5pm, I have classified it as an afternoon game. Games started between 5pm-7pm were classified as evening games, while games later than this were night games.</p><p>Additionally, the day of the week and the time_period were joined into one variable game_time to use in the model. For example, if it was a Saturday game starting at 5:30pm, it was classified as “Sat-Evening”.</p><p>Venues that were used infrequently, or more accurately, weren’t the teams primary “home” stadium were labelled as “Other”. There were 94 games played at these venues, and this includes at stadiums like <em>Eureka Stadium</em>, <em>Cazaly’s Stadium</em>, <em>Traeger Park</em>, and international venues <em>Wellington</em> in NZ and <em>Jiangwan Stadium</em> in China.</p><p>Another feature was created to determine rivalry games — the thinking being that when two fierce rivals face off against each other, the crowds will come. To be a rivalry game, a few different options existed. The first contains the four oldest traditional rivals, all from Melbourne — Collingwood, Richmond, Essendon and Carlton (hold your Carlton opinions, I know they’ve been horrible but their fans still turn up to these rivalry games). The Derby in SA (<em>The Showdown</em> between Adelaide and Port Adelaide) and in WA (<em>Western Derby</em> between West Coast and Fremantle) are also classed as rivalry games.</p><p>After hearing a key AFL staffer talk a few years back talk about betting markets being a strong predictor of attendances, I collected betting data to test this assumption. A number of features were created in addition to the raw odds and line data scraped. These included calculating the differences between the odds of the home and away team and also calculating the change in betting throughout the week.</p><p>Features were also created around the average experience of both the home and away teams, as were milestone games (where players of either team are playing in their 100, 200, 250 or 300th games).</p><p>Each team’s last game result was calculated, as was the number of wins each team had in their previous three games. Additionally, whether the team played finals the previous season was identified.</p><p>A ladder at each round was created and ladder position, percentage and score for and against was explored as possible features for the end model.</p><p>A feature was also created to identify games played by teams from the same state or not.</p><h3>Exploratory Data Analysis</h3><p>Before we start building the model, we need to understand both the response and predictor variables. The most effective way to do this is through visualisations, and what package is better to use than ggplot2!</p><p>This section will contain some Exploratory Data Analysis (more commonly known as ‘EDA’), which is a natural first step in any data analysis project. It is an extremely important step and allows the analyst to inspect the data — the shape the target and predictor variables have, are there any correlations between the variables that may be included in a predictive model, are there any outliers in the data, or any missing data… you get the point, EDA is very important!</p><p>Well, there is no missing data in the dataset being used… that has all been cleaned in the pre-processing stage that can be found in <a href="https://github.com/JaseZiv/AFL-Crowd-Analytics/blob/master/src/preprocessing.R">this</a> R code.</p><h3>The Response Variable — Attendance</h3><p>Let’s look at our response variable — or the thing we’re trying to predict — attendance at AFL premiership season games.</p><p>We can see that the attendance figures are somewhat positively skewed. We’ll see if the few higher attendance games cause any issues later on. This variable may need to undergo some form of transformation, but more on that later.</p><p>The median attendance to AFL games during the regular season is just under 32,500 for the 1,340 games played since 2013.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9yddTdcFFnT18bqK4VSr6A.png" /></figure><h3>Some Key Predictors</h3><pre>data_for_model &lt;- afl_premiership_season %&gt;%<br>  select(season, attendance, team1, team2, round, venue, game_time, home_median_experience, home_mean_experience, away_median_experience, away_mean_experience, home_300, home_milestone, away_300, away_milestone, count_milestones, rainfall_clean, min_temp, max_temp, last_five_days_rain, temperature_diff, HomeOddsOpen, AwayOddsOpen, HomeLineOpen, AwayLineOpen, odds_diff, rivalry_game, HomeTeamFav, team1_last_result, team2_last_result, split_round, home_team_finals_last_season, away_team_finals_last_season, season_stage, temperature, IsHomeMilestone, IsAwayMilestone, count_milestones, last_results, home_team_state, away_team_state, is_same_state, temperature_diff, home_odds_change, away_odds_change, home_line_change, away_line_change, second_home_game, home_season_points, home_score_for, home_percentage, home_ladder_pos, away_wins_last_three, away_season_points, away_score_for, away_percentage, away_ladder_pos, teams_in_eight, finals_last_season, home_wins_last_three, away_wins_last_three) %&gt;% na.omit()</pre><p>As was evident in the first post of this series, there was noticeable variation in attendances between the teams, and also whether teams won or lost their previous game. Additionally, the first post showed that the home team’s favouritism status also seemed to have a relationship with attendance.</p><p><strong>Categorical Features:</strong></p><p>Some of the key categorical variables created that we would expect to have an impact on crowd figures are displayed below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5HKl3SkueuecoNjvQg9obQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2qXCvNBVCs25zSFSeYOQ7Q.png" /></figure><p><strong>Some observations:</strong></p><ul><li>There appears to be a relationship between the venue the game is played at and the attendance. Stadiums classified as Other include secondary home grounds and some of the ovals not used in 2019. There were 238 of the 1,340 games at these stadiums</li><li>Whether either the home or away team played finals last year seems to have a weak relationship with attendance. It may be useful to include in our model</li><li>When the game is played and attendance appear to be related. Weekday Afternoon and Friday Afternoon are high because of ANZAC day clashes between Essendon and Collingwood no doubt. Friday and Sunday night games seem to draw better crowds. Could be a useful predictor</li><li>There appears to be some variance in attendance figures between the different rounds during a season. Could be a useful predictor</li><li>Games classed as “Rivalry Games” clearly draw larger attendances than non-rivalry games. Could be a useful predictor</li><li>Whether the round was a split round (where some teams have a bye, or week off) or not doesn’t appear to be related to attendances. Might not be a useful predictor</li><li>There appears to be slightly higher crowds when the home team has a player playing a milestone (either a 100, 200, 250 or 300th game), but the relationship certainly doesn’t appear strong. Might not be a useful predictor</li><li>When the two competing teams are from the same state, crowds appear to be higher. Could be a useful predictor</li><li>To be expected, games where the home team is playing the game at their second home ground (Hawthorn and North in Tasmania, Melbourne, Western Bulldogs in Darwin, etc) draw lower crowds. This is probably more to do with lower stadium capacities, but may still be a useful feature</li></ul><p><strong>Numerical Features:</strong></p><p>To explore the relationship between the numerical features and attendance, Pearson’s correlation will be used.</p><p>Using dplyr to select just the numerical features and then calculate the correlations on the converted matrix yields the below correlations with attendance.</p><pre>numeric_features &lt;- data_for_model %&gt;% select_if(is.numeric)<br><br>numeric_features %&gt;% <br>  as.matrix() %&gt;% <br>  cor() %&gt;% .[,&quot;attendance&quot;] %&gt;% sort(decreasing = T)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qLKxbJ1LmTEo-vni_FJD2w.png" /></figure><p>While all correlations appear to be fairly weak, the following observations can be made:</p><ul><li>The average experience (home_mean_experience) for the home team, calculated as the average games played by each player at each game they play in, yields the highest Pearson correlation with attendance of 0.264</li><li>The Home team’s percentage (scores for / scores against) also have a correlation above 0.2 at 0.236</li><li>The number of wins the home team had in their last three games had a Pearson correlation of 0.175, while the correlation of away team’s average playing experience was 0.159</li><li>Strangely, the home team’s ladder position had a negative correlation, with a Pearson correlation of -0.273. The away team’s ladder position also had a negative correlation of -0.150</li><li>Betting data shows some relationships; as the HomeOddsOpen increase, the attendance tends to decrease, which is somewhat expected. The odds_diff variable (the difference between the home teams opening odds and the away teams) also has a weak negative relationship (correlation -0.168)</li><li>Weather data shows some relationships. As expected, the maximum temperature and rainfall, both on the day and the accumulation of last five day’s rainfall has a negative relationship with attendance (-0.091, -0.081 and -0.65 respectively)</li></ul><h3>Building the model</h3><p>In any project that aims to build a machine learning model to predict the outcome of some response variable, it’s important to hold out a subset of your data when training the model. These two sets are generally referred to as training and test sets. The training set is what you pass to the model to build, and then the resulting predictions are made on the data the model hasn’t “seen”, to best mimic real life examples.</p><p>In this case, the model will be trained on data from the 2013 to 2018 seasons inclusive (afl_pre_2019), with the predictions then made on 2019 data that the model hasn’t seen, afl_2019. This is done to ensure the model doesn’t “overfit” the data. Models that overfit the data generally perform worse on unseen occurrences.</p><h3>Things that were attempted but not included in the model</h3><p>The following features were thought to have some validity but when run through the model were found to have no predictive power:</p><ul><li>The number of wins the teams had in their last three games didn’t contribute</li><li>The rainfall on the day had no predictive power (however the total rainfall for the five days leading up to game day did)</li><li>Milestone games made the model perform worse</li></ul><h3>What’s missing / would be good to have</h3><p>There is some information that wasn’t available publicly (or I didn’t try to get) which may prove handy in any future modelling work. These include:</p><ul><li>Ticket pricing for games during each season to explore the effect ticket pricing has on attendance</li><li>Whether tickets were given away to fans for games and whether that had an impact on attendances</li><li>Any social media posts / newspaper text analysis regarding teams / topical matters involving players playing in games, etc</li></ul><pre>afl_pre_2019 &lt;- data_for_model %&gt;% filter(season &lt; 2019, season &gt;= 2013)<br><br>afl_2019 &lt;- data_for_model %&gt;% filter(season == 2019)</pre><h3>Baseline Model</h3><p>Before building a model to predict games, we want a baseline model to be able to compare to. The temptation would be there to use the average crowd for each game as a baseline, but the different venue capacities make this less than relevant. As a result, to calculate the baseline model, I will use the median attendance for each venue (taking the average of the venues labelled “Other” as a whole), for each home-away team combo. For example, Collingwood(H) vs Essendon(A) is different to Essendon(H) vs Collingwood(A). The median is used because of the slightly skewed nature of AFL crowds.</p><pre>median_attendances &lt;- afl_pre_2019 %&gt;% <br>  group_by(team1, team2, venue) %&gt;% <br>  summarise(baseline_attendance = median(attendance)) %&gt;% ungroup()<br><br>baseline &lt;- afl_2019 %&gt;% select(team1, team2, venue, attendance) %&gt;% <br>  left_join(median_attendances, by = c(&quot;team1&quot;, &quot;team2&quot;, &quot;venue&quot;)) %&gt;% <br>  filter(!is.na(baseline_attendance))</pre><p>Using the average attendances per ground as our baseline, an RMSE of 6,816 and a mean absolute error of 4,958 is achieved. Any model that is built from here needs to be better than this.</p><h3>Transforming Response Variable</h3><p>Because of the somewhat skewed nature of our response variable, it might be wise to undergo some sort of transformation to normalise it more. A log transformation is routinely used to “un-skew” data. Below is the result of log transforming the attendance variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EaPWh3HtivcF2C2JAr8kEw.png" /></figure><p>It looks less skewed than the non-transformed variable, however (<em>spoiler alert</em>), the models performed worse with attendance being log transformed.</p><h3>Feature Selection Using Stepwise Method</h3><p>A popular method to select which features go into a model is using stepwise regression. This can either be performed:</p><ul><li><em>forward</em>, where variables are added one by one, with only those that improve the model’s explainability (R-squared) retained,</li><li><em>backward</em>, which starts with all variables, then removes them if they don’t add value to the model’s explainability, or</li><li><em>both</em>, which is an ensemble of both <em>forward</em> and <em>backward</em> feature selection.</li></ul><p>Using stepwise in <em>both</em> directions maximised the model’s Adjusted R-squared measure at 0.87.</p><p>When performing linear regression, we want to ensure there is no collinearity between the variables. One way to asses this is to calculate the Variable Inflation Factor (VIF). This is made easy with the vif function in the car package. A VIF above 5 is considered to be too high.</p><p>While the round the game is played is just over 5 (5.7), we will keep this in, however odds_diff and home_score_for will be removed from the full model selected using stepwise and re-run.</p><pre>full_model &lt;- lm(attendance ~ ., data = afl_pre_2019)<br><br><br><em># Set seed for reproducibility</em><br>set.seed(123)<br><br>step.model &lt;- MASS::stepAIC(full_model, direction = &quot;both&quot;, <br>                      trace = FALSE)</pre><pre>car::vif(step.model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WThvmKqjvh2N28yFL28v8Q.png" /></figure><pre><em># the backward selection model include a variable with hi collinearity (VIF = 12). Will recreate model without odds_diff</em><br>full_model_clean &lt;- lm(attendance ~ team1 + team2 + round + venue + game_time + <br>    home_milestone + rainfall_clean + min_temp + max_temp + last_five_days_rain + <br>    HomeOddsOpen + AwayOddsOpen + HomeLineOpen + <br>    rivalry_game + split_round + last_results + is_same_state + <br>    home_season_points + away_season_points + <br>    away_ladder_pos + teams_in_eight + home_team_finals_last_season, <br>    data = afl_pre_2019)<br><br><br>car::vif(full_model_clean)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*90Y-Pv91IJAiJ-vkl4nN8g.png" /></figure><p>The cleaned model’s Adjusted R-squared is 0.8652, while the RMSE was 6,474 and MAE 4,855. These results better the baseline model, but only just.</p><h3>Best model</h3><p>Determining the “best” model is an interesting exercise. If the explainability powers of the model is what we’re after, then we want to maximise the Adjusted R-squared metric. This analysis is aimed at maximising the success of predicting crowds, and as such, the Root Mean Squared Error (RMSE) or Mean Absolute Error (MAE) are the metrics we’re going to focus on. Both measures can be related back to the response variable — attendance — the difference being that MAE is the average absolute error of the model (difference between prediction and actual), while the RMSE penalises the model the larger the error.</p><p>For both RMSE and MAE, the below model results in the lowest RMSE and MAE when applied to the 2019 season games.</p><pre>set.seed(123)<br><br>fit_lm &lt;- lm(attendance ~ team1 + team2 + round + venue + game_time + I(HomeLineOpen * home_ladder_pos) + rivalry_game + last_results + last_five_days_rain + max_temp + min_temp + home_mean_experience + away_mean_experience + is_same_state + home_score_for + I(home_percentage * home_ladder_pos) + I(home_wins_last_three * round) + round * finals_last_season, data = afl_pre_2019)<br><br>summary(fit_lm)$adj.r.squared</pre><p>The Adjusted R-Squared of the model is 0.858, indicating that almost 86% of the variability in AFL attendances in the training set (2013–2018) is explained by the model.</p><p><strong>Interaction Terms</strong></p><p>Features were not only used on their own in this model. Interaction terms were also employed. The following features were combined to form a feature and gave the model a boost in predictive power:</p><ul><li>I(HomeLineOpen * home_ladder_pos)</li><li>I(home_percentage * home_ladder_pos)</li><li>I(home_wins_last_three * round)</li><li>round * finals_last_season</li></ul><h3>Analysing the best model</h3><p>The first thing we want to do is ensure the residuals are normally distributed to satisfy the condition of normality for linear models.</p><p>The below histogram suggests that the errors are fairly normally distributed and centred around zero. A good sign.</p><pre>ggplot(data = data.frame(fit_lm$residuals), aes(x= fit_lm.residuals)) + <br>  geom_histogram() +<br>  ggtitle(&quot;ERRORS ARE CENTRED AROUND ZERO&quot;) +<br>  scale_x_continuous(labels = comma, name = &quot;Model Residuals&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTl8zmtj-Qez5ZAruA8rKA.png" /></figure><p>Then we can look at the coefficients of each of the predictor variables.</p><p>To make inspecting these easier, the broom package was used. Using the tidy() function, the results of the summary lm output are nicely displayed in a DF.</p><p>The coefficients for each of the predictor variables (other than home and away teams) are displayed below.</p><pre>a &lt;- broom::tidy(fit_lm) %&gt;% mutate(p.value = round(p.value, 5))<br><br>a %&gt;% filter(!str_detect(term, &quot;team1&quot;), !str_detect(term, &quot;team2&quot;)) %&gt;%  <br>  kableExtra::kable(format = &quot;html&quot;, escape = F) %&gt;%<br>  kableExtra::kable_styling(&quot;striped&quot;, full_width = F)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*7bYsNoCBvYFA7KUS9pb2LQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*MblVrV9gtJt71gMTiyvukw.png" /></figure><pre>actual_2019crowds &lt;- afl_2019$attendance<br><br><em># fit linear model </em><br>afl_2019$predicted_attendance &lt;- predict(fit_lm, afl_2019)<br><br><em># put a floor on AFL attendances - surely the AFL doesn&#39;t let the crowd fall below 6,000</em><br>afl_2019$predicted_attendance &lt;- ifelse(afl_2019$predicted_attendance &lt; 6000, 6000, afl_2019$predicted_attendance)<br><br><em># calculate the errors</em><br>error &lt;- afl_2019$predicted_attendance - afl_2019$attendance</pre><p>Then we want to analyse how well the model fit the relationship. This can be achieved by plotting the actual vs predicted values on a scatterplot.</p><p>I can be seen that the linear model does a fairly good job on average. There model did under-predict some of the larger drawing games, while also predicting an attendance of over 30,000 to the Hawthorn vs GWS game at the MCG in round 8, which only drew 14,636 fans.</p><pre>afl_2019 %&gt;% <br>  ggplot(aes(x=attendance, y= predicted_attendance)) +<br>  geom_point() +<br>  geom_abline(slope = 1, intercept=0) +<br>  ggtitle(&quot;MODEL UNDER-PREDICTED SOME\nLARGE DRAWING GAMES&quot;) +<br>  scale_x_continuous(labels = comma, name = &quot;Actual Attendance&quot;) +<br>  scale_y_continuous(labels = comma, name = &quot;Predicted Attendance&quot;) +<br>  annotate(geom = &quot;text&quot;, x=83000, y= 55000, label = &quot;Model under-predicted\non these games&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogiCJ1bcGMm068M6nZXhVw.png" /></figure><p>We can also look at how the model performed predicting crowds at each of the venues.</p><pre>afl_2019 %&gt;% <br>  mutate(error = predicted_attendance - attendance) %&gt;% <br>  ggplot(aes(x= error)) +<br>  geom_density() +<br>  geom_vline(xintercept = 0, linetype = 2) +<br>  facet_wrap(~ venue, scales = &quot;free_y&quot;, ncol = 2) +<br>  scale_x_continuous(labels = comma, name = &quot;Error&quot;) +<br>  ggtitle(&quot;Exploring Errors by Venue&quot;) +<br>  theme(axis.text.y = element_blank(), axis.title.y = element_blank())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftpv9ntuPltoyEtI429GlQ.png" /></figure><p>The model tended to over-predict attendances at the Gabba, Perth Stadium, Sydney Showgrounds and even the MCG, while it under-predicted at Carrara and “Other”.</p><h3>Overall Performance of the Model</h3><p>The final model selected achieved an RMSE of 5,953 and MAE of 4,466, meaning that on average, the model is off by just under 4,500 fans when predicting attendances. When compared to the baseline model (RMSE of 6,816 and a MAE of 4,958), this model betters the baseline by over 500 people on average, but also doesn’t have as many extreme errors.</p><h3>Conclusion</h3><p>Hopefully you’ve made it to here!</p><p>I’d love to hear from you with any feedback / corrections / etc in the comments or get in <a href="https://www.linkedin.com/in/jason-zivkovic-4a0a1926/">touch</a>.</p><p><strong><em>This post was originally written and posted on the Don’t Blame the Data blog </em></strong><a href="https://www.dontblamethedata.com/"><strong><em>https://www.dontblamethedata.com</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=735b16a1f7c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-an-linear-regression-model-in-r-to-predict-afl-crowds-735b16a1f7c6">Building a Linear Regression Model in R to Predict AFL Crowds</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>