<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Craig Franklin on Medium]]></title>
        <description><![CDATA[Stories by Craig Franklin on Medium]]></description>
        <link>https://medium.com/@craigjfranklin?source=rss-157b8fa46c00------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5tFqG8_t9SV5d1TiX3YzeA.jpeg</url>
            <title>Stories by Craig Franklin on Medium</title>
            <link>https://medium.com/@craigjfranklin?source=rss-157b8fa46c00------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:11:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@craigjfranklin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Footy Tipping: Round 2]]></title>
            <link>https://medium.com/@craigjfranklin/footy-tipping-round-2-1bad1d94a7ca?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bad1d94a7ca</guid>
            <category><![CDATA[footy]]></category>
            <category><![CDATA[afl]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Thu, 28 Mar 2019 07:19:21 GMT</pubDate>
            <atom:updated>2019-03-28T07:19:21.513Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DBwpu3VydSVVwz67" /><figcaption>Photo by <a href="https://unsplash.com/@andredoesphoto?utm_source=medium&amp;utm_medium=referral">Andre Guerra</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I was planning on writing a post before the start of the AFL season with my tips for round one, but I was so busy performing emergency surgery on my prediction data, that I wasn’t even able to get my model’s predictions until around 9:00 pm the night before the first match, so this is a story about the challenges of using other people’s data and not knowing how sports websites will change when they work off that softness that accrues over the offseason and get back into leaping-mark-over-some-poor-bloke’s-shoulder shape.</p><h3>Data woes</h3><p>The scramble to fix my data began the weekend before round one, when I tried to run my tipping script (which pulls match and player data from the fitzRoy R package, which in turn gets the data from <a href="http://www.afltables.com/">AFL Tables</a>, and scrapes betting data from <a href="http://www.footywire.com/">Footywire</a>), and I got an error for trying to merge an empty data frame on columns that it obviously didn’t have. I soon found that the <a href="https://www.afl.com.au/news/teams">news page</a> on the official AFL website, from which I get rosters for upcoming matches, had been updated such that the default page showed information for upcoming rounds rather than for last year’s Grand Final. This was as it should be, but what I hadn’t accounted for in my code was that this page didn’t have any data yet, because rosters don’t get announced until about a day before the start of the round, so my scraper was returning a blank array. I fixed this by replacing the data-less void with a row-less one, adorned with decorative column labels. I was then able to fetch all my data without error, but when I ran my tipping script again, I got a new blank-data error: I had no data for upcoming matches on which to predict.</p><p>How could this be? Sure, I still didn’t know who was going to play that week, but I knew who was playing whom and when and where and how they performed previously. That was definitely enough to make a few haphazard guesses about match outcomes. When I examined my tipping data set, however, matches from 2019 were conspicuously absent. I examined my player data set, and, though there were copious zeros, there were at least rows for 2019, round-one matches, which means that missing player data wasn’t the cause of these match rows getting dropped. Finally, I took a step back and looked at each of my data sources (player, match, and betting) in isolation and realised that AFL Tables didn’t have any rows for upcoming matches. I often encounter this truth when coding and just as often forget it: it’s your assumptions about your code (and data) that cause bugs, and the more fundamental the assumption, the more difficult the debugging. Appending upcoming-match data scraped from Footywire resolved the problem, and by that time, the rosters for Richmond and Carlton had been announced, so I was able to enter my preliminary tips for the week.</p><h3>Round 1 retro</h3><p>This proved unfortunate. It turns out that I probably would have gotten more tips by running predicted_home_wins = np.random.randint(0, 2, 9).astype(bool) than I did by spending months developing and testing a machine-learning model trained on over a century’s worth of AFL/VFL match data. My tips and predicted margins (along with the match results) were as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yZu5AeQw8JNMl9FexD_XQ.png" /></figure><p>I sure was riding high after the opening match: off by just one point, top of the office ladder, and well on my way to being a two-time Footy Tipping Office Champ. But then everything fell apart, slowly at first, a close match just barely going the other way, little more than a crack in the edifice, but soon whole chunks were tumbling down and by the time we got to Sunday, I couldn’t bear to watch the rest of the Freo/North match, turning it off midway through the second quarter, because that monument that I had build with months of sweat and computer-screen-induced eye strain had been properly demolished. It was a rough round indeed: I got three tips and was off the margin by an average of 40.11 points. On the bright side, it was a rough week for most, leaving me around the middle of the ladder, a downright respectable 10th out of 16.</p><h3>Round 2 tips</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*a2VE912pPOX9tVZU3Jxuiw.png" /></figure><p>My model tends to follow the betting odds pretty closely, and I haven’t done any analysis of its inner workings yet, so I don’t have a good understanding of how it arrives at its predictions. One thing that I’ve noticed, is that my previous model had a strong home-team bias, almost always tipping the home team if the line odds were less than a goal or so. We’re only two rounds in, but Tipresias seems to have no qualms about tipping visiting teams in close matches, picking North Melbourne, Sydney, and West Coast to win in round one, and defying the betting odds to pick Brisbane to just edge out North at the Docklands. Once I get my tipping app settled and sufficiently automated, this will be something worth investigating.</p><p>I don’t have player data for any matches except Richmond v Collingwood yet, so these are subject to change, but I suspect that North v Brisbane is the only one that’s close enough to potentially change the predicted winner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bad1d94a7ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Footy Tipping with Machine Learning: Models, Assemble!]]></title>
            <link>https://medium.com/@craigjfranklin/footy-tipping-with-machine-learning-models-assemble-5f884a7e8538?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f884a7e8538</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Thu, 31 Jan 2019 10:18:21 GMT</pubDate>
            <atom:updated>2019-01-31T10:18:21.284Z</atom:updated>
            <content:encoded><![CDATA[<h4>Because if two models are better than one, then surely a whole heap of models are better still.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QWeSAA5Ay2QfoeHD" /><figcaption>Photo by <a href="https://unsplash.com/@themightymerlin?utm_source=medium&amp;utm_medium=referral">Benny Jackson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Part five of a series of posts chronicling my attempt to develop a data model for footy tipping. You can read part four </em><a href="https://towardsdatascience.com/footy-tipping-with-machine-learning-adding-player-data-9ccdab56a4c5"><em>here</em></a><em>. Jupyter notebooks used for this post can be found </em><a href="https://github.com/cfranklin11/tipresias/tree/master/notebooks"><em>here</em></a><em>.</em></p><h3>Of encapsulation and densification</h3><p>When I started on my project to build a new footy-tipping model, one of my principle ideas was to achieve greater accuracy by building a modular ensemble model, one whose individual pieces functioned as independent models, each with its own data source, whose predictions would be combined only at the very end of the data pipeline. I was inspired to try out this structure from studying the principles and best practices of object-oriented programming. (For those who are interested, Sandi Metz’s <a href="https://www.poodr.com/"><em>Practical Object-Oriented Design</em></a> is an excellent introduction.) I figured that reducing the interdependence of different sections of a codebase for greater flexibility and openness to change is as useful to a machine learning project as it is to a web application. Another influence was the limitations of the data itself: I have match data going back to 1897, (useful) player data going back to 1965, and betting data going back to 2010. Joining all of these together results in a sparse data table, especially since there’s a negative correlation between a data column’s length and its importance to a model’s predictions. The betting odds are some of the strongest predictors of which team will win, because they include the collected wisdom of experts and fans into a few numbers, but I only have these valuable figures for 12.2% of my data rows. My thinking was that reducing the sparsity of my data by splitting it up by type (betting, player, match) would reduce the amount of noise (i.e. zeros) and enhance my model’s ability to learn the values that were present. In pursuit of this idea, I’ve created a model trained on betting data only, one trained on player data only, one trained on match data only, and now I’m ready to bring them together into an ensemble and taste of the fruit of all my endeavours.</p><h3>Absolutely average</h3><p>There are two basic methods for creating an ensemble of multiple models for the sake of improving performance: averaging all the models’ predictions or stacking an extra model on top whose data inputs are the lower models’ predictions (not to mention mixing and matching various ensemble structures into a Frankenstein’s monster of math and algorithms). I’ve read a number of posts on <a href="https://www.kaggle.com/">Kaggle</a> and elsewhere about super fancy, multi-level stacked models that win competitions, but in my limited experience–and with my limited knowledge–stacked models are really difficult to do well, and a mediocre averaging model tends to outperform a mediocre stacked model, so that’s what I started with. One complication that I should have foreseen (because I ran into it last year), but failed to (because memory is like the bean bag in which I sit, losing a bean here and there through its porous hide, still supporting my weight, more or less, but sagging enough to allow the unyielding floor to press against my butt), was trying to create a pipeline for data that changed shape at various points during the process. <a href="https://scikit-learn.org/stable/">Scikit-learn</a> really doesn’t like that. I couldn’t figure out an easy way to combine differently-shaped data into one pipeline while staying within the scikit-learn ecosystem, so I created a gigantic wrapper class that contained the various model classes, paired with their respective data-generating classes, and manually trained and averaged them. I kept the scikit-learn interface in an attempt to preserve a modicum of dignity, but the solution was not particularly elegant. It would, however, serve to test my hypothesis. To get a sense of the ensemble’s performance, I compared it to the individual models that compose it as well as a basic model trained on all of the data joined together. Since yearly performance is key, because there’s no office tipping competition for the best average over multiple seasons, I took results from 2011 to 2016 broken down by year to see how each model performed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YOdSkzQXegNJ97tdyCaFIg.png" /></figure><p>Well, that was underwhelming. The averaging ensemble managed to get the lowest mean absolute error for one season (2015), whereas the betting-data model had the lowest error for three seasons, and the all-data model won the two remaining seasons. One of the main advantages of ensemble models is that they have lower variance, so even though the averaging model only wins once, it comes in second three times and is never the worst model. However, if you take the mean error scores of the betting-data model and averaging model, the former comes out ahead with 28.80 to the latter’s 29.11. I tried adding the all-data model into the averaging ensemble, which did improve its error and accuracy slightly, but not enough to justify the addition of a whole bunch of parameters that would need tuning or the much longer training time, not to mention the headaches from being unable to fully utilise scikit-learn’s pipelines to organise my data and models. For the sake of thoroughness, I tested a simple stacked model (all of the non-average models feeding predictions into a basic <a href="https://xgboost.readthedocs.io/en/latest/index.html">XGBoost</a> regression model), but, as expected, that performed even worse than the averaging model.</p><h3>Feature engineering made easy</h3><p>Though I must follow my boulder back down the hill, slapping the dust from my palms, rubbing the fatigue from my shoulders, I still manage to hum the happy tune that’s been stuck in my head for days. I recently read the book <a href="https://www.packtpub.com/big-data-and-business-intelligence/feature-engineering-made-easy"><em>Feature Engineering Made Easy</em></a> (FEME) by Sinan Ozdemir and Divya Susarla, and it gave me some ideas for my next attempt at the summit. I had already read a few books on machine learning, all of them doing a good job of teaching the fundamentals (the underlying math and algorithms, the basics of classification and regression, as well as useful tips and tricks for improving performance), but what I found particularly illuminating about FEME was in its change of emphasis: the authors largely (but not entirely) skipped over the fundamentals in order to focus on the trial-and-error workflow of developing machine learning models. Although I was already using cross-validation to compare the relative performance of different algorithms, model structures, and feature sets, in FEME, the authors add parameter tuning to their cross-validation process when comparing changes. This slows down the workflow of developing machine-learning models, but it also gets the results a little closer to the final performance, offering a more-accurate portrayal of each model’s error, accuracy, etc. once it has been tuned and is ready for production. When starting out, I would probably prefer the quicker feedback, but now that I’m working with ensemble models, with their myriad parameters, a little tuning can make a big difference, and the default values might not be giving me a good idea of how well these different models will actually perform.</p><h3>Another bite at the apple</h3><p>With this new strategy in mind, I went back to comparing different models, again using the basic all-data model as a benchmark, but with two important differences: using the joined data as a single input instead of juggling multiple data sources, and adding randomized search parameter tuning to the cross-validation. The simplified structure made it much easier to set up the averaging model and stacked model, such that I even added a basic bagging model (with XGBoost regressor as the estimator) to the mix. I chose randomized search over grid search, because, depending on how many iterations one uses, it’s much faster, and according to the <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html#sphx-glr-auto-examples-model-selection-plot-randomized-search-py">scikit-learn documentation</a>, under the right circumstances, it is almost as good as grid search for finding the best combination of parameter values. Doing k-fold cross-validation across the entire data set up to 2015 (as opposed to the yearly cross-validation of the previous trial), the performance of the averaging regressor, even with the sparse data, was now noticeably better than the all-data model, achieving an MAE of 26.53 while the latter got 26.88. Meanwhile, the bagging model scored 26.54 and the stacked model got 26.73. So, it looks like the averaging model and the bagging model have similar performance, but how close are they really, and are they really better than the much simpler all-data model?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TOLsxipMTCcGZotv" /><figcaption>Photo by <a href="https://unsplash.com/@amseaman?utm_source=medium&amp;utm_medium=referral">Andrew Seaman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Bringing statistics back</h3><p>Another good recent read for me has been the paper “<a href="https://arxiv.org/abs/1811.12808">Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning</a>” by Sebastian Raschka. Unsurprisingly, since I’ve read his book <a href="https://www.packtpub.com/big-data-and-business-intelligence/python-machine-learning-second-edition"><em>Python Machine Learning</em></a>, I was already familiar with most of the techniques and recommendations in the paper. His survey of statistical testing techniques, however, were new to me and of particular interest given my recent experiences with damn-near-impossible-to-parse performance differences among different algorithms. In this case, the question is one of the significance of differences in mean absolute error (MAE). How likely is a reduction of 0.25 in the MAE due to random chance? Does a difference of 0.01 MAE mean anything at all? Thankfully, I now know about the 5x2cv combined F test for measuring the statistical significance of differing performance scores for two models. What’s more, Raschka’s Python package <a href="http://rasbt.github.io/mlxtend/">mlxtend</a> includes a function to get the F-value and p-value from this test, and it plays nice with the scikit-learn API. Running the test on ensemble models, using a respectably large data set, was kind of slow, but yielded interesting results. First, I compared the averaging and bagging models, getting a p-score of 0.5782, meaning any difference in their respective performances has a 57.8% chance of being due to randomness. I’m just using this score as an additional datum in determining which model I want to proceed with, so I wasn’t planning on holding myself to the usual 0.05 threshold, but having less than a 50% chance of being a real effect is a strong indication that I should use the simpler of the two, namely the bagging regressor which will require less tuning. With the same principle in mind, I used the F test to compare the bagging model to the all-data model, getting a p-score of 0.0007, which indicates that the bagging model’s better performance is probably worth the extra complication and slower training. As a side note, Raschka mentions the potential pitfalls of using pairwise tests like the 5x2cv combined F test and recommends using the Bonferroni correction. Again, I’m not being super scientific here, but just out of curiosity, if we apply the correction to the p-scores (i.e. dividing the target p-score by the number of times we performed the test), the difference between bagging and all-data is still well below the threshold for significance, 0.0007 being far less than 0.025.</p><h3>The proof is in the (year-based cross-validation of) performance</h3><p>Although, I’ve basically made up my mind to use a bagging regressor, for the sake of completeness, I scored the predictions for the bagging and averaging models for seasons 2011 to 2016 broken down by year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ihZu28G1X2QxVc9kunvbg.png" /></figure><p>As expected, the models’ error scores are pretty close, with each winning three seasons. The bagging model performs a little better when looking at accuracy, winning four seasons, but the difference is small enough to be chalked up to chance. The overall averages across all of the measured seasons bear this out, with averaging getting a slightly lower error overall, and bagging getting a slightly higher accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*ixmH6wnhlhEq54iV" /></figure><p>These figures provide the final evidence that there is no meaningful difference between the two models’ performances, so I’m better off using the simpler of the two, which, as stated earlier, is the bagging regressor.</p><h3>For the umpteenth time: keep it simple, stupid</h3><p>Well, my big idea came to naught, but I continue to learn the value of starting out simple and expanding from there rather than aiming for a big escherian target only to find out the folly of toying with non-euclidean nightmares that will overwhelm my feeble mind’s capacity for comprehension. One of these days, that lesson will stick. I’ve also learned the utility of applying statistical tests to machine-learning model scores to help with differentiating among them, which I will be sure to keep doing when basic cross-validation leaves the winner in doubt. Now that I have my ensemble model, I’m getting close to unleashing version 1.0 of <a href="https://github.com/cfranklin11/tipresias">tipresias</a> on the world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f884a7e8538" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Footy Tipping with Machine Learning: Adding Player Data]]></title>
            <link>https://medium.com/data-science/footy-tipping-with-machine-learning-adding-player-data-9ccdab56a4c5?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ccdab56a4c5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Wed, 19 Dec 2018 23:15:53 GMT</pubDate>
            <atom:updated>2018-12-22T13:54:17.136Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I stopped worrying and learned to fetch, aggregate, and reshape AFL player data.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qrS1WlYJw-8yNkEP" /><figcaption>Photo by <a href="https://unsplash.com/@scottwebb?utm_source=medium&amp;utm_medium=referral">Scott Webb</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Part four of a series of posts chronicling my attempt to develop a data model for footy tipping. You can read part three </em><a href="https://blog.goodaudience.com/toward-a-better-footy-tipping-model-the-first-model-bbc8c67a472a"><em>here</em></a><em>. Jupyter notebooks used for this post can be found </em><a href="https://github.com/cfranklin11/tipresias/tree/master/notebooks"><em>here</em></a><em>.</em></p><p>When I started this project, one of my main goals was to incorporate player data into my footy-tipping model. <a href="https://github.com/cfranklin11/footy-tipper">Footy Tipper</a>, my model for last season, incorporated betting and team-match data into an ensemble, but I couldn’t quite figure out how to include player-based features in a way that enhanced performance. I tried sorting player lists alphabetically, treating player names like categorical variables, then adding embedding layers to my neural net, but that just added noise and actually lowered accuracy slightly. I didn’t have much time to experiment, as the AFL season was quickly approaching, so I dropped the idea of using player lists for the time being, vowing to return with more men, horses, and steel. With the advent of <a href="https://github.com/cfranklin11/tipresias">Tipresias</a>, I’m making another assault on the barricades with a few new weapons at my side.</p><h3>fitzRoy is good, but I’m a pythonista</h3><p>During the recent AFL season, I started looking into the robust AFL stats community that I, in my American chauvinism, had assumed didn’t exist. What? A regional professional sports league in a country with 7.6% of the population of the U.S. is big enough to support a slew of data-crazed fans capable of writing sufficient blog posts and tweets to inform my new-found interest in footy stats? Consider me humbled. I followed a few accounts on Twitter, read some posts on <a href="http://matterofstats.com/">Matter of Stats</a>, <a href="http://hpnfooty.com/">HPN</a>, and started comparing my model’s performance to those of the notables competing on <a href="http://squiggle.com.au/">Squiggle</a>.</p><p>In addition to teaching me about the finer points of footy, my research yielded this little gem: <a href="https://github.com/jimmyday12/fitzRoy">fitzRoy</a>. fitzRoy is an R package that scapes popular AFL stats sites, cleans the data, and serves up data frames on the good china. It even has decent documentation! I was quite excited at the prospect of getting tons of data with minimal effort, because using an open-source package is way easier than maintaining my own website scraper, especially when faced with the borgesian nightmare that is navigating <a href="http://afltables.com/">AFL Tables</a>. The only problem is that I do all my work in Python, and fitzRoy is strictly for R, the basics of which I learn every couple of years, then promptly forget from lack of practice. I have a great data source in R; I have a data-consuming, machine-learning model in Python; how on earth do I get the twain to meet?</p><p>Luckily, there’s a tool for everything, even tools for getting your tools to work better. A little googling led me to the Python package <a href="https://rpy2.readthedocs.io/en/version_2.8.x/">rpy2</a>, whose entire purpose is to let you run R in a Python context, whether that means importing R packages as Python modules, running your own R code directly in a .py file, or converting R data frames to pandas. Unfortunately, this adds an extra level of complication to my project dependencies: not only am I using a few more packages, but I now need two full languages available in my development environment. The best tool for solving this problem is definitely Docker (with code, it’s tools all the way down), especially if you plan on running code like this on different machines or collaborating with others. This isn’t a Docker tutorial (if you’re interested, you can can check out the Docker setup for <a href="https://github.com/cfranklin11/tipresias">Tipresias</a> on GitHub), but the basic idea is that it creates an isolated virtual environment (i.e. a ‘container’) in which you can run your code. This means that you can install exact versions of all of your dependencies without worrying about conflicts with other libraries on your computer. Also, a collaborator can easily create a development environment with everything already installed by just entering a few commands.</p><h3>I had not thought that footy had undone so many</h3><p>Over 121 years, a lot of people have played a lot of matches in the VFL and AFL: 619,751 (plus a hundred or so from a few round-robin rounds that were easier to drop than munge), give or take. The size of this data set presented new challenges that went beyond longer processing times. For one, my Jupyter notebook kept crashing while I was cleaning and transforming the data, which caused me to spend a whole weekend figuring out how to do it in batches only to realise that all I needed was to increase Docker’s memory limit.</p><p>As for figuring out how to get the player data into a useful shape for match predictions, in the spirit of always starting with an MVP (‘minimum viable product’ in this context), I settled on a simple approach: calculate a rolling mean of each player’s stats (e.g. kicks, marks, goals), then add up the mean stats for all the players on a given team who participated in a given match. This functions similarly to using a rolling mean of a team’s in-game stats, but with the added nuance of only including the recent performance of players who are actually playing. So, if a team’s best player is not playing in their next match, the aggregated stats will reflect this diminishment in the team’s expected performance.</p><h3>Getting a model on the board</h3><p>Now that I have my player data aggregated and organised into team-match rows, like the datasets for my other models, I can start looking at the relative performance of various machine-learning models. As with models based on other datasets, I found that including both a team’s stats and their opponent’s stats improves model performance, with the lowest cross-validation mean absolute error (MAE) being 31.79 without opposition stats and 30.95 with those stats. Below are the MAEs for various models using aggregated player data for teams and their opponents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0VTI20RCpvIp2ObE" /></figure><p>As with the betting data, though less pronounced this time, linear models are outperforming the fancy ensembles. However, the top five are all so close that if they weren’t in ascending order, it would be difficult to pick the best. The accuracy scores below further demonstrate just how close these models’ performances are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jtr0CvU6LNS4e97g" /></figure><p>When looking at model accuracy, LinearSVR and Ridge are still toward the top, but they’re now behind the ensemble models that are third through fifth for MAE scores. Thankfully, at least the same models are in the top five for both MAE and accuracy, which limits how many I need to compare in a year-by-year breakdown of performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*736zeDvSQg5N7jIZ" /></figure><p>Ridge and XGBoost seem to have the best yearly MAEs, with each having the lowest in two years, but as with the CV/test scores, there isn’t a clear winner as the error scores in 2015 and 2016 are very close, and neither XGBoost nor Ridge is consistently in the top two for any given year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fpjvaa1qJYcUMT42" /></figure><p>No matter how many different ways I slice the data, I keep getting mixed results. It’s a bit of a gut call at this point, but XGBoost tends to be one of the better models across different metrics and data segments. It’s not a clear winner (both Ridge and GradientBoost have roughly equal performance), but I’ve had good results from using XGBoost in the past, and that’s enough to give it the edge.</p><h3>Player rating model</h3><p>Inspired by HPN’s <a href="http://www.hpnfooty.com/?p=21810">PAV</a>, but not knowing enough about footy to apply the rigour necessary to creating my own player rating system, I wondered if I could get an algorithm to create one for me. Instead of aggregating player in-match stats as I did above, I fed them, along with some aggregated opposition stats, into a regressor to try to estimate each player’s contribution to the final score. It’s a simplistic approach, but I just wanted to see if it had any potential as a means of using player data to predict match results.</p><p>I put unaggregated player data through a few linear algorithms (when I tried to use scikit-learn ensembles, it took waaaay too long to train), to estimate the final score margin for each match based on each player’s in-game stats. I then took the best-performing model and used its predictions to create a player rating feature, which I aggregated by team by match (i.e. each match has two rows, one for each participating team) for the final model. The final performance of this stacked ensemble wasn’t too terrible, but the best-performing models still had a couple more points of MAE and a few percentage points lower accuracy than the simpler models based on aggregated player stats, and even if the relative performance were closer, the amount of time it took to train a model on raw player data was painful. Seriously, my notebook would occasionally hang midway through calculating cross-validation scores, and it it was like Sisyphus’s boulder rolling back down into the stygian depths. There is the possibility of using something like this to add a feature to the main player-data model, but I imagine I would get better performance out of adding PAV instead, so this idea is very low priority for future experimentation.</p><h3>Player data in 3D</h3><p>Given that player data adds an additional segment to my initial team-match data structure, I wondered if, instead of aggregating the stats, I couldn’t just break the players out into their own dimension, transforming matrices with dimensions [team-match x aggregated feature] into [team-match x player x feature]. Getting the data shape right required more work than I had hoped, but I was able to reuse some code from the recurrent neural network (RNN) in Footy Tipper to get it over the line. The biggest challenge for me is always getting the data in the correct sort order and figuring out the correct dimensions such that numpy’s reshape works the way I want it to. Once I get beyond two dimensions, my brain starts to break, but I got there eventually.</p><p>As with the player-rating model, this was just an experiment, so I implemented the simplest version possible, opting for one-dimensional convolutional layers interspersed with the usual pooling layers. Having more experience with RNNs, I was pleasantly surprised by how quickly the CNN models trained on the data, but overfitting was a major issue even with dropout or regularisation parameters. The error on the training set kept going down with each epoch while the validation error would peak two or three epochs in, then proceed to rise. This resulted in similar performance to the player-rating model, namely, at best, in the range of 30–32 for MAE and 64–66% for accuracy, both metrics being decidedly worse than linear models trained on aggregated player data. There is some potential for adding this to an ensemble or expanding the data somehow to reduce the overfitting, but for now, as with the player-rating model, this sort of experiment is a lower priority than working on other improvements, such as feature engineering and implementing an RNN on the full data set.</p><h3>What have we learned today?</h3><p>I feel like the refrain of this series of blog posts is “keep it simple, stupid”. Though I don’t regret spending time on experimenting with more-complicated methods, basic linear models have consistently performed as well or better than the fancy ensemble models. Even when the almighty XGBoost is the best model, there are always at least a couple of linear models that are just a hair worse and probably sometimes better when using the right random seeds. Having seen measurable gains from creating ensembles from a few different model types, however, I think the slightly more-nuanced lesson is to start with simple building blocks before moving on to constructing the whole castle. It’s important to have a good foundation against which you can compare the impact of future changes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ccdab56a4c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/footy-tipping-with-machine-learning-adding-player-data-9ccdab56a4c5">Footy Tipping with Machine Learning: Adding Player Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Toward a Better Footy Tipping Model: The First Model]]></title>
            <link>https://medium.com/good-audience/toward-a-better-footy-tipping-model-the-first-model-bbc8c67a472a?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbc8c67a472a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Sat, 15 Sep 2018 04:55:10 GMT</pubDate>
            <atom:updated>2018-09-20T15:13:58.365Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0Beeq_k7TefQ0uKq" /><figcaption>“person holding leafed plant” by <a href="https://unsplash.com/@jlanzarini?utm_source=medium&amp;utm_medium=referral">Joshua Lanzarini</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Part three of a series of posts chronicling my attempt to develop a data model for footy tipping. You can read part two </em><a href="https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-the-folly-of-memory-9351670abe19"><em>here</em></a><em>. The data and code used for this post is in the </em><a href="https://github.com/cfranklin11/tipresias"><em>Tipresias</em></a><em> GitHub repo (in the notebooks folder). All data is from AFL matches from 2010 to 2017.</em></p><p>Now that I’ve done some exploratory analysis of AFL data, I’m ready to create the first tipping model for my Campaign for Glory 2019. I’ve already mentioned some of the <a href="https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-mistakes-were-made-ee5a6738741f">mistakes</a> that I made in developing Footy Tipper, one of the biggest being reaching for the RNN right away. So, this time I’ll start with linear models for a faster test-and-revise cycle and properly verify whether my previous assumptions are correct or not.</p><h4>Do classifiers or regressors perform better?</h4><p>For <a href="https://github.com/cfranklin11/footy-tipper">Footy Tipper</a>, I used an ensemble of classifiers, because that was the goal of the model: classify a team’s result for a given match as a win or loss. The standard footy-tipping tie-breaker of predicting the margin of victory for the first match of every round, however, basically slammed a knee into my back as it jumped over me for the mark, and I was forced to eat the grass of relying on the pre-game line odds as determined by the betting site LadBrokes. It turned out alright, as I had one of the smallest margin errors in the office competition, but, just as I hope to build a model that is more accurate than the betting odds, I hope to build one that is better at predicting victory margins as well. Additionally, during the season, I discovered the bustling AFL statistics community and the excellent site <a href="https://squiggle.com.au/">Squiggle</a>, which runs a tipping competition specifically for statistical models and uses <a href="https://en.wikipedia.org/wiki/Mean_absolute_error">mean absolute error</a>, which only applies to regression models, as a secondary performance metric. If I want to play with the big lads and ladies, it seems that a regression model will be necessary. The main objective, however, is still to tip with greater accuracy than any of my coworkers, thus securing insufferable-bragging rights for six months. Achieving a low MAE and the smallest tie-breaker score in the comp aren’t worth much if I’m three tips off the lead. So, the question remains: are classifiers or regressors more likely to make me office footy tipping champion?</p><p>To test this, I imported the following models from S<a href="http://scikit-learn.org/stable/">cikit-learn</a>:</p><ul><li>Classifiers: LogisticRegression, RidgeClassifier, DecisionTreeClassifier, LinearSVC, KNeighborsClassifier</li><li>Regressors: Lasso, ElasticNet, Ridge, LinearSVR</li><li>I also included RandomForestClassifier and RandomForestRegressor to get one ensemble model comparison.</li></ul><p>I didn’t choose these models for any particular reason, I was just trying to get a good spread of the common linear models (plus RandomForest) that are often used in tutorials, blog posts, etc. I included StandardScaler in the pipeline to scale the features, because large differences in numbers impact some models more than others. This also makes the results a little more realistic, because I’m planning on using StandardScaler in the final model pipeline anyway. Since I noticed some issues with over-fitting in my initial tests, I measured model accuracy via 5-fold cross-validation (CV) and for a test set that I held out for performance measurement only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*NRWNa1YuxOPMznc7hbgmuQ.png" /><figcaption>Comparison of accuracy scores of basic regression and classification models from Scikit-learn sorted from highest accuracy to lowest</figcaption></figure><p><em>Note: I set the minimum value at 0.6 in order to make the differences among models more visible, because setting the axis at 0 made them all look about the same. It’s just important to keep in mind that Lasso isn’t four times as accurate as DecisionTreeClassifier.</em></p><p>Most of the models actually got a higher accuracy on the test set than the in CV, which isn’t typical and suggests this test set is, just by chance, a little easier to predict (randomising the train/test splits on subsequent trials yields more-typical performance, with CV accuracies being slightly higher than test accuracies). This also suggests that over-fitting the training data isn’t an issue for now. As stated earlier, regressors are better suited to tipping competitions, but if classifiers were significantly better at the core objective — predicting winners — then I’d be willing to focus on that type of model, leaving score prediction as an afterthought. Thankfully, I don’t have to make that compromise, because regressors, making up the top four models tested, clearly outperform classifiers in tipping accuracy. RandomForestRegressor is a notable exception to this, being the only regressor to be less accurate than a classifier, which foreshadows the substandard performance of ensemble models in general that we will see later.</p><h4>Should data be organised by match or team-match combination?</h4><p>This is something that I actually tested with my Footy Tipper RNN classifier last year, and I found that splitting home and away teams into separate rows (team-matches) did improve accuracy somewhat. However, in the interest of testing my assumptions, especially given that I’m now dealing with linear regressors instead of neural net classifiers, let’s look at whether this holds up.</p><p>For the previous test, I had to use accuracy as the performance metric, because I needed to be able to compare performance between regressors and classifiers, which use different error metrics. However, I recently read a <a href="http://www.matterofstats.com/mafl-stats-journal/2018/5/17/accuracy-versus-mae-for-assessing-forecaster-ability-in-a-finite-season">post</a> on the very-informative <a href="http://www.matterofstats.com/">Matter of Stats</a> that makes a compelling argument for judging models based on an error metric (in this case, mean absolute error) rather than accuracy, because it is more likely to give you the best model. This is due to MAE being more stable than accuracy, which makes the latter more susceptible to dumb luck. In other words, in any given season, the model with the lowest MAE has a better chance of actually being the best model over the long run, than the model with the highest accuracy. Therefore, now that I’m only looking at regressors, I will focus on MAE as the primary metric, though still taking accuracy into consideration, for judging model performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*TYfy6GFfdmGwLOBCSa4wVw.png" /><figcaption>Difference in MAEs of models trained on data with one row per match vs data with two rows (one per participating team) per match, sorted by lowest CV MAE to highest</figcaption></figure><p>Keeping in mind that, with regards to error, lower is better, reorganising match data into team-match rows (thus doubling the number of rows and nearly halving the number of columns) tends to improve the MAE by a very small amount. There are a couple of exceptions for the test sets, however, which could be due to some randomness or possibly overfitting by the better-performing models. As with the question of regressors vs classifiers, this slightly better performance comes with a configuration that is also a little more convenient. Organising the data by team-match is necessary for calculating various cumulative statistics, and it’s easier if I don’t have to reshape heaps of data just to do some calculations, then reshape them again to fit the match-row structure. Another argument in favor of adopting the team-match-row structure is that, as shown below, it unambiguously lowers the standard deviation of the MAE, meaning that the predictions are, on average, both slightly more accurate and decidedly more consistent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*2UzS1cQdDp-wTYhbmkiiYg.png" /></figure><h4><strong>A closer look at regression models</strong></h4><p>Now that I’ve decided to use a regression model, and I know the general shape of my data, I’ll add some relevant features based on my previous EDA (home vs away, ladder position, win vs loss streak, etc.) and look at a wider range of potential models, including a number of popular ensembles. In addition to the linear models already examined, I’ll add the following:</p><ul><li>AdaBoostRegressor</li><li>BaggingRegressor</li><li>ExtraTreesRegressor</li><li>GradientBoostingRegressor</li><li>SVR (with kernel=’rbf’)</li><li>XGBRegressor</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*kpnMVMrTFMuDTypvKA_kTA.png" /><figcaption>MAEs for each model (Note: the minimum is set to 27 to highlight differences in performance)</figcaption></figure><p>As when comparing regressors to classifiers, Lasso, ElasticNet, and Ridge are the best-performing models. Ranking by accuracy or standard deviation shuffles the order slightly. (SVR with rbf kernel has the highest accuracy for some reason), but the same three models are still among the best — and never among the worst — regardless of the metric used. Although ensembles look a little better when focusing on standard deviation (averaging a few different models has a tendency to mitigate extreme results), I was surprised at the extent to which they underperform linear models in both error and accuracy. To see if this was due to overfitting, on account of my larding up the data with too many features, I cut it down to the basics: team, opposing team, line odds (i.e. point spread), year, round number, and home vs away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*EiyX7TTfzUmvgOTSCDHZPQ.png" /></figure><p>Reducing the number of features has mixed results. With the exception of ElasticNet, it improves MAE by a negligible amount, and it actually hurts the performance of the already-high-MAE ensemble models, which are the models that this change was supposed to help. In the future, I’ll use some regularization and feature consolidation (e.g. ladder position difference instead of ladder position and opposition ladder position as separate features) to keep the dimensionality of the data reasonable, but it looks like the added features provide useful signals to the models.</p><h4><strong>Which is best, Lasso, ElasticNet, or Ridge?</strong></h4><p>Since Lasso has the lowest mean absolute error for both cross-validation and test sets, and MAE is my principle performance metric, I’m leaning toward using it for the first version of my footy-tipping model. With Ridge having the best CV accuracy and ElasticNet the best test accuracy, however, any of these models could be considered the best depending on how you weigh the different metrics in your assessment. As the final criteria for determining which model to use, I’ll do cross-validation by season — instead of random split — to see which model performs better, because in a tipping competition it doesn’t matter how much you win by, but whether you win. Destroying the competition one year, only to narrowly lose the next three in a row might give you the same average accuracy as barely winning all four seasons, but the latter wins you a lot more money. And glory. The charts below show model performance per season, with each model being trained on all previous seasons’ data (e.g. to predict match results for 2013, models are trained on data from 2010 through 2012, inclusive).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*KJgJ6HzH8W6nJqdKsZxPGA.png" /></figure><p>Technically, Lasso has the lowest MAE in five out of the six seasons, with ElasticNet having the lowest in 2011. Realistically, 2011, 2015, and 2016 are so close that they might as well be ties, meaning Lasso wins outright in 2012, 2013, and 2014, but Lasso and ElasticNet tie in 2011 and 2015, and Lasso and Ridge tie in 2016. Even with this caveat, Lasso still looks like the best model for working with betting data, as its overall lower MAE is consistent enough to make it the winning model for most seasons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*lSnpuRugA17neNN6pEdXag.png" /></figure><p>Accuracy continues to be the less-consistent measure of performance, spreading out first place a little more evenly among the top three models. Lasso is the most accurate in 2011, 2013, and 2016; ElasticNet the most accurate in 2012 and 2015; and Ridge ties Lasso in 2014. Even with accuracy’s variability as a performance metric, Lasso proves, yet again, to be the superior model.</p><p>Now that I’ve looked at regressors vs classifiers, organising data by match or team-match, and applied multiple metrics and data splits to a wide variety of regression models, I’m confident that the Lasso algorithm is the best choice for the first piece of my burgeoning footy tipping model, which, like one of Hannibal’s elephants, will carry me over the Alps deep into my foes’ homeland and stomp them into the muck that darkens their riversides.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F7b0181%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F7b0181%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/748a06e9019942f2677bba917ef55804/href">https://medium.com/media/748a06e9019942f2677bba917ef55804/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbc8c67a472a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/good-audience/toward-a-better-footy-tipping-model-the-first-model-bbc8c67a472a">Toward a Better Footy Tipping Model: The First Model</a> was originally published in <a href="https://medium.com/good-audience">Good Audience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing like a Developer]]></title>
            <link>https://medium.com/marketplacer-product/writing-like-a-developer-60f04edac14b?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/60f04edac14b</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[writing-tips]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Thu, 23 Aug 2018 23:44:04 GMT</pubDate>
            <atom:updated>2018-08-23T23:44:04.507Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prose and code aren’t so different after all</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eAb05VbkEMFdwDql" /><figcaption>“A hand taking a clear shot of hanging lights with the camera of a smartphone” by <a href="https://unsplash.com/@dustinlee?utm_source=medium&amp;utm_medium=referral">Dustin Lee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Having spent a fair bit of time both writing and coding, I’ve noticed that these two seemingly disparate crafts have a lot in common. They share some vocabulary: we <em>read</em> and <em>write</em> code; a coding <em>language</em> has a particular <em>syntax</em>; Python even has <em>dictionaries</em>. More importantly, however, both disciplines have best practices, communal knowledge gained over years that we must learn in order to consider ourselves skilled practitioners. These best practices have the common objective of more-effectively communicating one’s ideas to other people. So, before you write your next blog post, try approaching your prose with the same principles that you apply to your code, and the result will be clearer, more-persuasive writing.</p><h3><strong>Single Responsibility Principle</strong></h3><p>The hardest working principle in the dev business doesn’t just apply to coding. As we’ve all heard many times already, the single responsibility principle can be boiled down to the idea that each part of our code (class, method, etc.) should do only one thing and do it well. We don’t want to leave a bunch of sporks lying around. Each piece is aligned with, and contributes to, the whole, of course, but if we’re working on a class or method that is doing too much, the standard recommendation is to break it up into separate pieces.</p><p>A good piece of writing should have a similar organising principle, with the piece itself having one purpose, and each paragraph and sentence contributing to that purpose while only doing one thing at a time. If you’re writing a tutorial, don’t editorialise on your tool preferences or offer your opinion on the latest Twitter UI change. If you’re writing an opinion piece, don’t offer you’re top five tips for improving front-end performance. Writing an outline goes a long ways toward accomplishing this, as it becomes immediately clear if a paragraph deviates from the main idea when you’re only comparing a handful of topic sentences.</p><h3><strong>Sketch an Outline</strong></h3><p>Before coding a new feature, we should (but often don’t) plan out our classes, especially their interfaces, consider how our code will fit into the existing application. One doesn’t necessarily have to resort to intricate UML diagrams that would put the Cave of Altamira to shame, but a little planning goes a long way toward identifying potential problems and minimising backtracking due to going down the wrong path.</p><p>Similarly, before writing an essay or tutorial, organise your thoughts into a concrete structure to make sure they all fit together and flow in a logical order. In the heat of the moment, it’s easy to follow an idea only to realise that after a few paragraphs you are in full-on tangent mode and have left behind the main purpose of your post a few hundred words ago. Such digressions during the writing process aren’t always a waste of time: they can help clarify your thinking about other parts of your post, for example. The problem is when one becomes overly enamored of such tangents and decides to include them in the final draft, diluting the main purpose of the piece of writing.</p><h3><strong>Always keep the requirements in mind</strong></h3><p>Scope creep, the larding of new, non-essential requirements onto a feature branch, is anathema to us as devs: it delays deployment, often breaking real or imagined deadlines, resulting in the hapless dev being blamed for working so slowly. There is an inverted form of scope creep, however, that we should be equally vigilant against, that of the dev diving down a rabbit hole of paying off tech debt, spending weeks on refactoring old classes or components, while the project manager is just waiting around for that new drop-down menu. We shouldn’t go to the other extreme of doing the bare minimum for each ticket, of course, but being mindful of the requirements of the project that we’re working on gives us something concrete to guide our code. Do we need a brand new class, a new higher-order component, or can we extend existing code? How much clean-up of legacy code is required for us to accomplish the objectives of this ticket? Achieving the right balance is difficult and good, clear requirements give us reasonable boundaries to help with these decisions.</p><p>When writing, especially as an amateur, you’re unlikely to be assigned specific topics revealed unto thee from on high. You write whatever the hell you want, and anyone is free to read it or ignore it. Therefore, you’ll want to think hard about who your intended audience is, and what you want to communicate to them: these are your requirements. A blog post should have a more-casual tone than an academic article, for example. Otherwise, you’ll come off as stilted and overly dry to the general public (unless your intended audience are academics). Also, you’ll have to be conscious of your use of technical language, especially esoteric jargon that a general audience is unlikely to understand. People might be willing to look up the meaning of an acronym or two, but if a post is littered with them, the readers will eventually give up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vFeIF14wRcjZ_Pzb" /><figcaption>Photo by <a href="https://unsplash.com/@priscilladupreez?utm_source=medium&amp;utm_medium=referral">Priscilla Du Preez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Linting, testing, code reviews</strong></h3><p>Just as git add . &amp;&amp; git commit — amend — no-edit &amp;&amp; git push — force isn’t a great idea, especially while on master, simply because you’ve finished writing a post doesn’t mean you’re ready to hit ‘publish’ just yet. When writing code, we use linters to correct style and basic errors, tests to catch more-complicated errors, and finally we do code reviews to get a second opinion on everything we’ve done, all before we can even think about merging into master. When writing prose, you have similar tools at our disposal. Spell-check and grammar-check can catch the basic errors for you. Also, reading your piece aloud can make problems with flow or awkward sentences more obvious, but having someone else review your writing and offer suggestions is often the best way to make sure that you’re communicating your ideas clearly.</p><h3><strong>Don’t overdo best practices</strong></h3><p>I am a perfectionist by nature, and I have a feeling that I’m not alone among developers in this respect. The downside of perfectionism is that it can go too far and cause excessive procrastination. No code, no piece of writing will ever be as good as it is while it’s still a vague concept inside your head, which can inspire people to put off disappointment by keeping all that work hypothetical for as long as possible, whether it means spending weeks over-engineering a small feature request or thinking through a blog post rather than just writing down their ideas. It requires discipline, but if you have an idea for a piece, just write it, and don’t worry too much whether it’s good, because it’s definitely not. First drafts are never good. The important thing is to give your thoughts a definite shape in the form of words on a page or screen, because you can always go back later, revise and refine your writing, hopefully making it good in the process. But if you’re waiting for perfection, or sufficient inspiration, you’ll never write anything, good or bad.</p><h3><strong>Write with empathy</strong></h3><p>We talk a lot about best practices and come up with evocative acronyms for them, like DRY and SOLID, but I’ve also seen developers far smarter than I admit that a little duplication is preferable to the wrong abstraction, and we’ve all seen code optimised to the point unintelligibility. Best practices in coding are meant to make reading and writing code easier for humans, not machines, and when we forget that, viewing these principles as goods for their own sake rather than means toward the good of making things easier for the next developer who comes across our code, we are writing fundamentally bad code.</p><p>So, if you’re going to put your writing out into the world, try to consider it from the perspective of your potential readers. Will they find it interesting, useful? Will they be confused or bored? The cleverness of your arguments and the depth of your knowledge won’t amount to much if no one can understand you or be bothered to finish your piece. The ultimate goal of publishing one’s writing, regardless of the forum or style, is to have people read it and get some value out of it. Otherwise, you might as well just jot down your thoughts in a notebook that’s headed for the shredder as soon as you’re done.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60f04edac14b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/marketplacer-product/writing-like-a-developer-60f04edac14b">Writing like a Developer</a> was originally published in <a href="https://medium.com/marketplacer-product">Marketplacer Product Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Toward a Better Footy Tipping Model: The Folly of Memory]]></title>
            <link>https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-the-folly-of-memory-9351670abe19?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/9351670abe19</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[australian-football]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Mon, 13 Aug 2018 09:23:21 GMT</pubDate>
            <atom:updated>2018-08-27T12:23:59.773Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SbXiJCHsq1rYrEyx" /><figcaption>“A man in a formal suit climbing up white marble stairs” by <a href="https://unsplash.com/@theexplorerdad?utm_source=medium&amp;utm_medium=referral">Joshua Ness</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Part two of a series of posts chronicling my attempt to develop a data model for footy tipping. You can read part one </em><a href="https://towardsdatascience.com/toward-a-better-footy-tipping-model-an-analysis-of-basic-heuristics-80de4235e768"><em>here</em></a><em>. The data and code used for this post is in the </em><a href="https://github.com/cfranklin11/footy-tipper"><em>Footy Tipper</em></a><em> GitHub repo (in the experiments folder). All data is from AFL matches from 2010 to 2017.</em></p><p>I concluded the previous post with a promise to consider AFL match data as a series of events occurring in time, because that is what they are. Though aggregations can reveal patterns across non-temporal dimensions (e.g. home vs away, ladder position, team), they obscure the sport as participants and spectators experience it. In this post I will look at two competing forces in the realm of sports narratives: the long-term perceptions of individual team’s quality (are they good or bad?) and short-term perceptions of a team’s chances of winning their next match (are they hot or cold?).</p><h3><strong>Betting odds are slow to adjust</strong></h3><p>In the last post, I looked at oddsmakers’ biases toward different teams, aggregating performance and odds data by team, then by team and year. Teams get better and worse from season to season, however, and even over the course of a single season a team can get off to a hot start only to tumble down the ladder, like hapless Jack down his hill, except with no crown in sight. Perceptions of how good various teams are may be slow to change, but obscuring the passage of time leaves out the fact that they do eventually change: Carlton wasn’t always this bad after all. The question is how quickly does the general public’s perception of a team’s quality, the likelihood of their winning matches, change relative to their real win rate?</p><p>To examine the relevant trends, I plotted each team’s rolling win rate, which averages their winning percentage over the last 23 rounds, updating with each new round, and compared this plot to that of the team’s rolling win rate as predicted by oddsmakers. I chose a 23-round window for the rolling mean for two reasons: being the average season length for the current number of AFL teams, it’s a natural choice, and, perhaps more importantly, it provides a good balance between responsiveness to change and smooth, interpretable charts. Although a holistic, quantifiable summary proved elusive, I did notice three common patterns in the relationship between a team’s real win rate and its predicted win rate.</p><h4><strong>The Rollercoasters</strong></h4><p>For teams whose win rate bounces up and down from season to season, their predicted win rate has a tendency to lag their real win rate by 5–15 rounds, finally catching up at peaks and valleys only to continue on its trajectory for a few more rounds, then lagging again while going in the new direction. Good examples of this pattern are Adelaide, Richmond, and Western Bulldogs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ShfvkNKORg2umQ1nrtNQ5g.png" /></figure><h4>The Winners</h4><p>For teams with consistently high win rates that remain flat, or change slowly, over long periods of time, their predicted win rate is generally higher than their real win rate. This is essentially an extended version of the peaks of the rollercoasters. Good examples of this pattern are Hawthorn, Sydney, and Collingwood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GncBh5UmzhEZyAqNCj7WaQ.png" /></figure><h4>The Losers</h4><p>For teams with consistently low win rates that remain flat, or change slowly, over long periods of time, their predicted win rate is generally lower than their real win rate. This is an extended version of the valleys of the rollercoasters. Good examples of this pattern are Brisbane, Melbourne, and GWS (first few years).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zForXl60WHpCZG8_ln1fJg.png" /></figure><p>Although each pattern has at least a few exemplary teams, it’s important to keep in mind that the rollercoasters are much more common than the winners and losers, mostly because few teams remain consistently good or bad for more than a season or two: constantly changing win rates is the norm in a league with institutional rules designed to promote parity. Given these patterns, a model could potentially estimate the direction (and, to a lesser extent, the size) of the betting odds’ prediction error if it knew the type of team (rollercoaster, winner, or loser), the current average win rate, and the current slope of the win rate plot (are they getting better or worse?). With this in mind, I calculated the slopes of the different win rates and plotted them with the real and predicted win rates, but the resulting charts confused more than they illuminated. However, this will still be worth exploring in the context of developing a machine-learning (ML) model, as an algorithm is often better than I at synthesising whatever signal exists in derived characteristics like the slope of a plot.</p><h3>Momentum is not a thing</h3><p>When I created <a href="https://github.com/cfranklin11/footy-tipper">Footy Tipper</a> last year, I researched how people use machine learning for forecasting, with the most-common example being predicting future stock prices based on trends in their past values. There are competing theories as to the relevance of today’s share price to tomorrow’s, but the general conception of time as a sequence of causes and effects, of linear unbroken narratives that tell the story of all existence, is so powerful that people will believe in it regardless of the evidence for or against. With that assumption built into tutorials on ML forecasting, I quickly decided that a recurrent neural network (RNN) should be the basis for my tipping model.</p><p>The advantage of an RNN is that, unlike most ML models, it is able to work with three-dimensional data. Typically, data is organised into two-dimensional tables: rows of observations (e.g. people, places, events) and columns of features (characteristics), like an Excel spreadsheet. RNNs, on the other hand, process three-dimensional data, adding a sequence of time periods for a given observation and its features, which allows the model to learn how earlier versions of a given observation influence its current characteristics. In other words, RNNs have a sense of sequential time that simpler ML models lack. As stated earlier, both observers and participants experience AFL matches in a sequence, this trivial fact, along with the equally obvious fact that events do not occur in a vacuum — the past influences the present — made examining actual trends in teams’ win rates seem unnecessary and made RNN the obvious choice for taking historical performance into account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lf3bLAFnLmYCLM-Cd8iEJw.png" /></figure><p>Or maybe not. In Past Craig’s defence, there is a slight bump toward the extremes for winners and losers three weeks after a given round, but win rates quickly regress to their respective means and remain steady for about 10 weeks, sticking close to their overall win rates of 56.6% for winners and 43.9% for losers. What this means is that winning recently is less an indicator of future wins and more an indicator that a given team tends to win in general. In fact, always tipping the winners from a recent round would be a heuristic almost as accurate as always picking the home team (56.6% vs 57.3%).</p><p>Teams that regularly win are, of course, more likely to win in any given week, but even consistent losers get lucky and snag a win now and again, which can muddle these single-round win rate trends. What about winning streaks, though? The bottom of the ladder can win a few over the course of a season, but surely they’re not about to win a few in a row on their way to losing the rest through September. So, how predictive of future wins are hot andcold streaks?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_LpBW9doihfNUjZ1pwL7fQ.png" /><figcaption>Note: 0 win streak win rate is not 50% due to draws being counted as wins, because in tipping they are</figcaption></figure><p>It looks like winning teams are actually less likely to win — and losing teams less likely to lose — follow-up matches than they are to win or lose any given match selected at random. Again, good teams win more often than they lose, whether they’re on a hot streak or not, but they are actually less likely to win the next match after a streak of any length than they are to win in general. Given the prevalence of <a href="http://www.wikinvest.com/wiki/Recency_bias">recency bias</a>, it’s understandable that people have an intuitive sense of hot and cold teams having an above- or below-average win rate, ignoring their even-more-extreme overall win rate, which makes fans over-inflate or over-deflate the expected probability of a future win.</p><p>I recently found an example of this when I, anxious about how the upcoming round would affect my performance in the office footy tipping competition, read a <a href="http://www.sportingnews.com/au/afl/news/melbourne-demons-v-sydney-swans-full-preview-teams-odds-and-how-to-watch-sundays-afl-action-buddy-franklin/ijv38q458d5u1pmuyl51py6fv">preview</a> of the upcoming Melbourne-Sydney match, the first two lines of which are literally as follows:</p><blockquote>Melbourne has won four of it’s last five games but dropped three in a row before that.</blockquote><blockquote>Sydney is back in winning form after losing four of the previous five.</blockquote><p>Forgetting for a moment the arbitrary framing of these figures in the pursuit of an easy narrative, if teams’ recent results are at the top of professional commentators’ minds, then such trends are likely to be a preoccupation of the average fan as well. There is further evidence of this in the fact that ESPN’s footytips explicitly displays teams’ last five results (under the heading ‘Streak’) alongside other important information such as ladder position, percentage, and win points. If this recency bias represents the conventional wisdom, and, as I demonstrated in the last post, betting odds tend to amplify conventional wisdom about which teams should win and why, how does this belief affect oddsmakers’ predictions?</p><p>The table below gives us an idea of how biased those expectations are by comparing the real win rates to those predicted by betting odds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*jvHPw6x7jnVOG7YUQ9Y4qQ.png" /><figcaption>Note: 0 win streak win rate is not 50% due to draws being counted as wins, because in tipping they are</figcaption></figure><p>As we saw with other forms of conventional wisdom, the belief that a team that has won or lost recently is just going to keep on winning or losing is contradicted by the data. Betting odds are the chump who buys a stock that’s been rapidly increasing in value for weeks (because of course it’s going to keep going up forever) only to have that same hot stock crash the next day. Past results are not indicative of future returns, my friend. Despite this bias in favour of hot teams and against cold teams, the accuracy of betting odds — in line with earlier analysis — remains pretty consistent across all winning/losing streak lengths (about 70% to 73%, close to the overall accuracy of 72.3%). At this point, however, I’m less interested in finding specific situations in which betting odds are inaccurate, which is never the case when only considering one or two dimensions, but biases that can be exploited by an ML model that can learn under which combination of factors a betting-odds underdog may be more likely to win.</p><h3>In conclusion</h3><p>Just as a narrow-minded dedication to basing all knowledge on context-free data analysis can cause one to miss the biases inherent in that data, an obsession with narrative, the stories we tell about our favourite sports and athletes, the drama of it all, can blind us to the statistical reality of long-term trends taking precedence over short-term random walks up and down the boulevard. It’s clear from the above analysis that betting odds, representing the wisdom of the crowd, are particularly susceptible to this way of viewing winning and losing in sports. As bad teams get better and good teams get worse over the rounds and seasons, people still remember when they were bad or good, so fans have difficulty adjusting their perceptions to match the new reality, like parents who continue to buy their children dolls or toy cars well into the latters’ teenage years, because they just can’t believe how fast their little boy or girl is growing up.</p><p>Even more difficult than catching up to medium-term changes in teams’ win rates is disinterestedly considering teams’ recent performance and accurately predicting their likelihood of winning next week. It turns out that good teams usually win and bad teams usually lose, and when good teams win too much they are a little less likely, not more, to win again as they keep close to their medium-term win rates.</p><p>In starting work on a new, better footy tipping model, I wanted to explore the data more thoroughly in order to gain insight into useful paths of inquiry, as opposed to the spaghetti-against-the-wall approach that I took last time. I believe that I have now done my due diligence in exploring patterns in winning and the biases in betting odds, and am ready to get started on the actual model. And this time, I won’t start with an RNN that takes hours to train.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9351670abe19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Toward a Better Footy Tipping Model: An Analysis of Basic Heuristics]]></title>
            <link>https://medium.com/data-science/toward-a-better-footy-tipping-model-an-analysis-of-basic-heuristics-80de4235e768?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/80de4235e768</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[australian-football]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Sun, 05 Aug 2018 03:01:13 GMT</pubDate>
            <atom:updated>2018-08-13T09:27:58.489Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part one of a series of posts chronicling my attempt to develop a data model for footy tipping. You can read the prelude </em><a href="https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-mistakes-were-made-ee5a6738741f"><em>here</em></a><em> and part two </em><a href="https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-the-folly-of-memory-9351670abe19"><em>here</em></a><em>. The data and code used for this post is in the </em><a href="https://github.com/cfranklin11/footy-tipper"><em>Footy Tipper</em></a><em> GitHub repo (in the experiments folder). All data is from AFL matches from 2010 to 2017.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gwL2yPELD0SFhldgy4V0Mg.jpeg" /><figcaption>Photo by <a href="https://www.flickr.com/photos/charlot17/">Charles Van den Broek</a> on <a href="https://www.flickr.com/">Flickr</a></figcaption></figure><p>Footy tipping is the popular Australian past-time of picking which Australian Rules Football (AFL) teams will win their matches every week (like filling out a March Madness bracket, except it’s done a round at a time, and the umpires make their fingers into pistols when a team scores). Each tipper gets a point for correctly picking the winner and a point in the case of a draw. At the end of the season, the winner is the one with the most points, with predicting the margin of the first match of each round serving as a tie-breaker.</p><p>There are a variety of heuristics that one could use to pick the winners and losers for the week. In my conversations with tippers and footy fans, I’ve found that the average participant tends to take in data, such as which team is favoured, which is the home team, how well each team has been playing lately, and whom they expect fellow competitors to tip, then process the input in their gut, arriving in this way at their predictions for the round.</p><p>Below, I will present a few basic heuristics for predicting match results, comparing their respective accuracies, following with a deeper analysis of the biases contained in the betting odds for AFL match results.</p><h3><strong>Comparison of Accuracies of Basic Heuristics</strong></h3><p>Below is a brief survey of simple, but reasonable, rules for tipping AFL matches. This certainly isn’t a comprehensive list, but for a tipper who isn’t really paying attention to how teams are playing these days, each heuristic represents a means of achieving an accuracy greater than 50% while using a single, easy-to-find data point.</p><h4>Heuristic 1: Always Tip the Home Team</h4><p>Although few would pick the home team to win every match, it is a truth universally acknowledged that playing at home is preferable to playing away, so much so that playoff series are structured around the acceptance of this premise, and online tipping competitions punish non-tippers by defaulting to the visiting team.</p><p>Accuracy when always tipping the home team: 57.29%</p><h4>Heuristic 2: Always Tip the Higher-Ranked Team</h4><p>This, though simple, is probably closer to how a casual fan might tip: the parameters are a quick two-word Google search away (you don’t even have to go to another website), and one is always picking the historically more-successful team from the start of the season to the current round. Of course, upsets do happen, and the difference between #1 and #15 is much greater than it is between #7 and #8.</p><p>Accuracy when always tipping the higher-ranked team: 68.00%</p><h4>Heuristic 3: Always Tip the Odds-on Favourite</h4><p>From the perspective of the end-user, this is a heuristic almost as simple as tipping the higher-ranked team, but it acts almost like a sophisticated black-box ensemble model: as far as we know, outsiders have access to the same data as oddsmakers, and we know that they’re optimising for revenue, but we don’t know what processes — what secret sauces — produce the line odds and point spreads that people bet on. Maybe a group of cragged, old men in fedoras and cheap suits smoke cigars and argue over exactly how many points Sydney are going to beat St. Kilda by; maybe a distributed deep-learning model, running on GPUs day and night, calculates odds to 100 significant digits every 5 minutes; maybe an augur watches the skies for how the birds are flying on any given day. In any case, the betting odds definitely represent an aggregation of far more data than any of the previous heuristics described.</p><p>Accuracy when always tipping the odds-on favourite: 72.32%</p><h4>Summary of Heuristics</h4><p>As shown in the average accuracies above, a heuristic’s reliability increases with the number of data inputs that its source takes into account.</p><ul><li>Calculating the home team just needs one datum: which team is at home.</li><li>Calculating the higher-ranked team requires somewhat more: both teams’ cumulative wins, draws, scores, and opponents’ scores for the season.</li><li>Finally, creating the odds for each match potentially takes into account all of the above plus each team’s recent win percentage, their in-game stats, their players’ individual stats, if any players are injured, head-to-head history for each matchup, and so on.</li></ul><p>Since tipping the odds-on favourite is the most accurate heuristic by over 4%, I’ll use this as the benchmark against which I’ll measure the performance of the machine-learning model that I will develop later. Therefore, I’ll focus further analysis on examining the biases built into betting odds, with the goal of using this information to help my model identify when to contradict the odds, hopefully achieving greater accuracy as a result.</p><h3>Analysis of Betting Odds</h3><p>Now that we know that tipping based on betting odds alone offers the highest accuracy, let’s explore under what conditions betting odds are more, or less, accurate. Odds are determined through a feedback loop of betting companies setting them, bettors risking their money based on perceived inefficiencies (“There’s no way Carlton can lose to Melbourne by over 50!”), then the companies adjusting those odds based on aggregate betting behaviour in order to maximise revenue. This means that betting odds represent a pretty good proxy for footy fans’ conventional wisdom around each team’s win probability, as any discrepancy between oddsmakers and the gambling public gets resolved quickly. So, the question is if there are biases that tend to shape general perceptions of teams’ win probabilities. Under what conditions are upsets more likely? If we can understand when and why the betting odds tend to be less accurate, we can emphasise those features when training a model in the hopes that it will be able to predict match results more accurately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cx0wc8POQ231V9C7fyKc2w.png" /></figure><p>Betting odds are disproportionately wrong when the spread is small (roughly between -25 and 25 points). No surprise there. However, betting odds are correct more often than wrong at every point-spread value, meaning that even for close matches, you’re still better off tipping the favourite.</p><h4>Betting Odds for Home Teams</h4><p>As we’ve shown, all other things being equal, home teams win slightly more often (~57%) than visiting teams. So, how do the oddsmakers take this into account? Do they have any biases with regards to home teams vs visiting teams?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*KYX-qv1RRE4fMHovMdMM2A.png" /></figure><p>Odds-makers are reasonably balanced in favouring home teams, picking them to win 57.68% of the time, which is in line with home teams’ 57.29% win rate. The oddsmakers improve upon their overall accuracy by about 2% when picking the home team (and do ~4% worse than average when picking away), suggesting that they’re better at recognising when home-field advantage will be a deciding factor than when it won’t.</p><h4>Betting Odds for Home Teams: Cross-State Matches</h4><p>Beyond the widely-accepted impact of home-field advantage across sports and leagues, a quirk of the AFL’s origins adds a wrinkle to our analysis. Aussie rules football began as a regional sport in the country’s south and west, with rugby league being the dominant contact sport in the north and east. In fact, the Australian Football League used to be the Victorian Football League, and even now 10 of the league’s 18 teams reside in the state of Victoria, their home stadiums little more than an hour’s drive (1 hour, 1minute according to <a href="https://www.google.com/maps/dir/MCG,+Brunton+Avenue,+Richmond+VIC/GMHBA+Stadium,+370+Moorabool+St,+South+Geelong+VIC+3220/@-37.9887757,144.5273674,11z/data=!3m1!4b1!4m14!4m13!1m5!1m1!1s0x6ad64295571a6281:0x63575fd647a0b2f9!2m2!1d144.9834493!2d-37.8199669!1m5!1m1!1s0x6ad4140d007cc031:0x54b5f173afaf1677!2m2!1d144.3545702!2d-38.1579763!3e0">Google Maps</a>) away from each other. So, the conventional wisdom about home field advantage, based on various conversations that I’ve had, is that it is minor, maybe even nonexistent, for teams from the same area and much more significant when the teams are from different states.</p><p>With this in mind, I’ll look at cross-state matches to see if the impact of home field advantage is greater under these circumstances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tqf0ON95Vt0HYVNL19g-dQ.png" /></figure><p>Home teams win 58.16% of cross-state matches, about one point more often than home teams overall, indicating a real, if small, effect from the extra travel distance. The oddsmakers, however, favour cross-state home teams 61.51% of the time, about three points more often, meaning that the conventional wisdom says that the cross-state effect is about three times what it actually is. This is evidence that betting odds tend to amplify conventional wisdom about the factors that contribute to teams’ win probabilities. The accuracy of betting odds for cross-state matches is 72.65%, less than half a point higher than the their overall accuracy. The normalised confusion matrix above also shows similar balance among prediction errors to what we see for all matches.</p><h4>Betting Odds for Higher-Ranked Teams</h4><p>As demonstrated earlier, a stronger predictor of win probability than whether a team is playing at home is whether they are higher on the ladder than their opponent (~57% vs ~68%). As such, it makes sense for oddsmakers to generally favour higher-ranked teams. The question is how strong this tendency is and to what extent it is backed up by the actual performance of higher-ranked teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q_d2HoD8n4Z5Q9b_e16PLA.png" /></figure><p>Although higher-ranked teams only win 68.00% of their matches, oddsmakers pick them to win 82.45% of the time, which is a much greater imbalance than we’ve seen with other segments.</p><p>Looking at the confusion matrices, we see that betting odds incorrectly favour the higher-ranked team more often than they favour the lower-ranked team at all. Furthermore, as with picking the home team — only more so — oddsmakers are significantly better at identifying when a better team (per their rank) will win than when a worse team will win.</p><p>It seems that the stronger the real effect is (home-field advantage, home-field advantage for cross-state matches, ladder rank as a predictor of wins) the more imbalanced betting odds become in favouring the teams that are supposed to win. Even if these biases don’t seem to significantly affect the overall accuracy of betting odds, they do show some situations in which those odds are less accurate, namely when they are contrary to the conventional wisdom. This means that, at least for close matches, when the betting odds favour lower-ranked and/or away teams, it might be worthwhile to occasionally tip their opponents.</p><h4>Betting Odds by Team</h4><p>The AFL is closer to American sports leagues than European soccer leagues in how it is organised: There is a draft between seasons in which bad teams get first dibs on the top amateur players, there’s a salary cap, free agency, etc. Many of these rules are meant to promote parity among teams, preventing a situation in which one group of teams are going to the finals, while another group are languishing at the bottom of the ladder, year after year. Despite this, in talking to footy fans, I get the sense that some teams get a reputation for being perennial winners or losers, and that these perceptions are slow to change. This has the potential to influence how oddsmakers treat different teams, basing predictions on reputation more than actual performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*s8soKsguwlClpAk3UTER1w.png" /><figcaption>Team win rate vs predicted win rate, sorted by difference between the two (predicted win rate — win rate)</figcaption></figure><p>As represented by the percent difference between the win rate and predicted win rate, there does seem to be a tendency to over-favour teams with winning records (Hawthorn, Sydney) and under-favour teams with losing records (Brisbane, Gold Coast).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*-qXtsSeBQONRU75IVZOnhA.png" /></figure><p>The relationship between teams’ win rates and odds-makers’ tendency to over-favour good teams and under-favour bad teams exists but is weak. Also, the pro-good-team bias, like the other biases examined, doesn’t have much impact on the accuracy of betting odds, as high and low accuracies are spread among teams with varying prediction error values. However, there does seem to be a pattern of good and bad teams being associated with high betting accuracy and mediocre teams being associated with low accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*CflMJ4eG0w6pBsHeMPSs5w.png" /></figure><p>It is somewhat noisy, but the plot above shows that oddsmakers are somewhat better at predicting results for bad teams and good teams, but have trouble with the mediocre teams that win between 40% and 60% of their matches. This tendency fits with the general pattern of betting odds being less accurate when predicting the outcomes of close matches. Although I can sympathise with the frustration of trying to tip matches with those mid-tier teams that can beat a top team one week and lose to a bottom team the next, this isn’t a terribly interesting observation. Tipping the close matches is the hard part, but those are the ones that you need to get right in order to have a chance at winning the pot.</p><h3>In Conclusion</h3><p>Among the simple heuristics explored here, the ones based on a larger number of data inputs tend to be better at predicting AFL match results, with betting odds being the most accurate predictor of all. Breaking down matches by different dimensions demonstrated some biases and weaknesses in oddsmakers’ ability to predict winners and losers. Betting odds tend to amplify the conventional wisdom about the factors that contribute to victory, favouring teams that benefit from those factors at a higher rate than their win rate warrants. However, these biases have little impact on betting odds’ performance, with lower accuracy generally centred around close matches, which are difficult to predict regardless of the quality and sophistication of one’s methods. The one slight weakness that a model could exploit is that betting odds’ accuracy suffers when they defy conventional wisdom by favouring lower-ranked and/or away teams.</p><p>One important dimension that I haven’t covered is that of time. AFL match data can be aggregated by team or home vs away or higher- vs lower-ranked, but they represent discrete events that occur in a sequence, and past matches not only affect future matches, but trends in teams’ performance on the field impact the conventional wisdom about who will win next week, which in turn affects the betting odds for next week’s matches. Time is a large subject, however, so I will dig into it in the next post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80de4235e768" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/toward-a-better-footy-tipping-model-an-analysis-of-basic-heuristics-80de4235e768">Toward a Better Footy Tipping Model: An Analysis of Basic Heuristics</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Toward a Better Footy Tipping Model: Mistakes Were Made]]></title>
            <link>https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-mistakes-were-made-ee5a6738741f?source=rss-157b8fa46c00------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee5a6738741f</guid>
            <category><![CDATA[australian-football]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Craig Franklin]]></dc:creator>
            <pubDate>Mon, 30 Jul 2018 21:17:16 GMT</pubDate>
            <atom:updated>2018-08-05T03:03:25.385Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The prelude to a series of posts chronicling my attempt to develop a data model for footy tipping. You can read part one </em><a href="https://medium.com/@craigjfranklin/toward-a-better-footy-tipping-model-an-analysis-of-basic-heuristics-80de4235e768"><em>here</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hKoO6GMRyVdm4-Rg" /><figcaption>Photo by <a href="https://unsplash.com/@amiion?utm_source=medium&amp;utm_medium=referral">Andra Ion</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The meaning of ‘footy’, like ‘curry’, is heavily dependent on context. Sure, there may be a shared essence, the consistency of the sauce, the rice, the contact of foot against ball, but the details are so varied and so specific, going beyond nationality down to the level of regional cultures, local histories, so as to make one version strange, foreign to those who enjoy another, meanings divided by borders, not of nations, but of practices, ingredients, movements. As a child, I confidently declared that I did not like curry, to which my dad replied, “But that’s impossible: there are too many curries, and they’re too different from each other, to make a blanket statement about all of them. You don’t actually know whether you like curry.” I should have specified that I did not like <em>his</em> curry. Thankfully, my palate has since matured, and I now enjoy a wide variety of curries: red and green from Thailand, vindaloo and rogan josh from India, katsudon from Japan, etc. Footy, on the other hand, was not even a concept that occurred to me in my youth. Football was a sport in which the only players who touched the ball with their feet were two specialists who spent a grand total of maybe 10 minutes on the field per game, but footy meant nothing to me. Not until I arrived in Australia was I able to attach ‘footy’ to something that existed in external reality. And as with any new knowledge, my initial conception of footy was shallow, only encompassing the first instance that I encountered: Aussie rules football. Only with time, and cultural exchange, have I learned that there are those who refer to rugby and even soccer (referred to here as such to differentiate it from all the other footballs) as ‘footy’. An Englishman, Queenslander, and Victorian are capable of speaking the same language without understanding a single word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i36FYCez8p9YFxyU" /><figcaption>Photo by <a href="https://unsplash.com/@geelongaerialphotography?utm_source=medium&amp;utm_medium=referral">Daniel Anthony</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve lived in Melbourne, Victoria for over four years, and here ‘footy’ means Aussie rules football. I have vague memories of occasionally catching a game at odd hours as a child, perhaps the result of a young ESPN desperate to fill airtime, but the only aspect of the game that stuck was the referees shooting finger guns in the most-serious way possible whenever a team scored. Over the last few years, however, I’ve attended and watched enough matches and had enough conversations with footy fans to get a sense of the game, its rules, and a genuine appreciation for a sport that combines the dynamic action of American football with the uninterrupted flow of soccer. One traditional fan activity that intrigued me was the office footy tipping competition. Like the American practice of filling out a March Madness bracket, a group of coworkers pay a flat entrance fee, then compete to see who can more-accurately predict the winners of Australian Football League (AFL) matches, with the winner taking the pot. Predictions are ‘tips’, and to make a prediction is ‘to tip’. I didn’t participate in such competitions for the first few seasons I lived in Australia, because I didn’t know enough about the sport to have any confidence in my ability to pick winners and losers. Last year, however, I started studying machine learning in my free time and, searching of an engaging — but light — practice project, I thought it might be fun to use data in order to teach a computer predict AFL match results for me. More importantly, I found the idea of the ignorant American rocking up and, like the scrawny, awkward nerd doing physics calculations in order to hit the game-deciding home run, winning the office footy tipping competition too amusing. I had to at least try.</p><p>I spent months researching machine-learning methods, especially for time-series data, which presented challenges well beyond classifying irises or predicting who would survive the Titanic; reading papers on applying machine learning to sports; and working on my own models. I remember the irrepressible excitement at reaching almost 80% accuracy on my validation data set, which was the 2016 season. I also remember the crushing despair that came from realising that my recent performance gains had been due to a coding error that resulted in the validation data being mixed in with the training data. Obviously, it’s much easier to predict the results of matches that you’ve already seen. Despite the setbacks, the newb mistakes, the weekends lost to getting Keras and Vowpal Wabbit models to just freaking pickle already, by mid-March I had a model that fed AFL data into a weighted ensemble of a multi-layer recurrent neural network (RNN) classifier, Adaboost classifier, Gradient Boost classifier, Random Forest classifier, Vowpal Wabbit classifier, and XGBoost classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*hBX2ht1QA6q-5siUqCIQIw.png" /><figcaption>Jupyter notebook output from a test run: tipping accuracies of betting odds, tipping higher-ranked teams, and my machine-learning model.</figcaption></figure><p>In anticipation of the 2018 AFL season, I ran a test that covered 2012 (when the AFL expanded to its current number of teams) to 2017, and my model achieved an overall accuracy ~0.5<strong>% </strong>higher than that of the betting odds.</p><p>I dubbed the app that runs my model <a href="https://github.com/cfranklin11/footy-tipper">Footy Tipper</a>. Over the preceding six months, I had learned a lot about footy, machine learning, data science, and even web development, and I was ready to out-tip my coworkers to glory.</p><p>Unfortunately, this being my first big project, I made a number of methodological and architectural mistakes that go well beyond the errant slicing of arrays, and I hope to rectify them for Footy Tipper 2.0.</p><ul><li>From the beginning, I was impatient to get to the good stuff, skipping the kind of exploratory data analysis that could have informed the development of my prediction model and given me insight into which ideas were worth pursuing (rather than just throwing all the data I had at a handful of models).</li><li>Another consequence of my impatience was my going straight to working with “the best algorithm”, which, after reading about forecasting models, seemed to be a recurrent neural network (RNN). Only later, as I started to build an ensemble model, did I discover that there were algorithms that were simpler, quicker to train, and capable of achieving comparable performance to my multi-layered RNN.</li><li>I’ve learned a lot about programming over the past year, the principles of Object-Oriented Programming (OOP) in particular. I now realise that development-by-messy-scripts that are later turned into classes without a clear purpose beyond “Well, I guess it can go here” severely hindered my ability to experiment and test out ideas, because I got really hairy bugs in my pipeline every time I made the smallest change to the shape of my data.</li><li>I was too busy with the technical details to be creative, but ‘Footy Tipper’ is a particularly uninspired name. I didn’t spend four years and 55-credits’ worth of class time earning a B.A. in English literature with a specialisation in creative writing, eventually graduating with departmental honours, just to give my machine-learning-based app such a literal name. There’s not even a literary reference in it. I can do better.</li></ul><p>Despite these missteps, and an embarrassingly brutal first half of the season in which, at my nadir, I fell to 11th place out of 15 and a full five tips behind the leader, I’m currently in 2nd place in the office footy tipping competition and only missing out on the lead from the tiebreaker. Not exactly worthy of glory, but not bad, within shouting distance of glory with four rounds to play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ujmQyE2kYHV31Cin" /><figcaption>Photo by <a href="https://unsplash.com/@yulokchan?utm_source=medium&amp;utm_medium=referral">Joseph Chan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Looking forward, I’m going to take a more-methodical approach to creating my next model, exploring the data a little more and starting with basic models before testing the performance of more complicated ones. There are too many things that I want to do differently to make tweaking my current model and app a practical solution, but there is a lot of code for data cleaning and transformation that I want to reuse, so I’m working from the same codebase, but with the goal of rebuilding both the model and app from scratch. So, for my next post, I will begin my journey of data-science renewal by analysing simple heuristics for footy tipping that aren’t based on statistical or machine-learning models in order to establish a benchmark against which to judge the performance of any future models I develop. Because if I can’t consistently beat tippers who just look up the favourites in the morning paper, what am I doing with all this time and effort spent?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee5a6738741f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>