<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Paul Solomon on Medium]]></title>
        <description><![CDATA[Stories by Paul Solomon on Medium]]></description>
        <link>https://medium.com/@sol.paul?source=rss-4bfbfb6c1cc2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*VpTwLRtZ1uwf1P3RelkwKw.jpeg</url>
            <title>Stories by Paul Solomon on Medium</title>
            <link>https://medium.com/@sol.paul?source=rss-4bfbfb6c1cc2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:30:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sol.paul/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build your own WhatsApp Text Generator(and learn all about language models)]]></title>
            <link>https://medium.com/analytics-vidhya/build-your-own-whatsapp-text-generator-and-learn-all-about-language-models-a1d6d96cd32e?source=rss-4bfbfb6c1cc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1d6d96cd32e</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[google-colaboratory]]></category>
            <dc:creator><![CDATA[Paul Solomon]]></dc:creator>
            <pubDate>Mon, 14 Oct 2019 14:27:10 GMT</pubDate>
            <atom:updated>2019-10-17T13:12:36.344Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build your own WhatsApp Text Generator (and learn all about language models)</h3><h4>A practical end-to-end Deep Learning NLP example</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*Hfrb7wIY4jymRrpQGXs0vA.png" /></figure><p>A normal conversation on WhatsApp, but all is not what it seems. The people are real; the chat is fake. It was generated by a language model trained on a real conversation history. In this post I will take you through the steps to build your own version using the power of <em>recurrent neural networks</em> and <em>transfer learning</em>.</p><h4>Requirements</h4><p>I have used the <a href="https://docs.fast.ai/">fastai library</a> inside of Google’s free research tool for data science, <a href="https://colab.research.google.com/">Colab</a>. This means very little time (and no money) getting set up. All you need to build your own model is the code laid out in this post (also <a href="https://colab.research.google.com/drive/1UcX4p1Icw0quNrhuA2B912DRTwcIDEQA">here</a>) and the following:</p><ul><li>A device to access the internet</li><li>A google account</li><li>WhatsApp chat histories</li></ul><p>I discuss some theory and delve into some of the source code, but for more detail there are various links to academic papers and documentation. If you want to learn more I also strongly recommend you look into the excellent <a href="https://course.fast.ai/">fastai course</a>.</p><h4>Drive and Colab initial set up</h4><p>First, we’re going to create a space on <a href="http://drive.google.com/">Google Drive</a> for your notebook. Click “New” and give the folder a name (I used “whatsapp”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gHqT4bjPGHrtCmkkfLdBXA.png" /></figure><p>Then go into your new folder, click “new” again, open up a Colab notebook, and give it a suitable name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*JpzNpzv-_JkYvA_uLAc77Q.png" /></figure><p>Finally, we want to enable a GPU for the notebook. This will speed up the training and text generation process significantly (GPUs are more efficient than CPUs for matrix multiplications, the main computation under the hood in neural networks).</p><p>Click “Runtime” from the top menu, then “Change runtime type”, and select “GPU” for the hardware accelerator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8eZDBSp5r7acnwkYx7FnPg.png" /></figure><h4>WhatsApp data</h4><p>Now let’s get some data. The more the better, so you’ll want to pick a chat with a reasonably long history. Also, explain what you’re doing to anyone else involved in the conversation and get their permission first.</p><p>To download the chat, click options (three vertical dots on the top right), select “More”, then “Export chat”, “Without media” and if you have Drive installed on your mobile device you should have an option to save to your newly created folder (otherwise, save the file and manually add to Drive).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gLjtcke30xn5_lzQ798hnw.png" /></figure><h4>Data preparation</h4><p>Back to the notebook. Let’s start by updating the fastai library.</p><pre>!curl -s <a href="https://course.fast.ai/setup/colab">https://course.fast.ai/setup/colab</a> | bash</pre><p>Then some standard magic commands and we bring in three libraries: fastai.text (for the model), pandas (for data preparation) and re (for regular expressions).</p><pre>## magic commands<br>%reload_ext autoreload<br>%autoreload 2<br>%matplotlib inline</pre><pre>## import required packages<br>from fastai.text import *<br>import pandas as pd<br>import re</pre><p>We want to link this notebook to Google Drive in order to use the data we just exported from WhatsApp and save any models we create. To do so run the following code, go to the provided link, select your google account, and copy the authorization code back into your notebook.</p><pre>## Colab google drive stuff<br>from google.colab import drive<br>drive.mount(&#39;/content/gdrive&#39;, force_remount=True)<br>root_dir = &quot;/content/gdrive/My Drive/&quot;<br>base_dir = root_dir + &#39;whatsapp/&#39;</pre><p>We have some cleaning to do, but the data is currently in .txt format. Not ideal. So here’s a function to take the text file and convert it into a pandas dataframe with one row for each chat entry, along with a timestamp and the name of the sender.</p><pre>## function to parse the whatsapp extract file<br>def parse_file(text_file):<br>    &#39;&#39;&#39;Convert WhatsApp chat log text file to a Pandas dataframe.&#39;&#39;&#39;<br>    <br>    # some regex to account for messages taking up multiple lines<br>    pat = re.compile(r&#39;^(\d\d\/\d\d\/\d\d\d\d.*?)(?=^^\d\d\/\d\d\/\d\d\d\d|\Z)&#39;,<br>                     re.S | re.M)<br>    with open(text_file) as f:<br>        data = [m.group(1).strip().replace(&#39;\n&#39;, &#39; &#39;) <br>                for m in pat.finditer(f.read())]</pre><pre>    sender = []; message = []; datetime = []<br>    for row in data:<br>        <br>        # timestamp is before the first dash<br>        datetime.append(row.split(&#39; - &#39;)[0])</pre><pre>        # sender is between am/pm, dash and colon<br>        try:<br>            s = re.search(&#39; - (.*?):&#39;, row).group(1)<br>            sender.append(s)<br>        except:<br>            sender.append(&#39;&#39;)<br>            <br>        # message content is after the first colon<br>        try:<br>            message.append(row.split(&#39;: &#39;, 1)[1])<br>        except:<br>            message.append(&#39;&#39;)<br>    <br>    df = pd.DataFrame(zip(datetime, sender, message), <br>                      columns=[&#39;timestamp&#39;,<br>                               &#39;sender&#39;,<br>                               &#39;text&#39;])    <br>    <br>    # exclude any rows where format does not match <br>    # proper timestamp format<br>    df = df[df[&#39;timestamp&#39;].str.len() == 17]<br>    df[&#39;timestamp&#39;] = pd.to_datetime(df.timestamp, <br>                                     format=&#39;%d/%m/%Y, %H:%M&#39;)</pre><pre>    # remove events not associated with a sender<br>    df = df[df.sender != &#39;&#39;].reset_index(drop=True)<br>    <br>    return df</pre><p>Let’s see how it works. Create the path to your data, apply the function to the chat export, and take a look at the resulting dataframe.</p><pre>## path to directory with your file<br>path = Path(base_dir)</pre><pre>## parse whatsapp extract, replace chat.txt with your <br>## extract filename<br>df = parse_file(path/&#39;chat.txt&#39;)</pre><pre>## take a look at the result<br>df[205:210]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*ehaebVJ-amIbpL_dg3_O1A.png" /></figure><p>Perfect! This is a small snippet of conversation between me and my lovely wife. One of the advantages of this format is I can easily create a list of participant names in lower case, replacing any spaces with underscores. This will help later.</p><pre>## list of conversation participants<br>participants = list(df[&#39;sender&#39;].str.lower().<br>                    str.replace(&#39; &#39;, &#39;_&#39;).unique())<br>participants</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*FWvcfQ0DNYIvmSJGrRjNPw.png" /></figure><p>In this case there are only two names, but it will work with any number of participants.</p><p>Finally, we need to think about how we want this text to be fed into our model network. Normally, we would have multiple stand-alone pieces of text (e.g. wikipedia articles or IMDB reviews), but what we have here is a single ongoing stream of text. One continuous conversation. That’s what we create. One long string, including the sender names.</p><pre>## concatenate names and text into one string<br>text = [(df[&#39;sender&#39;].str.replace(&#39; &#39;, &#39;_&#39;) + &#39; &#39; + df[&#39;text&#39;]).str.cat(sep = &#39; &#39;)]</pre><pre>## show part of string<br>text[0][8070:8150]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4nAiPT_4BG4A5GNeNYlow.png" /></figure><p>Looks good. We’re ready to get this into a learner.</p><h4>Learner creation</h4><p>To use the fastai API we now need to create a <a href="https://docs.fast.ai/data_block.html#The-data-block-API">DataBunch</a>. This is an object that can then be used inside a Learner to train a model. In this case, it has three key inputs: the data (split into training and validation sets), labels for the data, and the batch size.</p><p><em>Data</em></p><p>To split training and validation let’s just pick a point somewhere in the middle of our long conversation string. I went for the first 90% for training, last 10% for validation. Then we can create a couple of <a href="https://docs.fast.ai/text.data.html#TextList">TextList</a> objects, and quickly check they look the same as previously.</p><pre>## find the index for 90% through the long string<br>split = int(len(text[0])*0.9)</pre><pre>## create TextLists for train/valid<br>train = TextList([text[0][:split]])<br>valid = TextList([text[0][split:]])</pre><pre>## quick look at the text<br>train[0][8070:8150]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k4nAiPT_4BG4A5GNeNYlow.png" /></figure><p>It’s worth going a little deeper on this TextList.</p><p>It is, more or less, a list of text (with just one element in this case), but let’s take a quick look at the source code to see what else is there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*xb7UkdrJtaSxcLA0iYDtMQ.png" /></figure><p>Ok, a TextList is a class with a bunch of methods (functions, anything in the above starting with “def”, all minimized). It inherits from an <a href="https://docs.fast.ai/data_block.html#ItemList">ItemList</a>, in other words it’s a kind of ItemList. By all means go and look up an ItemList, but I’m most interested in the “_processor” variable. The processor is a list with a TokenizeProcessor and a NumericalizeProcessor. These sound familiar in an NLP context:</p><p><em>Tokenize</em> — process text and break it up into its individual words</p><p><em>Numericalize</em> — replace those tokens with numbers that correspond to the position of the word in a vocab</p><p>Why am I highlighting this? Well, it certainly helps to understand the rules being used to process your text, and digging into this part of the source code and documentation will help you do that. But, specifically, I want to add my own new rule. I feel we should show that the sender names in the text are similar in some way. Ideally, I’d like a token before each sender name that tells the model “this is a sender name”.</p><p>How can we do this? That’s where _processor comes in handy. The documentation tells us we can use it to pass in a custom tokenizer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*eCAGsfSWAreAdTXdRm5wzg.png" /></figure><p>We can therefore create our own rule and pass it in with a custom processor. I still want to keep the previous defaults, so all I need to do is add my new function to the existing list of default rules, and add this new list in to our custom processor.</p><pre>## new rule<br>def add_spk(x:Collection[str]) -&gt; Collection[str]:<br>  res = []<br>  for t in x:<br>        if t in participants: res.append(&#39;xxspk&#39;); res.append(t)<br>        else: res.append(t)<br>  return res</pre><pre>## add new rule to defaults and pass in customer processor<br>custom_post_rules = defaults.text_post_rules + [add_spk]<br>tokenizer = Tokenizer(post_rules = custom_post_rules)</pre><pre>processor = [TokenizeProcessor(tokenizer=tokenizer), <br>             NumericalizeProcessor(max_vocab=30000)]</pre><p>The function adds the token ‘xxspk’ before every name.</p><p><strong>Before processing:</strong> “…eggs, milk Paul_Solomon Ok…”</p><p><strong>After processing:</strong> “…eggs , milk xxspk paul_solomon xxmaj ok…”</p><p>Note that I’ve applied some of the other default rules, namely identifying capitalised words (adds ‘xxmaj’ before capitalized words), and separating out punctuation.</p><p><em>Labels</em></p><p>We’re going to create something called a <em>language model</em>. What is this? Simple, it’s a model that predicts the next word in a sequence of words. To do this accurately the model needs to understand language rules and the context. In some ways, it needs to <em>learn the language.</em></p><p>So what’s the label? Easy, it’s the next word. More specifically, in the model architecture we’re using, for a sequence of words we can create a target sequence by taking that same sequence of tokens and shifting it one word to the right. At any point in the input sequence, we can look at that same point in the target sequence and find the correct word to predict (i.e. the label).</p><p><strong>Input Sequence:</strong> “… eggs , milk spkxx paul_solomon xxmaj …”</p><p><strong>Label/Next Word:</strong> “ok”</p><p><strong>Target Sequence:</strong> “… , milk spkxx paul_solomon xxmaj ok …”</p><p>We do this by using the label_for_lm method (one of the functions in TextList class above).</p><pre>## take train and valid and label for language model<br>src = ItemLists(path=path, train=train, valid=valid).label_for_lm()</pre><p><em>Batchsize</em></p><p>Neural networks are trained by passing in batches of data in parallel, so the final input for the databunch is our batchsize. We use 48, meaning 48 text sequences are passed through the network at a time. Each of these text sequences is 70 tokens long by default.</p><pre>## create databunch with batch size of 48<br>bs = 48<br>data = src.databunch(bs=bs)</pre><p>We now have our data! Let’s create the learner.</p><pre>## create learner<br>learn = language_model_learner(data, AWD_LSTM, drop_mult=0.3)</pre><p>Fastai gives us an option to quickly create a <a href="https://docs.fast.ai/text.learner.html#language_model_learner">language model learner</a>. All we need is our data (we have it already) and an existing model. This object has an argument ‘pretrained’ set to ‘True’ by default. This means that we’re going to take a <em>pre-trained</em> language model and <em>fine-tune</em> it to our data.</p><p>This is called <em>transfer learning</em>, and I love it. Language models need a lot of data to work well but we don’t have anywhere near enough in this case. To solve this problem, we can take an existing model, trained on massive amounts of data, and fine-tune it to our text.</p><p>In this case we use an <a href="https://arxiv.org/abs/1708.02182">AWD_LSTM</a> model which has been pre-trained on the <a href="https://arxiv.org/abs/1609.07843">WikiText-103</a> dataset. AWD LSTM is a language model that uses a type of architecture called a recurrent neural network. It’s trained on text, and in this case it has been trained on a whole load of wikipedia data. We can look up how much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L2oZ7TkQlyT7UaMbbUZkjw.png" /></figure><p>This model has been trained on over <em>100m</em> tokens from <em>28k</em> Wikipedia articles with state-of-the-art performance. Sounds like a great starting point for us!</p><p>Let’s get a quick sense of the model architecture.</p><pre>learn.model</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*ntSn-DnLwWbtJPi_T_Zkyg.png" /></figure><p>I’ll break this down.</p><ol><li><strong>Encoder </strong>— The vocab for our text will have any word that has been used more than twice. In this case that’s 2,864 words (yours will be different). Each of these words is represented using a vector of length 2,864 with a 1 in the appropriate position and all zeros elsewhere. Encoding takes this vector and multiplies it by a weight matrix to squash it down into a length 400 <em>word embedding</em>.</li><li><strong>LSTM cells</strong> — The length 400 word embedding is then fed into a LSTM cell. I won’t go into the detail of the cell, all you need to know is that a length 400 vector goes in to the first cell, and a length 1,152 vector comes out. Two other things worth noting: this cell has a <em>memory</em> (it’s remembering previous words) and the output of the cell is fed back into itself and combined with the next word (that’s the recurrent part), as well as being pushed into the next layer. There are three of these cells in a row.</li><li><strong>Decoder </strong>— The output of the third LSTM cell is a length 400 vector, this is expanded out again into a vector with the same length as your vocab (2,864 in my case). This gives us the prediction for the next word, and can be compared with the actual next word to calculate our loss and accuracy.</li></ol><p>Remember that this is a <em>pre-trained</em> model, so wherever possible the weights are exactly as were trained using the WikiText data. This will be the case for the LSTM cells, and for any words that are in both vocabs. Any new words are initialized by the mean of all embeddings.</p><p>Now let’s fine-tune it with our own data so that the text it generates sounds like our WhatsApp chat and not a Wikipedia article.</p><h4>Training</h4><p>First, we’re going to do some <em>frozen</em> training. This means we only update certain parts of the model. Specifically, we’re only going to train the <em>last layer group</em>. We can see above that the last layer group is “(1): LinearDecoder”, the decoder. All the word embeddings and LSTM cells will remain the same during training, it’s just the final decoding stage that will be updated.</p><p>One of the most important hyper parameters is the learning rate. Fastai gives us a useful little tool to quickly find a good value.</p><pre>## run lr finder<br>learn.lr_find()</pre><pre>## plot lr finder<br>learn.recorder.plot(skip_end=15)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*XZ7VY98pcCknAcZWZhqg9g.png" /></figure><p>Rule of thumb is to find the steepest part of the curve (i.e. the point of fastest learning). 1.3e-2 looks to be about right here.</p><p>Let’s go ahead and train for one epoch (once through all the training data).</p><pre>## train for one epoch frozen<br>learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*BeEwKnDYnmJCAj04vgllhg.png" /></figure><p>At the end of the epoch we can see the loss on the training and validation sets, and the accuracy on the validation set. We’re correctly predicting 41% of the next words in the validation set. Not bad.</p><p>Frozen training is a great way to start with transfer learning, but now we can open up the entire model by <em>unfreezing</em>. This means the encoder and LSTM cells will now be included in our training updates. It also means that the model will be more sensitive, so we reduce our learning rate to 1e-3.</p><pre>## train for four further cycles unfrozen<br>learn.fit_one_cycle(4, 1e-3, moms=(0.8,0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*69MhEMMXFMlhQ21gRD1n4Q.png" /></figure><p>Accuracy up to 44.4%. Note that the training loss is now lower than validation, that’s what we want to see, and the validation loss has bottomed out.</p><p>Note that you will almost certainly find that your loss and accuracy are different to the above (some conversations are more predictable than others) so I suggest you play around with the parameters (learning rates, training protocol, etc.) to try and get the best performance from your model.</p><h4>Text generation</h4><p>We now have a language model, fine-tuned to your WhatsApp conversation. To generate text all we need to do is set it running and it’ll start predicting the next word over and over for as long as you ask it to.</p><p>Fastai gives us a useful predict method to do exactly this, all we need to do is give it some text to get it started, and tell it how long to run for. The output will still be in tokenised format, so I wrote a function to clean up the text and print it out nicely in the notebook.</p><pre>## function to generate text<br>def generate_chat(start_text, n_words):<br>  text = learn.predict(start_text, n_words, temperature=0.75)<br>  text = text.replace(&quot; xxspk &quot;, <br>                      &quot;\n&quot;).replace(&quot; \&#39;&quot;, <br>                                    &quot;\&#39;&quot;).replace(&quot; n\&#39;t&quot;, <br>                                                  &quot;n\&#39;t&quot;)<br>  text = re.sub(r&#39;\s([?.!&quot;](?:\s|$))&#39;, r&#39;\1&#39;, text)<br>  <br>  for participant in participants:<br>    text = text.replace(participant, participant + &quot;:&quot;)<br>  <br>  print(text)</pre><p>Let’s go ahead and start it off.</p><pre>## generate some text of length 200 words<br>generate_chat(participants[0] + &quot; are you ok ?&quot;, 200)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*j6mC3t3bt5uMGeqneKKPzw.png" /></figure><p>Nice! It certainly reads like one of my conversations (focused largely on travelling home after work each day), the context is sustained (that’s the LSTM memory at work), and the text even looks to be tailored to each participant.</p><h4>Final thoughts</h4><p>I’ll finish with a word of caution. As with many other AI applications, fake text generation can be used <em>at scale</em> for unethical purposes (e.g. spreading messages designed to harm on the internet). I’ve used it here to provide a fun and hands-on way to learn about language models, but I encourage you to think about how the methods described above can be used for other purposes (e.g. as an input to a text classification system) or to other kinds of sequence-like data (e.g. music composition).</p><p>This is an exciting and fast-moving field with the potential to build powerful tools that create value and benefit society. I hope this post shows you that <em>anyone</em> can get involved.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1d6d96cd32e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/build-your-own-whatsapp-text-generator-and-learn-all-about-language-models-a1d6d96cd32e">Build your own WhatsApp Text Generator(and learn all about language models)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to win at Fantasy Premier League using data — Part 2 — Picking your team]]></title>
            <link>https://medium.com/@sol.paul/how-to-win-at-fantasy-premier-league-using-data-part-2-picking-your-team-d68759498fe3?source=rss-4bfbfb6c1cc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d68759498fe3</guid>
            <category><![CDATA[sports]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[excel]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[optimisation]]></category>
            <dc:creator><![CDATA[Paul Solomon]]></dc:creator>
            <pubDate>Wed, 07 Aug 2019 13:57:57 GMT</pubDate>
            <atom:updated>2019-09-29T12:34:05.809Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to win at Fantasy Premier League using data — Part 2 — Picking your team</h3><p><em>EDIT: Updated player forecasts, team optimisation tool and team of the week can now be found each gameweek at </em><a href="https://twitter.com/solpaul7"><em>https://twitter.com/solpaul7</em></a></p><p>In <a href="https://medium.com/@sol.paul/how-to-win-at-fantasy-premier-league-using-data-part-1-forecasting-with-deep-learning-bf121f38643a?source=friends_link&amp;sk=8bdf70639d7966fbeb3d58108539ba84">Part 1</a> I went through building a model to project player scores for the upcoming season. In this part I discuss how to use these projections in practice. I share:</p><ul><li>My general philosophy of decision making to maximise expectation</li><li>A tool to select optimal teams and interrogate player projections</li><li>(For those who are interested) a step-by-step guide to team optimisation in Microsoft Excel using the Solver Add-in</li></ul><h4>It’s all about maximising expectation</h4><p>FPL is all about points — I want to pick the combination of players that delivers the most expected points.</p><p>Let’s be clear about what <em>expected points</em> means.</p><p>A trick I often use is to imagine thousands of simulations of the season, each with slightly different outcomes. I want to pick the team that scores highest <em>on average </em>across all those seasons.</p><p>Now, I could be unlucky and start the season badly. If that happens, so be it. My <em>actual points</em> ended up on the low end of the scale in those thousands of seasons, but so long as I was right about my <em>expected points</em> then I’m happy. The FPL season is very long, so if I consistently make decisions that maximise expectation then the luck is likely to even itself out.</p><p>Selecting an opening team using this principle is straightforward, assuming that we think our forecast model is good. All we need to do is pick the team with the highest predicted score over, say, the first 10 weeks.</p><p>How do I figure out this optimal team?</p><p>I can use trial and error, looking at the predicted total scores for different combinations of players, but unless I’ve got a few million years to spare then I’m not going to find the highest scoring team. There is a much better way: optimisation. We can use an algorithm to find the top scoring combination of players almost instantly.</p><p>I’ve built a tool in Excel to do exactly this. I’ve also added functionality to allow you to make manual adjustments. My principle is to use data to <em>augment</em>, not replace, human expertise. There will be information that the model cannot capture, or clear biases in the data — I always want to create tools that allow me to recognise and correct these issues.</p><p>Download it <a href="https://drive.google.com/uc?export=download&amp;id=1WXpVNrOwkT1RQWzZEnfEv3sPdyez9Ubf">here</a> and have a play. I’ll be using it over the next two days to get my team ready for the season opener this Friday. I also plan to update the predictions and share the tool each week. It’s accuracy will improve significantly as the season progresses and it consumes more information.</p><p>If you’re interested in how I actually performed the optimisation in Excel, read on. Otherwise, good luck for gameweek 1!</p><h4>Team optimisation in Microsoft Excel, step by step</h4><p>Let’s start by opening the raw forecasts in Excel. We have one row for each player and week, so first we need to create another sheet with just one row per player. We’ll also turn this data into named tables (side note, if you’re not using named ranges, tables and pivot tables in Excel, then <a href="https://www.youtube.com/watch?v=0nbkaYsR94c">you should be</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*z2y8Ejg9kb8tgWWAW8uUZg.gif" /></figure><p>We want to look at predictions for gameweeks 1 to 10, but let’s make it adjustable by creating named cells for the starting and ending gameweek, and calculating total predicted score for each player using the SUMIFS function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*F27MhXy89YUpAh5O6hZaVg.gif" /></figure><p>One more bit of data cleaning, we need to create columns for each position (1 if it’s the players position, 0 if not) and a final selection column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*_U6WYrLEHdX7Y2o2lJ3gaw.gif" /></figure><p>Solver will select players by putting a 1 in the ‘Selected’ column. This means that we can use the SUMPRODUCT function to calculate total points, total price and number of players in each position (I set a few players to 1 to make sure it’s working).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*W8a3AmuUG8Pse09DXquZmA.gif" /></figure><p>Ok, we’re ready to use Solver to find our optimal team. Here’s what we need to tell it:</p><ul><li>Which cell to maximise: Total Points</li><li>Which cells to change: top 200 in the Selection column (200 is the maximum amount of cells Solver can handle, so we order the players by predicted score, and take the top 200)</li><li>Constraints: let’s pick eleven players in a 4–4–2 with a £78m budget, we also need to tell it to make our selection cells binary (1 or 0)</li><li>Solving method: Simplex LP</li></ul><p>Click solve and voila! We have the selections that give the maximum points based on these predictions and criteria (you may need to <a href="https://support.office.com/en-us/article/load-the-solver-add-in-in-excel-612926fc-d53b-46b4-872c-e24772f078ca">enable Solver</a> first).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*9swQS0SzR41OAvjlSC1cjg.gif" /></figure><p>Note: The final team shown in the GIF highlights an issue I mentioned in Part 1. Players are selected who we do not expect to start this weekend. I’ve got around this in the final tool shared above by setting a zero point prediction for players who are not expected to start. Over the next few weeks I plan to replace this with an approach that takes into account injury status and recent minutes. Watch this space.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d68759498fe3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to win at Fantasy Premier League using data — Part 1 — forecasting with deep learning]]></title>
            <link>https://medium.com/@sol.paul/how-to-win-at-fantasy-premier-league-using-data-part-1-forecasting-with-deep-learning-bf121f38643a?source=rss-4bfbfb6c1cc2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf121f38643a</guid>
            <category><![CDATA[sports-data]]></category>
            <category><![CDATA[premier-league]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Paul Solomon]]></dc:creator>
            <pubDate>Fri, 02 Aug 2019 10:35:53 GMT</pubDate>
            <atom:updated>2019-09-29T12:33:42.817Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to win at Fantasy Premier League using data — Part 1 — Forecasting with Deep Learning</h3><p><em>EDIT: Updated player forecasts, team optimisation tool and team of the week can now be found each gameweek at </em><a href="https://twitter.com/solpaul7"><em>https://twitter.com/solpaul7</em></a></p><p>With the start of August the Premier League juggernaut begins to cast its shadow on the British (and global) sporting calendar, drawing millions of players once again into the nine-month fantasy marathon.</p><p>As with all games that deal with uncertainty, the crystal balls are quickly deployed to divine the future and get the season off to a good start. Cue a host of experts telling you how Harry Kane hates August, where to find the best ‘differentials’, and the latest intelligence on a player’s new <a href="https://www.youtube.com/watch?v=6naO8n6HsqE">ugly girlfriend</a>.</p><p>I prefer to use data to cut through the noise.</p><p>This season I’ll be sharing a number of tools that have helped my team finish top 1% in four of the last seven years, and comfortably top 10% in the other three. These are:</p><ul><li>Player score forecasts</li><li>Team selection optimisation</li><li>Look ahead fixture strength visualisation</li></ul><p>This first part will focus on building a solid view of expected player performance, the bedrock of team selection throughout the season.</p><h4>Player Forecasting</h4><p>I don’t trust myself. I’m too easily fooled. Even worse, when I make mistakes things sometimes turn out nicely, producing some not-so-nice habits. It’s a disaster. This is why I need an emotionless, bias-less guide to tell me straight, what can I <em>expect </em>from a player?</p><p>In other words, I need a well built predictive model. Here are the steps I followed to build one:</p><ul><li><strong>Get historical data:</strong> the hardest bit, luckily <a href="https://github.com/vaastav/Fantasy-Premier-League">this repository</a> on GitHub contains three years worth of FPL data (credit to <a href="https://github.com/vaastav">vaastav</a> who has put together a fantastic resource)</li><li><strong>Build training set: </strong>some wrangling to get the above into the format I need, and add in some additional data points</li><li><strong>Train model:</strong> I’ve just finished part one of the <a href="https://course.fast.ai/">fast.ai course</a> (which is fabulous), so this was a great opportunity to try out their tabular learner (a neat way of applying deep learning concepts to non-vision/text problems)</li><li><strong>Forecast:</strong> Finally, I built a dataset with every player’s 2019/20 season fixtures, and applied the model to obtain forecasts</li></ul><p>I give a brief overview of the process below, but for more detail you can find the code to build the dataset, train the model, and generate forecasts in <a href="https://github.com/solpaul/fpl-prediction">this GitHub repository</a>.</p><h4>The Dataset</h4><p>I like to keep things simple to start with, so the current version of the model uses the following historical FPL data (one row for each player/gameweek):</p><ul><li>Player</li><li>Position</li><li>Team</li><li>Opposition</li><li>Home/away</li><li>Gameweek</li><li>Season</li><li>Minutes</li></ul><p>I considered calculating trend data (e.g. the player’s recent form), but after watching a time series worked example on the fast.ai course I decided not to include it for now. One of the stated advantages of using neural networks as opposed to other approaches such as random forest or XGBoost is a reduced need for feature engineering. I like quick. I like simple. I went with it for now.</p><p>I still felt uncomfortable about big off-season team changes though. Squads change drastically and entirely new teams are promoted into the league. To address this issue I took inspiration from <a href="https://fivethirtyeight.com/methodology/how-our-club-soccer-predictions-work/">fivethirtyeight’s club soccer predictions</a> methodology, bringing in team market value from <a href="https://www.transfermarkt.com/">Transfermarkt</a> for the start of each season. So the following was also included in the model:</p><ul><li>Team market value at start of season (relative to all teams)</li><li>Opposition market value at start of season (relative to all teams)</li></ul><h4>The Model</h4><p>I won’t go into the finer details of the neural network architecture and training, but I will mention the interesting use of the concept of embeddings. Here’s the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*jCiOTCxK1Xg5DMlCSBkklg.jpeg" /></figure><p>Those seven ‘embeddings’ at the top are for each of the seven categorical features (player, position, team, etc.). Each feature is represented by a vector of a certain length. For example, the first embedding represents each player (1,055 in total) with a vector of length 79. The vector is <em>learned</em> during the training process, and it can be thought of as representing <em>features </em>of that player. Somewhere in Mohamed Salah’s vector is encoded his ability to score lots of goals, among many other more abstract but useful characteristics.</p><p>(As an aside, this is a concept that lies at the heart of a common approach to personalised content, collaborative filtering. It’s very likely that companies like Netflix are representing you in the same way — but instead of your vector telling them about goal scoring prowess, it’s describing how much you like horror movies.)</p><p>These embeddings were combined with the continuous variables (minutes and team market values) and then fed into a standard neural network. After training (with lots of tweaking and checking), I ended up with a model to predict player scores each gameweek.</p><h4>Forecast</h4><p>Armed with a model I was ready to create the first forecast for the season ahead. Here I encountered a problem — the model was trained knowing the number of minutes each player played each week, not something we know for the future. For now I got around this by assuming each player will play the full 90 minutes. By no means perfect, but on this basis, here is gameweek 1 ordered by projected points:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fsheetsu.com%2Ftables%2F3cc7969852&amp;dntp=1&amp;url=https%3A%2F%2Fsheetsu.com%2Ftables%2F3cc7969852&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=sheetsu" width="700" height="679" frameborder="0" scrolling="no"><a href="https://medium.com/media/f49cd260ad66f5adc8b428e90cabf7ff/href">https://medium.com/media/f49cd260ad66f5adc8b428e90cabf7ff/href</a></iframe><p>Not bad. Salah at the top makes sense — he is the top scorer in the game two years running, and playing at home to the newly promoted Norwich. However, the 90 minute assumption is clearly causing some issues. For example, Phil Foden is not a regular starter for Manchester City. If he is unlikely to start then this forecast is far too optimistic. The same could be said for Origi and perhaps, to a lesser extent, Son. In the past these players have been very efficient at scoring in limited minutes, so the model is rightly projecting high scores for them, but to improve the predictions we clearly need some way to introduce information about likely minutes.</p><p>As an aside, Guardiola just described Foden as “<a href="https://en.as.com/en/2019/07/26/football/1564154721_083002.html">the most talented player</a>” he’s ever worked with (a group of players that includes Lionel Messi!). I’m cautiously optimistic to see the model recognising this talent based on data alone.</p><h4>Next Steps</h4><p>In the next few days the likely gameweek 1 starting line-ups will take shape. I’ll be using these to address the minutes issue. In the meantime, we now have a set of forecasts, but so what? In <a href="https://medium.com/@sol.paul/how-to-win-at-fantasy-premier-league-using-data-part-2-picking-your-team-d68759498fe3?sk=b0efb9e8f49829bfaadef1ed475ab88d">Part 2</a>, I show how to use these forecasts to create optimal team selections.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf121f38643a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>