<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bernhard Pfann, CFA on Medium]]></title>
        <description><![CDATA[Stories by Bernhard Pfann, CFA on Medium]]></description>
        <link>https://medium.com/@bernhard-pfann?source=rss-b2add0f92c53------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LzJ5AyExvNrVozRljVkvAQ.png</url>
            <title>Stories by Bernhard Pfann, CFA on Medium</title>
            <link>https://medium.com/@bernhard-pfann?source=rss-b2add0f92c53------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:21:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bernhard-pfann/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build a Language Model on your WhatsApp Chats]]></title>
            <link>https://medium.com/data-science/build-a-language-model-on-your-whatsapp-chats-31264a9ced90?source=rss-b2add0f92c53------2</link>
            <guid isPermaLink="false">https://medium.com/p/31264a9ced90</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[whatsapp]]></category>
            <dc:creator><![CDATA[Bernhard Pfann, CFA]]></dc:creator>
            <pubDate>Tue, 21 Nov 2023 05:26:53 GMT</pubDate>
            <atom:updated>2023-11-22T18:05:48.196Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build a Language Model on Your WhatsApp Chats</h3><h4>A visual guide through the GPT architecture with an application</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*crPIGRkoSbmYYuvF" /><figcaption>Photo by <a href="https://unsplash.com/@lunarts?utm_source=medium&amp;utm_medium=referral">Volodymyr Hryshchenko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Chatbots have undeniably transformed our interaction with digital platforms. Despite the impressive advancements in the capabilities of underlying language models to handle complex tasks, the user experience often falls short, feeling impersonal and detached.</p><p>To make conversations more relatable, I envisioned a chatbot that could emulate my casual writing style, akin to texting a friend over WhatsApp.</p><p>In this article, I’ll walk you through my journey of building a (small) language model that generates synthetic conversations, using my WhatsApp chat messages as input data. Along the way, I try to unravel the inner workings of the GPT architecture in a visual and hopefully digestible way, complemented by the actual Python implementation. You can find the full project on my <a href="https://github.com/bernhard-pfann/lad-gpt">GitHub</a>.</p><p><strong>Note:</strong> The <a href="https://github.com/bernhard-pfann/lad-gpt/blob/main/src/model.py">model class</a> itself is in large chunks taken from the <a href="https://www.youtube.com/watch?v=VMj-3S1tku0&amp;list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ&amp;pp=iAQB">video series of Andrej Karpathy</a><strong> </strong>and adapted for my needs. I can highly recommend his tutorials.</p><p><a href="https://github.com/bernhard-pfann/lad-gpt">GitHub - bernhard-pfann/lad-gpt</a></p><h4>Table of Contents</h4><ol><li>Selected Approach</li><li>Data Source</li><li>Tokenization</li><li>Indexing</li><li>Model Architecture</li><li>Model Training</li><li>Chat-Mode</li></ol><h3>1. Selected Approach</h3><p>When it comes to tailoring a language model to a specific corpus of data, there are several approaches one can take:</p><ol><li><strong>Model building: </strong>This involves constructing and training a model from scratch, providing the utmost flexibility in terms of model architecture and training data selection.</li><li><strong>Fine-tuning:</strong> This approach leverages an existing pre-trained model, adjusting its weights to align more closely with the specific data at hand.</li><li><strong>Prompt engineering:</strong> This also utilizes an existing pre-trained model, but here, the unique corpus is directly incorporated into the prompt, without changing the model’s weights.</li></ol><p>As my motivation for this project is mainly self-educational, and I am rather interested in the architecture of modern language models, I opted for the first approach. Yet, this choice comes with obvious limitations. Given the size of my data and available computational resources, I didn’t anticipate results on par with any state-of-the-art pre-trained models.</p><p>Nevertheless, I was hopeful that my model would pick up some interesting linguistic patterns, which it ultimately did. Exploring the second option (fine-tuning) might be the focus of a future article.</p><h3>2. Data Source</h3><p>WhatsApp, my primary communication channel, was an ideal source for capturing my conversational style. Exporting over six years of group chat history, totaling more than 1.5 million words was straightforward.</p><p>The data was parsed using a regex pattern into a list of tuples containing the date, contact name, and chat message.</p><pre>pattern = r&#39;\[(.*?)\] (.*?): (.*)&#39;<br>matches = re.findall(pattern, text)<br>text = [(x1, x2.lower()) for x0, x1, x2 in matches]</pre><pre>[<br>    (2018-03-12 16:03:59, &quot;Alice&quot;, &quot;Hi, how are you guys?&quot;),<br>    (2018-03-12 16:05:36, &quot;Tom&quot;, &quot;I am good thanks!&quot;),<br>    ...<br>]</pre><p>Now, each element was processed separately.</p><ul><li><strong>Sending date:</strong> Aside from converting it into a datetime object, I have not utilized this information. However, one could look at the time deltas to differentiate the start and end of topic discussions.</li><li><strong>Contact name:</strong> When tokenizing the text, each contact name is treated as a unique token. This ensures that the combination of first and last names is still considered a single entity.</li><li><strong>Chat message: </strong>A special “&lt;END&gt;” token is added at the end of each message.</li></ul><h3>3. Tokenization</h3><p>To train a language model, we need to break language into pieces (so-called tokens) and feed them to the model incrementally. Tokenization can be performed on multiple levels.</p><ul><li><strong>Character-level:</strong> Text is perceived as a sequence of individual characters (including white spaces). This granular approach allows every possible word to be formed from a sequence of characters. However, it is more difficult to capture semantic relationships between words.</li><li><strong>Word-level:</strong> Text is represented as a sequence of words. However, the model’s vocabulary is limited by the existing words in the training data.</li><li><strong>Sub-word-level:</strong> Text is broken down into sub-word units, which are smaller than words but larger than characters.</li></ul><p>While I started off with a character-level tokenizer, I felt that training time was wasted, learning character sequences of repetitive words, rather than focusing on the semantic relationship between words across the sentence.</p><p>For the sake of conceptual simplicity, I decided to switch to a word-level tokenizer, keeping aside the available libraries for more sophisticated tokenization strategies.</p><pre>from nltk.tokenize import RegexpTokenizer<br><br>def custom_tokenizer(txt: str, spec_tokens: List[str], pattern: str=&quot;|\d|\\w+|[^\\s]&quot;) -&gt; List[str]:<br>    &quot;&quot;&quot;<br>    Tokenize text into words or characters using NLTK&#39;s RegexpTokenizer, considerung <br>    given special combinations as single tokens.<br><br>    :param txt: The corpus as a single string element.<br>    :param spec_tokens: A list of special tokens (e.g. ending, out-of-vocab).<br>    :param pattern: By default the corpus is tokenized on a word level (split by spaces).<br>                    Numbers are considered single tokens.<br><br>    &gt;&gt; note: The pattern for character level tokenization is &#39;|.&#39;<br>    &quot;&quot;&quot;<br>    pattern = &quot;|&quot;.join(spec_tokens) + pattern<br>    tokenizer = RegexpTokenizer(pattern)<br>    tokens = tokenizer.tokenize(txt)<br>    return tokens</pre><pre>[&quot;Alice:&quot;, &quot;Hi&quot;, &quot;how&quot;, &quot;are&quot;, &quot;you&quot;, &quot;guys&quot;, &quot;?&quot;, &quot;&lt;END&gt;&quot;, &quot;Tom:&quot;, ... ]</pre><p>It turned out that my training data has a vocabulary of ~70,000 unique words. However, as many words appear only once or twice, I decided to replace such rare words by a “&lt;UNK&gt;” special token. This had the effect of reducing vocabulary to ~25,000 words, which leads to a smaller model that needs to be trained later.</p><pre>from collections import Counter<br><br>def get_infrequent_tokens(tokens: Union[List[str], str], min_count: int) -&gt; List[str]:<br>    &quot;&quot;&quot;<br>    Identify tokens that appear less than a minimum count.<br>    <br>    :param tokens: When it is the raw text in a string, frequencies are counted on character level.<br>                   When it is the tokenized corpus as list, frequencies are counted on token level.<br>    :min_count: Threshold of occurence to flag a token.<br>    :return: List of tokens that appear infrequently. <br>    &quot;&quot;&quot;<br>    counts = Counter(tokens)<br>    infreq_tokens = set([k for k,v in counts.items() if v&lt;=min_count])<br>    return infreq_tokens<br><br>def mask_tokens(tokens: List[str], mask: Set[str]) -&gt; List[str]:<br>    &quot;&quot;&quot;<br>    Iterate through all tokens. Any token that is part of the set, is replaced by the unknown token.<br><br>    :param tokens: The tokenized corpus.<br>    :param mask: Set of tokens that shall be masked in the corpus.<br>    :return: List of tokenized corpus after the masking operation.<br>    &quot;&quot;&quot;<br>    return [t.replace(t, unknown_token) if t in mask else t for t in tokens]<br><br><br>infreq_tokens = get_infrequent_tokens(tokens, min_count=2)<br>tokens = mask_tokens(tokens, infreq_tokens)</pre><pre>[&quot;Alice:&quot;, &quot;Hi&quot;, &quot;how&quot;, &quot;are&quot;, &quot;you&quot;, &quot;&lt;UNK&gt;&quot;, &quot;?&quot;, &quot;&lt;END&gt;&quot;, &quot;Tom:&quot;, ... ]</pre><h3>4. Indexing</h3><p>After tokenization, the next step is to convert the words and special tokens into numerical representations. Using a fixed vocabulary list, each word was indexed by its position. The encoded words were then prepared as PyTorch tensors.</p><pre>import torch<br><br>def encode(s: list, vocab: list) -&gt; torch.tensor:<br>    &quot;&quot;&quot;<br>    Encode a list of tokens into a tensor of integers, given a fixed vocabulary. <br>    When a token is not found in the vocabulary, the special unknown token is assigned. <br>    When the training set did not use that special token, a random token is assigned.<br>    &quot;&quot;&quot;<br>    rand_token = random.randint(0, len(vocab))<br><br>    map = {s:i for i,s in enumerate(vocab)}<br>    enc = [map.get(c, map.get(unknown_token, rand_token)) for c in s]<br>    enc = torch.tensor(enc, dtype=torch.long)<br>    return enc</pre><pre>torch.tensor([8127, 115, 2363, 3, ..., 14028])</pre><p>As we need to evaluate the quality of our model against some unseen data, we split the tensor into two parts. And voila, we have our training and validation sets, ready to feed to the language model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Z3wQkv74qMbYsZz0OlZ2A.png" /><figcaption>Image by author</figcaption></figure><h3>5. Model Architecture</h3><p>I decided to apply the GPT architecture, which has been promoted by the influential paper “<a href="https://arxiv.org/abs/1706.03762">Attention is All you Need</a>”. Since I tried to build a language generator and not a question-answer bot, the decoder-only (right side) architecture was sufficient for this purpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZmfpUQS6B9FjgOEj7_WvIg.png" /><figcaption>“<a href="https://arxiv.org/abs/1706.03762">Attention is All you Need</a>” by A. Vaswani et. al. (Retrieved from arXiv: 1706.03762)</figcaption></figure><p>In the following sections, I will break down each component of the GPT architecture, explaining its role and the underlying matrix operations. Starting off with the prepared training test, I will trace an exemplary context of 3 words through the model, until it leads to a prediction of the next token.</p><h4>5.1. Model Objective</h4><p>Before delving into technical specifics, it&#39;s crucial to understand our model&#39;s primary objective. In a decoder-only setup, our aim is to decode the structure of language to accurately predict the next token in a sequence, given the context of preceding tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyPssV1RzLo4rHCS531-zg.png" /><figcaption>Image by author</figcaption></figure><p>As we feed our indexed token sequence into the model, it undergoes a series of matrix multiplications with various weight matrices. The output is a vector representing the probability of each token being the next in the sequence, based on the input context.</p><p><strong>Model Evaluation:</strong></p><p>Our model’s performance is evaluated against the training data, where the actual next token is known. The objective is to maximize the probability of correctly predicting this next token.</p><p>However, in machine learning, we often focus on the concept of “loss”, which quantifies the error or the likelihood of incorrect predictions. To calculate this, we compare the model’s output probabilities with the actual next token (using <a href="https://en.wikipedia.org/wiki/Cross-entropy">cross-entropy</a>).</p><p><strong>Optimization:</strong></p><p>With an understanding of our current loss, we aim to minimize it through <a href="https://en.wikipedia.org/wiki/Backpropagation">backpropagation</a>. This process involves iteratively feeding token sequences into the model and adjusting the weight matrices to enhance performance.</p><p>In each figure, I will highlight in yellow the weight matrices that will be optimized during that procedure.</p><h4>5.2. Output Embedding</h4><p>Up to this point, each token in our sequence has been represented by an integer index. However, this simplistic form doesn’t reflect word relationships or similarities. To address this, we elevate our one-dimensional indices into higher-dimensional spaces through embedding.</p><ul><li><strong>Word-embeddings:</strong> The essence of a word is captured by an n-dimensional vector of floats.</li><li><strong>Positional-embeddings: </strong>These embeddings highlight the importance of a word’s position within a sentence, also represented as n-dimensional vectors of floats.</li></ul><p>For each token, we look up its word-embedding and positional embedding and then sum them up element-wise. This results in the output embedding of each token in the context.</p><p>In the below example, the context consists of 3 tokens. At the end of the embedding process, each token is represented by an n-dimensional vector (where n is the embedding size, a tunable hyperparameter).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CoEhjiKiRhu6SSO5n7gCEQ.png" /><figcaption>Image by author</figcaption></figure><p>PyTorch offers dedicated classes for such embeddings. Within our model class, we define the word-, and positional embeddings as follows (passing matrix dimensions as arguments):</p><pre>self.word_embedding = nn.Embedding(vocab_size, embed_size)<br>self.pos_embedding = nn.Embedding(block_size, embed_size)</pre><h4>5.3. Self-Attention Head</h4><p>While word embeddings provide a general sense of word similarity, the true meaning of a word often hinges on its surrounding context. For example, “bat” could refer to either an animal or a sports equipment, depending on the sentence. This is where the self-attention mechanism, a key component of the GPT architecture, comes into play.</p><p>The self-attention mechanism focuses on three main concepts: Query (Q), Key (K), and Value (V).</p><ol><li><strong>Query (Q): </strong>The query is essentially a representation of the current token for which the attention needs to be calculated. It’s like asking, “What should I, as the current token, pay attention to in the rest of the context?”</li><li><strong>Keys (K):</strong> Keys are representations of each token in the input sequence. They are paired with the Query to determine the attention scores. This comparison measures how much focus the query token should put on other tokens in the context. High scores mean that more attention should be paid.</li><li><strong>Value (V):</strong> Values are also representations of each token in the input sequence. However, their role is different, as they apply a final weighting to the attention-scores.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6XvWvcDtNyzyFNhLpdAp9w.png" /><figcaption>Image by author</figcaption></figure><p><strong>Example:</strong></p><p>In our example, each token of the context is already in embedded form, as n-dimension vectors (e1, e2, e3). The self-attention head takes them as input, to output a contextualized version for each of them, one at a time.</p><ol><li>When evaluating the token “name”, a query vector <strong>q</strong> is obtained by multiplying its embedded vector <strong>v2 </strong>with the trainable matrix <strong>M_Q</strong>.</li><li>At the same time key vectors <strong>(k1, k2, k3)</strong> are calculated for each token in the context, by multiplying each embedded vector <strong>(e1, e2, e3) </strong>with the trainable matrix <strong>M_K.</strong></li><li>The value vectors <strong>(v1, v2, v3)</strong> are obtained in the same way, just multiplied by a different trainable matrix <strong>M_V</strong>.</li><li>The attention-scores <strong>w</strong> are calculated as dot-product between the query vector and each key vector separately.</li><li>Finally we stack all value vectors to a matrix, and multiply that by the attention scores, to obtained the <strong>contextualized vector</strong> for the token “name”.</li></ol><pre>class Head(nn.Module):<br>    &quot;&quot;&quot;<br>    This module performs self-attention operations on the input tensor, producing <br>    an output tensor with the same time-steps but different channels. <br>    <br>    :param head_size: The size of the head in the multi-head attention mechanism.<br>    &quot;&quot;&quot;<br>    def __init__(self, head_size):<br>        super().__init__()<br>        self.key = nn.Linear(embed_size, head_size, bias=False)<br>        self.query = nn.Linear(embed_size, head_size, bias=False)<br>        self.value = nn.Linear(embed_size, head_size, bias=False)<br><br>    def forward(self, x):<br>        &quot;&quot;&quot;<br>        # input of size (batch, time-step, channels)<br>        # output of size (batch, time-step, head size)<br>        &quot;&quot;&quot;<br>        B,T,C = x.shape<br>        k = self.key(x)                                     <br>        q = self.query(x)                                   <br><br>        # compute attention scores<br>        wei = q @ k.transpose(-2,-1)                        <br>        wei /= math.sqrt(k.shape[-1])                       <br>        <br>        # avoid look-ahead<br>        tril = torch.tril(torch.ones(T, T))<br>        wei = wei.masked_fill(tril == 0, float(&quot;-inf&quot;))    <br>        wei = F.softmax(wei, dim=-1)<br>        <br>        # weighted aggregation of the values<br>        v = self.value(x)    <br>        out = wei @ v<br>        return out</pre><h4>5.4. Masked Multi-Head Attention</h4><p>Language is complex, and capturing all its nuances isn’t straightforward. A single set of attention calculations often isn’t enough to catch the subtleties of how words work together. That’s where the idea of multi-head attention in the GPT model comes in handy.</p><p>Think of multi-head attention as having several pairs of eyes looking at the data in different ways, each noticing unique details. These separate observations are then put together into one big picture. To keep this big picture manageable and compatible with the rest of our model, we use a linear layer (trainable weights) to compress it back to our original embedding size.</p><p>Finally, to make sure our model doesn’t just memorize the training data but also gets good at making predictions on new text, we use a dropout layer. This randomly turns off parts of the data during training, which helps the model become more adaptable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mmXsDel2tHKz3kW0GFTjwg.png" /><figcaption>Image by author</figcaption></figure><pre>class MultiHeadAttention(nn.Module):<br>    &quot;&quot;&quot;<br>    This class contains multiple `Head` objects, which perform self-attention <br>    operations in parallel.<br>    &quot;&quot;&quot;<br>    def __init__(self):<br>        super().__init__()<br>        head_size = embed_size // n_heads<br>        heads_list = [Head(head_size) for _ in range(n_heads)]<br>        <br>        self.heads = nn.ModuleList(heads_list)<br>        self.linear = nn.Linear(n_heads * head_size, embed_size)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, x):<br>        heads_list = [h(x) for h in self.heads]<br>        out = torch.cat(heads_list, dim=-1)<br>        out = self.linear(out)<br>        out = self.dropout(out)<br>        return out</pre><h4>5.5. Feed Forward</h4><p>The multi-head attention layer initially captures the contextual relationships within the sequence. More depth is added to the network via two consecutive linear layers, which collectively constitute the feed-forward neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eL-AAKDlBBWbjXoLM2369w.png" /><figcaption>Image by author</figcaption></figure><p>In the <strong>initial linear layer</strong>, we increase the dimensionality (by a factor of 4 in our case) which effectively broadens the network’s capacity to learn and represent more complex features. A <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">ReLU</a> function is applied on each element of the resulting matrix, which enables non-linear pattern be to recognized.</p><p>Subsequently, the<strong> second linear layer</strong> acts as a compressor, reducing the expanded dimensions back to the original shape (block-size x embedding-size). A <strong>dropout layer</strong> concludes the process, randomly deactivating elements of the matrix, for the sake of model generalization.</p><pre>class FeedFoward(nn.Module):<br>    &quot;&quot;&quot;<br>    This module passes the input tensor through a series of linear transformations <br>    and non-linear activations.<br>    &quot;&quot;&quot;<br>    def __init__(self):<br>        super().__init__()<br>        self.net = nn.Sequential(<br>            nn.Linear(embed_size, 4 * embed_size), <br>            nn.ReLU(),<br>            nn.Linear(4 * embed_size, embed_size),<br>            nn.Dropout(dropout),<br>        )<br><br>    def forward(self, x):<br>        return self.net(x)</pre><h4>5.6. Add &amp; Norm</h4><p>Now we link together the multi-head attention and feed-forward component, by introducing two more crucial elements:</p><ul><li><strong>Residual Connections (Add): </strong>These connections perform an element-wise addition of a layer’s output to its unaltered input. During training, the model adjusts the emphasis on layer transformations based on their usefulness. If a transformation is deemed nonessential, its weights and consequently its layer output tend towards zero. In this case at least the unaltered input is passed through the residual connection. This technique helps mitigating the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing gradient problem</a>.</li><li><strong>Layer Normalization (Norm):</strong> This method normalizes each embedded vector in the context by subtracting its mean and dividing by its standard deviation. This process also ensures that the gradients during backpropagation neither explode nor vanish.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WpFTz1PcKoL_kcXWW-FJvw.png" /><figcaption>Image by author</figcaption></figure><p>The chain of multi-head attention and feed-forward layers, linked with “Add &amp; Norm” is consolidated into a block. This modular design allows us to form a sequence of blocks. The number of these blocks is a hyper-parameter, which determines the depth of the model architecture.</p><pre>class Block(nn.Module):<br>    &quot;&quot;&quot;<br>    This module contains a single transformer block, which consists of multi-head <br>    self-attention followed by feed-forward neural networks.<br>    &quot;&quot;&quot;<br>    def __init__(self):<br>        super().__init__()<br><br>        self.sa = MultiHeadAttention()<br>        self.ffwd = FeedFoward()<br>        self.ln1 = nn.LayerNorm(embed_size)<br>        self.ln2 = nn.LayerNorm(embed_size)<br><br>    def forward(self, x):<br>        x = x + self.sa(self.ln1(x))<br>        x = x + self.ffwd(self.ln2(x))<br>        return x</pre><h4>5.7. Softmax</h4><p>Upon traversing multiple block components, we obtain a matrix of dimensions (block-size x embed-size). To reshape this matrix to the required dimensions (block-size x vocab size), we pass it through a final linear layer. This shape represents an entry for each word in the vocabulary at each position in the context.</p><p>Finally, we apply the soft-max transformation to these values, converting them into probabilities. We have successfully obtained a probability distribution for the next token at every position in the context.</p><h3>6. Model Training</h3><p>To train the language model, I selected token sequences from random positions within my training data. Given the fast-paced nature of WhatsApp conversations, I determined a context length of 32 words to be sufficient. Thus, I chose random 32-word chunks as the context input and used the corresponding vectors, shifted by one word, as the targets for comparison.</p><p>The training process was looping through the following steps:</p><ol><li>Sample multiple batches of context.</li><li>Feed these samples into the model to calculate the current loss.</li><li>Apply backpropagation based on the current loss and model weights.</li><li>Evaluate the loss more comprehensively every 500th iteration.</li></ol><p>Once all other model hyperparameters (like embedding size, number of self-attention heads, etc.) were fixed, I finalized a model with 2.5 million parameters. Given my limitations on input data size and computational resources, I found this to be the optimal setup for me.</p><p>The training process took approximately 12 hours for 10,000 iterations. One can see that training could have been stopped earlier, as the spread between the loss on the validation and training sets widens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVu0z3s-9ySpIMgLi9TUXg.png" /><figcaption>Image by author</figcaption></figure><pre>import json<br><br>import torch<br><br>from config import eval_interval, learn_rate, max_iters<br>from src.model import GPTLanguageModel<br>from src.utils import current_time, estimate_loss, get_batch<br><br><br>def model_training(update: bool) -&gt; None:<br>    &quot;&quot;&quot;<br>    Trains or updates a GPTLanguageModel using pre-loaded data.<br><br>    This function either initializes a new model or loads an existing model based<br>    on the `update` parameter. It then trains the model using the AdamW optimizer<br>    on the training and validation data sets. Finally the trained model is saved.<br><br>    :param update: Boolean flag to indicate whether to update an existing model.<br>    &quot;&quot;&quot;<br>    # LOAD DATA -----------------------------------------------------------------<br><br>    train_data = torch.load(&quot;assets/output/train.pt&quot;)<br>    valid_data = torch.load(&quot;assets/output/valid.pt&quot;)<br><br>    with open(&quot;assets/output/vocab.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        vocab = json.loads(f.read())<br><br>    # INITIALIZE / LOAD MODEL ---------------------------------------------------<br><br>    if update:<br>        try:<br>            model = torch.load(&quot;assets/models/model.pt&quot;)<br>            print(&quot;Loaded existing model to continue training.&quot;)<br>        except FileNotFoundError:<br>            print(&quot;No existing model found. Initializing a new model.&quot;)<br>            model = GPTLanguageModel(vocab_size=len(vocab))<br>        <br>    else:<br>        print(&quot;Initializing a new model.&quot;)<br>        model = GPTLanguageModel(vocab_size=len(vocab))<br><br>    # initialize optimizer<br>    optimizer = torch.optim.AdamW(model.parameters(), lr=learn_rate)<br><br>    # number of model parameters<br>    n_params = sum(p.numel() for p in model.parameters())<br>    print(f&quot;Parameters to be optimized: {n_params}\n&quot;, )<br><br>    # MODEL TRAINING ------------------------------------------------------------<br><br>    for i in range(max_iters):<br><br>        # evaluate the loss on train and valid sets every &#39;eval_interval&#39; steps<br>        if i % eval_interval == 0 or i == max_iters - 1:<br>            train_loss = estimate_loss(model, train_data)<br>            valid_loss = estimate_loss(model, valid_data)<br><br>            time = current_time()<br>            print(f&quot;{time} | step {i}: train loss {train_loss:.4f}, valid loss {valid_loss:.4f}&quot;)<br><br>        # sample batch of data<br>        x_batch, y_batch = get_batch(train_data)<br><br>        # evaluate the loss<br>        logits, loss = model(x_batch, y_batch)<br>        optimizer.zero_grad(set_to_none=True)<br>        loss.backward()<br>        optimizer.step()<br><br>    torch.save(model, &quot;assets/models/model.pt&quot;)<br>    print(&quot;Model saved&quot;)</pre><h3>7. Chat-Mode</h3><p>To interact with the trained model, I created a function that allows selecting a contact name via a dropdown menu and inputting a message for the model to respond to. The parameter “n_chats” determines the number of responses the model generates at once. The model concludes a generated message when it predicts the &lt;END&gt; token as the next token.</p><pre>import json<br>import random<br><br>import torch<br>from prompt_toolkit import prompt<br>from prompt_toolkit.completion import WordCompleter<br><br>from config import end_token, n_chats<br>from src.utils import custom_tokenizer, decode, encode, print_delayed<br><br>def conversation() -&gt; None:<br>    &quot;&quot;&quot;<br>    Emulates chat conversations by sampling from a pre-trained GPTLanguageModel.<br><br>    This function loads a trained GPTLanguageModel along with vocabulary and <br>    the list of special tokens. It then enters into a loop where the user specifies <br>    a contact. Given this input, the model generates a sample response. The conversation <br>    continues until the user inputs the end token.<br>    &quot;&quot;&quot;<br>    with open(&quot;assets/output/vocab.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        vocab = json.loads(f.read())<br><br>    with open(&quot;assets/output/contacts.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        contacts = json.loads(f.read())   <br><br>    spec_tokens = contacts + [end_token]<br>    model = torch.load(&quot;assets/models/model.pt&quot;)<br>    completer = WordCompleter(spec_tokens, ignore_case=True)<br>    <br>    input = prompt(&quot;message &gt;&gt; &quot;, completer=completer, default=&quot;&quot;)<br>    output = torch.tensor([], dtype=torch.long)<br>    print()<br><br>    while input != end_token:<br>        for _ in range(n_chats):<br><br>            add_tokens = custom_tokenizer(input, spec_tokens)<br>            add_context = encode(add_tokens, vocab)<br>            context = torch.cat((output, add_context)).unsqueeze(1).T<br><br>            n0 = len(output)<br>            output = model.generate(context, vocab)<br>            n1 = len(output)<br><br>            print_delayed(decode(output[n0-n1:], vocab))<br>            input = random.choice(contacts)<br><br>        input = prompt(&quot;\nresponse &gt;&gt; &quot;, completer=completer, default=&quot;&quot;)<br>        print()</pre><p><strong>Conclusion:</strong></p><p>Due to the privacy of my personal chats, I am unable to present example prompts and conversations here.</p><p>Nonetheless, you can expect a model of this scale to successfully learn the general structure of sentences, producing meaningful outputs in terms of word order. In my case, it also picked up context for certain topics that were prominent in the training data. For instance, as my personal chats often revolve around tennis, the names of tennis players and tennis-related words were typically output together.</p><p>However, when evaluating the coherence of the generated sentences, I concede that the results did not quite meet my already modest expectations. But of course, I could also blame my friends for chatting too much nonsense, limiting the models&#39; ability to learn something useful...</p><p>To show at least some example output at the end, you can see how the dummy model performs on 200 trained dummy messages ;)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*xzdH3i12s9jteq80h3OAGg.png" /><figcaption>Image by author</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31264a9ced90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/build-a-language-model-on-your-whatsapp-chats-31264a9ced90">Build a Language Model on your WhatsApp Chats</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling Traffic Density of the City of Vienna]]></title>
            <link>https://medium.com/data-science/modeling-traffic-density-of-the-city-of-vienna-c41480c35523?source=rss-b2add0f92c53------2</link>
            <guid isPermaLink="false">https://medium.com/p/c41480c35523</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[traffic-optimization]]></category>
            <category><![CDATA[network-analysis]]></category>
            <category><![CDATA[vienna]]></category>
            <dc:creator><![CDATA[Bernhard Pfann, CFA]]></dc:creator>
            <pubDate>Tue, 11 May 2021 14:48:59 GMT</pubDate>
            <atom:updated>2021-05-11T16:51:26.679Z</atom:updated>
            <content:encoded><![CDATA[<h4>Network Analysis / Machine Learning</h4><h4>Connecting official records of the road network with Uber data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jIS9dHFOpMxC9Xxm" /><figcaption>Photo by <a href="https://unsplash.com/@ahmgeek?utm_source=medium&amp;utm_medium=referral">Ahmad Tolba</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a citizen of a larger European city, I have perceived urban area traffic to be quite messy and unpredictable. However, based on personal experience each of us has built up some gut-feeling about certain busy streets and quickest routes through our hometowns, maybe even dependent on the time of the day.</p><p>In this article, I attempt to test these conjectures by analytically deriving the traffic density of the City of Vienna on a granular level. I do this by combining two publicly available sources:</p><ul><li><strong>Road network:</strong> The official records of the road network of the City of Vienna containing ~30,000 road segments, each with its respective geo-location, length, and street type (<a href="https://www.data.gv.at/">https:\\data.gov.at</a>).</li><li><strong>Uber:</strong> Aggregated data on Uber rides, showing travel-time between any combination of sub-districts in Vienna (<a href="https://movement.uber.com/?lang=en-US">https://movement.uber.com</a>).</li></ul><h3><strong>Selected Approach</strong></h3><p>As a first step, the road network enables the simulation of the shortest path between any two points based on the given maximum speed limit on each street. Such a path results in a hypothetical “traffic-free” travel time.</p><p>Next, these “traffic-free” travel times can be contrasted with real-world “traffic-included” travel times sourced from Uber rides. By framing a constrained optimization problem, coefficients representing traffic density at each segment of the network are thereby obtained. The traffic density and average travel speed during different scenarios can be observed as a result.</p><p>The Python implementation which is underlying this article can be accessed and forked from my respective repository.</p><p><a href="https://bernhard-pfann.github.io/">My Project Portfolio</a></p><h3>1. Simulating Paths</h3><h4>1.1. Road Network</h4><p>The official records of the road network contain all street segments and crossings within the city. After a few steps of cleansing, they can be depicted in the form of a DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*6iZYYznY-7Kn5M1arAak8w.png" /><figcaption>Crossings represent “network nodes”</figcaption></figure><p>Since network analysis will be applied, the data is loaded as “nodes” and “edges” into a network graph. The indexes of both DataFrames are identifiers to connect the objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*tGmL43yG0AdKSrybbinj7w.png" /><figcaption>Streets represent “network edges”, that connect nodes</figcaption></figure><p>Within this project, I rely on several functionalities from the NetworkX package, which is designed to analyze complex network graphs. See below the graph initialization of nodes and edges from a DataFrame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4ef63344eabf8a2d06fbf3b0778e6d8/href">https://medium.com/media/c4ef63344eabf8a2d06fbf3b0778e6d8/href</a></iframe><p>By plotting the graph and adding color codes for different street types, the full road network is made visible. The data source distinguishes between three different types of roads: local streets (green), main streets (yellow), and federal streets (red).¹ The street type serves as a proxy for the allowed max. speed limit, although it might not map 1:1 with reality in every case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJayQ5NDV7V2CbkM994rXg.png" /><figcaption>Network graph color-coded by street type</figcaption></figure><p>Since we will, later on, try to find paths from various starting- to ending-points through the network, we have to ensure that all nodes are connected to the main network graph.</p><pre>nodes_main = list(max(nx.connected_components(self.G)))<br>nodes_all  = list(self.G.nodes())<br>nodes_disconnect = set(nodes_all) - set(nodes_main)</pre><p>Pruning the network by the identified disconnected nodes and corresponding edges leaves a fully connected network to be analyzed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjVBYaIrmhnHUkS3dzK4qg.png" /><figcaption>Network graph color-coded by disconnected nodes &amp; edges</figcaption></figure><h4>1.2. Uber Areas</h4><p>Uber splits the City of Vienna into 1370 sub-districts (“areas”) in order to aggregate sensitive data on individual rides. Based on actual rides, the mean travel time between any combination of two districts is being synthetically derived and results in an overwhelming 14 Mio. measurements of daily travel times during Q1 2020. For more details on the aggregation methodology by Uber see the <a href="https://movement.uber.com/_static/76002ded222a46a02ae89f207e91e335.pdf">official manual</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*kLY_5b_YB_hfJMSad3H8fw.png" /><figcaption>Uber travel times during Q1 2020 in Vienna</figcaption></figure><p>Anyways, to connect this data on Uber rides with the introduced road network, each node of the network needs to be mapped to its respective area it lies in. The district boundaries are provided in the form of polygons and can be plotted as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rwtk4YjZIkQOJZwGLRh6rg.png" /><figcaption>Uber areas are segmenting the urban region of Vienna</figcaption></figure><p>By inspecting a single node and iteratively checking with all polygons if the node lies within its boundaries, a mapping for all nodes can be achieved.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f4548b611ecfb9d25119329bb300dba/href">https://medium.com/media/3f4548b611ecfb9d25119329bb300dba/href</a></iframe><h4>1.3. Shortest Paths</h4><p>After the successful mapping of network components to Uber defined areas, the available attributes for each edge have the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*oLhw1DLyD8jxZd1sofl_zg.png" /><figcaption>Attributes of a random network edge stored as a dictionary</figcaption></figure><p>The travel time attribute is a simple division of distance (meters) by speed (km/h) and transformed into seconds. It represents the hypothetical time spent on a street by constantly driving at the maximum speed limit. Thus, it can also be referred to as the “traffic-free” travel time.</p><p>To estimate the shortest path through the network between any of the areas, the NetworkX algorithm looks for a route that minimizes a specified edge attribute. In this case, we minimize the traffic-free travel time.</p><p>Since paths can only be calculated between nodes, and not between areas, a random node from the source- and destination-area needs to be drawn. To avoid noise from unlucky draws, several of these random nodes are tested, while only the path of median travel time is selected as output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a1cde95fa1787c3b897fe26efdc211f/href">https://medium.com/media/8a1cde95fa1787c3b897fe26efdc211f/href</a></iframe><p>A visual representation of such a shortest path can be seen below. While the dark blue line marks the actual route, areas that are being passed through are highlighted in light blue and labeled with their area number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*da7XE1lMl9Q9A7wA1-igIw.png" /></figure><p>Various statistics can be extracted from the chosen path. The relevant metric that will serve as an input for the optimization problem is the meter distance traveled by areas on the respective path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*AcC-ifnD1q39yDLiIfdnrg.png" /><figcaption>Traveled meter distance by areas on the path</figcaption></figure><h3>2. Constrained Optimization</h3><h4>2.1. Identifying Coefficients</h4><p>We are already very close to finding out how traffic looks like in each area of the city. After iterating through all source- and destination-areas from the Uber data set and collecting meter distances for areas on each path, we can set the obtained data matrix into relation with actual travel times from Uber rides.² A split of the DataFrame into X and y variables is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-V7FPmyKaWwRI5hIFQJ1A.png" /><figcaption>Meter distances per area serve as independent variables to estimate mean travel time</figcaption></figure><p>To put the relationship into simple mathematical notation, the actual Uber travel time (y) should equal the sum of meter distances (x) multiplied with area-specific coefficients (beta).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*6u62759gLk3HPSKMfdudAQ.png" /></figure><p>Logic thereby implies, that a higher beta-term of area “j”, allocates more of the total travel time to that specific area, meaning higher traffic density. Also, since the units of measurement for this equation are in seconds (y) and meters (X), the beta-term is expressed as seconds-per-meter. This of course can be transformed into km/h, to give it a real-world meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*ExI3tkHhRS2l5xKpeZ-PbQ.png" /></figure><h4>2.2. Running the Optimization</h4><p>Since the problem is now specified, a model needs to be fit, that minimizes the RMSE for the given function. In order to obtain sensible beta values, the coefficients are limited to stay within fixed boundaries. Translated into units of km/h, these correspond to a bandwidth between 5–120km/h.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8f00ceaefb1472a16c96187d366249b/href">https://medium.com/media/b8f00ceaefb1472a16c96187d366249b/href</a></iframe><p>After 10 iterations a minimum seems to be present at ~130 RMSE. By running the model on an unseen test-set, the RMSE increases only slightly to 134, which indicates the robustness of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3rlSjmPIOEOhy1Du6t2AqA.png" /><figcaption>RMSE of optimization at each iteration</figcaption></figure><p>The 1370 coefficients, which are initialized as a normal distribution centered around 0.2, seem to find a smooth distribution after the 10th iteration as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*QKxlVg3oLIMmbWHnMEVD6A.gif" /><figcaption>Distribution of coefficients at each iteration</figcaption></figure><p>Plotting the coefficients on a map draws a picture of the geospatial traffic density. While brighter areas indicate low traffic, darker areas are associated with more busy streets and slower traffic. For gray areas, none of the simulated paths intersected their boundaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ck2W_OR99BBbiFcrp0ro8w.png" /><figcaption>Traffic density by area</figcaption></figure><p>By interpreting the results, two observations turned out as expected. Firstly, the city center is exposed to the highest traffic volume. This effect is also spreading out to the more western districts, where high population density is coupled with strongly ramified smaller streets. Secondly, the larger brighter areas in the south and along the Danube (main river flowing through the city) map quite well with the existing high-way network, which allows for higher travel speed.</p><h4>2.3. Scenario Analysis</h4><p>While modeling the overall traffic density can already identify hot spots in the city, it might also be insightful to compare the traffic situation at different times of the day or compare the dynamics during weekdays vs. weekends.</p><p>To get scenario results, the Uber rides which train the model are simply filtered for the selected time- or day-category. See below a comparison of the distribution of traffic density coefficients in the early morning (0–7 AM) versus the coefficients during the afternoon (4–7 PM). As expected, the coefficients in the morning are on average lower, corresponding to less traffic and higher average travel speed (as shown in the lower subplot).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPuq3c6LBkuKZn2zTyv0Yg.png" /><figcaption>Distribution of coefficients and speed per each for different scenarios</figcaption></figure><p>A plot of the coefficient differences between the two scenarios on a map supports this finding in a spatial dimension as well since red areas mean higher traffic density in the afternoon compared to the early morning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuIfZFEqX5LCdI74_ZM-Bg.png" /><figcaption>Differences in traffic density (morning vs. afternoon)</figcaption></figure><h3>3. Final Considerations</h3><p>With the presented approach we were able to combine two different publicly available data sources, by mapping the street network to the “Uber-defined” segmentation of the city. This enabled us to enrich the source- and destination areas of each Uber ride with information about the path during the ride. By fitting a model that puts the traveled path in relation to the travel time, sensible estimators of traffic density were derived.</p><p>However, building on this, various extensions come to mind, that could make insights more actionable:</p><ul><li>Including data on one-way streets and traffic lights into the network (at least the later one is publicly available)</li><li>Introducing an artificial traffic jam, to observe its implications on alternative routes</li><li>Estimating traffic density coefficients for each street segment</li><li>Comparing traffic dynamics pre-Covid vs. post-Covid</li></ul><p>If you find any other interesting applications or you want to rebuild the project for your own hometown, please feel free to make use of my work. You find the source code in my respective GitHub repository.</p><p><a href="https://bernhard-pfann.github.io/">My Project Portfolio</a></p><p>[1] Loosely translated from German</p><p>[2] Due to limited computational capacity, only 100,000 out of 14 Mio. paths have been simulated</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c41480c35523" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/modeling-traffic-density-of-the-city-of-vienna-c41480c35523">Modeling Traffic Density of the City of Vienna</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decomposing & Predicting the Euro Yield Curve]]></title>
            <link>https://medium.com/data-science/decomposing-predicting-the-euro-yield-curve-b3ad1670fdbb?source=rss-b2add0f92c53------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3ad1670fdbb</guid>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[yield-curve]]></category>
            <category><![CDATA[financial-modelling]]></category>
            <category><![CDATA[financial-risk-management]]></category>
            <dc:creator><![CDATA[Bernhard Pfann, CFA]]></dc:creator>
            <pubDate>Sat, 20 Feb 2021 23:26:13 GMT</pubDate>
            <atom:updated>2021-02-21T15:03:14.633Z</atom:updated>
            <content:encoded><![CDATA[<h4>Applying Principal Component Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ag9o4xKyg-T-DOIz" /><figcaption>Photo by <a href="https://unsplash.com/@pluyar?utm_source=medium&amp;utm_medium=referral">Shane Aldendorff</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The term structure of interest rates (“yield curve”) is a representation that plots bonds of the same type (e.g. credit quality, sector) in terms of their prices, expressed as yields, over different maturity dates. For financial institutions is it crucial to understand its behavior and the direct implications it has on the firms capital base.</p><p>In this article I study the underlying dynamics of the yield curve by decomposing it into its main drivers. Therefore I will apply Principal Component Analysis (PCA), as proposed in several academic papers. The following topics will be explored:</p><ol><li>Introduce the data</li><li>Interpret the main drivers of variance (principal components) in contrast to traditional yield curve factors “level”, ”slope” and “curvature”</li><li>Evaluate the fit of generated model yield curves from a reduced set of dimensions against actual yields (in-sample vs. out-of-sample)</li><li>Derive non-linear stress scenarios for 1-month ahead</li><li>Test predictability of the main principal components with an autoregressive time series model against a benchmark</li></ol><p>The Python implementation of the described ideas can be accessed and forked from my respective repository.</p><p><a href="https://bernhard-pfann.github.io/">My Project Portfolio</a></p><p><strong><em>Note from Towards Data Science’s editors:</em></strong><em> While we allow independent authors to publish articles in accordance with our </em><a href="https://towardsdatascience.com/questions-96667b06af5"><em>rules and guidelines</em></a><em>, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our </em><a href="https://towardsdatascience.com/readers-terms-b5d780a700a4"><em>Reader Terms</em></a><em> for details.</em></p><h3>1. Introducing the Data</h3><p>The underlying data represents spot rates of Euro Area AAA-rated government bonds on a daily basis. The selected time horizon spans from Jan-2005 to Dec-2020 and can be freely accessed <a href="https://www.ecb.europa.eu/stats/financial_markets_and_interest_rates/euro_area_yield_curves/html/index.en.html">here</a>.</p><p>After cleaning &amp; filtering the data for the relevant information, yield curves for different points in time can be plotted. The chart below makes a decline in level as well as steepness quite prominent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*80FFVAyFHJQOoKu9dJbIaA.png" /></figure><p>Inspecting the yield curve shifts in more granular time steps, uncovers the volatile dynamics in level, slope and curvature underneath macroeconomic long-term drifts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*mq_2HTm1dcKcX20HOUKF6Q.gif" /><figcaption>Chart 1: Actual yield curve vs. one-year lagged version | Chart 2: Difference between depicted yield curves</figcaption></figure><p>In order to better understand what drives each point on the curve, the volatility, as well as the varying correlation between maturities needs to be assessed. Consequently a yield curve that is formed of spot rates from 30 different maturities, would require a 30x30 covariance matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*EGiMrozrCF79y_9qhg-61Q.png" /><figcaption>Rolling volatility of yields for different maturities</figcaption></figure><h3>2. Interpretation of Principal Components</h3><p>Building statistical models with such a large number of parameters, increases model complexity as well as uncertainty stemming from spurious inputs. Principal Component Analysis (PCA) is a well-known technique to reduce this high dimensionality by compressing the variability of all features into a limited set of transformed features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*kZZ9QpvEEi5OdzODEAivJg.png" /><figcaption>Input data matrix, where each “x” represents the “n-th“ observations of feature “f”</figcaption></figure><p>In this case, each feature of X is a vector of daily yields for a given maturity. In order to capture the maximum variability of all maturities, Eigenvectors are derived. These vectors are based on the covariance matrix of X and minimize the sum of distances when being projected onto the vector itself. For reference see a more in-depth explanation of the topic in the following video...</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPFDu9oVAE-g%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPFDu9oVAE-g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPFDu9oVAE-g%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/580d87668dd96a73d942395581bdea72/href">https://medium.com/media/580d87668dd96a73d942395581bdea72/href</a></iframe><p>The Eigenvalue decomposition results in the same number of vectors, as the initial data matrix has. However, the vectors are ordered by importance, so that only the most important ones need to be retained for further analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*IpSJnmM4Wnhjv8WrZvYRvA.png" /><figcaption>Each Eigenvector “w” states vector directions into all dimensions. Lambda states the importance of each w.</figcaption></figure><p>When plotting the first 3 Eigenvectors, the pronunciation of specific maturities can be observed, which gives an initial intuition about the vector’s economic meaning (e.g. PC2 captures yield curve shifts with divergent directions in short- and long-term maturities)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*7369WBd34QchXWgNmOTLtQ.png" /><figcaption>Eigenvectors for the first 3 PCs</figcaption></figure><p>After retaining only the first 3 Eigenvectors, a simple matrix multiplication with the original yield curves, transforms each yield curve into a data point of 3 dimensions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*X6zG-cqoE_RVuq50JsLqMQ.png" /><figcaption>Matrix-multiplication of input matrix “X” with retained Eigenvectors “W” results in transformed Scores “T”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*M3dDNTKt4a_gCaZSVg_SIg.png" /><figcaption>Each point represents a full yield curve transformed into 2-dimensional space</figcaption></figure><p>It can be observed that a high value in PC1 corresponds to a generally low level of interest rates (violet/blue points in 2019/20) and vice versa. Also the increased steepness of the yield curve is tracked by PC2 (lime green/orange points in 2009/10).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*3vAYOeLkCWsu50RCrTv4Sw.gif" /><figcaption>Chart 1: Actual yield curve | Chart 2: Yield curve in reduced dimensionality</figcaption></figure><p>To finalize my conclusion about the interpretability of the derived Eigenscores (Principal Components), each can be compared to a proxy of the traditional factors “level”, “slope” and “curvature”. These are usually observed as some kind of spread between maturities (e.g. 10Y-6M spread as proxy for steepness)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*DFXXCJZQZU37bqtGAWK1_A.png" /></figure><p>In order to model and predict future behavior of the yield curve, the derived scores need to be back-transformed into meaningful original units. This is done by multiplying the Eigenscores with the inverse of Eigenvectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*XPPlPO16IxyixY1DdG4P-Q.png" /><figcaption>Matrix multiplication of Scores “t” with the inverse of the Eigenvector matrix “w” results in generated model yield curves</figcaption></figure><p>The resulting matrix, represents generated model yield curves, where each curve can be described by only 3 factors.</p><h3>3. Goodness of Fit</h3><p>To see if model yield curves generated from low-dimensionality are actually helpful, it is necessary of evaluate their fit against the actually realized curves. A direct comparison of these curves is depicted below, together with the squared deviations at each point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*w3We9MTDvGZorEPl1XsFKA.gif" /><figcaption>Evaluation of goodness-of-fit for model yield curves based on 2 &amp; 3 dimensions.</figcaption></figure><p>It is obvious that the model-curves based on 3-dimenions are closer to the actual yield curves, than the 2-dimensional ones, since a compression into less dimensions always implies some loss of information.</p><p>For predictive modeling it is furthermore important to assess the out-of-sample fit, since Eigenvectors can only be derived ex-post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*QNgqKr1TPE3uQSxr43wvHQ.png" /><figcaption>RMSE of differences between 3-dimensional model vs. actual curves.</figcaption></figure><p>Unsurprisingly the out-of-sample fit shows an increased RMSE, compared to the yield curve model that is fitted over the whole time horizon. However the yield curve shape is the overlaying driver for overall RMSE.</p><h3>4. Deriving Stress Scenarios</h3><p>The assessment of stress scenarios and their implications on the capital base are an integral part of the financial risk management function in banks and insurances. Its simplest form are parallel upward-, or downward shocks.</p><p>However since the explored 3 PCs’ capture the yield curve variability that well, stressing the components directly can give realistic non-linear stress scenarios. The intuitive historical Value at Risk concept can be applied, to find maximum upward-, and downward changes for a selected rolling time window. By considering the 5% largest deviations within that window, a 95% confidence interval can be established.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*pJisw27vC5OFZ0y7-Nf5nA.png" /></figure><p>Again, the back-transformation of Eigenscores gives model yield curves, which in this case are the stressed curves. Consistent to the proven economic interpretation of each PC, a shock in level, slope and curvature is the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*xVKIbdO8BrsOBGJwMWVqYA.png" /></figure><h3>5. Yield Curve Prediction</h3><p>The most-interesting question in almost every financial modeling task is concerning its potential for making predictions about the future. In this case I propose a simple Autoregressive (AR) model to predict each vector of Eigenscores separately by its own lags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*QRAF-GTI8gPa7qkfZGXcog.png" /><figcaption>Autoregressive model with n-lags</figcaption></figure><p>By definition the one-step ahead forecast of x shall be a dot-product of its own lags and the fitted corresponding coefficients. To fully specify model parameters, a train &amp; test horizon as well as the maximum number of considered lags is defined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*LkddbivAjLzI-Bw9-ExJiw.png" /></figure><p>Additionally a valid time series model requires the stationarity in mean, variance and covariance of the underlying data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*--ZVErHzfA6GQN15Cc2oYg.png" /><figcaption>Eigenscores over time</figcaption></figure><p>It is clearly observable that the first two vectors of Eigenscores are clearly non-stationary (also supported by Augmented Dickey-Fuller test). Therefore a transformation of the time series into first differences is necessary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*sW56M0NLuI-UhDoCsXyfMA.png" /><figcaption>First differences Eigenscores over time</figcaption></figure><p>After fulfilling these relevant specifications and transformations, the actual model fit can be assessed. It seems promising that the first lag of each component shows statistical significant (p-value &gt; 0.05), even though the coefficients size is only marginal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*N0czwaIdWHanRwue8Sq4qA.png" /></figure><p>To evaluate how well the model predictions perform, its errors need to be compared to a benchmark model. Its most commonly used implementation is a naive forecaster, that predicts the value in t+1 to be equal to the value in the previous period t+0.</p><p>Comparing the prediction errors of both models makes clear, that the AR-model almost resembles the benchmark model due to its small coefficient values close to 0. However the small but significant first-lag causes a marginal out-performance in RMSE over the whole test data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*1Ufc3e3vaPhne0yOit9lzw.png" /><figcaption>Prediction errors as RMSE of all maturities against actually realized yield curves</figcaption></figure><h3>Conclusion</h3><p>Altogether, the application of PCA helps to decompose the seemingly high-dimensional yield curve data, into its few main drivers. With this limited set of dimensions, that also holds an economic meaning, causality and explainabilty of financial models can be improved. The derivation of stress scenarios or one-step-ahead forecasts, are use cases in financial risk management or portfolio construction that can be further extended.</p><p>In case you want to build up on my work, feel free to check out the respective Github repository.</p><p><a href="https://bernhard-pfann.github.io/">My Project Portfolio</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3ad1670fdbb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/decomposing-predicting-the-euro-yield-curve-b3ad1670fdbb">Decomposing &amp; Predicting the Euro Yield Curve</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tackling UNO Card Game with Reinforcement Learning]]></title>
            <link>https://medium.com/data-science/tackling-uno-card-game-with-reinforcement-learning-fad2fc19355c?source=rss-b2add0f92c53------2</link>
            <guid isPermaLink="false">https://medium.com/p/fad2fc19355c</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[card-game]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[uno]]></category>
            <dc:creator><![CDATA[Bernhard Pfann, CFA]]></dc:creator>
            <pubDate>Fri, 22 Jan 2021 19:17:55 GMT</pubDate>
            <atom:updated>2021-01-23T23:50:06.292Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tackling the UNO Card Game with Reinforcement Learning</h3><h4>A complete walk-through starting from scratch</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_a9iQ4WGX0z0YYvV" /><figcaption>Photo by <a href="https://unsplash.com/@thirdserving?utm_source=medium&amp;utm_medium=referral">Third Serving</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>During quarantine times like these, the classic UNO card game emerged to be one of the most frequently played games between my girlfriend and me. After countless rounds and several major winning streaks on each side, I was curious if an optimal game strategy could be analytically derived, that would significantly outperform in the long-run.</p><p><strong>My journey consisted of the following parts:</strong></p><ol><li>Creating a game engine of the UNO card game in Python from scratch</li><li>Obtaining game statistics from simulating a series of 100,000 games</li><li>Implementing basic Reinforcement Learning techniques in order to discover an optimal game strategy</li></ol><h3>1. UNO Card Game Engine</h3><p>In order to train a Reinforcement Learning (RL) agent how to play intelligently, a fully-fledged game environment needs to be in place. It has to capture all the mechanics and rules so that the agent can interact with the game like a real human player would do.</p><p>The object-oriented nature of Python allows structuring the code intuitively into class objects for cards, deck, player, turn, and game. Here are the main ideas…</p><h4>1.1 Card</h4><p>The most granular element of the game is the card itself, which has a “color” and “value” property. A card can also be evaluated if it is playable or not, dependent on the currently open card.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b3c86207294be0979f940e5306c1af8/href">https://medium.com/media/1b3c86207294be0979f940e5306c1af8/href</a></iframe><h4>1.2 Deck</h4><p>A fixed list of UNO cards represents the card deck. It is initialized and shuffled at the beginning of each game. Cards can be drawn from it and a card refill is implemented, in case the deck runs out of cards.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/314a34ff79a8076d8bdcbda63c197151/href">https://medium.com/media/314a34ff79a8076d8bdcbda63c197151/href</a></iframe><h4>1.3 Player</h4><p>A player has a <em>name</em>, <em>hand cards,</em> and <em>playable hand cards</em> as properties and needs to perform a variety of functions. He has to:</p><ul><li>Identify the current <em>state</em> he is in (cards he holds etc.)</li><li>Recognize the possible <em>actions</em> he can take (cards he can play)</li><li>Incorporate the decision making logic from the RL-agent</li></ul><h4>1.4 Turn</h4><p>This class clumps together a players’ action sequence during a turn. After the initialization which distributes the 7 starting cards to each player, the active player (the player whose turn it is) either draws or plays a card. In this setting, <em>player_1</em> is utilizing the RL-agent, while <em>player_2</em> follows a random strategy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44116dfb47883e0448ea630426d707c5/href">https://medium.com/media/44116dfb47883e0448ea630426d707c5/href</a></iframe><h4>1.5 Game</h4><p>Finally, a card game comprises of nothing else than a series of turns. Therefore an iteration of turns is executed until any of the players meets the winning condition (holding no hand cards).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3d30bbc2b348dce4560492e7998e3e9/href">https://medium.com/media/c3d30bbc2b348dce4560492e7998e3e9/href</a></iframe><p>The depicted code snippets try to convey the basic game mechanism. The full script can be accessed on my Github repo <a href="https://github.com/bernhard-pfann/uno-card-game_rl">https://github.com/bernhard-pfann/uno-card-game_rl</a>.</p><h3>2. Game Statistics from Simulations</h3><p>With a fully functioning game engine, we are now able to run as many simulations of the game, as we like. This is helpful to get insights into the statistical features of the game, but it also facilitates the training of an RL-agent. I was mainly interested in the following questions:</p><ul><li>How many turns do games last?</li><li>How big is the advantage of the player making the first turn?</li><li>What are the most likely situations in the course of a game?</li></ul><h4>2.1 Game Length</h4><p>Within the generated sample of 100,000 simulations, the average game length is 41 turns. The significantly lower mode of 13 turns results from the right skewness (the portion of games with more than 100 turns).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*AZftOsPtfxjSOA3hzJmXfg.png" /><figcaption>Distribution of turns per game from 100,000 simulated games</figcaption></figure><p>The longest game in the sample lasted 327 turns, while the shortest only took 3 turns, which is only possible through the combination of multiple special cards (<em>Skip, Reverse,</em> <em>+2, +4</em>).</p><h4>2.2 Advantage of the First Mover</h4><p>A player who starts the game has a certain edge over his opponent. To quantify this advantage, the cumulative long-run winning-rate of two randomly playing agents is observed in the following two scenarios:</p><ol><li>Player 1 always gets to make the first move (red simulations)</li><li>Player 1 gets to make the first move every 2nd game (violet simulations)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*I2f5IsKeQzXlQKjqNyvtfA.png" /><figcaption>Cumulative Win-Rate from simulated 30,000 games</figcaption></figure><p>As expected, the winning rate of two randomly playing agents converges to 50% in a fair set-up with alternating first movers. This is an indication that the game engine works as expected.</p><p>More interesting is the fact that the player who always makes the first move consistently wins 52%<strong> </strong>of all games.<strong> </strong>This corresponds to a ca. 4% (52% vs. 48%) higher likelihood to win against the opponent.</p><h4>2.3 The Course of Play</h4><p>Like in sports analytics, a heatmap can be used to display where the course of play had its focus. By keeping track of every state and action a player has experienced, we can identify the most likely situations in the game.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*-NvZlkF4gd-cosCQF9xQuw.png" /><figcaption>Number of state-action occurrences during 100,000 simulated games</figcaption></figure><p>The axes of the heatmap denote a players’ number of hand cards, as well as the action taken at the respective point in time. It stands out, that most of the time players are lingering in the “mid-game” by holding 4–5 cards. It makes sense that normal cards (0–9) from all colors are played the most often since they are the most common cards in a deck.</p><h3>3. Application of Reinforcement Learning</h3><p>Searching for the optimal strategy in the UNO card game is a classical use case for Reinforcement Learning. Thereby the game itself can be framed as a finite Markov Decision Process (MDP), which implies the following characteristics:</p><ul><li><strong>States:</strong> Each step of the game can be described by a state</li><li><strong>Actions:</strong> The decision-maker interacts with the game by taking actions<strong> </strong>based on the state he is in</li><li><strong>Reward:</strong> Taking certain actions can lead to a desirable terminal state (e.g winning the game), which is rewarded</li></ul><p>The stochastic element, which is inherent through the randomly drawn cards as well as the opponents’ moves, require numerous simulations, to identify a long-run optimum. The basic techniques, which I applied are Monte Carlo and Q-Learning with a discrete state-action matrix.</p><h4>3.1 Defining State-Space</h4><p>In a supervised machine learning set-up, a function is fitted to map the available features to the output variable. RL on the other hand evaluates each action sequentially and individually at each step. Therefore states, actions, and rewards need to be defined.</p><p>A state can represent any information available to the decision-maker, that is useful to describe the current situation of the game. This could be the type of cards he holds, the number of cards the opponent holds, or information regarding cards that have already been played.</p><p>At first sight, UNO might appear to be a very simplistic card game, due to its limited set of rules. However, when figuring out the possible combinations of cards a player can hold, it quickly gets out of hand. To be precise there are about 10¹⁶³ combinations. Since the RL-agent has to learn an optimal action for each state, it makes sense to limit the number of states.</p><p>Thus, I formed a 16-digit state identification, that captures which cards the agent holds and which he can play, differentiating only between colors, and special cards (skip, reverse, etc.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*w2mUiihNrQAf5k3oknCm4w.png" /><figcaption>Identification of a state</figcaption></figure><p>By capping each property of the state-identification with a maximum value of 2, state-space has been successfully limited to 270,000 possibilities.</p><h4>3.2 Possible Actions</h4><p>The cards, that the agent decides to play, represent the actions he is taking. Applying the same aggregation logic as above results in 9 different actions: <em>Red, Green, Blue, Yellow, Skip, Reverse, +2, +4, Wild Color</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*llyT05aRWYiJNUbrj0Atgg.png" /><figcaption>Q-table initialized at zero</figcaption></figure><h4>3.3 Monte Carlo Agent</h4><p>Given the discrete state-action matrix, the agent navigates through the fields by simulating multiple games. While the matrix is initialized with all values at zero, Monte Carlo (MC) updates all visited state-action values after every completed game.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*hfvIf2i-ielftHizMqtqqA.png" /><figcaption>q-value update function</figcaption></figure><p>The q-value at state <em>s</em> taking action <em>a</em> is updated dependent on the achieved reward in this episode <em>R</em>, and the step size parameter <em>alpha</em>. In order to decide which action to take in a respective state, the epsilon-greedy form of the algorithm chooses:</p><ul><li><strong>With epsilon probability:</strong> Random action</li><li><strong>With 1-epsilon probability:</strong> Action with maximum q-values</li></ul><h4>3.4 Q-Learning</h4><p>In its basic form, Q-learning works in a similar way. However, while MC waits for the completion of each episode before updating q-values, Q-learning updates them with a lag of one step, at each step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*sXCFPqnY2LAj4SwfKJ4MHg.png" /><figcaption>q-value update function</figcaption></figure><p>The q-value is thereby dependent on the step-size parameter, the reward of the next step <em>r,</em> and the q-value of the next step at state <em>s-hat</em> and <em>action-hat</em>.</p><p>Both algorithms consequently take the same 2 parameters which have the following effects:</p><ul><li><strong>Alpha:</strong> A higher step size parameter increases the change in q-values at each update while prohibiting values to converge closer to their true optimum</li><li><strong>Epsilon:</strong> A higher epsilon grants more exploration of actions, which do not appear profitable at first sight. At the same time, this dilutes the optimal game strategy when it has been picked up by the agent.</li></ul><h3>4. Results</h3><p>Now that we have specified the models, let us see if they are useful. Therefore games are simulated with one random player and the other one leveraging the RL-algorithm. As an evaluation criterion, I use the cumulative long-run winning rate of the RL-player.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*LSKvFAy9B2PfY8hZEM4_Hw.png" /><figcaption>Cumulative Win-Rate from 100,000 simulated games</figcaption></figure><p>After running 100,000 simulations for each of the two RL-models, a consistent outperformance over the random agent can be testified. Given the enormous number of run episodes, statistical significance can be ensured, even though the margin of outperformance seems to be limited.</p><p>A win-rate of 51.5% means, that the RL-algorithm is 3% more likely to win a game against a random player.</p><h4>4.1 Q-Table</h4><p>To see how well the algorithms have learned the game, a closer inspection of the Q-table can be helpful. Since the full table is of size 270,000 x 8, states have been aggregated by the number of hand cards.</p><p>The steadily declining violet curve depicts the fact that states with fewer hand cards are closer to win a game. Therefore these states carry higher Q-values, which helps the RL-agent to choose actions that reduce their number of hand cards as fast as possible, in order to achieve a reward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*qNulaxsbiJ0lB-oKo51tIQ.png" /><figcaption>Aggregated Q-Values from Q-Learning Model</figcaption></figure><h4>4.2 State-Action Coverage</h4><p>To judge if the models have been trained on enough simulations, the state-action coverage can be assessed. Therefore the different state-action pairs that have already occurred during 100,000 games can be counted.</p><p>In total, both models have seen ca. 60,000 different combinations. At first sight, this is lower than expected, since we now know that 100,000 games have had on average 41 turns (41 x 100,000 = 4.1 Mio. turns). However, we also showcased how concentrated the course of play is towards certain card combinations, that appear multiple times within the sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*gMqpDSeOdRLFJt0n1ge2eg.png" /><figcaption>State-Action Coverage during 100,000 simulated games</figcaption></figure><p>The quicker initial coverage increase of the Monte-Carlo model does not come by surprise, due to the nature of its update function.</p><h3>5. Conclusion</h3><p>Even though the RL-models are able to consistently outperform a random player by some margin, it remains unclear how the performance would turn out against a real human player. There are several limitations that are hindering the algorithm from achieving more impressive results:</p><ul><li><strong>There are only a few moves in each game, that require strategic evaluation</strong>, since most of the time, players do not hold more than one playable card in their hand. In only ca. 32% of playable turns, players have multiple options available, where intelligent behavior could lead to outperformance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*hhVCEKNFEP3YHszV3qKxLg.png" /></figure><ul><li><strong>Not all relevant state information has been included </strong>since this simple form of RL with a discrete state-action matrix only works for a limited set of states. Information about the number of cards an opponent holds, or the color distribution of already played cards could be helpful in order to fine-tune a game strategy.</li><li><strong>Finally, it is a game of luck!</strong> Compared to other board games like Monopoly or Chess, UNO brings in a large amount of stochasticity, which diminishes the potential for strategic supremacy.</li></ul><p>The bottom line is that I will continue to trust my gut feeling when being challenged in the next game of UNO. My showcased work is available in my Github repository, where anybody more acquainted with advanced RL-methods can pick it up from here.</p><p><a href="https://github.com/bernhard-pfann/uno-card-game_rl">bernhard-pfann/uno-card-game_rl</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fad2fc19355c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/tackling-uno-card-game-with-reinforcement-learning-fad2fc19355c">Tackling UNO Card Game with Reinforcement Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>