<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Miguel Romero Calvo on Medium]]></title>
        <description><![CDATA[Stories by Miguel Romero Calvo on Medium]]></description>
        <link>https://medium.com/@mromerocalvo?source=rss-4e4973afd99------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3hQBk5ZsFUiC0O1k_kCRSw.jpeg</url>
            <title>Stories by Miguel Romero Calvo on Medium</title>
            <link>https://medium.com/@mromerocalvo?source=rss-4e4973afd99------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:56:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mromerocalvo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Dissecting Transformer-XL]]></title>
            <link>https://medium.com/@mromerocalvo/dissecting-transformer-xl-90963e274bd7?source=rss-4e4973afd99------2</link>
            <guid isPermaLink="false">https://medium.com/p/90963e274bd7</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Miguel Romero Calvo]]></dc:creator>
            <pubDate>Sun, 23 Jun 2019 05:31:21 GMT</pubDate>
            <atom:updated>2019-06-23T09:29:05.135Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*633HlVQwSSlvnRwO1cyjmQ.png" /></figure><p>In this blog post, the mechanism used to develop Transformer-XL will be explained. This is done by explaining and providing the intuition behind the changes that were necessary to build this architecture upon the Transformer encoder. We will start where we left when <a href="https://medium.com/dissecting-bert/dissecting-bert-part-1-d3c3d495cdb3">explaining the Transformer</a>.</p><p>The Transformer-XL is built upon the Transformer an introduces to major changes. This blog-post will is divided into 3 main sections to reach a wider range of readers. Depending on your familiarity with the topic you may want to jump directly to a specific section.</p><ol><li><strong><em>The problem with Transformer, why Transformer-XL?: </em></strong><em>The motivation for Transformer-XL.</em></li><li><strong>Transformer-XL: the proposed solution: </strong>Basic idea.</li><li><strong>Absolute Positional Encoding &amp; Memory: </strong>Technical problems with the proposed solution and workaround.</li></ol><h3>The problem with Transformer, why Transformer-XL?</h3><p>While <strong>the Transformer</strong> was originally built for Translation tasks, <strong>recent trends</strong> indicate that its application on Language Modeling can yield significant improvements. However, the architecture needs to be slightly modified for an optimal application.</p><p>Why? is there any problem with it? A major improvement of the Transformer over RNN is its ability to naturally capture long-term dependencies. However, the number of intermediate steps (gradients) that need to be stored is higher than those in most RNN and it increases as the sequence length increases.</p><p>From a more visual perspective, if we take a look at the matrices in the attention section of the transformer blog-post the &quot;height&quot; and &quot;width&quot; of those will increase for every new word that we are considering and all those intermediate numbers need to be stored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XPhY6qWClZKD4g7QC8mjmA.png" /></figure><p>This will happen for all the matrices for all the encoder blocks and we need to store the gradients to efficiently compute backpropagation later on.</p><p>In other words, if you try to fit a whole document at once, as would be ideal for that architecture, your computer may explode (BOOOOM!💥) or you can get this lovely message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjLTq2f9ceL0TCAH3i7A3Q.jpeg" /></figure><p>To prevent this issue an early approach was to split the documents into <strong>fixed-size</strong> segments (i.e. chunks) segment_len of text and train the model with one segment at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0B4xRDeXNKJQ5JGZX2EaPw.png" /></figure><p>This solves the memory problem but it would kill the ability of the model to capture long-term dependencies. A word at a specific segment will only be allowed to attend other words in the same segments but nothing from previous segments. This problem is called <em>context fragmentation</em>.</p><p>For example in the sentence:</p><blockquote>The daughter had a nice umbrella that her mother gave her.</blockquote><p>if &quot;<em>daughter&quot; </em>and<em> &quot;her&quot; </em>fall in different segments</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5yu-sgoV8yMxUcpKJGnFw.png" /></figure><p>the encoded representation (i.e. learned vector) for the word “<em>her”</em> won’t be able to incorporate the information from the word “<em>daughter” </em>what makes it extremely difficult for the model to guess that the next word is more likely to be “<em>mother”</em> or “<em>father”</em>.</p><h3>Transformer-XL: the proposed solution</h3><p>Let&#39;s introduce some kind of &quot;memory&quot; between segments. Under that first approach, we were training our model by processing one segment at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5qVjdleJWjKe2T9Z5t1sg.png" /></figure><p>Ideally, we would like to have some kind of &quot;memory&quot; those segments encoding information from the previous segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0T7c3gdgKywI1w0m9ZEEA.png" /></figure><p>But, how can we go about doing that? What would be a way of making sure that information from previous segments is properly passed to the next one? Let&#39;s dig a little bit more into that &quot;Model&quot;, more specifically remember that this is composed by an Embedding layer, a positional encoding layer and a series of sequential encoder blocks following the notation from the <strong>previous blog-post </strong>we have this high-level schema:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*P7ilTfb48TE1Jp-v-ZPlLQ.png" /></figure><p>Where 1 and 2 are referring to the embedding and positional encoding layers respectively. The way the authors go about transferring the memory is passing along the hidden states after each block. Visually:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3rxi7H06Ir-q_W2Q2zSIg.png" /></figure><p>In terms of pseudo-code, if we call h_{t,i}<em> </em>the output from the i-th encoder block when computing segment t the procedures can be roughly summarized in the following steps.</p><pre>&gt; For encoder i in all encoder blocks except the last one do:</pre><pre>&gt; &gt; Set h_{-1,i } to a tensor of zeros such that the size/shape matches the hidden states in upcoming segments.*</pre><pre><strong>&gt; When computing segment 0:</strong></pre><pre>&gt; &gt;  For encoder i in all encoder blocks except the first one do:</pre><pre>&gt; &gt; &gt; Combine the saved hidden states: h_{-1,i-1} and h_{0,i-1}.</pre><pre>&gt; &gt;  For encoder i in all encoder blocks except the last one do:</pre><pre>&gt; &gt; &gt; Make a copy of its output h_{0,i }(hidden state).</pre><pre><strong>&gt; When computing segment 1:</strong></pre><pre>&gt; &gt; For encoder i in all encoder blocks except the first one do:</pre><pre>&gt; &gt; &gt; Combine the saved hidden states: h_{0,i-1} and h_{1,i-1}.</pre><pre>&gt; &gt; For encoder i in all encoder blocks except the last one do:</pre><pre>&gt; &gt; &gt; Make a copy of its output h_{1,i }(hidden state).</pre><pre><strong>…</strong></pre><pre><strong>&gt; When computing segment t:</strong></pre><pre>&gt; &gt; For encoder i in all encoder blocks except the first one do:</pre><pre>&gt; &gt; &gt; Combine the saved hidden states: h_{t-1,i-1} and h_{t,i-1}.</pre><pre>&gt; &gt; For encoder i in all encoder blocks except the last one do:</pre><pre>&gt; &gt; &gt; Make a copy of its output h_{t,i }(hidden state).</pre><pre>* This shape will be (batch_size, segment_len, emb_dim).</pre><p>Now let&#39;s dig into on how are those combined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*TUlFNALGuLylkV6HJg557Q.png" /></figure><p>It is nothing fancy, both hidden states will be concatenated through the segment_dim . This is, imagine we have two tensors, h_{t-1,i} and h_{t,i} , each of them with shapes (batch_size, segment_len, emb_dim). Then, we will be combining them through the segment_dim resulting in one tensor of shape (batch_size, 2*segment_len, emb_dim). Let&#39;s illustrate that with an example in the sentence above leaving aside thebatch_dim.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPt7gPA2JO7mOGaJvov5Qg.png" /></figure><p>We call the result of this operation cat_h_{t,i} processing segment t and &quot;entering&quot; block i .</p><p>An observation worth mentioning is that during training we <strong>deactivate backpropagation </strong>for the portion of the <strong>memory</strong>. Otherwise, we will still have the original problem regarding computational resources. Practically, one way to do that in PyTorch is setting the .requires_grad=False for a deep copy of the previous hidden states.</p><p>A fact worth mentioning is that the encoder block i &#39;s inputs are both h_{t,i} and cat_h_{t,i} both of those will play a critical role during attention. We will be using the notation from the <strong>original paper </strong>and the <strong>previous blog post (go to that last one if you need a deeper review)</strong>.</p><blockquote><strong>QUICK REMINDER OF SELF-ATTENTION</strong></blockquote><blockquote>Let&#39;s quickly remember a big picture of self-attention. The formula is as follows:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/1*QKx8cEs2JIF9ZrXhbdZlAA.png" /></figure><blockquote>The juicy part of the formula, where the distribution of attention for a specific word through a set of other words happens, is:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*a9o18gftLzt-pwg4qryCJw.png" /></figure><blockquote>For a sentence such as &quot;Hello, how are you?&quot; this will look as:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*QDMedAr4D-GS78kMyWUmfg.png" /></figure><blockquote>And the way to read this is:</blockquote><blockquote>The representation of &quot;how&quot; is paying 99.9%* of attention to the representation of &quot;?&quot;</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lWIu8wd164yvVP6XSAvT8w.png" /></figure><blockquote>* 9.99•10^(-1) over 1</blockquote><p>Borrowing the notation from the attention section of the <a href="https://medium.com/dissecting-bert/dissecting-bert-part-1-d3c3d495cdb3">transformer blog post</a>, the query, key and value matrices (Q, K, V) for a specific head of attention will be computed using h_{t,i} for Q and cat_h_{t,i} for K and V. This will yield to the following shapes for those matrices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*dJOJfKpyhXpenaZFBIR3Cw.png" /></figure><p>In this case the term:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*a9o18gftLzt-pwg4qryCJw.png" /></figure><p>Will have dimensions (batch_size, segment_length, 2*segment_length) and, a priori, we can think about it as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*fb66XywT4pX2j1LZqsnzJQ.png" /></figure><p>The words in the rows are able to attend to the memory! This is great but we are not finished. Remember that the whole point of doing all that was training language models. i.e. predicting the next word. This means that &quot;that&quot; will be trained to output a high probability for &quot;her&quot; at the end of the model, &quot;her&quot; will be trained to output a high probability for “mother”, etc.</p><p>We <strong>can not </strong>allow the representation of the row words to attend anything that follows because that won&#39;t be the case in testing time. In other words, we won&#39;t be using the following words if we have to predict them.</p><p>To solve that, we use masking in the upper triangular matrix to avoid &quot;row words&quot; attending to &quot;column words&quot; that they won&#39;t have access to in testing time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V6MRmFDS4DEdL8D0zV8lIg.png" /></figure><p>I have seen several ways of doing masking. In my opinion, the best way of doing so is setting to -∞ the scalars that need to be zero before applying the softmax. If you set the scalars to zero after the softmax you will end up with a distribution of attention per &quot;row word&quot; that don&#39;t add to 1. I.e. the proportions of attention being distributed won&#39;t be correct.</p><p>Once we do that and multiply by V the result of attention will have the same dimensions as in the Transformer (batch_size, segment_length, d_v) for one head and (batch_size, segment_length, emb_dim) for multi-head self-attention.</p><h3>Absolute Positional Encoding &amp; Memory</h3><h4>Problem</h4><p>If we were to implement the above we could probably make it run seamlessly, however, we will be introducing fundamental biases because of the positional encoding (also referred to as absolute positional encoding).</p><p>The positional encoding is a technique that was initially introduced to allow the attention mechanism to account for the relative position of words. Under the original set-up of just having one segment of data, simply encoding that via the absolute position, i.e. position, 0, 1, 2, …., was enough and the model was able to extract information about the relative positions of two words (if you want to go deeper this is thanks to that positional encoding being defined with sinusoidal functions).</p><p>If we use multiple segments, as it is the case for the Transformer XL, adding absolute positional encoding before the encoder blocks every time a segment is processed represents a problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjCLaKORX821NKmWAVfzlg.png" /></figure><p>Why?</p><p>Let&#39;s take a look at how the positional encodings for each segment will look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*Q7NdN76A3-G9s-CSSiaWoQ.png" /></figure><p>The positional encoding for &quot;The&quot; &amp; &quot;that&quot;, &quot;daughter&quot; &amp; &quot;her&quot;, … are the same! In other words, if we have to learn the distribution of attention for the first &quot;her&quot; (2nd segment) the representation of daughter being immediately before or 6 positions apart is exactly the same. The model does not have enough information to distinguish between cases.</p><h4>Solution</h4><p>To solve this issue the authors erase the absolute positional encoding layer before the encoder blocks and add &quot;relative positional encoding&quot; which we will explain in a moment.</p><p>Using the previous notation we can analyze how the attention distribution was computed with respect to the embeddings and the positional encodings by doing some math. The notation (i, •) refers to the entire row i and (•, j) to the entire column j .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lZf5GokP3-2gluQ-ZRT6nQ.png" /></figure><p>If we now look at the last part of the derivation we will see four terms. Let me give a quick sense of what they are doing:</p><p>a) Is encoding how much attention the word in row i should pay to word in col j regardless of the position. For example, how often is &quot;she&quot; related to &quot;her&quot;.</p><p>b) Encapsulates the global attention that a word in a specific position i should pay to a row in a position j . For example if i and j were to be far apart we would expect the result of this term to be particularly low.</p><p>c) Capture the attention that a specific word in row i should pay to other words in position j . A clear example is if the word in column i is &quot;dog&quot;. In that case, we may want to pay special attention to words in position j=i-1 for cases such as &quot;hot dog&quot;.</p><p>d) Is the reciprocal of c), how often is the word that we want to attend to related to other words in position i .</p><p>By understanding what those terms mean we realize that what is really important is the relative position of one word with respect to the other and because of the decomposition that we just made we can modify the computation of the attention matrix in our benefit. The replace parts of expressions b), c) and d) as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*7Fp1KEZnTuoO1ej_r3bE4w.png" /></figure><p>We can see three major changes:</p><ol><li>We are using the relative positional encoding: i.e. rather than having P_{•, j} we are now using P_{•, i — j} to account for the relative position of one another.</li><li>Whenever using P_{•, i — j} we have replaced W^k by WˆR which have the same dimensions. This accounts for two sets of independent weights W^k by WˆRto capture meaningful and positional information without &quot;fighting&quot; to each other at training time.</li><li>We have replaced P_{i,•}W^Q by u and v both with dimensions (1, d_k). Observe that because we are taking relative positions above we don&#39;t need to provide the absolute position i in terms of positional encoding anymore. Thus we can just make the term P_{i,•}W^Q a vector. Moreover, if we use two vectors we allow for each of them to focus on the general position and the position given the word we attend to as its the case of u and v respectively.</li></ol><p>Thus, we end up replacing the above product ( the matrix to compute attention) with a matrix where every term is given by the formula below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*TIO_Jr0xRADdTLCa38dPUg.png" /></figure><p>And that&#39;s it! Those are the two major modifications made from Transformer to Transformer-XL: Memory between segments &amp; Change from Absolute to Relative Positional Encoding.</p><p><em>If you have any feedback please let me know in the comment section!</em></p><p><strong>References:</strong></p><p><a href="https://arxiv.org/pdf/1706.03762.pdf">Attention Is All You Need</a>; Vaswani et al., 2017.</p><p><a href="https://arxiv.org/pdf/1901.02860.pdf">Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context</a>; Zihang Dai et al, 2019.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90963e274bd7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set-up a remote machine for Deep Learning with Jupyter Notebook.]]></title>
            <link>https://medium.com/@mromerocalvo/set-up-a-remote-machine-for-deep-learning-with-jupyter-notebook-1946729f9fc?source=rss-4e4973afd99------2</link>
            <guid isPermaLink="false">https://medium.com/p/1946729f9fc</guid>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Miguel Romero Calvo]]></dc:creator>
            <pubDate>Sun, 02 Jun 2019 16:01:00 GMT</pubDate>
            <atom:updated>2019-08-08T17:33:02.434Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Some people doing Deep Learning may need to use a remote machine with a GPU. Sometimes the hardest part is the set-up. When I started using this I spent two days fighting with the configuration just to get a nice <strong>conda</strong> set-up and a remote Jupyter Notebook up and running. This blog aims to make it less painful for those in the same position I was back then.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R2L91mI2h2gAueL1995MUw.png" /></figure><blockquote>You do not need to read the whole blog post. Find what you are looking for in the index and just jump to that section.</blockquote><p>As a quick summary, this post has the following sections:</p><blockquote><strong>1. BASIC SET-UP</strong></blockquote><blockquote>1.1.<em> </em><strong>Installing Anaconda3</strong></blockquote><blockquote>1.2.<em> </em><strong>Setting up Jupyter Notebook</strong></blockquote><blockquote>1.3.<strong> Installing PyTorch</strong></blockquote><blockquote><strong>2. TMUX</strong></blockquote><blockquote><strong>3. WHAT COOL KIDS DO: EXTRAS</strong></blockquote><blockquote>3.1.<strong><em> Custom Password</em></strong></blockquote><blockquote>3.2.<strong><em> Ip address -&gt; name (MAC)</em></strong></blockquote><blockquote>3.3.<strong><em> SSH TUNEL</em></strong></blockquote><blockquote><strong><em>4 RUNNING &amp; OPENING THE NOTEBOOK</em></strong></blockquote><p>In <strong>Section 1</strong> the basic set-up is explained. In <strong>Section 2</strong> we show a recommended way of running your Jupyter Notebook using Tmux. In <strong>Section 3</strong> we close up with some recommendations that can make your life easier when working in a remote machine. Lastly, in <strong>Section 4</strong> we review how to open Jupyter notebook after launching it.</p><p>Throughout the blog post, I am going to use <em>IP</em> to refer to the IP address of the remote machine, for instance, 127.83.75.12 ( <em>or the ec2…. in case you are using AWS </em>) and <em>user </em>to refer to the username you have to access to access the remote machine, for example, miguel.</p><p>I will be using</p><pre>$</pre><p>if the code is run in the terminal. We will be using <strong>vi </strong>as a file editor. Let&#39;s review some basic functionality that we are going to use:</p><ul><li>To use the &quot;edit&quot; mode press <strong>i</strong> and to exit it press <strong>esc</strong>.</li><li>To save and exit press <strong>esc</strong> and type in <strong>:wq</strong></li></ul><p>I am assuming that you have your ssh private key in ~/.ssh/ and you are able to connect to the remote machine by running</p><pre>ssh user@IP</pre><p>in the terminal.</p><blockquote><strong>Note for those using AWS machines</strong>: Take in account that you are going to add an inbound rule for the port where the jupyter notebook is running. We will set the specific port in sections 1.2. and 3.3.</blockquote><h3><strong>1. BASIC SET-UP</strong></h3><ul><li>Ssh to your remote machine</li></ul><pre>ssh user@IP</pre><h3>1.1.<em> </em><strong>Installing Anaconda3</strong></h3><p>If you are using AWS check first if the instance you are working on has anaconda installed. You can check with:</p><pre>$ conda --version</pre><ul><li>Download Anaconda3 from one of <a href="https://repo.continuum.io/archive/">those</a> <em>.sh</em> links. Here, I will assume the remote machine is a Linux-x86 64.</li></ul><pre>$ wget -c <a href="https://repo.continuum.io/archive/Anaconda3-5.3.0-Linux-x86_64.sh">https://repo.continuum.io/archive/Anaconda3-5.3.0-Linux-x86_64.sh</a></pre><ul><li>Execute the script</li></ul><pre>$ bash Anaconda3–5.3.0-Linux-x86_64.sh</pre><p>While executing you are going to need to answers some questions. Here I am giving the answers that you are (most often) going to want.</p><ul><li>Say yes or press enter to have your anaconda installed in the remote home directory <em>(default)</em>.</li><li>You want to initialize it in <em>.bashrc</em>. Say <em>yes</em> to that.</li><li>Say <em>no </em>to install vscode.</li></ul><p>The installation will finish. To &quot;refresh&quot; your current terminal window run</p><pre>$ source ~/.bashrc</pre><blockquote>Congratulations, Anaconda3 has been installed!</blockquote><h3>1.2.<em> </em><strong>Setting up Jupyter notebook</strong></h3><p>I am assuming you are in your <em>home </em><strong><em>remote</em></strong><em> directory.</em></p><pre>$ cd ~/</pre><p>Let&#39;s move on to setting up Jupyter Notebook. First things first, let&#39;s make sure you have the latest version.</p><pre>$ conda update jupyter notebook</pre><p>Once this has finished refresh the current terminal window.</p><pre>$ source ~/.bashrc</pre><p>Now it&#39;s time to tell Jupyter Notebook that you want to use it remotely. To do so we have to create a configuration file for Jupyter Notebook.</p><pre>$ jupyter notebook --generate-config</pre><p>The configuration file has been created. Now let&#39;s edit it.</p><pre>$ vi .jupyter/jupyter_notebook_config.py</pre><p>Now that you are inside it. Add the following lines at the top of the file (without the &gt;).</p><pre>c = get_config()<br>c.NotebookApp.ip = &#39;0.0.0.0&#39;<br>c.NotebookApp.open_browser = False<br>c.NotebookApp.port = 8888</pre><p>Exit the script and return to your home remote directory.</p><blockquote>Great! You have set up your Jupyter Notebook to work remotely!</blockquote><h3>1.3.<strong> Installing PyTorch</strong></h3><p>This is straight forward. They provide a <a href="https://pytorch.org/get-started/locally/">nice GUI</a> for you to know what command to run. You want to use the command from the &quot;start locally&quot; tab. The main things you need to know are:</p><ul><li><em>Am I using an Nvidia GPU? If so what version?</em> You can check by running the command below and looking at the top-right of the table that appears.</li></ul><pre>$ nvcc --version</pre><ul><li><em>What python version you are using?</em> You can check it by running the command below.</li></ul><pre>$ python — version</pre><h3><strong>2. TMUX</strong></h3><p>To explain why <a href="https://linuxize.com/post/getting-started-with-tmux/">tmux</a> is useful, let me first give you some context. If you ssh to your remote machine and start a Jupyter Notebook, run a script or spin-up any process they may work great.. but… they are killed if your ssh breaks for any reason.</p><blockquote>Imagine that you have been 7 hours training a model, only 30 min to finish and all of a sudden your internet connection suddenly goes down 😭. You just lost those 7 hours!</blockquote><blockquote>Not cool, no?</blockquote><p>Tmux works great in those scenarios. It allows you to create sessions so if you run your Jupyter Notebooks, scripts or anything in them they won&#39;t be killed if you are disconnected from the ssh.</p><p>From my experience, it is usually installed but if it&#39;s not please look at <a href="https://linuxize.com/post/getting-started-with-tmux/">this</a> webpage to see how to install it.</p><p>Let me review some basic tmux commands:</p><ul><li>To <strong>create a new session</strong> <em>k</em>. Where <em>k</em> is the index of the session you want to create. For example, <em>k</em> can be 0,1, 2, ….</li></ul><pre>$ tmux new -s k</pre><ul><li>To <strong>exit a session</strong> without killing it. Press <strong>ctrl+b</strong> (at the same time) and then press <strong>d </strong>(for detach).</li><li>To <strong>go/attach</strong> a session already created run the command below where <em>k </em>is the index of the session.</li></ul><pre>$ tmux attach -t k</pre><ul><li>To <strong>kill </strong>a session run the command below where <em>k </em>is the index of the session to be killed.</li></ul><pre>$ tmux kill-session -t k</pre><h3><strong>3. WHAT COOL KIDS DO: EXTRAS</strong></h3><p>In this section, some extra configuration will be provided. By no mean the following sub-sections are necessary to run your Jupyter Notebook. However, in my experience, they are really handy if you work with the same machine on a daily basis.</p><h3>3.1.<strong><em> Custom Password</em></strong></h3><p>Up until now, you had to copy paste the URL with the token whenever initializing Jupyter Notebook. I find that to be annoying. There is a much better way to deal with it. You can set Jupyter to work with your custom password rather than a token. Let&#39;s see how to do so.</p><p>I am assuming you are in your <em>home </em><strong><em>remote</em></strong><em> directory.</em></p><p>Let&#39;s first generate a hash password from your custom password. To do so run the following:</p><pre>$ ipython</pre><p>Then a python console will appear and you have to run the following:</p><pre>[0] from notebook.auth import passwd</pre><p>And then run the following command to get the hashed version of your CUSTOM_PASSWORD.</p><pre>[1] passwd(CUSTOM_PASSWORD)</pre><p>A string such as ‘asfsadfasdfdgasdf’ will appear. You to copy it and paste it afterwards in the Jupyter configuration file. Let&#39;s do that step by step. Assuming that you have already copied the string your first need to exit the console.</p><pre>[3] exit</pre><p>Assuming that you are in your remote home directory run the following.</p><pre>$ vi .jupyter/jupyter_notebook_config.py</pre><p>Then, insert the following line somewhere under <em>c = get_config().</em></p><pre>c.NotebookApp.password = u’asfsadfasdfdgasdf’</pre><p>where ’asfsadfasdfdgasdf’ is the string you have copied (i.e. your hashed password).</p><p><strong>Note</strong>: The hashed password is wrapped around <strong>u&#39; &#39;</strong></p><p>Now save and extit the configuration file and run the following line in the terminal.</p><pre>$ source ~/.bashrc</pre><blockquote>Done!</blockquote><h3>3.2.<strong><em> Ip address -&gt; name (MAC)</em></strong></h3><p>Every time you were accessing the remote machine you had to do the following:</p><pre>ssh user@IP</pre><p>I don&#39;t know you, but I am pretty bad at remembering IP addresses and I don&#39;t find it that useful. Thankfully, we can replace the IP by some name, for example &quot;<em>gpu&quot; </em>after some minor configurations. Then, every time we access the remote machine we do the command below rather than the one above.</p><pre>ssh user@gpu</pre><p>I am going to assume you are using a mac as a remote machine. You have to run:</p><pre>$ sudo chmod 777 /etc/hosts<br>$ vi /etc/hosts</pre><p>then, assuming that you want to use the name &quot;gpu&quot; and your IP is 172.28.38.12, add the following line at the bottom (you can use any name you like and you are going to need to change the IP for your own). Even if Medium does not allow me to, <strong>there should be 3 spaces between the ip address and the name</strong>.</p><pre>172.28.38.12 gpu</pre><p>Great, now save the file and exit. And run the following command.</p><pre>$ chmod 400 /etc/hosts</pre><blockquote>Done!</blockquote><h3>3.3.<strong><em> SSH TUNEL</em></strong></h3><p>Here we assume that you have successfully completed 3.1.</p><p>Up until now, you had to access your Jupyter Notebook typing:</p><p><a href="https://ip:8888/">https://IP:</a>PORT</p><p>where PORT is the port where the Jupyter Notebook is running (most often 8888). I find that to be less handy than &quot;having&quot; it in my localhost:8888. To get around that we can use what it&#39;s call an ssh tunnel.</p><p>The idea of an ssh tunnel is that you are going to see in one of your local ports (e.g. localhost:8889 ) what is happening in a specific port of another remote machine where you can ssh to. In our case, this is translated to you being able to type</p><pre>localhost:8889</pre><p>in your browser and see the notebook that is being executed in the remote machine.</p><p>How do we do that?</p><p>A requirement of doing that is that the port in the remote machine has to be always the same. If you are the only one using the machine you can leave the port 8888 in your Jupyter Notebook configuration. Otherwise, we have to change the port. To change the port ssh to your remote machine and edit the Jupyter Notebook configuration file.</p><pre>$ ssh user@IP</pre><pre>$ vi .jupyter/jupyter_notebook_config.py</pre><p>And replace 8888 in the line</p><pre>c.NotebookApp.port = 8888</pre><p>with some other number. One of the things to take into account when chosing that number is that you don&#39;t want that port to be used by anyone else or any other process but your Jupyter notebook. For this example assume that we choose port 1234. Then the above line should look as follows:</p><pre>c.NotebookApp.port = 1234</pre><p>Save the changes and exit the script. Then run:</p><pre>$ source ./.bashrc</pre><p>Now we are going to swap and use the <strong>local machine </strong>(i.e. you should be in your local terminal). We want to edit your .bash_profile or .bashrc for Mac or Linux users respectively. For those who are not familiar with them, those are scripts that are being executed every time you open the terminal.</p><p>To automate the tunnel creation we will edit those .bash_profile or .bash_rc</p><pre>$ vi ./.bash_profile</pre><p>and add the following lines</p><pre>ssh -N -f -L localhost:LOCAL_PORT:localhost:1234 [user]@[IP/NAME(if configurated)]</pre><pre>clear</pre><p>Where LOCAL_PORT is the port we want to use when tying</p><p>localhost:LOCAL_PORT</p><p>in the browser and 1234 is the remote port we previously set up.</p><blockquote><strong>Note</strong>: When opening the terminal for the first time it may take some time but it gets faster after that.</blockquote><blockquote>Done!</blockquote><h3><strong><em>4 RUNNING &amp; OPENING THE NOTEBOOK</em></strong></h3><p>After launching Jupyter notebook</p><pre>$ jupyter notebook</pre><p>the way in which you open it depends on how many of the previous steps you have completed.</p><p>If you haven&#39;t done 3.1 you have to copy the URL that appears in the output of running the command above and replace the <em>localhost </em>by the<em> IP </em>of the remote machine (or ec2-…. in case of using AWS).</p><p><strong>Note: </strong>Make sure you are doing http<strong>s </strong>when accessing that URL.</p><p>If you have done 3.1. you don&#39;t need to copy the token. and if you have done 3.3. you only have to type localhost:LOCAL_PORT and you are good to go.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1946729f9fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dissecting BERT Appendix: The Decoder]]></title>
            <link>https://medium.com/dissecting-bert/dissecting-bert-appendix-the-decoder-3b86f66b0e5f?source=rss-4e4973afd99------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b86f66b0e5f</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Miguel Romero Calvo]]></dc:creator>
            <pubDate>Mon, 26 Nov 2018 22:08:32 GMT</pubDate>
            <atom:updated>2018-11-27T23:45:55.205Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*HQhbF6EOR6cL_ZAiHSkqmA.jpeg" /><figcaption>Unleash the force of self-attention</figcaption></figure><p><em>This is the appendinx of Understanding BERT written by Miguel Romero and Francisco Ingham. Each article was written jointly by both authors. If you did not already, please refer to </em><a href="https://medium.com/@mromerocalvo/6dcf5360b07f"><em>Part 1</em></a><em> to understand the Encoder architecture in depth since this post assumes its prior understanding.</em></p><p><em>Many thanks to Yannet Interian for her revision and feedback.</em></p><h3>Notation</h3><p>Before we begin, let’s define the notation we will use throughout the article:</p><blockquote><strong>emb_dim</strong>: Dimension of the embedding</blockquote><blockquote><strong>input_length</strong>: Length of the input sequence</blockquote><blockquote><strong>target_length</strong>: Length of the target sequence + 1. The +1 is a consequence of the shift.</blockquote><blockquote><strong>vocab_size</strong>: Amount of words in your vocabulary (deviated from the corpus).</blockquote><blockquote><strong>target input</strong>: We will use this term interchangeably to describe the input string (set of sentences) or sequence in the decoder.</blockquote><h3>Introduction</h3><p>The <strong>Transformer</strong> is an attention-based architecture for Natural Language Processing (NLP) that was introduced in the paper <strong>Attention Is All You Need </strong>a year ago.</p><p>In this blog post, we are going to examine the Decoder in depth; the part of the Transformer architecture<strong> </strong>that are not used in <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a>. We will reference the <a href="https://medium.com/dissecting-bert/encoder-architecture-d3c3d495cdb3">Encoder</a> to explain the full <strong>Transformer</strong>’s architecture.</p><blockquote><strong>Note: </strong>If you are only interested in understanding how BERT works, the parts of the <strong>Transformer</strong> described in this blog post are not relevant.</blockquote><p>This article is organized as follows:</p><ol><li>The Problem the <strong>Transformer</strong> aims to solve.</li><li>The Information Flow.</li><li>The Decoder.</li></ol><h3>Problem</h3><p>The problem that the <strong>Transformer</strong> addresses is translation. To translate a sentence into another language, we want our model to:</p><ul><li>Be able to capture the relationships between the words in the input sentence.</li><li>Combine the information contained in the input sentence and what has already been translated at each step.</li></ul><p>Imagine that the goal is to translate a sentence from English to Spanish and we are given the following sequences of tokens:</p><blockquote>X = [‘Hello’, ‘,’, ‘how’, ‘are’, ‘you’, ‘?’] (Input sequence)<br>Y = [‘Hola’, ‘,’, ‘como’, ‘estas’, ‘?’] (Target sequence)</blockquote><p>First, we want to process the information in the input sequence <em>X</em> by combining the information in each of the words of the sequence. This is done inside the Encoder.</p><p>Once we have this information in the output of the Encoder we want to combine it with the target sequence. This is done in the Decoder.</p><p>Encoder and Decoder are specific parts of the <strong>Transformer</strong> architecture as illustrated in Figure 1. We will investigate the Decoder in detail in <em>Level 3-b: Layers</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ekte5vEsR54slWTy6wrhoA.png" /><figcaption>Figure 1: <strong>Transformer</strong> architecture divided into Encoder and Decoder</figcaption></figure><h3>Information Flow</h3><p>The data flow through the architecture is as follows:</p><ol><li>The model represents each token as a vector of dimension <em>emb_dim</em>. We then have a matrix of dimensions <em>(input_length) x (emb_dimb)</em> for a specific input sequence.</li><li>It then adds positional information (positional encoding). This step will return a matrix of dimensions <em>(input_length) x (emb_dim)</em>, just as in the previous step.</li><li>The data goes through N encoder blocks. After that, we obtain a matrix of dimensions <em>(input_length) x (emb_dim)</em>.</li><li>The target sequence is masked and sent through the decoder’s equivalent of 1) and 2). The output has dimensions <em>(target_length) x (emb_dim)</em>.</li><li>The result of 4) goes through N decoder blocks. In each of the iterations, the decoder is using the encoder’s output 3). This is represented in Figure 2 by the arrows from the encoder to the decoder. The dimensions of the output are <em>(target_length) x (emb_dim)</em>.</li><li>Finally, it applies a fully connected layer and a row-wise softmax. The dimensions of the output are<em> (target_length) x (vocab_size).</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P2cHPZmuKncBhNmybAlynQ.png" /></figure><p>The dimensions of the input and the output of the decoder blocks are the same. Hence, it makes sense to use the output of one decoder block as the input of the next decoder block.</p><p><strong>Note</strong>: In the Attention Is All You Need experiments, N was chosen to be 6.</p><blockquote>Note: The blocks do not share weights</blockquote><h3>Inputs</h3><p>Remember that the described algorithm is processing both the input sentence and the target sentence to train the network. The input sentence will be encoded as described in <a href="https://medium.com/dissecting-bert/encoder-architecture-d3c3d495cdb3">The Encoder’s architecture</a>. In this section, we discuss how given a target sentence (e.g. “Hola, como estás ?”) we obtain a matrix representing the target sentence for the decoder blocks.</p><p>The process is exactly the same. It is also composed of two general steps:</p><ol><li>Token embeddings</li><li>Encoding of the positions.</li></ol><p>The main difference is that the target sentence is shifted. That is, before padding, the target sequence will be as follows:</p><blockquote>[“Hola”, “, “, “como”, “estás”, “?”]→[“&lt;SS&gt;”, “Hola”, “, “, “como”, “estás”, “?”]</blockquote><p>The rest of the process to vectorize the target sequence will be exactly as the one described for input sentences in <a href="https://medium.com/dissecting-bert/encoder-architecture-d3c3d495cdb3">The Encoder’s architecture</a>.</p><h3>Decoder</h3><p>In this section, we will cover those parts of the decoder that differ from those covered in the encoder.</p><h3>Decoder block — Training vs Testing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*3VAGzPTi7auz_oBv7-Xf9g.png" /></figure><p>During test time we don’t have the ground truth. The steps, in this case, will be as follows:</p><ol><li>Compute the embedding representation of the input sequence.</li><li>Use a starting sequence token, for example ‘&lt;SS&gt;’ as the first target sequence: [&lt;SS&gt;]. The model gives as output the next token.</li><li>Add the last predicted token to the target sequence and use it to generate a new prediction [‘&lt;SS&gt;’, Prediction_1,…,Prediction_n]</li><li>Do step 3 until the predicted token is the one representing the End of the Sequence, for example &lt;EOS&gt;.</li></ol><p>During training we have the ground truth, i.e. the tokens we would like the model to output for every iteration of the above process. Since we have the target in advance, we will give the model the whole shifted target sequence at once and ask it to predict the non-shifted target.</p><p>Following up with our previous examples we would input:</p><blockquote>[‘&lt;SS&gt;’,’Hola’, ‘,’, ‘ como’, ‘estas’, ‘?’]</blockquote><p>and the expected prediction would be:</p><blockquote>[’Hola’, ‘,’, ‘ como’, ‘estas’, ‘?’,’&lt;EOS&gt;’]</blockquote><p>However, there is a problem here. What if the model sees the expected token and uses it to predict itself? For example, it might see <em>‘estas’</em> at the right of <em>‘como’</em> and use it to predict<em> ‘estas’</em>. That’s not what we want because the model will not be able to do that a testing time.</p><p>We need to modify some of the attention layers to prevent the model of seeing information on the right (or down in the matrix of vector representation) but allow it to use the already predicted words.</p><p>Let’s illustrate this with an example. Given:</p><blockquote>[‘&lt;SS&gt;’,’Hola’, ‘,’, ‘ como’, ‘estas’, ‘?’]</blockquote><p>we will transform it into a matrix as described above and add positional encoding. This would result in a matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*sYbUKTi-k6o5smZlBbzU5w.png" /></figure><p>And just as in the encoder the output of the decoder block will be also a matrix of sizes <em>(target_length) x (emb_dim)</em>. After a row-wise linear (a linear layer in the form of matrix product on the right) and a Softmax per row this will result in a matrix for which the maximum element per row indicates the next word.</p><p>That means that the row assigned to “&lt;SS&gt;” is in charge of predicting “Hola”, the row assigned to “Hola” is in charge of predicting “,” and so on. Hence, to predict “estas” we will allow that row to directly interact with the green region but not with the red region in Figure 13.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*a4V1WXN-CDwoJE5vyVR4NQ.png" /></figure><p>Observe that we don’t have problems in the linear layers because they are defined to be token-wise/row-wise in the form of a matrix multiplication through the right.</p><p>The problem will be in <strong>Multi-Head Attention </strong>and the input will need to be masked. We will talk more about masking in the next section.</p><p>At training time, the prediction of all rows matter. Given that at prediction time we are doing an iterative process we are just going to care about the prediction of the next word of the last token in the target/output sequence.</p><h4>Masked Multi-Head Attention</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*B_pAfVZEU-7np0Y7LBrZdw.png" /></figure><p>This will work exactly as the <strong>Multi-Head Attention</strong> mechanism but adding masking to our input.</p><p>The only <strong>Multi-Head Attention</strong> block where masking is required is the first one of each decoder block. This is because the one in the middle is used to combine information between the encoded inputs and the outputs inherited from the previous layers. There is no problem in combining every target token’s representation with any of the input token’s representations (since we will have all of them at test time).</p><p>The modification will take place after computing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*VGKAJY_w1KKUYn4L0eGdkA.png" /></figure><p>Observe that this is a matrix such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*GysQ0m1lL4awcymVpCxc6Q.png" /></figure><p>Now, the masking step is just going to set to minus infinity all the entries in the strictly upper triangular part of the matrix. In our example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*qWPzSPMytnv5zS89cciFkA.png" /></figure><p>That’s it! The rest of the process is identical as described in the <strong>Multi-Head Attention</strong> for the encoder.</p><p>Let’s now dig in what does it means mathematically to set those elements to minus infinity. Observe that if those entries are relative attention measures per each row, the larger they are, the more attention we need to pay to that token. So setting those elements to minus infinity is mainly saying: “ For the row assigned to predicting <em>“estás” </em>(the one with input <em>“como”)</em>, ignore <em>“estás”</em> and “<em>?</em>”. Our Softmax output would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xu2X2zrGfTVAp9rvjVq4aA.png" /></figure><p>The relative attention of those tokens that we were trying to ignore has indeed gone to zero.</p><p>When multiplying this matrix with <em>V_i</em> the only elements that will be accounted for to predict the next word are the ones into its right, i.e. the ones that the model will have access to during test time.</p><p>Observe that this time the output of the modified <strong>Multi-Head Attention</strong> layer will be a matrix of dimensions <em>(target_length) x (emb_dim)</em> because the sequence from which it has been calculated has a sequence length of <em>target_length.</em></p><h3>Multi-Head Attention — Encoder output and target</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*TTpBMsvq_8tmttJji1TWSg.png" /></figure><p>Observe that in this case we are using different inputs for that layer. More specifically, instead of deriving <em>Q_i</em>, <em>K_i</em> and <em>V_i</em> from <em>X</em> as we have been doing in previous <strong>Multi-Head Attention</strong> layers, this layer will use both the Encoder’s final output <em>E</em> (final result of all encoder blocks) and the Decoder’s previous layer output <em>D</em> (the masked <strong>Multi-Head Attention</strong> after going through the <strong>Dropout,</strong> <strong>Add &amp; Norm </strong>layer).</p><p>Let’s first clarify the shape of those inputs and what they represent:</p><ol><li><em>E</em>, the encoded input sequence, is a matrix of dimensions <em>(input_length) x (emb_dim)</em> which has encoded, by going through 6 encoder blocks, the relationships between the input tokens.</li><li><em>D</em>, the output from the masked <strong>Multi-Head Attention</strong> after going through the Add &amp; Norm, is a matrix of dimensions<em> (target_length) x (emb_dim)</em>.</li></ol><p>Let’s now dive into what to do with those matrices. We will use weighted matrices with the same dimensions as before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*UILj3J57TLdYcySUvSRX4w.png" /></figure><p>But this time the projection generating <em>Q_i</em> will be done using <em>D</em> (target information), while the ones generating <em>K</em> and <em>V</em> will be created using <em>E</em> (input information).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*8j2ylRN4IfJV56NYUKSdog.png" /></figure><p>for every head i=1,…, h.</p><p>The matrix <em>W_0</em> used after the concatenation of the heads will have dimensions <em>(d_v*h) x (emb_dim)</em> just like the one used in the encoder block.</p><p>Apart from that, the <strong>Multi-Head Attention</strong> block is exactly the same as the one in the encoder.</p><p>Observe that in this case, the matrix resulting from:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*T4MCL9SdNaQUC5PVLr-PVQ.png" /></figure><p>is describing relevant relationships between “encoded input tokens” and “encoded target tokens”. Moreover notice that this matrix will have dimensions <em>(target_length)x(input_length)</em>.</p><p>Following up on our example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*O9OkEHqFK6ffKtdgsX7R3g.png" /></figure><p>As we can see, every projected row-token in the target attends to all the positions in the (encoded) input sequence. This matrix encodes the relationship between the input sequence and the target sequence.</p><p>Repeating the notation we used in the <strong>Multi-Head Attention</strong> of the Encoder the multiplication with <em>V_i</em> results in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*k8kVGcHvKy3_ItXzYyWItQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*61XBHADgigA6igSXTF93-Q.png" /></figure><p>As we can see, every token in the target sequence is represented in every head as a combination of encoded input tokens. Moreover, this will happen for multiple heads and just as before, that is going to allow each token of the target sequence to be represented by multiple relationships with the tokens in the input sequence.</p><p>Just like in the encoder block, once the concatenation has been carried out, we will take the product of that with <em>W_0:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*K50k7y_ThfTudF-8FUR5Bg.png" /></figure><h3>Linear and Softmax</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*ccrz7fNllR2mmIc5lSeGSw.png" /></figure><p>This is the final step before being able to get the predicted token for every position in the target sequence. If you are familiar with Language Models, this is identical to their last layers. The output from the last Add &amp; Norm layer of the last Decoder block is a matrix X of dimensions <em>(target_length)x(emb_dim).</em></p><p>The idea of the linear layer is for every row in <em>x</em> of <em>X </em>to compute:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/1*BJU_slrWn-h1_0q9oU3Yhw.png" /></figure><p>where <em>W_1</em> is a matrix of learned weights of dimensions <em>(emb_dim) x (vocab_size)</em>. Therefore, for a specific row the result will be a vector of length <em>vocab_size.</em></p><p>Finally, a softmax is applied to this vector resulting in a vector describing the probability of the next token. Therefore, taking the position corresponding to the maximum probability returns the most likely next word according to the model.</p><p>In matrix form this looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/70/1*2ozBz8_LLxku9uk5ZPTVig.png" /></figure><p>And applying a Softmax in each resulting row.</p><p><em>If you have any feedback please let us know in the comment section!</em></p><h3>References</h3><p><a href="https://arxiv.org/pdf/1706.03762.pdf">Attention Is All You Need</a>; Vaswani et al., 2017.</p><p><a href="https://arxiv.org/pdf/1810.04805.pdf">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>; Devlin et al., 2018.</p><p><a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html#embeddings-and-softmax">The Annotated Transformer</a>; Alexander Rush, Vincent Nguyen and Guillaume Klein.</p><p><a href="https://arxiv.org/pdf/1801.06146.pdf">Universal Language Model Fine-tuning for Text Classification</a>; Howard et al., 2018.</p><p><a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">Improving Language Understanding by Generative Pre-Training</a>; Radford et al., 2018.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b86f66b0e5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dissecting-bert/dissecting-bert-appendix-the-decoder-3b86f66b0e5f">Dissecting BERT Appendix: The Decoder</a> was originally published in <a href="https://medium.com/dissecting-bert">Dissecting BERT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dissecting BERT Part 1: The Encoder]]></title>
            <link>https://medium.com/dissecting-bert/dissecting-bert-part-1-d3c3d495cdb3?source=rss-4e4973afd99------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3c3d495cdb3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Miguel Romero Calvo]]></dc:creator>
            <pubDate>Mon, 26 Nov 2018 22:02:42 GMT</pubDate>
            <atom:updated>2019-05-07T18:03:04.546Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*SdiFbDnvGZWRUhev7ynU9Q.jpeg" /><figcaption>A meaningful representation of the input, you must encode</figcaption></figure><p><em>This is Part 1/2 of Dissecting BERT written by Miguel Romero and Francisco Ingham. Each article was written jointly by both authors. If you already understand the Encoder architecture from Attention is All You Need and you are interested in the differences that make BERT awesome, head on to </em><a href="https://medium.com/dissecting-bert/dissecting-bert-part2-335ff2ed9c73"><em>BERT Specifics</em></a><em>.</em></p><p><em>Many thanks to Yannet Interian for her review and feedback.</em></p><p>In this blog post, we are going to examine the <strong>Encoder </strong>architecture in depth (see Figure 1) as described in <a href="https://arxiv.org/pdf/1706.03762.pdf">Attention Is All You Need</a>. In <a href="https://medium.com/dissecting-bert/dissecting-bert-part2-335ff2ed9c73">BERT Specifics</a> we will dive into the novel modifications that make <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a> particularly effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*VtfbRAAiQhb0IUi7fSKTaQ.png" /><figcaption>Figure 1: The Encoder</figcaption></figure><h3>Notation</h3><p>Before we begin, let’s define the notation we will use throughout the article:</p><blockquote><strong>emb_dim</strong>: Dimension of the token embeddings.</blockquote><blockquote><strong>input_length</strong>: Length of the input sequence (the same in all sequences in a specific batch due to padding).</blockquote><blockquote><strong>hidden_dim</strong>: Size of the Feed-Forward network’s hidden layer.</blockquote><blockquote><strong>vocab_size</strong>: Amount of words in the vocabulary (derived from the corpus).</blockquote><h3>Introduction</h3><p>The <strong>Encoder</strong> used in <strong>BERT</strong> is an attention-based architecture for Natural Language Processing (NLP) that was introduced in the paper <strong>Attention Is All You Need </strong>a year ago. The paper introduces an architecture called the <strong>Transformer </strong>which is composed of two parts, the <strong>Encoder</strong> and the <strong>Decoder</strong>. Since <strong>BERT </strong>only uses the <strong>Encoder </strong>we are only going to explain that in this blog post (if you want to learn about the <strong>Decoder </strong>and how it is integrated with the <strong>Encoder</strong>, we wrote a separate <a href="https://medium.com/dissecting-bert/transformer-architecture-where-the-encoder-comes-from-3b86f66b0e5f">blog post</a> on this).</p><p>Transfer learning has quickly become a standard for state of the art results in NLP since the release of <a href="https://arxiv.org/pdf/1801.06146.pdf"><strong>ULMFiT</strong></a><strong> </strong>earlier this year. After that, remarkable advances have been achieved by combining the <strong>Transformer</strong> with transfer learning.<strong> </strong>Two iconic examples of this combination are OpenAI’s <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf"><strong>GPT</strong></a> and Google AI’s <a href="https://arxiv.org/pdf/1810.04805.pdf"><strong>BERT</strong></a><strong>.</strong></p><p>This series aims to:</p><ol><li>Provide an intuitive understanding of the <strong>Transformer</strong> and <strong>BERT</strong>’s underlying architecture.</li><li>Explain the fundamental principles of what makes <strong>BERT</strong> so successful in NLP tasks.</li></ol><p>To explain this architecture we will adopt the <em>general to specifics </em>approach. We will start by looking at the information flow in the architecture and we will dive into the inputs and outputs of the <strong>Encoder</strong> as presented in the paper<strong>. </strong>Next, we will look into each of the encoder blocks and understand how <em>Multi-Head Attention</em> is used. Don&#39;t worry if you don&#39;t know what that is yet; we will make sure you understand it by the end of this article.</p><h3>Information Flow</h3><p>The data flow through the architecture is as follows:</p><ol><li>The model represents each token as a vector of <em>emb_dim</em> size. With one embedding vector for each of the input tokens, we have a matrix of dimensions <em>(input_length) x (emb_dim)</em> for a specific input sequence.</li><li>It then adds positional information (positional encoding). This step returns a matrix of dimensions <em>(input_length) x (emb_dim)</em>, just like in the previous step.</li><li>The data goes through N encoder blocks. After this, we obtain a matrix of dimensions <em>(input_length) x (emb_dim)</em>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*YkQYLsEZdRHJdGBLmNws8w.png" /><figcaption>Figure 2: Information flow in the Encoder</figcaption></figure><blockquote><strong>Note: </strong>The dimensions of the input and output of the encoder block are the same. Hence, it makes sense to use the output of one encoder block as the input of the next encoder block.</blockquote><blockquote><strong>Note:</strong> In <strong>BERT&#39;</strong>s experiments, the number of blocks N (or L, as they call it) was chosen to be 12 and 24.</blockquote><blockquote><strong>Note:</strong> The blocks do not share weights with each other</blockquote><h3>From words to vectors</h3><h4>Tokenization, numericalization and word embeddings</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*SvQNJV3n-6WlBHC25z5QVg.png" /><figcaption>Figure 3: Where tokenization, numericalization and embeddings happen.</figcaption></figure><p>Tokenization, numericalization and embeddings do not differ from the way it is done with RNNs. Given a sentence in a corpus:</p><blockquote>“ Hello, how are you?”</blockquote><p>The first step is to tokenize it:</p><blockquote>“ Hello, how are you?” → [“Hello”, “,” , “how”, “are”, “you”, “?”]</blockquote><p>This is followed by numericalization, mapping each token to a unique integer in the corpus’ vocabulary.</p><blockquote>[“Hello”, “, “, “how”, “are”, “you”, “?”] → [34, 90, 15, 684, 55, 193]</blockquote><p>Next, we get the embedding for each word in the sequence. Each word of the sequence is mapped to a <em>emb_dim </em>dimensional vector that the model will learn during training. You can think about it as a vector look-up for each token. The elements of those vectors are treated as model parameters and are optimized with back-propagation just like any other weights.</p><p>Therefore, for each token, we look up the corresponding vector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*29pl2Key0h0ioFYFPN-KxA.png" /></figure><p>Stacking each of the vectors together we obtain a matrix Z of dimensions <em>(input_length) x (emb_dim)</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*ntuXg7wgd7jrYf8otvXc0w.png" /></figure><p>It is important to remark that padding was used to make the input sequences in a batch have the same length. That is, we increase the length of some of the sequences by adding ‘<em>&lt;pad&gt;’ </em>tokens. The sequence after padding might be:</p><blockquote>[“&lt;pad&gt;”, “&lt;pad&gt;”, “&lt;pad&gt;”, “Hello”, “, “, “how”, “are”, “you”, “?”] →</blockquote><blockquote>[5, 5, 5, 34, 90, 15, 684, 55, 193]</blockquote><p>if the <em>input_length</em> was set to 9.</p><h4>Positional Encoding</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*wJp9_l0l6yofumPYC3PMaQ.png" /><figcaption>Figure 4: Where Positional Encoding is computed.</figcaption></figure><blockquote><strong>Note: </strong>In BERT the authors used learned positional embeddings. If you are only interested in BERT you can skip this section where we explain the functions used to calculate the positional encodings in Attention is All You Need</blockquote><p>At this point, we have a matrix representation of our sequence. However, these representations are not encoding the fact that words appear in different positions.</p><p>Intuitively, we aim to be able to modify the represented meaning of a specific word depending on its position. We don&#39;t want to change the full representation of the word but we want to modify it a little to encode its position.</p><p>The approach chosen in the paper is to add numbers between <em>[-1,1] </em>using predetermined (non-learned) sinusoidal functions to the token embeddings. Observe that now, for the rest of the <strong>Encoder,</strong> the word will be represented slightly differently depending on the position the word is in (even if it is the same word).</p><p>Moreover, we would like the <strong>Encoder</strong> to be able to use the fact that some words are in a given position while, in the same sequence, other words are in other specific positions. That is, we want the network to able to understand relative positions and not only absolute ones. The sinuosidal functions chosen by the authors allow positions to be represented as linear combinations of each other and thus allow the network to learn relative relationships between the token positions.</p><p>The approach chosen in the paper to add this information is adding to Z a matrix P with positional encodings.</p><blockquote>Z + P</blockquote><p>The authors chose to use a combination of sinusoidal functions. Mathematically, using <em>i</em> for the position of the token in the sequence and <em>j </em>for the position of the embedding feature:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*xCeAOFp17t-NcWWpF2k9Gw.png" /></figure><p>More specifically, for a given sentence P, the positional embedding matrix would be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i4k32A-DJhdrtuB4Ty76Wg.png" /></figure><p>The authors explain that the result of using this deterministic method instead of learning positional representations (just like we did with the embeddings) lead to similar performance. Moreover, this approach had some specific advantages over learned positional representations:</p><ul><li>The<em> input_length</em> can be increased indefinitely since the functions can be calculated for any arbitrary position.</li><li>Fewer parameters needed to be learned and the model trained quicker.</li></ul><p>The resulting matrix:</p><blockquote>X = Z + P</blockquote><p>is the input of the first encoder block and has dimensions <em>(input_length) x (emb_dim)</em>.</p><h3>Encoder block</h3><p>A total of N encoder blocks are chained together to generate the <strong>Encoder’s </strong>output. A specific block is in charge of<em> finding relationships between the input representations and encode them </em>in its output<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*EblTBhM-9mOqYWMARk6ajQ.png" /><figcaption>Figure 5: Encoder block.</figcaption></figure><p>Intuitively, this iterative process through the blocks will help the neural network capture more complex relationships between words in the input sequence. You can think about it as iteratively building the meaning of the input sequence as a whole.</p><h3>Multi-Head Attention</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*9W5_CpuM3Iq09kOYyK9CeA.png" /><figcaption>Figure 6: Where Multi-Head Attention happens.</figcaption></figure><p>The <strong>Transformer</strong> uses <em>Multi-Head Attention</em>, which means it computes attention <em>h</em> different times with different weight matrices and then concatenates the results together.</p><p>The result of each of those parallel computations of attention is called a <em>head</em>. We are going to denote a specific head and the associated weight matrices with the subscript <em>i</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*m-NRoagK_I5fFvBjjS7TZg.png" /><figcaption>Figure 7: Illustration of the parallel heads computations and their concatenation</figcaption></figure><p>As shown in Figure 7, once all the heads have been computed they will be concatenated. This will result in a matrix of dimensions <em>(input_length) x (h*d_v)</em>. Afterwards, a linear layer with weight matrix W⁰ of dimensions <em>(h*d_v) x (emb_dim)</em> will be applied leading to a final result of dimensions <em>(input_length) x (emb_dim)</em>. Mathematically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*KOP_pGoin2Q8HM63IHp0SQ.png" /></figure><p>Where <em>Q</em>,<em>K</em> and <em>V</em> are placeholders for different input matrices. In particular, for this case <em>Q</em>,<em>K</em> and <em>V</em> will be replaced by the output matrix of the previous step X.</p><h3>Scaled Dot-Product Attention</h3><h4>Overview</h4><p>Each head is going to be characterized by three different projections (matrix multiplications) given by matrices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*gboq9CniDQypmzjJMI07fg.png" /></figure><p>To compute a <em>head</em> we will take the input matrix <em>X</em> and separately project it with the above weight matrices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*XZZ1vsDFlxSsCbQOFsX7EQ.png" /></figure><blockquote><strong>Note</strong>: In the paper d_k and d_v are set such that d_k = d_v = emb_dim/h</blockquote><p>Once we have <em>K_i</em>, <em>Q_i</em> and <em>V_i</em> we use them to compute the <em>Scaled Dot-Product Attention</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*V6LGUR-0NmlOGmm0TDAa5g.png" /></figure><p>Graphically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*nCznYOY-QtWIm8Y4jyk2Kw.png" /><figcaption>Figure 8: Illustration of the Dot-Product Attention.</figcaption></figure><blockquote><strong>Note</strong>: In the encoder block the computation of attention does not use a mask. In our Decoder post we explain how the decoder uses masking.</blockquote><h4>Going Deeper</h4><p>This is the key of the architecture (the name of the paper is no coincidence) so we need to understand it carefully. Let’s start by looking at the matrix product between <em>Q_i</em> and <em>K_i</em> transposed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*szTtSJSZBfej5q-KpLmf3Q.png" /></figure><p>Remember that <em>Q_i</em> and <em>K_i</em> were different projections of the tokens into a <em>d_k</em> dimensional space. Therefore, <em>we can think about the dot product of those projections as a measure of similarity between tokens projections</em>. For every vector projected through <em>Q_i</em> the dot product with the projections through <em>K_i</em> measures the similarity between those vectors. If we call <em>v_i</em> and <em>u_j</em> the projections of the <em>i-th</em> token and the <em>j-th</em> token through <em>Q_i</em> and <em>K_i </em>respectively, their dot product can be seen as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*sVzqg63PXk1iBuZZKKmxjg.png" /></figure><p>Thus, this is a measure of how similar are the directions of <em>u_i </em>and<em> v_j</em> and how large are their lengths (the closest the direction and the larger the length, the greater the dot product).</p><p>Another way of thinking about this matrix product is as the encoding of a specific relationship between each of the tokens in the input sequence (the relationship is defined by the matrices <em>K_i</em>, <em>Q_i</em>).</p><p>After this multiplication, the matrix is divided element-wise by the square root of <em>d_k </em>for scaling purposes.</p><p>The next step is a <strong>Softmax applied row-wise </strong>(one softmax computation for each row):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*T4MCL9SdNaQUC5PVLr-PVQ.png" /></figure><p>In our example, this could be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*pYmeuVvDGqw78yqDfa5A5A.png" /><figcaption>Before Softmax</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j5zenq5fGm9XN2nrFakf4w.png" /><figcaption>After Softmax</figcaption></figure><p>The result would is rows with numbers between zero and one that sum to one. Finally, the result is multiplied by <em>V_i </em>to get the result of the head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*zGdOr4l45qLv0ourRZhdXw.png" /></figure><h4>Example 1</h4><p>For the sake of understanding let’s propose a dummy example. Suppose that the resulting first row of:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*T4MCL9SdNaQUC5PVLr-PVQ.png" /></figure><p>is [0,0,0,0,1,0]. Hence, because 1 is in the 5th position of the vector, the result will then be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4rw40C2zM5LNYJeQsq1fA.png" /></figure><p>Where <em>v_{token}</em> is the projection through <em>V_i</em> of the token’s representation. Observe that in this case the word <em>“hello”</em> ends up with a representation based on the 4th token <em>“you”</em> of the input for that head.</p><p>Supposing an equivalent example for the rest of the heads. The word <em>“Hello”</em> will be now represented by the concatenation of the different projections of other words. <em>The network will learn over training time which relationships are more useful and will relate tokens to each other based on these relationships.</em></p><h4>Example 2</h4><p>Let us now complicate the example a little bit more. Suppose now our previous example in the more general scenario where there isn’t just a single 1 per row but decimal positive numbers that sum to 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*yYjI8hoMkGi0bt4YbXWfTA.png" /></figure><p>If we do as in the previous example and multiply that by <em>V_i</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*_emEwPm4BVw8jIz0chFowg.png" /></figure><p>This results in a matrix where each row is a composition of the projection of the token’s representations through <em>V_i</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*UPGkH-C2Fhs1pNqkBmRIAQ.png" /></figure><p>Observe that we can think about the resulting representation of “Hello” as a weighted combination (centroid) of the projected vectors through <em>V_i</em> of the input tokens.</p><p>Thus, a specific head captures a specific relationship between the input tokens. Now, if we do that <em>h</em> times (a total of <em>h</em> heads) each encoder block is capturing <em>h </em>different relationships between input tokens.</p><p>Following up, assume that the example above referred to the first head. Then the first row would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzvV8LWBUKdscKt-MyD_iQ.png" /></figure><p>Then the first row of the result of the <em>Multi-Head Attention</em> layer, i.e. the representation of “Hello” at this point, would be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*K50k7y_ThfTudF-8FUR5Bg.png" /></figure><p>Which is a vector of length <em>emb_dim</em> given that the matrix <em>W_0</em> has dimensions <em>(d_v*h) x (emb_dim)</em>. Applying the same logic in the rest of the rows/tokens representations we obtain a matrix of dimensions <em>(input_length) x (emb_dim)</em>.</p><p>Thus, at this point, the representation of the token is the concatenation of <em>h</em> weighted combinations of token representations (centroids) through the <em>h</em> different learned projections.</p><h3>Position-wise Feed-Forward Network</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*CQLvEk4zNr_02c8FwwSwCg.png" /><figcaption>Figure 9: Feed Forward</figcaption></figure><p>This step is composed of the following layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*waxVSGTevWvyLjVZM0-Qpg.png" /><figcaption>Figure 10: Scheme of the Feed Forwards Neural Netwrok</figcaption></figure><p>Mathematically, for each row in the output of the previous layer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*Gnmd9gomuuPYZWC1mrGcMg.png" /></figure><p>where <em>W_1</em> and <em>W_2</em> are <em>(emb_dim) x (d_F)</em> and <em>(d_F) x (emb_dim) </em>matrices respectively.</p><p>Observe that during this step, vector representations of tokens don’t “interact” with each other. It is equivalent to run the calculations row-wise and stack the resulting rows in a matrix.</p><p>The output of this step has dimension<em> (input_length) x (emb_dim)</em>.</p><h3>Dropout, Add &amp; Norm</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*gL6twzkQNKw0f4ZkxdrtWw.png" /><figcaption>Figure 11: Where Dropout, Addition and normalization happens.</figcaption></figure><p>Before this layer, there is always a layer for which inputs and outputs have the same dimensions (<em>Multi-Head Attention </em>or <em>Feed-Forward</em>). We will call that layer <em>Sublayer </em>and its input <em>x.</em></p><p>After each <em>Sublayer</em>, dropout is applied with 10% probability. Call this result <em>Dropout(Sublayer(x))</em>. This result is added to the <em>Sublayer</em>’s input <em>x, </em>and we get <em>x + Dropout(Sublayer(x)).</em></p><p>Observe that in the context of a <em>Multi-Head Attention</em> layer, this means adding the original representation of a token <em>x </em>to the representation based on the relationship with other tokens. It is like telling the token:</p><blockquote>“Learn the relationship with the rest of the tokens, but don’t forget what we already learned about yourself!”</blockquote><p>Finally, a token-wise/row-wise normalization is computed with the mean and standard deviation of each row. This improves the stability of the network.</p><p>The output of these layers is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*NIC_d2mmrCPwVlOu1qfDEA.png" /></figure><p>And that’s it! This is the architecture behind all of the magic in state of the art NLP.</p><p><em>If you have any feedback please let us know in the comment section!</em></p><h4>References</h4><p><a href="https://arxiv.org/pdf/1706.03762.pdf">Attention Is All You Need</a>; Vaswani et al., 2017.</p><p><a href="https://arxiv.org/pdf/1810.04805.pdf">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a>; Devlin et al., 2018.</p><p><a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html#embeddings-and-softmax">The Annotated Transformer</a>; Alexander Rush, Vincent Nguyen and Guillaume Klein.</p><p><a href="https://arxiv.org/pdf/1801.06146.pdf">Universal Language Model Fine-tuning for Text Classification</a>; Howard et al., 2018.</p><p><a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">Improving Language Understanding by Generative Pre-Training</a>; Radford et al., 2018.</p><p>Source of cover picture: <a href="https://docplayer.it/40515264-Crittografia-e-numeri-primi-tfa-a059-anna-nobili-ottaviano-rosi.html">Cripttografia e numeri primi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3c3d495cdb3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dissecting-bert/dissecting-bert-part-1-d3c3d495cdb3">Dissecting BERT Part 1: The Encoder</a> was originally published in <a href="https://medium.com/dissecting-bert">Dissecting BERT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>