<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by George Mihaila on Medium]]></title>
        <description><![CDATA[Stories by George Mihaila on Medium]]></description>
        <link>https://medium.com/@gmihaila?source=rss-9c419ab1d0e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-cUabxPv5G_LD-tCOYD2rw.jpeg</url>
            <title>Stories by George Mihaila on Medium</title>
            <link>https://medium.com/@gmihaila?source=rss-9c419ab1d0e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:50:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gmihaila/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[⚙️ Bert Inner Workings]]></title>
            <link>https://gmihaila.medium.com/%EF%B8%8F-bert-inner-workings-1c3054cd1591?source=rss-9c419ab1d0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c3054cd1591</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[George Mihaila]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 17:44:01 GMT</pubDate>
            <atom:updated>2021-01-10T17:48:22.171Z</atom:updated>
            <content:encoded><![CDATA[<h4>Let’s look at how an input flows through Bert.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-eRrWkjtGQxlkv42QhqU9Q.jpeg" /><figcaption>Picture of when I replaced my car’s cylinder head</figcaption></figure><figure><a href="https://colab.research.google.com/github/gmihaila/ml_things/blob/master/notebooks/pytorch/bert_inner_workings.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/458/1*HPyYAbxKrRLVetMtNZmcOQ.png" /></a></figure><figure><a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/bert_inner_workings.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/379/1*xgK8v-H9I8IbshyQdGEwyw.png" /></a></figure><figure><a href="https://www.dropbox.com/s/jeftyo6cebfkma2/bert_inner_workings.ipynb?dl=1"><img alt="" src="https://cdn-images-1.medium.com/max/477/1*_WWiquUwuSrsOHfnlgPHBQ.png" /></a></figure><p><strong>Disclaimer:</strong> <em>The format of this tutorial notebook is very similar to my other tutorial notebooks. This is done intentionally in order to keep readers familiar with my format.</em></p><p><strong>Main idea:</strong></p><p>I created this notebook to better understand the inner workings of Bert. I followed a lot of tutorials to try to understand the architecture, but I was never able to really understand what was happening under the hood. For me it always helps to see the actual code instead of just simple abstract diagrams that a lot of times don’t match the actual implementation. If you’re like me than this tutorial will help!</p><p>I went as deep as you can go with Deep Learning — all the way to the tensor level. For me it helps to see the code and how the tensors move between layers. I feel like this level of abstraction is close enough to the core of the model to perfectly understand the inner workings.</p><p>I will use the implementation of Bert from one of the best NLP library out there — <a href="https://huggingface.co/">HuggingFace</a> <a href="https://github.com/huggingface/transformers">Transformers</a>. More specifically, I will show the inner working of <a href="https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification">Bert For Sequence Classification</a>.</p><p>The term <strong>forward pass</strong> is used in Neural Networks and it refers to the calculations involved from the input sequence all the way to output of the last layer. It’s basically the flow of data from input to output.</p><p>I will follow the code from an example input sequence all the way to the final output prediction.</p><h3>What should I know for this notebook?</h3><p>Some prior knowledge of Bert is needed. I won’t go into any details of how Bert works. For this there is plenty of information out there.</p><p>Since I am using the PyTorch implementation of Bert any knowledge on PyTorch is very useful.</p><p>Knowing a little bit about the <a href="https://github.com/huggingface/transformers">transformers</a> library helps too.</p><h3>How deep are we going?</h3><p>I think the best way to understand such a complex model as Bert is to see the actual layer components that are used. I will dig in the code until I see the actual PyTorch layers used torch.nn. In my opinion there is no need to go deeper than the torch.nn layers.</p><h3>Tutorial Structure</h3><p>Each section contains multiple subsections.</p><p>The order of each section matches the order of the model’s layers from input to output.</p><p>At the beginning of each section of code I created a diagram to illustrate the flow of tensors of that particular code.</p><p>I created the diagrams following the model’s implementation.</p><p>The major section <strong>Bert For Sequence Classification</strong> starts with the <strong>Class Call</strong> that shows how we normally create the Bert model for sequence classification and perform a forward pass. <strong>Class Components</strong> contains the components of BertForSequenceClassification implementation.</p><p>At the end of each major section, I assemble all components from that section and show the output and diagram.</p><p>At the end of the notebook, I have all the code parts and diagrams assembled.</p><h3>Terminology</h3><p>I will use regular deep learning terminology found in most Bert tutorials. I’m using some terms in a slightly different way:</p><ul><li><strong>Layer</strong> and <strong>layers</strong>: In this tutorial when I mention layer it can be an abstraction of a group of layers or just a single layer. When I reach torch.nn you know I refer to a single layer.</li><li>torch.nn: I&#39;m referring to any PyTorch layer module. This is the deepest I will go in this tutorial.</li></ul><h3>How to use this notebook?</h3><p>The purpose of this notebook is purely educational. This notebook is to be used to align known information on how Bert woks with the code implementation of Bert. I used the Bert implementation from <a href="https://github.com/huggingface/transformers">Transformers</a>. My contribution is on arranging the code implementation and creating associated diagrams.</p><h3>Dataset</h3><p>For simplicity I will only use two sentences as our data input: I love cats! and He hates pineapple pizza.. I&#39;ll pretend to do binary sentiment classification on these two sentences.</p><h3>Coding</h3><p>Now let’s do some coding! We will go through each coding cell in the notebook and describe what it does, what’s the code, and when is relevant — show the output.</p><p>I made this format to be easy to follow if you decide to run each code cell in your own python notebook.</p><p>When I learn from a tutorial, I always try to replicate the results. I believe it’s easy to follow along if you have the code next to the explanations.</p><h3>Installs</h3><ul><li><a href="https://github.com/huggingface/transformers"><strong>transformers</strong></a> library needs to be installed to use all the awesome code from Hugging Face. To get the latest version I will install it straight from GitHub.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26d66833de474c102bcbc33cbb149f6f/href">https://medium.com/media/26d66833de474c102bcbc33cbb149f6f/href</a></iframe><pre>Installing build dependencies ... done<br>Getting requirements to build wheel ... done<br>Preparing wheel metadata ... done<br> |████████████████████████████████| 2.9MB 6.7MB/s <br> |████████████████████████████████| 890kB 48.9MB/s <br> |████████████████████████████████| 1.1MB 49.0MB/s <br>Building wheel for transformers (PEP 517) ... done<br>Building wheel for sacremoses (setup.py) ... done<br> |████████████████████████████████| 71kB 5.2MB/s</pre><h3>Imports</h3><p>Import all needed libraries for this notebook.</p><p>Declare parameters used for this notebook:</p><ul><li>set_seed(123) - Always good to set a fixed seed for reproducibility.</li><li>n_labels - How many labels are we using in this dataset. This is used to decide size of classification head.</li><li>ACT2FN - Dictionary for special activation functions used in Bert. We&#39;ll only need the gelu activation function.</li><li>BertLayerNorm - Shortcut for calling the PyTorch normalization layer torch.nn.LayerNorm.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32ed67ad0e091d0a8d4262c839d1ffd6/href">https://medium.com/media/32ed67ad0e091d0a8d4262c839d1ffd6/href</a></iframe><h3>Define Input</h3><p>Let’s define some text data on which we will use Bert to classify as positive or negative.</p><p>We encoded our positive and negative sentiments into:</p><ul><li>0 — for negative sentiments.</li><li>1 — for positive sentiments.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c7546edf42445801ff4eb119baa92fd/href">https://medium.com/media/6c7546edf42445801ff4eb119baa92fd/href</a></iframe><h3>Bert Tokenizer</h3><p>Creating the tokenizer is pretty standard when using the Transformers library.</p><p>Using our newly created tokenizer we&#39;ll use it on our two sentence dataset and create the input_sequence that will be used as input for our Bert model.</p><p>Show Bert Tokenizer Diagram</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6da8959eb2400c642a02eb665404eb3f/href">https://medium.com/media/6da8959eb2400c642a02eb665404eb3f/href</a></iframe><pre>Downloading: 100% |████████████████████████████████| 213k/213k [00:00&lt;00:00, 278kB/s]<br><br>PRETTY PRINT OF `input_sequences` UPDATED WITH `labels`:<br>input_ids : tensor([[  101,   146,  1567, 11771,   106,   102,     0,     0,     0],<br>        [  101,  1124, 18457, 10194, 11478,  7136, 13473,   119,   102]])<br><br>token_type_ids : tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0],<br>        [0, 0, 0, 0, 0, 0, 0, 0, 0]])<br><br>attention_mask : tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0],<br>        [1, 1, 1, 1, 1, 1, 1, 1, 1]])<br><br>labels : tensor([1, 0])<br><br>ORIGINAL TEXT:<br>I love cats!<br>He hates pineapple pizza.<br><br>TEXT AFTER USING `BertTokenizer`:<br>[CLS] I love cats! [SEP] [PAD] [PAD] [PAD]<br>[CLS] He hates pineapple pizza. [SEP]</pre><h3>Bert Configuration</h3><p>Predefined values specific to Bert architecture already defined for us by Hugging Face.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdfa22dd3557d8f4465523b165219974/href">https://medium.com/media/bdfa22dd3557d8f4465523b165219974/href</a></iframe><pre>Downloading: 100% |████████████████████████████████| 433/433 [00:00&lt;00:00, 15.5kB/s]<br><br>NUMBER OF LAYERS: 12<br>EMBEDDING SIZE: 768<br>ACTIVATIONS: gelu</pre><h3>Bert For Sequence Classification</h3><p>I will go over the Bert for Sequence Classification model. This is a Bert language model with a classification layer on top.</p><p>If you plan on looking at other transformers models his tutorial will be very similar.</p><h3>Class Call</h3><p>Let’s start with doing a forward pass using the whole model call from Hugging Face Transformer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d6f8c8d77b41de4d9af45865dc24a3e/href">https://medium.com/media/5d6f8c8d77b41de4d9af45865dc24a3e/href</a></iframe><pre>Downloading: 100% |████████████████████████████████| 436M/436M [00:07&lt;00:00, 61.3MB/s]<br><br>Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ...<br>You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.<br><br>FORWARD PASS OUTPUT: SequenceClassifierOutput(loss=tensor(0.7454), logits=tensor([[ 0.2661, -0.1774],<br>        [ 0.2223, -0.0847]]), hidden_states=None, attentions=None)</pre><h3>Class Components</h3><p>Now let’s look at the code implementation and break down each part of the model and check the outputs.</p><p>Start with the BertForSequenceClassification found in <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L1449">transformers/src/transformers/models/bert/modeling_bert.py#L1449</a>.</p><p>The forward pass uses the following layers:</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L815">BertModel</a> layer:</li></ul><p>self.bert = BertModel(config)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#dropout">torch.nn.Dropout</a> layer for dropout:</li></ul><p>self.dropout = nn.Dropout(config.hidden_dropout_prob)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> layer used for classification:</li></ul><p>self.classifier = nn.Linear(config.hidden_size, config.num_labels)</p><h3>BertModel</h3><p>This is the core Bert model that can be found at: <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L815">transformers/src/transformers/models/bert/modeling_bert.py#L815</a>.</p><p>Hugging Face was nice enough to mention a small summary: <em>The bare Bert Model transformer outputting raw hidden-states without any specific head on top.</em></p><p>The forward pass uses the following layers:</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L165">BertEmbeddings</a> layer:</li></ul><p>self.embeddings = BertEmbeddings(config)</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L512">BertEncoder</a> layer:</li></ul><p>self.encoder = BertEncoder(config)</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L601">BertPooler</a> layer:</li></ul><p>self.pooler = BertPooler(config)</p><h4>Bert Embeddings</h4><p>This is where we feed the input_sequences created under <strong>Bert Tokenizer</strong> and get our first embeddings.</p><p>Implementation can be found at: <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L165">transformers/src/transformers/models/bert/modeling_bert.py#L165</a>.</p><p>This layer contains actual PyTorch layers. I won’t go into farther details since this is how far we need to go.</p><p>The forward pass uses following layers:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#embedding">torch.nn.Embedding</a> layer for word embeddings:</li></ul><p>self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#embedding">torch.nn.Embedding</a> layer for position embeddings:</li></ul><p>self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#embedding">torch.nn.Embedding</a> for token type embeddings:</li></ul><p>self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#layernorm">torch.nn.LayerNorm</a> layer for normalization:</li></ul><p>self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#dropout">torch.nn.Dropout</a> layer for dropout:</li></ul><p>self.dropout = nn.Dropout(config.hidden_dropout_prob)</p><figure><img alt="Bert Embeddings Diagram" src="https://cdn-images-1.medium.com/max/1024/1*aDnh89Sd4j1gjlc6rO-EDA.png" /><figcaption>Bert Embeddings Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/060531e8e34c53e58d1afc702f809bbb/href">https://medium.com/media/060531e8e34c53e58d1afc702f809bbb/href</a></iframe><pre>Created Tokens Positions IDs:<br> tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8]])<br><br>Tokens IDs:<br> torch.Size([2, 9])<br><br>Tokens Type IDs:<br> torch.Size([2, 9])<br><br>Word Embeddings:<br> torch.Size([2, 9, 768])<br><br>Position Embeddings:<br> torch.Size([1, 9, 768])<br><br>Token Types Embeddings:<br> torch.Size([2, 9, 768])<br><br>Sum Up All Embeddings:<br> torch.Size([2, 9, 768])<br><br>Embeddings Layer Nromalization:<br> torch.Size([2, 9, 768])<br><br>Embeddings Dropout Layer:<br> torch.Size([2, 9, 768])</pre><h4>Bert Encoder</h4><p>This layer contains the core of the bert model where the self-attention happens.</p><p>The implementation can be found at: <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L512">transformers/src/transformers/models/bert/modeling_bert.py#L512</a>.</p><p>The forward pass uses:</p><ul><li>12 of the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L429">BertLayer</a> layers ( in this setup config.num_hidden_layers=12):</li></ul><p>self.layer = nn.ModuleList([BertLayer(config) for _ in range(config.num_hidden_layers)])</p><p><strong>BERT LAYER</strong></p><p>This layer contains basic components of the self-attention implementation.</p><p>Implementation can be found at <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L429">transformers/src/transformers/models/bert/modeling_bert.py#L429</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L351">BertAttention</a> layer:</li></ul><p>self.attention = BertAttention(config)</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L400">BertIntermediate</a> layer:</li></ul><p>self.intermediate = BertIntermediate(config)</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L415">BertOutput</a> layer:</li></ul><p>self.output = BertOutput(config)</p><p><strong>Bert Attention</strong></p><p>This layer contains basic components of the self-attention implementation.</p><p>Implementation can be found at <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L351">transformers/src/transformers/models/bert/modeling_bert.py#L351</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L212">BertSelfAttention</a> layer:</li></ul><p>self.self = BertSelfAttention(config)</p><ul><li><a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L337">BertSelfOutput</a> layer:</li></ul><p>self.output = BertSelfOutput(config)</p><p><strong>BertSelfAttention</strong></p><p>This layer contains the torch.nn basic components of the self-attention implementation.</p><p>Implementation can be found at <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L212">transformers/src/transformers/models/bert/modeling_bert.py#L212</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> used for the Query layer:</li></ul><p>self.query = nn.Linear(config.hidden_size, self.all_head_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> used for the Key layer:</li></ul><p>self.key = nn.Linear(config.hidden_size, self.all_head_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> used for the Value layer:</li></ul><p>self.value = nn.Linear(config.hidden_size, self.all_head_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#dropout">torch.nn.Dropout</a> layer for dropout:</li></ul><p>self.dropout = nn.Dropout(config.attention_probs_dropout_prob)</p><figure><img alt="BertSelfAttention Diagram" src="https://cdn-images-1.medium.com/max/1024/1*zJfSP5ypukW7oJMF-Br-xA.png" /><figcaption>BertSelfAttention Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4103a23927ae58e44d77ba1527baa5b4/href">https://medium.com/media/4103a23927ae58e44d77ba1527baa5b4/href</a></iframe><pre>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Query Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Key Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Value Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Query:<br> torch.Size([2, 12, 9, 64])<br><br>Key:<br> torch.Size([2, 12, 9, 64])<br><br>Value:<br> torch.Size([2, 12, 9, 64])<br><br>Key Transposed:<br> torch.Size([2, 12, 64, 9])<br><br>Attention Scores:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Scores Divided by Scalar:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Softmax Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Dropout Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Context:<br> torch.Size([2, 12, 9, 64])<br><br>Context Permute:<br> torch.Size([2, 9, 12, 64])<br><br>Context Reshaped:<br> torch.Size([2, 9, 768])</pre><p><strong>BertSelfOutput</strong></p><p>This layer contains the torch.nn basic components of the self-attention implementation.</p><p>Implementation can be found at <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L337">transformers/src/transformers/models/bert/modeling_bert.py#L337</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> layer:</li></ul><p>self.dense = nn.Linear(config.hidden_size, config.hidden_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#layernorm">torch.nn.LayerNorm</a> layer for normalization:</li></ul><p>self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#dropout">torch.nn.Dropout</a> layer for dropout:</li></ul><p>self.dropout = nn.Dropout(config.hidden_dropout_prob)</p><figure><img alt="BertSelfOutput Diagram" src="https://cdn-images-1.medium.com/max/1021/1*1vtEMMS1LH0wvYIsZrUQbA.png" /><figcaption>BertSelfOutput Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/879b7d30e82d644673cafc1076d9b078/href">https://medium.com/media/879b7d30e82d644673cafc1076d9b078/href</a></iframe><pre>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Normalization Layer:<br> torch.Size([2, 9, 768])</pre><p><strong>Assemble BertAttention</strong></p><p>Put together <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L212"><strong>BertSelfAttention</strong></a> layer and <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L337"><strong>BertSelfOutput</strong></a> layer to create the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L351"><strong>BertAttention</strong></a><strong> layer</strong>.</p><p>Now perform a forward pass using previous output layer as input.</p><figure><img alt="BertAttention Diagram" src="https://cdn-images-1.medium.com/max/1024/1*8mx-hAMQ7lHXe8MqqG0ADQ.png" /><figcaption>BertAttention Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91468b10ae70b24eead1589f5dfd2910/href">https://medium.com/media/91468b10ae70b24eead1589f5dfd2910/href</a></iframe><pre>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Query Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Key Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Value Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Query:<br> torch.Size([2, 12, 9, 64])<br><br>Key:<br> torch.Size([2, 12, 9, 64])<br><br>Value:<br> torch.Size([2, 12, 9, 64])<br><br>Key Transposed:<br> torch.Size([2, 12, 64, 9])<br><br>Attention Scores:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Scores Divided by Scalar:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Softmax Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Dropout Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Context:<br> torch.Size([2, 12, 9, 64])<br><br>Context Permute:<br> torch.Size([2, 9, 12, 64])<br><br>Context Reshaped:<br> torch.Size([2, 9, 768])<br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Normalization Layer:<br> torch.Size([2, 9, 768])</pre><p><strong>BertIntermediate</strong></p><p>This layer contains the torch.nn basic components of the Bert model implementation.</p><p>Implementation can be found at <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L400">transformers/src/transformers/models/bert/modeling_bert.py#L400</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> layer:</li></ul><p>self.dense = nn.Linear(config.hidden_size, config.intermediate_size)</p><figure><img alt="BertIntermediate Diagram" src="https://cdn-images-1.medium.com/max/1024/1*bnLcgjiNwa2VhZX8QUielQ.png" /><figcaption>BertIntermediate Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c16169574727b4450efb82d38d2345db/href">https://medium.com/media/c16169574727b4450efb82d38d2345db/href</a></iframe><pre>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 3072])<br><br>Hidden States Gelu Activation Function:<br> torch.Size([2, 9, 3072])</pre><p><strong>BertOutput</strong></p><p>This layer contains the torch.nn basic components of the Bert model implementation.</p><p>Implementation can be found at <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L415">transformers/src/transformers/models/bert/modeling_bert.py#L415</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> layer:</li></ul><p>self.dense = nn.Linear(config.intermediate_size, config.hidden_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#layernorm">torch.nn.LayerNorm</a> layer for normalization:</li></ul><p>self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#dropout">torch.nn.Dropout</a> layer for dropout:</li></ul><p>self.dropout = nn.Dropout(config.hidden_dropout_prob)</p><figure><img alt="BertOutput Diagram" src="https://cdn-images-1.medium.com/max/1024/1*lvraYHH-rQMWIA45s0cHEw.png" /><figcaption>BertOutput Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0b257b8528cc1ca159f650324e01255/href">https://medium.com/media/f0b257b8528cc1ca159f650324e01255/href</a></iframe><pre>Hidden States:<br> torch.Size([2, 9, 3072])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Layer Normalization:<br> torch.Size([2, 9, 768])</pre><p><strong>Assemble BertLayer</strong></p><p>Put together <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L351"><strong>BertAttention</strong></a> layer, <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L400"><strong>BertIntermediate</strong></a> layer and <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L415"><strong>BertOutput</strong></a> layer to create the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L429"><strong>BertLayer</strong></a><strong> layer</strong>.</p><p>Now perform a forward pass using previous output layer as input.</p><figure><img alt="BertLayer Diagram" src="https://cdn-images-1.medium.com/max/1024/1*ZyvPk0KO-pF5-f1lUHi6FQ.png" /><figcaption>BertLayer Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49b0fa5439ceb3885f8b92d5abcd5dff/href">https://medium.com/media/49b0fa5439ceb3885f8b92d5abcd5dff/href</a></iframe><pre>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Query Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Key Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Value Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Query:<br> torch.Size([2, 12, 9, 64])<br><br>Key:<br> torch.Size([2, 12, 9, 64])<br><br>Value:<br> torch.Size([2, 12, 9, 64])<br><br>Key Transposed:<br> torch.Size([2, 12, 64, 9])<br><br>Attention Scores:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Scores Divided by Scalar:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Softmax Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Dropout Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Context:<br> torch.Size([2, 12, 9, 64])<br><br>Context Permute:<br> torch.Size([2, 9, 12, 64])<br><br>Context Reshaped:<br> torch.Size([2, 9, 768])<br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Normalization Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 3072])<br><br>Hidden States Gelu Activation Function:<br> torch.Size([2, 9, 3072])<br><br>Hidden States:<br> torch.Size([2, 9, 3072])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Layer Normalization:<br> torch.Size([2, 9, 768])</pre><p><strong>Assemble BertEncoder</strong></p><p>Put together 12 of the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L429"><strong>BertLayer</strong></a> layers ( in this setup config.num_hidden_layers=12) to create the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L512"><strong>BertEncoder</strong></a> layer.</p><p>Now perform a forward pass using previous output layer as input.</p><figure><img alt="BertEncoder Diagram" src="https://cdn-images-1.medium.com/max/1024/1*1xQGl0nVC7_8STUg3RJhvw.png" /><figcaption>BertEncoder Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b066916db360e3deaa59809f988bf840/href">https://medium.com/media/b066916db360e3deaa59809f988bf840/href</a></iframe><pre>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br><br>----------------- BERT LAYER 1 -----------------<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Query Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Key Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Value Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Query:<br> torch.Size([2, 12, 9, 64])<br><br>Key:<br> torch.Size([2, 12, 9, 64])<br><br>Value:<br> torch.Size([2, 12, 9, 64])<br><br>Key Transposed:<br> torch.Size([2, 12, 64, 9])<br><br>Attention Scores:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Scores Divided by Scalar:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Softmax Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Attention Probabilities Dropout Layer:<br> torch.Size([2, 12, 9, 9])<br><br>Context:<br> torch.Size([2, 12, 9, 64])<br><br>Context Permute:<br> torch.Size([2, 9, 12, 64])<br><br>Context Reshaped:<br> torch.Size([2, 9, 768])<br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Normalization Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 3072])<br><br>Hidden States Gelu Activation Function:<br> torch.Size([2, 9, 3072])<br><br>Hidden States:<br> torch.Size([2, 9, 3072])<br><br>Hidden States Linear Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>Hidden States Layer Normalization:<br> torch.Size([2, 9, 768])<br><br>----------------- BERT LAYER 2 -----------------<br><br>...<br><br>----------------- BERT LAYER 12 -----------------</pre><h4>BertPooler</h4><p>This layer contains the core of the bert model where the self-attention happens.</p><p>The implementation can be found at: <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L601">transformers/src/transformers/models/bert/modeling_bert.py#L601</a>.</p><p>The forward pass uses:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear">torch.nn.Linear</a> layer:</li></ul><p>self.dense = torch.nn.Linear(config.hidden_size, config.hidden_size)</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Tanh.html#tanh">torch.nn.Tanh</a> activation function layer:</li></ul><p>self.activation = torch.nn.Tanh()</p><figure><img alt="BertPooler Diagram" src="https://cdn-images-1.medium.com/max/1024/1*BdH6U7GpMFj1AWN1_SKMOg.png" /><figcaption>BertPooler Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b74a3c001f56b1e0ecbd31af6979199e/href">https://medium.com/media/b74a3c001f56b1e0ecbd31af6979199e/href</a></iframe><pre>Hidden States:<br> torch.Size([2, 9, 768])<br><br>First Token [CLS]:<br> torch.Size([2, 768])<br><br>First Token [CLS] Linear Layer:<br> torch.Size([2, 768])<br><br>First Token [CLS] Tanh Activation Function:<br> torch.Size([2, 768])</pre><h4>Assemble BertModel</h4><p>Put together <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L165"><strong>BertEmbeddings</strong></a> layer, <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L512"><strong>BertEncoder</strong></a> layer and <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L601"><strong>BertPooler</strong></a> layer to create the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L815"><strong>BertModel</strong></a> layer.</p><p>Now perform a forward pass using previous output layer as input.</p><figure><img alt="BertModel Diagram" src="https://cdn-images-1.medium.com/max/1024/1*wzED9ex4RIDqdU9M2Bf9qQ.png" /><figcaption>BertModel Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62f729571a968680af2299276873f3bb/href">https://medium.com/media/62f729571a968680af2299276873f3bb/href</a></iframe><pre>Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Attention Head Size:<br>  64<br> <br> Combined Attentions Head Size:<br>  768<br> Created Tokens Positions IDs:<br>  tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8]])<br> <br> Tokens IDs:<br>  torch.Size([2, 9])<br> <br> Tokens Type IDs:<br>  torch.Size([2, 9])<br> <br> Word Embeddings:<br>  torch.Size([2, 9, 768])<br> <br> Position Embeddings:<br>  torch.Size([1, 9, 768])<br> <br> Token Types Embeddings:<br>  torch.Size([2, 9, 768])<br> <br> Sum Up All Embeddings:<br>  torch.Size([2, 9, 768])<br> <br> Embeddings Layer Nromalization:<br>  torch.Size([2, 9, 768])<br> <br> Embeddings Dropout Layer:<br>  torch.Size([2, 9, 768])<br> <br> ----------------- BERT LAYER 1 -----------------<br> <br> ...<br> <br> ----------------- BERT LAYER 12 -----------------<br> <br>...<br> <br> Hidden States:<br>  torch.Size([2, 9, 768])<br> <br> First Token [CLS]:<br>  torch.Size([2, 768])<br> <br> First Token [CLS] Linear Layer:<br>  torch.Size([2, 768])<br> <br> First Token [CLS] Tanh Activation Function:<br>  torch.Size([2, 768])</pre><h3>Assemble Components</h3><p>Put together <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L815"><strong>BertModel</strong></a> layer, <a href="https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html#dropout"><strong>torch.nn.Dropout</strong></a> layer and <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear"><strong>torch.nn.Linear</strong></a> layer to create the <a href="https://github.com/huggingface/transformers/blob/d944966b19a4d6860bddc7cdc1ba928ca8a0da91/src/transformers/models/bert/modeling_bert.py#L1449"><strong>BertForSequenceClassification</strong></a> model.</p><p>Now perform a forward pass using previous output layer as input.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/484704606241829a4603009eb92211fc/href">https://medium.com/media/484704606241829a4603009eb92211fc/href</a></iframe><pre>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Attention Head Size:<br> 64<br><br>Combined Attentions Head Size:<br> 768<br>Created Tokens Positions IDs:<br> tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8]])<br><br>Tokens IDs:<br> torch.Size([2, 9])<br><br>Tokens Type IDs:<br> torch.Size([2, 9])<br><br>Word Embeddings:<br> torch.Size([2, 9, 768])<br><br>Position Embeddings:<br> torch.Size([1, 9, 768])<br><br>Token Types Embeddings:<br> torch.Size([2, 9, 768])<br><br>Sum Up All Embeddings:<br> torch.Size([2, 9, 768])<br><br>Embeddings Layer Nromalization:<br> torch.Size([2, 9, 768])<br><br>Embeddings Dropout Layer:<br> torch.Size([2, 9, 768])<br><br>----------------- BERT LAYER 1 -----------------<br><br>...<br><br>----------------- BERT LAYER 12 -----------------<br><br>...<br><br>Hidden States:<br> torch.Size([2, 9, 768])<br><br>First Token [CLS]:<br> torch.Size([2, 768])<br><br>First Token [CLS] Linear Layer:<br> torch.Size([2, 768])<br><br>First Token [CLS] Tanh Activation Function:<br> torch.Size([2, 768])</pre><h3>Complete Diagram</h3><ul><li>If you want a <strong>.pdf</strong> version of this diagram: <a href="https://github.com/gmihaila/ml_things/raw/master/notebooks/pytorch/bert_inner_workings/bert_inner_workings.pdf">bert_inner_workings.pdf</a>.</li><li>If you want a <strong>.png</strong> version of this diagram: <a href="https://github.com/gmihaila/ml_things/raw/master/notebooks/pytorch/bert_inner_workings/bert_inner_workings.png">bert_inner_workings.png</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zrsCmjlPZZNFRF52.png" /></figure><h3>Final Note</h3><p>If you made it this far <strong>Congrats!</strong> 🎊 and <strong>Thank you!</strong> 🙏 for your interest in my tutorial!</p><p>I’ve been using this code for a while now and I feel it got to a point where is nicely documented and easy to follow.</p><p>Of course is easy for me to follow because I built it. That is why any feedback is welcome and it helps me improve my future tutorials!</p><p>If you see something wrong please let me know by opening an issue on my <a href="https://github.com/gmihaila/ml_things/issues">ml_things GitHub repository</a>!</p><p>A lot of tutorials out there are mostly a one-time thing and are not being maintained. I plan on keeping my tutorials up to date as much as I can.</p><h3>Contact 🎣</h3><p>🦊 GitHub: <a href="https://github.com/gmihaila">gmihaila</a></p><p>🌐 Website: <a href="https://gmihaila.github.io/">gmihaila.github.io</a></p><p>👔 LinkedIn: <a href="https://www.linkedin.com/in/mihailageorge">mihailageorge</a></p><p>📬 Email: georgemihaila@my.unt.edu.com</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href">https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href</a></iframe><p><em>Originally published at </em><a href="https://gmihaila.github.io/tutorial_notebooks/gpt2_finetune_classification/"><em>https://gmihaila.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c3054cd1591" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ GPT2 For Text Classification using Hugging Face  Transformers]]></title>
            <link>https://gmihaila.medium.com/gpt2-for-text-classification-using-hugging-face-transformers-574555451832?source=rss-9c419ab1d0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/574555451832</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[George Mihaila]]></dc:creator>
            <pubDate>Thu, 26 Nov 2020 04:02:42 GMT</pubDate>
            <atom:updated>2020-11-27T22:13:24.784Z</atom:updated>
            <content:encoded><![CDATA[<h4>Complete tutorial on how to use GPT2 for text classification!</h4><figure><img alt="Background image" src="https://cdn-images-1.medium.com/max/1024/1*4outcWur4FrNtEFprir6Cg.jpeg" /></figure><figure><a href="https://colab.research.google.com/github/gmihaila/ml_things/blob/master/notebooks/pytorch/gpt2_finetune_classification.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/458/1*HPyYAbxKrRLVetMtNZmcOQ.png" /></a></figure><figure><a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/gpt2_finetune_classification.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/379/1*xgK8v-H9I8IbshyQdGEwyw.png" /></a></figure><figure><a href="https://www.dropbox.com/s/6t6kvlewoabwxqw/gpt2_finetune_classification.ipynb?dl=1"><img alt="" src="https://cdn-images-1.medium.com/max/477/1*_WWiquUwuSrsOHfnlgPHBQ.png" /></a></figure><p><strong>Disclaimer:</strong> <em>The format of this tutorial notebook is very similar to my other tutorial notebooks. This is done intentionally in order to keep readers familiar with my format.</em></p><p>This notebook is used to fine-tune GPT2 model for text classification using <a href="https://huggingface.co/transformers/">Huggingface</a> <a href="https://github.com/huggingface/transformers">transformers</a> library on a custom dataset.</p><p>Hugging Face is very nice to us to include all the functionality needed for GPT2 to be used in classification tasks. Thank you Hugging Face!</p><p>I wasn’t able to find much information on how to use GPT2 for classification so I decided to make this tutorial using similar structure with other transformers models.</p><p><strong>Main idea:</strong> Since GPT2 is a decoder transformer, the last token of the input sequence is used to make predictions about the next token that should follow the input. This means that the last token of the input sequence contains all the information needed in the prediction. With this in mind we can use that information to make a prediction in a classification task instead of generation task.</p><p>In other words, instead of using first token embedding to make prediction like we do in Bert, we will use the last token embedding to make prediction with GPT2.</p><p>Since we only cared about the first token in Bert, we were padding to the right. Now in GPT2 we are using the last token for prediction so we will need to pad on the left. Because of a nice upgrade to HuggingFace Transformers we are able to configure the GPT2 Tokenizer to do just that.</p><h3>What should I know for this notebook?</h3><p>Since I am using PyTorch to fine-tune our transformers models any knowledge on PyTorch is very useful.</p><p>Knowing a little bit about the <a href="https://github.com/huggingface/transformers">transformers</a> library helps too.</p><h3>How to use this notebook?</h3><p>Like with every project, I built this notebook with reusability in mind.</p><p>All changes will happen in the data processing part where you need to customize the PyTorch Dataset, Data Collator and DataLoader to fit your own data needs.</p><p>All parameters that can be changed are under the <strong>Imports</strong> section. Each parameter is nicely commented and structured to be as intuitive as possible.</p><h3>Dataset</h3><p>This notebook will cover pretraining transformers on a custom dataset. I will use the well known movies reviews positive — negative labeled <a href="https://ai.stanford.edu/~amaas/data/sentiment/">Large Movie Review Dataset</a>.</p><p>The description provided on the Stanford website:</p><p><em>This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. See the README file contained in the release for more details.</em></p><p><strong>Why this dataset?</strong> I believe is an easy to understand and use dataset for classification. I think sentiment data is always fun to work with.</p><h3>Coding</h3><p>Now let’s do some coding! We will go through each coding cell in the notebook and describe what it does, what’s the code, and when is relevant — show the output.</p><p>I made this format to be easy to follow if you decide to run each code cell in your own python notebook.</p><p>When I learn from a tutorial I always try to replicate the results. I believe it’s easy to follow along if you have the code next to the explanations.</p><h3>Downloads</h3><p>Download the <em>Large Movie Review Dataset</em> and unzip it locally.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href">https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href</a></iframe><h3>Installs</h3><ul><li><a href="https://github.com/huggingface/transformers"><strong>transformers</strong></a> library needs to be installed to use all the awesome code from Hugging Face. To get the latest version I will install it straight from GitHub.</li><li><a href="https://github.com/gmihaila/ml_things"><strong>ml_things</strong></a> library used for various machine learning related tasks. I created this library to reduce the amount of code I need to write for each machine learning project.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9adde78f0b99c5e8e998b5dc23e42bb4/href">https://medium.com/media/9adde78f0b99c5e8e998b5dc23e42bb4/href</a></iframe><pre>Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done  <br> |████████████████████████████████| 2.9MB 6.7MB/s   <br> |████████████████████████████████| 890kB 48.9MB/s   <br> |████████████████████████████████| 1.1MB 49.0MB/s  <br>Building wheelfor transformers (PEP 517) ... done <br>Building wheel for sacremoses (setup.py) ... done  <br> |████████████████████████████████| 71kB 5.2MB/s  <br>Building wheel for ml-things (setup.py) ... done <br>Building wheel for ftfy (setup.py) ... done</pre><h3>Imports</h3><p>Import all needed libraries for this notebook.Declare parameters used for this notebook:</p><ul><li>set_seed(123) - Always good to set a fixed seed for reproducibility.</li><li>epochs - Number of training epochs (authors recommend between 2 and 4).</li><li>batch_size - Number of batches - depending on the max sequence length and GPU memory. For 512 sequence length a batch of 10 USUALY works without cuda memory issues. For small sequence length can try batch of 32 or higher. max_length - Pad or truncate text sequences to a specific length. I will set it to 60 to speed up training.</li><li>device - Look for gpu to use. Will use cpu by default if no gpu found.</li><li>model_name_or_path - Name of transformers model - will use already pretrained model. Path of transformer model - will load your own model from local disk. In this tutorial I will use gpt2 model.</li><li>labels_ids - Dictionary of labels and their id - this will be used to convert string labels to numbers.</li><li>n_labels - How many labels are we using in this dataset. This is used to decide size of classification head.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0209e890c074c8616b88968d4917933e/href">https://medium.com/media/0209e890c074c8616b88968d4917933e/href</a></iframe><h3>Helper Functions</h3><p>I like to keep all Classes and functions that will be used in this notebook under this section to help maintain a clean look of the notebook:</p><p><strong>MovieReviewsDataset(Dataset)</strong></p><p>If you worked with PyTorch before, this is pretty standard. We need this class to read in our dataset, parse it and return texts with their associated labels.</p><p>In this class I only need to read in the content of each file, use fix_text to fix any Unicode problems and keep track of positive and negative sentiments.</p><p>I will append all texts and labels in lists.</p><p>There are three main parts of this PyTorch Dataset class:</p><ul><li><strong>init()</strong> where we read in the dataset and transform text and labels into numbers.</li><li><strong>len()</strong> where we need to return the number of examples we read in. This is used when calling len(MovieReviewsDataset()).</li><li><strong>getitem()</strong> always takes as an input an int value that represents which example from our examples to return from our dataset. If a value of 3 is passed, we will return the example form our dataset at position 3.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbd554df452738bc0f256ebdbc9b6e07/href">https://medium.com/media/cbd554df452738bc0f256ebdbc9b6e07/href</a></iframe><p><strong>Gpt2ClassificationCollator</strong></p><p>I use this class to create the Data Collator. This will be used in the DataLoader to create the bathes of data that get fed to the model. I use the tokenizer and label encoder on each sequence to convert texts and labels to number.</p><p>Lucky for us, Hugging Face thought of everything and made the tokenizer do all the heavy lifting (split text into tokens, padding, truncating, encode text into numbers) and is very easy to use!</p><p>There are two main parts of this Data Collator class:</p><ul><li><strong>init()</strong> where we initialize the tokenizer we plan to use, how to encode our labels and if we need to set the sequence length to a different value.</li><li><strong>call()</strong> used as function collator that takes as input a batch of data examples. It needs to return an object with the format that can be fed to our model. Luckily our tokenizer does that for us and returns a dictionary of variables ready to be fed to the model in this way: model(**inputs). Since we are fine-tuning the model I also included the labels.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/754306925c2b98db63fb2389163f4886/href">https://medium.com/media/754306925c2b98db63fb2389163f4886/href</a></iframe><p><strong>train(dataloader, optimizer_, scheduler_, device_)</strong></p><p>I created this function to perform a full pass through the DataLoader object (the DataLoader object is created from our Dataset* type object using the **MovieReviewsDataset class). This is basically one epoch train through the entire dataset.</p><p>The dataloader is created from PyTorch DataLoader which takes the object created from MovieReviewsDataset class and puts each example in batches. This way we can feed our model batches of data!</p><p>The optimizer_ and scheduler_ are very common in PyTorch. They are required to update the parameters of our model and update our learning rate during training. There is a lot more than that but I won’t go into details. This can actually be a huge rabbit hole since A LOT happens behind these functions that we don’t need to worry. Thank you PyTorch!</p><p>In the process we keep track of the actual labels and the predicted labels along with the loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9344e858d82ace7096ff6fbfbe00fd81/href">https://medium.com/media/9344e858d82ace7096ff6fbfbe00fd81/href</a></iframe><p><strong>validation(dataloader, device_)</strong></p><p>I implemented this function in a very similar way as train but without the parameters update, backward pass and gradient decent part. We don’t need to do all of those VERY computationally intensive tasks because we only care about our model’s predictions.</p><p>I use the DataLoader in a similar way as in train to get out batches to feed to our model.</p><p>In the process I keep track of the actual labels and the predicted labels along with the loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e55d28c407534b832266f48f5d988144/href">https://medium.com/media/e55d28c407534b832266f48f5d988144/href</a></iframe><h3>Load Model and Tokenizer</h3><p>Loading the three essential parts of the pretrained GPT2 transformer: configuration, tokenizer and model.</p><p>For this example I will use gpt2 from HuggingFace pretrained transformers. You can use any variations of GP2 you want.</p><p>In creating the model_config I will mention the number of labels I need for my classification task. Since I only predict two sentiments: positive and negative I will only need two labels for num_labels.</p><p>Creating the tokenizer is pretty standard when using the Transformers library. After creating the tokenizer it is critical for this tutorial to set padding to the left tokenizer.padding_side = &quot;left&quot; and initialize the padding token to tokenizer.eos_token which is the GPT2&#39;s original end of sequence token. This is the most essential part of this tutorial since GPT2 uses the last token for prediction so we need to pad to the left.</p><p>HuggingFace already did most of the work for us and added a classification layer to the GPT2 model. In creating the model I used GPT2ForSequenceClassification. Since we have a custom padding token we need to initialize it for the model using model.config.pad_token_id. Finally we will need to move the model to the device we defined earlier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57e7bfeb2c162e08bcaf72d5adad56b0/href">https://medium.com/media/57e7bfeb2c162e08bcaf72d5adad56b0/href</a></iframe><pre>Loading configuraiton... <br>Loading tokenizer... <br>Loading model... <br>Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at gpt2 and are newly initialized: [&#39;score.weight&#39;] <br>You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. <br>Model loaded to `cuda`</pre><h3>Dataset and Collator</h3><p>This is where I create the PyTorch Dataset and Data Loader with Data Collator objects that will be used to feed data into our model.</p><p>This is where I use the <strong>MovieReviewsDataset</strong> class to create the PyTorch Dataset that will return texts and labels.</p><p>Since we need to input numbers to our model we need to convert the texts and labels to numbers. This is the purpose of a collator! It takes data outputted by the PyTorch Dataset and passed through the Data Collator function to output the sequence for our model.</p><p>I’m keeping the tokenizer away from the PyTorch Dataset to make the code cleaner and better structured. You can obviously use the tokenizer inside the PyTorch Dataset and output sequences that can be used straight into the model without using a Data Collator.</p><p>I strongly recommend to use a validation text file in order to determine how much training is needed in order to avoid overfitting. After you figure out what parameters yield the best results, the validation file can be incorporated in train and run a final train with the whole dataset.</p><p>The data collator is used to format the PyTorch Dataset outputs to match the inputs needed for GPT2.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/88da744feed0060231448ef99bf3482f/href">https://medium.com/media/88da744feed0060231448ef99bf3482f/href</a></iframe><pre>Dealing with Train... <br>pos files: 100%|████████████████████████████████|12500/12500 [01:17&lt;00:00, 161.19it/s] <br>neg files: 100%|████████████████████████████████|12500/12500 [01:05&lt;00:00, 190.72it/s]  </pre><pre>Created `train_dataset` with 25000 examples! <br>Created `train_dataloader` with 782 batches! <br>Reading pos files... <br>pos files: 100%|████████████████████████████████|12500/12500 [00:54&lt;00:00, 230.93it/s] <br>neg files: 100%|████████████████████████████████|12500/12500 [00:42&lt;00:00, 291.07it/s] <br> <br>Created `valid_dataset` with 25000 examples! <br>Created `eval_dataloader` with 782 batches!</pre><h3>Train</h3><p>I created optimizer and scheduler use by PyTorch in training. I used most common parameters used by transformers models.</p><p>I looped through the number of defined epochs and call the <strong>train</strong> and <strong>validation</strong> functions.</p><p>I’m trying to output similar info after each epoch as Keras: <em>train_loss: — val_loss: — train_acc: — valid_acc</em>.</p><p>After training, plot train and validation loss and accuracy curves to check how the training went.</p><p><strong>Note:</strong> <em>The training plots might look a little weird: The validation accuracy starts higher than training accuracy and the validation loss starts lower than the training loss. Normally this will be the opposite. I assume the data split just happen to be easier for the validation part or too hard for training part or both. Since this tutorial is about using GPT2 for classification I will not worry about the results of the model too much.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fc357c3aaabd733121bf2bd12737ea5/href">https://medium.com/media/6fc357c3aaabd733121bf2bd12737ea5/href</a></iframe><pre>Epoch <br>100%|████████████████████████████████|4/4 [15:11&lt;00:00, 227.96s/it]  </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782 [02:42&lt;00:00, 4.82it/s]  </pre><pre>Validation on batches... <br>100%|████████████████████████████████|782/782 [02:07&lt;00:00, 6.13it/s]    <br>  <br>train_loss: 0.54128 - val_loss: 0.38758 - train_acc: 0.75288 - valid_acc: 0.81904   </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782 [02:36&lt;00:00, 5.00it/s]  </pre><pre>Validation on batches... <br>100%|████████████████████████████████|782/782 [01:41&lt;00:00, 7.68it/s]    </pre><pre>train_loss: 0.36716 - val_loss: 0.37620 - train_acc: 0.83288 -valid_acc: 0.82912   </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782 [02:36&lt;00:00, 5.00it/s]  </pre><pre>Validation on batches... <br>100%|████████████████████████████████|782/782 [01:24&lt;00:00, 9.24it/s]    </pre><pre>train_loss: 0.31409 - val_loss: 0.39384 - train_acc: 0.86304 - valid_acc: 0.83044   </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782 [02:36&lt;00:00, 4.99it/s]  </pre><pre>Validation on batches... <br>100%|████████████████████████████████|782/782 [01:09&lt;00:00, 11.29it/s]    </pre><pre>train_loss: 0.27358 - val_loss: 0.39798 - train_acc: 0.88432 - valid_acc: 0.83292</pre><figure><img alt="Train and validation loss." src="https://cdn-images-1.medium.com/max/857/1*4Nfd0atckiTsCmy5mDKqGQ.png" /><figcaption>Train and validation loss.</figcaption></figure><figure><img alt="Train and validation accuracy." src="https://cdn-images-1.medium.com/max/857/1*BBDRT3hNWI8MqYSGP6_WbQ.png" /><figcaption>Train and validation accuracy.</figcaption></figure><h3>Evaluate</h3><p>When dealing with classification is useful to look at precision recall and F1 score.</p><p>A good gauge to have when evaluating a model is the confusion matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0604ec850939853f9b9414fd20a2445b/href">https://medium.com/media/0604ec850939853f9b9414fd20a2445b/href</a></iframe><pre>Training on batches... <br>100%|████████████████████████████████|782/782 [01:09&lt;00:00, 11.24it/s]  <br>              <br>              precision    recall  f1-score   support           <br>         neg       0.84      0.83      0.83     12500          <br>         pos       0.83      0.84      0.83     12500      </pre><pre>    accuracy                           0.83     25000    <br>   macro avg       0.83      0.83      0.83     25000 <br>weighted avg       0.83      0.83      0.83     25000</pre><figure><img alt="Confusion matrix normalized." src="https://cdn-images-1.medium.com/max/490/1*JxWmly1upn2r_7xHxBvJYA.png" /><figcaption>Confusion matrix normalized.</figcaption></figure><h3>Final Note</h3><p>If you made it this far <strong>Congrats!</strong> 🎊 and <strong>Thank you!</strong> 🙏 for your interest in my tutorial!</p><p>I’ve been using this code for a while now and I feel it got to a point where is nicely documented and easy to follow.</p><p>Of course is easy for me to follow because I built it. That is why any feedback is welcome and it helps me improve my future tutorials!</p><p>If you see something wrong please let me know by opening an issue on my <a href="https://github.com/gmihaila/ml_things/issues">ml_things GitHub repository</a>!</p><p>A lot of tutorials out there are mostly a one-time thing and are not being maintained. I plan on keeping my tutorials up to date as much as I can.</p><h3>Contact 🎣</h3><p>🦊 GitHub: <a href="https://github.com/gmihaila">gmihaila</a></p><p>🌐 Website: <a href="https://gmihaila.github.io/">gmihaila.github.io</a></p><p>👔 LinkedIn: <a href="https://www.linkedin.com/in/mihailageorge/">mihailageorge</a></p><p>📓 Medium: <a href="https://gmihaila.medium.com/">@gmihaila</a></p><p>📬 Email: <a href="/service/mailto:georgemihaila@my.unt.edu.com?subject=GitHub%20Website">georgemihaila@my.unt.edu.com</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href">https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href</a></iframe><p><em>Originally published at </em><a href="https://gmihaila.github.io/tutorial_notebooks/gpt2_finetune_classification/"><em>https://gmihaila.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=574555451832" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Better Batches with PyTorchText BucketIterator]]></title>
            <link>https://gmihaila.medium.com/better-batches-with-pytorchtext-bucketiterator-12804a545e2a?source=rss-9c419ab1d0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/12804a545e2a</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[batching]]></category>
            <dc:creator><![CDATA[George Mihaila]]></dc:creator>
            <pubDate>Fri, 13 Nov 2020 17:43:12 GMT</pubDate>
            <atom:updated>2020-11-13T17:51:05.466Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to use PyTorchText BucketIterator to sort text data for better batching.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MlQUdNGktfwewjYEnvCylQ.jpeg" /></figure><figure><a href="https://colab.research.google.com/github/gmihaila/ml_things/blob/master/notebooks/pytorch/pytorchtext_bucketiterator.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/458/1*HPyYAbxKrRLVetMtNZmcOQ.png" /></a></figure><figure><a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/pytorchtext_bucketiterator.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/379/1*xgK8v-H9I8IbshyQdGEwyw.png" /></a></figure><figure><a href="https://www.dropbox.com/s/7gyq6qup6y43z9b/pytorchtext_bucketiterator.ipynb?dl=1"><img alt="" src="https://cdn-images-1.medium.com/max/477/1*_WWiquUwuSrsOHfnlgPHBQ.png" /></a></figure><p>This notebook is a simple tutorial on how to use the powerful <strong>PytorchText</strong> <strong>BucketIterator</strong> functionality to group examples ( <strong>I use examples and sequences interchangeably</strong>) of similar lengths into batches. This allows us to provide the most optimal batches when training models with text data.</p><p>Having batches with similar length examples provides a lot of gain for recurrent models (RNN, GRU, LSTM) and transformers models (bert, roBerta, gpt2, xlnet, etc.) where padding will be minimal.</p><p>Basically any model that takes as input variable text data sequences will benefit from this tutorial.</p><p><strong>I will not train any models in this notebook!</strong> I will release a tutorial where I use this implementation to train a transformer model.</p><p>The purpose is to use an example text datasets and batch it using <strong>PyTorchText</strong> with <strong>BucketIterator</strong> and show how it groups text sequences of similar length in batches.</p><p>This tutorial has two main parts:</p><ul><li><strong>Using PyTorch Dataset with PyTorchText Bucket Iterator</strong>: Here I implemented a standard PyTorch Dataset class that reads in the example text datasets and use PyTorch Bucket Iterator to group similar length examples in same batches. I want to show how easy it is to use this powerful functionality form PyTorchText on a regular PyTorch Dataset workflow which you already have setup.</li><li><strong>Using PyTorch Text TabularDataset with PyTorchText Bucket Iterator</strong>: Here I use the built-in PyTorchText TabularDataset that reads data straight from local files without the need to create a PyTorch Dataset class. Then I follow same steps as in the previous part to show how nicely text examples are grouped together.</li></ul><p><em>This notebooks is a code adaptation and implementation inspired from a few sources:</em> <a href="https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html">torchtext_translation_tutorial</a>, <a href="https://github.com/pytorch/text">pytorch/text — GitHub</a>, <a href="https://torchtext.readthedocs.io/en/latest/index.html">torchtext documentation</a> and <a href="https://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/">A Comprehensive Introduction to Torchtext</a>.</p><h3>What should I know for this notebook?</h3><p>Some basic PyTorch regarding Dataset class and using DataLoaders. Some knowledge of PyTorchText is helpful but not critical in understanding this tutorial. The BucketIterator is similar in applying Dataloader to a PyTorch Dataset.</p><h3>How to use this notebook?</h3><p>The code is made with reusability in mind. It can be easily adapted for other text datasets and other NLP tasks in order to achieve optimal batching.</p><p>Comments should provide enough guidance to easily adapt this notebook to your needs.</p><p>This code is designed mostly for <strong>classification tasks</strong> in mind, but it can be adapted for any other Natural Language Processing tasks where batching text data is needed.</p><h3>Dataset</h3><p>I will use the well known movies reviews positive — negative labeled <a href="https://ai.stanford.edu/~amaas/data/sentiment/">Large Movie Review Dataset</a>.</p><p>The description provided on the Stanford website:</p><p><em>This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. See the README file contained in the release for more details.</em></p><p><strong>Why this dataset?</strong> I believe is an easy to understand and use dataset for classification. I think sentiment data is always fun to work with.</p><h3>Coding</h3><p>Now let’s do some coding! We will go through each coding cell in the notebook and describe what it does, what’s the code, and when is relevant — show the output.</p><p>I made this format to be easy to follow if you decide to run each code cell in your own python notebook.</p><p>When I learn from a tutorial I always try to replicate the results. I believe it’s easy to follow along if you have the code next to the explanations.</p><h3>Downloads</h3><p>Download the IMDB Movie Reviews sentiment dataset and unzip it locally.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href">https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href</a></iframe><h3>Installs</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1db2d27da05cc664dcf938cd9a462a78/href">https://medium.com/media/1db2d27da05cc664dcf938cd9a462a78/href</a></iframe><h3>Imports</h3><p>Import all needed libraries for this notebook.</p><p>Declare basic parameters used for this notebook:</p><ul><li>device - Device to use by torch: GPU/CPU. I use CPU as default since I will not perform any costly operations.</li><li>train_batch_size - Batch size used on train data.</li><li>valid_batch_size - Batch size used for validation data. It usually is greater than train_batch_size since the model would only need to make prediction and no gradient calculations is needed.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a8d811155f0773a17aa22efde501721/href">https://medium.com/media/7a8d811155f0773a17aa22efde501721/href</a></iframe><h3>Using PyTorch Dataset</h3><p>This is where I create the PyTorch Dataset objects for training and validation that <strong>can</strong> be used to feed data into a model. This is standard procedure when using PyTorch.</p><h3>Dataset Class</h3><p>Implementation of the PyTorch Dataset class.</p><p>Most important components in a PyTorch Dataset class are:</p><ul><li>__len__(self, ) where it returns the number of examples in our dataset that we read in __init__(self, ). This will ensure that len() will return the number of examples.</li><li>__getitem__(self, item) where given an index item will return the example corresponding to the item position.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f965659fff932f4d995880a125f8f49/href">https://medium.com/media/3f965659fff932f4d995880a125f8f49/href</a></iframe><h3>Train — Validation Datasets</h3><p>Create PyTorch Dataset for train and validation partitions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7bc8f8486ee9bcf01c0b0159d8eea99/href">https://medium.com/media/a7bc8f8486ee9bcf01c0b0159d8eea99/href</a></iframe><h3>PyTorch DataLoader</h3><p>In order to group examples from the PyTorch Dataset into batches we use PyTorch DataLoader. This is standard when using PyTorch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/501e0151e62e12c07348036095fca0cd/href">https://medium.com/media/501e0151e62e12c07348036095fca0cd/href</a></iframe><h3>PyTorchText Bucket Iterator Dataloader</h3><p>Here is where the magic happens! We pass in the <strong>train_dataset</strong> and <strong>valid_dataset</strong> PyTorch Dataset splits into <strong>BucketIterator</strong> to create the actual batches.</p><p>It’s very nice that PyTorchText can handle splits! No need to write same line of code again for train and validation split.</p><p><strong>The </strong><strong>sort_key parameter is very important!</strong> It is used to order text sequences in batches. Since we want to batch sequences of text with similar length, we will use a simple function that returns the length of an data example ( len(x[&#39;text&#39;)). This function needs to follow the format of the PyTorch Dataset we created in order to return the length of an example, in my case I return a dictionary with text key for an example.</p><p><strong>It is important to keep </strong><strong>sort=False and </strong><strong>sort_with_batch=True to only sort the examples in each batch and not the examples in the whole dataset!</strong></p><p>Find more details in the PyTorchText <strong>BucketIterator</strong> documentation <a href="https://torchtext.readthedocs.io/en/latest/data.html#bucketiterator">here</a> — look at the <strong>BPTTIterator</strong> because it has same parameters except the <strong>bptt_len</strong> argument.</p><p><strong>Note:</strong><em>If you want just a single DataLoader use </em><em>torchtext.data.BucketIterator instead of </em><em>torchtext.data.BucketIterator.splits and make sure to provide just one PyTorch Dataset instead of tuple of PyTorch Datasets and change the parameter </em><em>batch_sizes and its tuple values to </em><em>batch_size with single value: </em><em>dataloader = torchtext.data.BucketIterator(dataset, batch_size=batch_size, ).</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54c01570f15c46db64c458d2b81112ea/href">https://medium.com/media/54c01570f15c46db64c458d2b81112ea/href</a></iframe><h3>Compare DataLoaders</h3><p>Let’s compare the PyTorch DataLoader batches with the PyTorchText BucketIterator batches. We can see how nicely examples of similar length are grouped in same batch with PyTorchText.</p><p><strong>Note:</strong> <em>When using the PyTorchText BucketIterator, make sure to call </em><em>create_batches() before looping through each batch! Else you won&#39;t get any output form the iterator.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95bfa022540b9eeae5aaeb9fdc7d3fc5/href">https://medium.com/media/95bfa022540b9eeae5aaeb9fdc7d3fc5/href</a></iframe><pre>PyTorch DataLoader</pre><pre>Batch size: 10</pre><pre>LABEL LENGTH TEXT<br>  pos   1037 Fascinating movie, based on a true story, about an...<br>  neg   1406 Or maybe that&#39;s what it feels like. Anyway, &quot;The B...<br>  pos   679  Far by my most second favourite cartoon Spielberg ...<br>  neg   922  This movie reminds me of &quot;Irréversible (2002)&quot;, an...<br>  pos   214  There&#39;s never a dull moment in this movie. Wonderf...<br>  neg   1288 I don&#39;t think any player in Hollywood history last...<br>  pos   605  The thing I remember most about this film is that ...<br>  pos   1411 Fabulous, fantastic, probably Disney&#39;s best musica...<br>  neg   604  Just another film that exploits gratuitous frontal...<br>  pos   368  What can i say about the first film ever?&lt;br /&gt;&lt;br...</pre><pre><br>PyTorchText BuketIterator</pre><pre>Batch size: 10</pre><pre>LABEL LENGTH TEXT<br>  pos    609 That&#39;s My Bush is a live action project made by So...<br>  neg    610 Terminus Paradis was exceptional, but &quot;Niki ardele...<br>  neg    612 Awesomely improbable and foolish potboiler that at...<br>  pos    613 The events of September 11 2001 do not need extra ...<br>  pos    613 Okay, first of all I got this movie as a Christmas...<br>  neg    617 I have been known to fall asleep during films, but...<br>  pos    625 Fragglerock is excellent in the way that Schindler...<br>  neg    625 Sure I&#39;ve seen bad movies in my life, but this one...<br>  neg    626 Even 20+ years later, Ninja Mission stands out as ...<br>  pos    626 This film is excellently paced, you never have to ...</pre><h3>Train Loop Examples</h3><p>Now let’s look at a model training loop would look like. I printed the first 10 batches list of examples lengths to show how nicely they are grouped throughout the dataset!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c83ff77a86c6b6620fc195758a4d7e88/href">https://medium.com/media/c83ff77a86c6b6620fc195758a4d7e88/href</a></iframe><pre>Batch examples lengths: [848, 848, 849, 849, 850, 852, 853, 854, 856, 857] <br>Batch examples lengths: [779, 780, 780, 781, 781, 782, 782, 782, 783, 784] <br>Batch examples lengths: [2100, 2103, 2104, 2109, 2114, 2135, 2147, 2151, 2158, 2164] <br>Batch examples lengths: [903, 905, 910, 910, 910, 910, 914, 915, 916, 919] <br>Batch examples lengths: [968, 968, 970, 970, 971, 972, 973, 975, 981, 982] <br>Batch examples lengths: [806, 806, 807, 807, 808, 809, 810, 810, 811, 811] <br>Batch examples lengths: [731, 733, 734, 735, 736, 736, 737, 737, 738, 739] <br>Batch examples lengths: [357, 357, 358, 361, 362, 362, 362, 364, 366, 371] <br>Batch examples lengths: [2330, 2335, 2337, 2350, 2351, 2353, 2367, 2374, 2376, 2383] <br>Batch examples lengths: [1916, 1920, 1921, 1936, 1951, 1953, 1967, 1970, 1981, 1985] <br>Batch examples lengths: [1395, 1398, 1399, 1402, 1403, 1412, 1412, 1413, 1414, 1414]</pre><h3>Using PyTorchText TabularDataset</h3><p>Now I will use the TabularDataset functionality which creates the PyTorchDataset object right from our local files.</p><p>We don’t need to create a custom PyTorch Dataset class to load our dataset as long as we have tabular files of our data.</p><h3>Data to Files</h3><p>Since our dataset is scattered into multiple files, I created a function files_to_tsv which puts our dataset into a .tsv file (Tab-Separated Values).</p><p>Since I’ll use the <strong>TabularDataset</strong> from pytorch.data I need to pass tabular format files.</p><p>For text data I find the Tab Separated Values format easier to deal with.</p><p>I will call the files_to_tsv function for each of the two partitions <strong>train</strong> and <strong>test</strong>.</p><p>The function will return the name of the .tsv file saved so we can use it later in PyTorchText.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1934d3f186406abef17b2bd14ff8322e/href">https://medium.com/media/1934d3f186406abef17b2bd14ff8322e/href</a></iframe><pre>/content/aclImdb/train <br>pos Files: 100% |████████████████████████████████| 12500/12500 [00:34&lt;00:00, 367.26it/s] <br>neg Files: 100% |████████████████████████████████| 12500/12500 [00:21&lt;00:00, 573.00it/s]  </pre><pre>/content/aclImdb/test <br>pos Files: 100% |████████████████████████████████| 12500/12500 [00:11&lt;00:00, 1075.80it/s] <br>neg Files: 100% |████████████████████████████████| 12500/12500 [00:12&lt;00:00, 1037.94it/s]</pre><h3>TabularDataset</h3><p>Here I setup the data fields for PyTorchText. We have to tell the library how to handle each column of the .tsv file. For this we need to create data.Field objects for each column.</p><p>text_tokenizer: For this example I don&#39;t use an actual tokenizer for the text column but I need to create one because it requires as input. I created a dummy tokenizer that returns same value. Depending on the project, here is where you will have your own tokenizer. It needs to take as input text and output a list.</p><p>label_tokenizer The label tokenizer is also a dummy tokenizer. This is where you will have a encoder to transform labels to ids.</p><p>Since we have two .tsv files it&#39;s great that we can use the .split function from <strong>TabularDataset</strong> to handle two files at the same time one for train and the other one for test.</p><p>Find more details about <strong>torchtext.data</strong> functionality <a href="https://torchtext.readthedocs.io/en/latest/data.html#dataset-batch-and-example">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff383b06a46fe157ded78323f8b57f0c/href">https://medium.com/media/ff383b06a46fe157ded78323f8b57f0c/href</a></iframe><h3>PyTorchText Bucket Iterator Dataloader</h3><p>I’m using same setup as in the <strong>PyTorchText Bucket Iterator Dataloader</strong> code cell section. The only difference is in the sort_key since there is different way to access example attributes (we had dictionary format before).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fb44fd0c11bf8cc2548a96b7b1d34b4/href">https://medium.com/media/6fb44fd0c11bf8cc2548a96b7b1d34b4/href</a></iframe><h3>Compare DataLoaders</h3><p>Let’s compare the PyTorch DataLoader batches with the PyTorchText BucketIterator batches created with TabularDataset. We can see how nicely examples of similar length are grouped in same batch with PyTorchText.</p><p><strong>Note:</strong> <em>When using the PyTorchText BucketIterator, make sure to call </em><em>create_batches() before looping through each batch! Else you won&#39;t get any output form the iterator.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49aeb8f5e576819b45ceef9f1ae60d93/href">https://medium.com/media/49aeb8f5e576819b45ceef9f1ae60d93/href</a></iframe><pre>PyTorch DataLoader</pre><pre>Batch size: 10</pre><pre>LABEL LENGTH TEXT<br>  neg   1205 This movie is bad news and I&#39;m really surprised at...<br>  pos   762  Micro-phonies is a classic Stooge short. The guys ...<br>  pos   782  After becoming completely addicted to Six Feet Und... <br>  neg   1708 You do realize that you&#39;ve been watching the EXACT...<br>  neg   2341 Okay, as a long time Disney fan, I really -hate- d...<br>  neg   705  This movie is simply not worth the time or money s... <br>  neg   1370 I&#39;m sorry to say that there isn&#39;t really any way, ...<br>  pos   681  Something about &quot;Paulie&quot; touched my heart as few m...<br>  neg   1401 It really is that bad of a movie. My buddy rented ... <br>  neg   656  my friend bought the movie for 5€ (its is not even...</pre><pre>PyTorchText BuketIterator</pre><pre>Batch size: 10</pre><pre>LABEL LENGTH TEXT<br>  pos    609 That&#39;s My Bush is a live action project made by So...<br>  neg    610 Terminus Paradis was exceptional, but &quot;Niki ardele...<br>  neg    612 Awesomely improbable and foolish potboiler that at... <br>  pos    613 Okay, first of all I got this movie as a Christmas...<br>  neg    617 I have been known to fall asleep during films, but... <br>  pos    625 Fragglerock is excellent in the way that Schindler...<br>  neg    625 Sure I&#39;ve seen bad movies in my life, but this one...<br>  pos    626 This film is excellently paced, you never have to ...</pre><h3>Train Loop Examples</h3><p>Now let’s look at a model training loop would look like. I printed the first 10 batches list of examples lengths to show how nicely they are grouped throughout the dataset!</p><p>We see that we get same exact behavior as we did when using PyTorch Dataset. Now it depends on which way is easier for you to use PyTorchText BucketIterator: with PyTorch Dataset or with PyTorchText TabularDataset</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/587f0cab358656c8382ef7b62d970f15/href">https://medium.com/media/587f0cab358656c8382ef7b62d970f15/href</a></iframe><pre>Batch examples lengths: [848, 848, 849, 849, 850, 852, 853, 854, 856, 857]<br>Batch examples lengths: [779, 780, 780, 781, 781, 782, 782, 782, 783, 784]<br>Batch examples lengths: [2100, 2103, 2104, 2109, 2114, 2135, 2147, 2151, 2158, 2164]<br>Batch examples lengths: [903, 905, 910, 910, 910, 910, 914, 915, 916, 919]<br>Batch examples lengths: [968, 968, 970, 970, 971, 972, 973, 975, 981, 981]<br>Batch examples lengths: [806, 806, 807, 807, 808, 809, 810, 810, 811, 811]<br>Batch examples lengths: [731, 733, 734, 735, 736, 736, 737, 737, 738, 739]<br>Batch examples lengths: [357, 357, 358, 361, 362, 362, 362, 364, 366, 371]<br>Batch examples lengths: [2330, 2335, 2337, 2350, 2351, 2353, 2367, 2374, 2376, 2381]<br>Batch examples lengths: [1916, 1920, 1921, 1936, 1951, 1953, 1967, 1970, 1981, 1985]<br>Batch examples lengths: [1395, 1398, 1399, 1402, 1403, 1412, 1412, 1413, 1414, 1414]</pre><h3>Final Note</h3><p>If you made it this far <strong>Congrats!</strong> 🎊 and <strong>Thank you!</strong> 🙏 for your interest in my tutorial!</p><p>I’ve been using this code for a while now and I feel it got to a point where is nicely documented and easy to follow.</p><p>Of course is easy for me to follow because I built it. That is why any feedback is welcome and it helps me improve my future tutorials!</p><p>If you see something wrong please let me know by opening an issue on my <a href="https://github.com/gmihaila/ml_things/issues">ml_things GitHub repository</a>!</p><p>A lot of tutorials out there are mostly a one-time thing and are not being maintained. I plan on keeping my tutorials up to date as much as I can.</p><p>🦊 GitHub: <a href="https://github.com/gmihaila">gmihaila</a></p><p>🌐 Website: <a href="https://gmihaila.github.io/">gmihaila.github.io</a></p><p>👔 LinkedIn: <a href="https://www.linkedin.com/in/mihailageorge">mihailageorge</a></p><p>📬 Email: <a href="/service/mailto:georgemihaila@my.unt.edu.com?subject=GitHub%20Website">georgemihaila@my.unt.edu.com</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href">https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href</a></iframe><p><em>Originally published at </em><a href="https://gmihaila.github.io/tutorial_notebooks/pytorchtext_bucketiterator/"><em>https://gmihaila.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12804a545e2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Pretrain Transformers Models in PyTorch using  Transformers]]></title>
            <link>https://gmihaila.medium.com/pretrain-transformers-models-in-pytorch-using-transformers-ecaaec00fbaa?source=rss-9c419ab1d0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecaaec00fbaa</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[George Mihaila]]></dc:creator>
            <pubDate>Sun, 25 Oct 2020 23:04:04 GMT</pubDate>
            <atom:updated>2020-10-25T23:10:01.300Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pretrain or train from scratch 67 transformers models on your custom dataset.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*leGN_4O83ay9TS_iELQ-tQ.jpeg" /><figcaption>Reading a book is the best training for your mind.</figcaption></figure><figure><a href="https://colab.research.google.com/github/gmihaila/ml_things/blob/master/notebooks/pytorch/pretrain_transformers_pytorch.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/458/1*HPyYAbxKrRLVetMtNZmcOQ.png" /></a></figure><figure><a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/pretrain_transformers_pytorch.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/379/1*xgK8v-H9I8IbshyQdGEwyw.png" /></a></figure><figure><a href="https://www.dropbox.com/s/rkq79hwzhqa6x8k/pretrain_transformers_pytorch.ipynb?dl=1"><img alt="" src="https://cdn-images-1.medium.com/max/477/1*_WWiquUwuSrsOHfnlgPHBQ.png" /></a></figure><p><strong>Disclaimer:</strong> <em>The format of this tutorial notebook is very similar with my other tutorial notebooks. This is done intentionally in order to keep readers familiar with my format.</em></p><p>This notebook is used to pretrain transformers models using <a href="https://huggingface.co/transformers/">Huggingface</a> on your own custom dataset.</p><p><strong>What do I mean by <em>pretrain transformers</em>? The </strong><a href="https://www.merriam-webster.com/dictionary/pretraining"><strong>definition of pretraining</strong></a><strong> is <em>to train in advance</em>. That is exactly what I mean! Train a transformer model to use it as a pretrained transformers model which can be used to fine-tune it on a specific task!</strong></p><p><strong>I also use the term <em>fine-tune</em> where I mean to continue training a pretrained model on a custom dataset. I know it is confusing and I hope I’m not making it worse. <em>At the end of the day you are training a transformer model that was previously trained or not!</em></strong></p><p>With the AutoClasses functionality we can reuse the code on a large number of transformers models!</p><p>This notebook is designed to:</p><ul><li><strong>Use an already pretrained transformers model and fine-tune (continue training) it on your custom dataset.</strong></li><li><strong>Train a transformer model from scratch on a custom dataset.</strong> This requires an already trained (pretrained) tokenizer. This notebook will use by default the pretrained tokenizer if an already trained tokenizer is no provided.</li></ul><p>This notebook is <strong>heavily inspired</strong> from the Hugging Face script used for training language models: <a href="https://github.com/huggingface/transformers/tree/master/examples/language-modeling">transformers/tree/master/examples/language-modeling</a>. I basically adapted that script to work nicely in a notebook with a lot more comments.</p><p><strong>Notes from </strong><a href="https://github.com/huggingface/transformers/tree/master/examples/language-modeling"><strong>transformers/tree/master/examples/language-modeling</strong></a><strong>:</strong> <em>Fine-tuning the library models for language modeling on a text file (GPT, GPT-2, CTRL, BERT, RoBERTa, XLNet). GPT, GPT-2 and CTRL are fine-tuned using a causal language modeling (CLM) loss. BERT and RoBERTa are fine-tuned using a masked language modeling (MLM) loss. XLNet is fine-tuned using a permutation language modeling (PLM) loss.</em></p><h3>What should I know for this notebook?</h3><p>Since I am using PyTorch to fine-tune our transformers models any knowledge on PyTorch is very useful.</p><p>Knowing a little bit about the <a href="https://github.com/huggingface/transformers">transformers</a> library helps too.</p><p>In this notebook <strong>I am using raw text data to pretrain / train / fine-tune transformers models</strong> . There is no need for labeled data since we are not doing classification. The Transformers library handles the text files in same way as the original implementation of each model did.</p><h3>How to use this notebook?</h3><p>Like with every project, I built this notebook with reusability in mind. This notebook uses a custom dataset from.txt files. Since the dataset does not come in a single .txt file I created a custom function movie_reviews_to_file that reads the dataset and creates the text file. The way I load the .txt files can be easily reused for any other dataset.</p><p>The only modifications needed to use your own dataset will be in the paths provided to the train .txt file and evaluation .txt file.</p><p>All parameters that need to be changed are under the <strong>Parameters Setup</strong> section. Each parameter is nicely commented and structured to be as intuitive as possible.</p><h3>What transformers models work with this notebook?</h3><p>A lot of people will probably use it for Bert. When there is a need to run a different transformer model architecture, which one would work with this code? Since the name of the notebooks is <strong>pretrain_transformers</strong> it should work with more than one type of transformers.</p><p>I ran this notebook across all the pretrained models found on Hugging Face Transformer. This way you know ahead of time if the model you plan to use works with this code without any modifications.</p><p>The list of pretrained transformers models that work with this notebook can be found <a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/pretrain_transformers_pytorch_status_models.md">here</a>. There are <strong>67 models that worked</strong> 😄 and 39 models that failed to work 😢 with this notebook. <em>Remember these are pretrained models and fine-tuned on custom dataset.</em></p><h3>Dataset</h3><p>This notebook will cover pretraining transformers on a custom dataset. I will use the well known movies reviews positive — negative labeled <a href="https://ai.stanford.edu/~amaas/data/sentiment/">Large Movie Review Dataset</a>.</p><p>The description provided on the Stanford website:</p><p><em>This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. See the README file contained in the release for more details.</em></p><p><strong>Why this dataset?</strong> I believe is an easy to understand and use dataset for classification. I think sentiment data is always fun to work with.</p><h3>Coding</h3><p>Now let’s do some coding! We will go through each coding cell in the notebook and describe what it does, what’s the code, and when is relevant — show the output.</p><p>I made this format to be easy to follow if you decide to run each code cell in your own python notebook.</p><p>When I learn from a tutorial I always try to replicate the results. I believe it’s easy to follow along if you have the code next to the explanations.</p><h3>Downloads</h3><p>Download the <em>Large Movie Review Dataset</em> and unzip it locally.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href">https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href</a></iframe><h3>Installs</h3><ul><li><a href="https://github.com/huggingface/transformers"><strong>transformers</strong></a> library needs to be installed to use all the awesome code from Hugging Face. To get the latest version I will install it straight from GitHub.</li><li><a href="https://github.com/gmihaila/ml_things"><strong>ml_things</strong></a> library used for various machine learning related tasks. I created this library to reduce the amount of code I need to write for each machine learning project.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9adde78f0b99c5e8e998b5dc23e42bb4/href">https://medium.com/media/9adde78f0b99c5e8e998b5dc23e42bb4/href</a></iframe><h3>Imports</h3><p>Import all needed libraries for this notebook.</p><p>Declare basic parameters used for this notebook:</p><ul><li>set_seed(123) - Always good to set a fixed seed for reproducibility.</li><li>device - Look for gpu to use. I will use cpu by default if no gpu found.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5c6114bd9a6557056afb19eb19adbab/href">https://medium.com/media/c5c6114bd9a6557056afb19eb19adbab/href</a></iframe><h3>Helper Functions</h3><p>I like to keep all Classes and functions that will be used in this notebook under this section to help maintain a clean look of the notebook:</p><p><strong>movie_reviews_to_file(path_data: str, path_texts_file: str)</strong></p><p>As I mentioned before, we will need .txt files to run this notebook. Since the Large Movie Review Dataset comes in multiple files with different labels I created this function to put together all data in a single .txt file. Examples are saved on each line of the file. The path_data points to the path where data files are present and path_texts_file will be the .txt file containing all data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee7507727e157f2f039f56a781a9f665/href">https://medium.com/media/ee7507727e157f2f039f56a781a9f665/href</a></iframe><p><strong>ModelDataArguments</strong></p><p>This class follows similar format as the <a href="https://github.com/huggingface/transformers">transformers</a> library. The main difference is the way I combined multiple types of arguments into one and used rules to make sure the arguments used are correctly set. Here are all argument detailed (they are also mentioned in the class documentation):</p><ul><li>train_data_file: <em>Path to your </em><em>.txt file dataset. If you have an example on each line of the file make sure to use </em><em>line_by_line=True. If the data file contains all text data without any special grouping use </em><em>line_by_line=False to move a </em><em>block_size window across the text file.</em></li><li>eval_data_file: <em>Path to evaluation </em><em>.txt file. It has the same format as </em><em>train_data_file.</em></li><li>line_by_line: <em>If the </em><em>train_data_file and </em><em>eval_data_file contains separate examples on each line set </em><em>line_by_line=True. If there is no separation between examples and </em><em>train_data_file and </em><em>eval_data_file contains continuous text then </em><em>line_by_line=False and a window of </em><em>block_size will be moved across the files to acquire examples.</em></li><li>mlm: <em>Is a flag that changes loss function depending on model architecture. This variable needs to be set to </em><em>True when working with masked language models like bert or roberta and set to </em><em>False otherwise. There are functions that will raise ValueError if this argument is not set accordingly.</em></li><li>whole_word_mask: <em>Used as flag to determine if we decide to use whole word masking or not. Whole word masking means that whole words will be masked during training instead of tokens which can be chunks of words.</em></li><li>mlm_probability: <em>Used when training masked language models. Needs to have </em><em>mlm=True. It represents the probability of masking tokens when training model.</em></li><li>plm_probability: <em>Flag to define the ratio of length of a span of masked tokens to surrounding context length for permutation language modeling. Used for XLNet.</em></li><li>max_span_length: <em>Flag may also be used to limit the length of a span of masked tokens used for permutation language modeling. Used for XLNet.</em></li><li>block_size: <em>It refers to the windows size that is moved across the text file. Set to -1 to use maximum allowed length.</em></li><li>overwrite_cache: <em>If there are any cached files, overwrite them.</em></li><li>model_type: <em>Type of model used: bert, roberta, gpt2. More details </em><a href="https://huggingface.co/transformers/pretrained_models.html"><em>here</em></a><em>.</em></li><li>model_config_name: <em>Config of model used: bert, roberta, gpt2. More details </em><a href="https://huggingface.co/transformers/pretrained_models.html"><em>here</em></a><em>.</em></li><li>tokenizer_name: <em>Tokenizer used to process data for training the model. It usually has same name as </em><em>model_name_or_path: bert-base-cased, roberta-base, gpt2 etc.</em></li><li>model_name_or_path: <em>Path to existing transformers model or name of transformer model to be used: bert-base-cased, roberta-base, gpt2 etc. More details </em><a href="https://huggingface.co/transformers/pretrained_models.html"><em>here</em></a><em>.</em></li><li>model_cache_dir: <em>Path to cache files. It helps to save time when re-running code.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99bf40fc409c79c0160b2c4bdcff8e84/href">https://medium.com/media/99bf40fc409c79c0160b2c4bdcff8e84/href</a></iframe><p><strong>get_model_config(args: ModelDataArguments)</strong></p><p>Get model configuration. Using the ModelDataArguments to return the model configuration. Here are all argument detailed:</p><ul><li>args: <em>Model and data configuration arguments needed to perform pretraining.</em></li><li>Returns: <em>Model transformers configuration.</em></li><li>Raises: <em>ValueError: If </em><em>mlm=True and </em><em>model_type is NOT in [&quot;bert&quot;, &quot;roberta&quot;, &quot;distilbert&quot;, &quot;camembert&quot;]. We need to use a masked language model in order to set </em><em>mlm=True.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ae88e6bff086fc33096e77d549b2a69/href">https://medium.com/media/1ae88e6bff086fc33096e77d549b2a69/href</a></iframe><p><strong>get_tokenizer(args: ModelDataArguments)</strong></p><p>Get model tokenizer.Using the ModelDataArguments return the model tokenizer and change block_size form args if needed. Here are all argument detailed:</p><ul><li>args: <em>Model and data configuration arugments needed to perform pretraining.</em></li><li>Returns: <em>Model transformers tokenizer.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e1f737f6548c2ae25b02440cfafd6f6/href">https://medium.com/media/9e1f737f6548c2ae25b02440cfafd6f6/href</a></iframe><p><strong>get_model(args: ModelDataArguments, model_config)</strong></p><p>Get model. Using the ModelDataArguments return the actual model. Here are all argument detailed:</p><ul><li>args: <em>Model and data configuration arguments needed to perform pretraining.</em></li><li>model_config: <em>Model transformers configuration.</em></li><li>Returns: <em>PyTorch model.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c4aeb03d7435644e8f91cf88afd2a7a/href">https://medium.com/media/0c4aeb03d7435644e8f91cf88afd2a7a/href</a></iframe><p><strong>get_dataset(args: ModelDataArguments, tokenizer: PreTrainedTokenizer, evaluate: bool=False)</strong></p><p>Process dataset file into PyTorch Dataset. Using the ModelDataArguments return the actual model. Here are all argument detailed:</p><ul><li>args: <em>Model and data configuration arguments needed to perform pretraining.</em></li><li>tokenizer: <em>Model transformers tokenizer.</em></li><li>evaluate: <em>If set to </em><em>True the test / validation file is being handled. If set to </em><em>False the train file is being handled.</em></li><li>Returns: <em>PyTorch Dataset that contains file’s data.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18bad5330b8478989947c70cf4b6193d/href">https://medium.com/media/18bad5330b8478989947c70cf4b6193d/href</a></iframe><p><strong>get_collator(args: ModelDataArguments, model_config: PretrainedConfig, tokenizer: PreTrainedTokenizer)</strong></p><p>Get appropriate collator function. Collator function will be used to collate a PyTorch Dataset object. Here are all argument detailed:</p><ul><li>args: <em>Model and data configuration arguments needed to perform pretraining.</em></li><li>model_config: <em>Model transformers configuration.</em></li><li>tokenizer: <em>Model transformers tokenizer.</em></li><li>Returns: <em>Transformers specific data collator.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dec09cee62d18797acaca5bbde38fa18/href">https://medium.com/media/dec09cee62d18797acaca5bbde38fa18/href</a></iframe><h3>Parameters Setup</h3><p>Declare the rest of the parameters used for this notebook:</p><ul><li>model_data_args contains all arguments needed to setup dataset, model configuration, model tokenizer and the actual model. This is created using the ModelDataArguments class.</li><li>training_args contain all arguments needed to use the <a href="https://colab.research.google.com/drive/1u_J6YQ1mfHmR-nGPhrOPt4g9ZSgl9-TL">Trainer</a> functionality from Transformers that allows us to train transformers models in PyTorch very easy. You can find the complete documentation <a href="https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments">here</a>. There are a lot of parameters that can be set to allow multiple functionalities. I only used the following parameters (the comments are inspired from the HuggingFace documentation of <a href="https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments">TrainingArguments</a>:</li><li>output_dir: <em>The output directory where the model predictions and checkpoints will be written. I set it up to </em><em>pretrained_bert_model where the model and will be saved.</em></li><li>overwrite_output_dir: <em>Overwrite the content of the output directory. I set it to </em><em>True in case I run the notebook multiple times I only care about the last run.</em></li><li>do_train: <em>Whether to run training or not. I set this parameter to </em><em>True because I want to train the model on my custom dataset.</em></li><li>do_eval: <em>Whether to run evaluation on the evaluation files or not. I set it to </em><em>True since I have test data file and I want to evaluate how well the model trains.</em></li><li>per_device_train_batch_size: <em>Batch size GPU/TPU core/CPU training. I set it to </em><em>2 for this example. I recommend setting it up as high as your GPU memory allows you.</em></li><li>per_device_eval_batch_size: <em>Batch size GPU/TPU core/CPU for evaluation.I set this value to </em><em>100 since it&#39;s not dealing with gradients.</em></li><li>evaluation_strategy: <em>Evaluation strategy to adopt during training: </em><em>no: No evaluation during training; </em><em>steps: Evaluate every </em><em>eval_steps;epoch`: Evaluate every end of epoch. I set it to &#39;steps&#39; since I want to evaluate model more often.</em></li><li>logging_steps: <em>How often to show logs. I will se this to plot history loss and calculate perplexity. I set this to </em><em>20 just as an example. If your evaluate data is large you might not want to run it that often because it will significantly slow down training time.</em></li><li>eval_steps: <em>Number of update steps between two evaluations if evaluation_strategy=&quot;steps&quot;. Will default to the same value as logging_steps if not set. Since I want to evaluate model ever</em><em>logging_steps I will set this to </em><em>None since it will inherit same value as </em><em>logging_steps.</em></li><li>prediction_loss_only: <em>Set prediction loss to </em><em>True in order to return loss for perplexity calculation. Since I want to calculate perplexity I set this to </em><em>True since I want to monitor loss and perplexity (which is exp(loss)).</em></li><li>learning_rate: <em>The initial learning rate for Adam. Defaults is set to </em><em>5e-5.</em></li><li>weight_decay: <em>The weight decay to apply (if not zero)Defaults is set to </em><em>0.</em></li><li>adam_epsilon: <em>Epsilon for the Adam optimizer. Defaults to </em><em>1e-8.</em></li><li>max_grad_norm: <em>Maximum gradient norm (for gradient clipping). Defaults to </em><em>0.</em></li><li>num_train_epochs: <em>Total number of training epochs to perform (if not an integer, will perform the decimal part percents of the last epoch before stopping training). I set it to </em><em>2 at most. Since the custom dataset will be a lot smaller than the original dataset the model was trained on we don&#39;t want to overfit.</em></li><li>save_steps: <em>Number of updates steps before two checkpoint saves. Defaults to </em><em>500.</em></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d2adf8fdbe28169c64903dd92f0d947/href">https://medium.com/media/6d2adf8fdbe28169c64903dd92f0d947/href</a></iframe><h3>Load Configuration, Tokenizer and Model</h3><p>Loading the three essential parts of the pretrained transformers: configuration, tokenizer and model.</p><p>Since I use the AutoClass functionality from Hugging Face I only need to worry about the model’s name as input and the rest is handled by the transformers library.</p><p>I will be calling each three functions created in the <strong>Helper Functions</strong> tab that help return config of the model, tokenizer of the model and the actual PyTorch model.</p><p>After model is loaded is always good practice to resize the model depending on the tokenizer size. This means that the tokenizer&#39;s vocabulary will be aligned with the models embedding layer. This is very useful when we have a different tokenizer that the pretrained one or we train a transformer model from scratch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c32c10c997f6640f1674ef5ff686d69c/href">https://medium.com/media/c32c10c997f6640f1674ef5ff686d69c/href</a></iframe><pre>Loading model configuration... <br>Downloading: 100%|████████████████████████████████|433/433 [00:01&lt;00:00, 285B/s]  </pre><pre>Loading model`s tokenizer... <br>Downloading: 100%|████████████████████████████████|433/433 [00:01&lt;00:00, 285B/s]  </pre><pre>Loading actual model... <br>Downloading: 100% |████████████████████████████████|436M/436M [00:36&lt;00:00, 11.9MB/s]  </pre><pre>Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForMaskedLM: [&#39;cls.seq_relationship.weight&#39;, &#39;cls.seq_relationship.bias&#39;] - This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model). - This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). <br>Embedding(28996, 768, padding_idx=0)</pre><h3>Dataset and Collator</h3><p>This is where I create the PyTorch Dataset and data collator objects that will be used to feed data into our model.</p><p>This is where I use the MovieReviewsDataset text files created with the movie_reviews_to_file function. Since data is partitioned for both train and test I will create two text files: one used for train and one used for evaluation.</p><p>I strongly recommend to use a validation text file in order to determine how much training is needed in order to avoid overfitting. After you figure out what parameters yield the best results, the validation file can be incorporated in train and run a final train with the whole dataset.</p><p>The data collator is used to format the PyTorch Dataset outputs to match the output of our specific transformers model: i.e. for Bert it will created the masked tokens needed to train.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/938c6da4e5ec945f772b11f4e495c9ce/href">https://medium.com/media/938c6da4e5ec945f772b11f4e495c9ce/href</a></iframe><pre>Reading `train` partition... neg: 100%|████████████████████████████████|12500/12500 [00:55&lt;00:00, 224.11files/s] pos: 100%|████████████████████████████████|12500/12500 [00:55&lt;00:00, 224.11files/s] `.txt` file saved in `/content/train.txt`  Reading `test` partition... neg: 100%|████████████████████████████████|12500/12500 [00:55&lt;00:00, 224.11files/s] pos: 100%|████████████████████████████████|12500/12500 [00:55&lt;00:00, 224.11files/s] `.txt` file saved in `/content/train.txt`  Creating train dataset... Creating evaluate dataset...</pre><h3>Train</h3><p>Hugging Face was very nice to us for creating the Trainer class. This helps make PyTorch model training of transformers very easy! We just need to make sure we loaded the proper parameters and everything else is taking care of!</p><p>At the end of the training the tokenizer is saved along with the model so you can easily re-use it later or even load in on Hugging Face Models.</p><p>I configured the arguments to display both train and validation loss at every logging_steps. It gives us a sense of how well the model is trained.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1d2cc85567615ceadd3a0e941eec8af/href">https://medium.com/media/d1d2cc85567615ceadd3a0e941eec8af/href</a></iframe><pre>Loading `trainer`... <br>Start training... |████████████████████████████████|[5000/5000 09:43, Epoch 2/2]  <br>Step    Training Loss   Validation Loss <br> 700         2.804672          2.600590 <br>1400         2.666996          2.548267 <br>2100         2.625075          2.502431 <br>2800         2.545872          2.485056 <br>3500         2.470102          2.444808 <br>4200         2.464950          2.420487 <br>4900         2.436973          2.410310</pre><h3>Plot Train</h3><p>The Trainer class is so useful that it will record the log history for us. I use this to access the train and validation losses recorded at each logging_steps during training.</p><p>Since we are training / fine-tuning / extended training or pretraining (depending what terminology you use) a language model, we want to compute the perplexity.</p><p>This is what <a href="https://en.wikipedia.org/wiki/Perplexity">Wikipedia</a> says about perplexity: <em>In information theory, perplexity is a measurement of how well a probability distribution or probability model predicts a sample. It may be used to compare probability models. A low perplexity indicates the probability distribution is good at predicting the sample.</em></p><p>We can look at the perplexity plot in the same way we look at the loss plot: the lower the better and if the validation perplexity starts to increase we are starting to overfit the model.</p><p><strong>Note:</strong> It looks from the plots that the train loss is higher than validation loss. That means that our validation data is too easy for the model and we should use a different validation dataset. Since the purpose of this notebook is to show how to train transformers models and provide tools to evaluate such process I will leave the results <em>as is</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23b8a4e5414d59d6123cfbd3b70ed416/href">https://medium.com/media/23b8a4e5414d59d6123cfbd3b70ed416/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-GfqxtIZiIKoxhwi_5m6UA.png" /><figcaption>Train and Evaluation Loss history.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IzhMxA_q-n06350URrK9vA.png" /><figcaption>Train and Evaluation Perplexity history.</figcaption></figure><h3>Evaluate</h3><p>For the final evaluation we can have a separate test set that we use to do our final perplexity evaluation. For simplicity I used the same validation text file for the final evaluation. That is the reason I get the same results as the last validation perplexity plot value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8757c4f1016570163186912ed1d6642/href">https://medium.com/media/e8757c4f1016570163186912ed1d6642/href</a></iframe><pre>|████████████████████████████████|[250/250 00:25]  Evaluate Perplexity:      11.01</pre><h3>Final Note</h3><p>If you made it this far <strong>Congrats!</strong> 🎊 and <strong>Thank you!</strong> 🙏 for your interest in my tutorial!</p><p>I’ve been using this code for a while now and I feel it got to a point where is nicely documented and easy to follow.</p><p>Of course is easy for me to follow because I built it. That is why any feedback is welcome and it helps me improve my future tutorials!</p><p>If you see something wrong please let me know by opening an issue on my <a href="https://github.com/gmihaila/ml_things/issues">ml_things GitHub repository</a>!</p><p>A lot of tutorials out there are mostly a one-time thing and are not being maintained. I plan on keeping my tutorials up to date as much as I can.</p><h3>Contact</h3><p>🦊 GitHub: <a href="https://github.com/gmihaila">gmihaila</a></p><p>🌐 Website: <a href="https://gmihaila.github.io/">gmihaila.github.io</a></p><p>👔 LinkedIn: <a href="https://www.linkedin.com/in/mihailageorge">mihailageorge</a></p><p>📬 Email: <a href="/service/mailto:georgemihaila@my.unt.edu.com?subject=GitHub%20Website">georgemihaila@my.unt.edu.com</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href">https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href</a></iframe><p><em>Originally published at </em><a href="https://gmihaila.github.io/tutorial_notebooks/pretrain_transformers_pytorch/"><em>https://gmihaila.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecaaec00fbaa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-tune Transformers in PyTorch using  Transformers]]></title>
            <link>https://gmihaila.medium.com/fine-tune-transformers-in-pytorch-using-transformers-57b40450635?source=rss-9c419ab1d0e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/57b40450635</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[George Mihaila]]></dc:creator>
            <pubDate>Fri, 09 Oct 2020 18:40:58 GMT</pubDate>
            <atom:updated>2020-10-20T22:49:53.182Z</atom:updated>
            <content:encoded><![CDATA[<h4>Complete tutorial on how to fine-tune <strong>73 transformer models</strong> for text classification — no code changes necessary!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3tqgi9nO0aGDZU-l7kQccQ.jpeg" /><figcaption>Me trying to look like I’m ‘fine-tuning’ a multimeter.</figcaption></figure><figure><a href="https://colab.research.google.com/github/gmihaila/ml_things/blob/master/notebooks/pytorch/finetune_transformers_pytorch.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/458/1*HPyYAbxKrRLVetMtNZmcOQ.png" /></a></figure><figure><a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/finetune_transformers_pytorch.ipynb"><img alt="" src="https://cdn-images-1.medium.com/max/379/1*xgK8v-H9I8IbshyQdGEwyw.png" /></a></figure><figure><a href="https://www.dropbox.com/s/tsqicfqgt8v87ae/finetune_transformers_pytorch.ipynb?dl=1"><img alt="" src="https://cdn-images-1.medium.com/max/477/1*_WWiquUwuSrsOHfnlgPHBQ.png" /></a></figure><h3>Info</h3><p>This notebook is designed to use a pretrained transformers model and fine-tune it on a classification task. The focus of this tutorial will be on the code itself and how to adjust it to your needs.</p><p>This notebook is using the <a href="https://huggingface.co/transformers/model_doc/auto.html"><strong>AutoClasses</strong></a> from <a href="https://github.com/huggingface/transformers"><strong>transformer</strong></a> by <a href="https://huggingface.co/"><strong>Hugging Face</strong></a><strong> </strong>functionality. This functionality can guess a model’s configuration, tokenizer and architecture just by passing in the model’s name. <em>This allows for code reusability on a large number of transformers models!</em></p><h3>What should I know for this notebook?</h3><p>I provided enough instructions and comments to be able to follow along with minimum Python coding knowledge.</p><p>Since I am using PyTorch to fine-tune our transformers models any knowledge on PyTorch is very useful. Knowing a little bit about the <strong>transformers</strong> library helps too.</p><h3>How to use this notebook?</h3><p>I built this notebook with reusability in mind. The way I load the dataset into the PyTorch <strong>Dataset</strong> class is pretty standard and can be easily reused for any other dataset.</p><p>The only modifications needed to use your own dataset will be in reading in the dataset inside the <strong>MovieReviewsDataset </strong>class which uses PyTorch <strong>Dataset</strong>. The <strong>DataLoader </strong>will return a dictionary of <strong><em>batch</em></strong> inputs format so that it can be fed straight to the model using the statement: outputs = model(**<strong><em>batch</em></strong>). <em>As long as this statement holds, the rest of the code will work!</em></p><h3>What transformers models work with this notebook?</h3><p>There are rare cases where I use a different model than Bert when dealing with classification from text data. When there is a need to run a different transformer model architecture, which one would work with this code?</p><p>Since the name of the notebooks is <em>finetune_transformers</em> it should work with more than one type of transformers.</p><p>I ran this notebook across <a href="https://huggingface.co/transformers/pretrained_models.html#pretrained-models"><strong>all the pretrained models</strong></a> found on Hugging Face Transformer. This way you know ahead of time if the model you plan to use works with this code without any modifications.</p><p>The list of pretrained transformers models that work with this notebook can be found <a href="https://github.com/gmihaila/ml_things/blob/master/notebooks/pytorch/finetune_transformers_pytorch_status_models.md#pretrained-models-that-work-with-finetune_transformers_pytorchipynb"><strong>here</strong></a>. There are <strong>73 models that worked 😄 </strong>and <strong>33 models that failed to work 😢 </strong>with this notebook.</p><h3>Dataset</h3><p>This notebook will cover fine-tune transformers for binary classification task. I will use the well known movies reviews positive — negative labeled <a href="https://ai.stanford.edu/~amaas/data/sentiment/"><strong>Large Movie Review Dataset</strong></a>.</p><p>The description provided on the Stanford website:</p><p><em>This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. See the README file contained in the release for more details.</em></p><p><strong>Why this dataset?</strong> I believe is an easy to understand and use dataset for classification. I think sentiment data is always fun to work with.</p><h3>Coding</h3><p>Now let’s do some coding! We will go through each coding cell in the notebook and describe what it does, what’s the code, and when is relevant — show the output.</p><p>I made this format to be easy to follow if you decide to run each code cell in your own python notebook.</p><p>When I learn from a tutorial I always try to replicate the results. I believe it’s easy to follow along if you have the code next to the explanations.</p><h4>Downloads</h4><p>Download the <em>Large Movie Review Dataset</em> and unzip it locally.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href">https://medium.com/media/8514c5ee76e59688a799f591b1e9a460/href</a></iframe><h4>Installs</h4><ul><li><a href="https://github.com/huggingface/transformers"><strong>transformers</strong></a> library needs to be installed to use all the awesome code from Hugging Face. To get the latest version I will install it straight from GitHub.</li><li><a href="https://github.com/gmihaila/ml_things"><strong>ml_things</strong></a> library used for various machine learning related tasks. I created this library to reduce the amount of code I need to write for each machine learning project. Give it a try!</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9adde78f0b99c5e8e998b5dc23e42bb4/href">https://medium.com/media/9adde78f0b99c5e8e998b5dc23e42bb4/href</a></iframe><h4>Imports</h4><p>Import all needed libraries for this notebook.</p><p>Declare parameters used for this notebook:</p><ul><li>set_seed(123) - Always good to set a fixed seed for reproducibility.</li><li>epochs - Number of training epochs (authors recommend between 2 and 4).</li><li>batch_size - Number of batches - depending on the max sequence length and GPU memory. For 512 sequence length a batch of 10 <em>USUALY</em> works without cuda memory issues. For small sequence length can try batch of 32 or higher.</li><li>max_length - Pad or truncate text sequences to a specific length. I will set it to 60 tokens to speed up training.</li><li>device - Look for gpu to use. I will use cpu by default if no gpu found.</li><li>model_name_or_path - Name of transformers model - will use already pretrained model. Path of transformer model - will load your own model from local disk. I always like to start off with bert-base-cased: <em>12-layer, 768-hidden, 12-heads, 109M parameters. Trained on cased English text.</em></li><li>labels_ids - Dictionary of labels and their id - this will be used to convert string labels to numbers.</li><li>n_labels - How many labels are we using in this dataset. This is used to decide size of classification head.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43c4f368880a700494427e66a235cf9a/href">https://medium.com/media/43c4f368880a700494427e66a235cf9a/href</a></iframe><h4>Helper Functions</h4><p>I like to keep all Classes and functions that I will use in this notebook under this section to help maintain a clean look of the notebook:</p><p><strong>MovieReviewsDataset</strong></p><p>If you worked with PyTorch before this is pretty standard. We need this class to read in our dataset, parse it, use tokenizer that transforms text into numbers and get it into a nice format to be fed to the model.</p><p>Lucky for use, <strong>Hugging Face</strong> thought of everything and made the <strong>tokenizer</strong> do all the heavy lifting (split text into tokens, padding, truncating, encode text into numbers) and is very easy to use!</p><p>In this class I only need to read in the content of each file, use <a href="https://pypi.org/project/ftfy/"><strong>fix_text</strong></a> to fix any Unicode problems and keep track of positive and negative sentiments.</p><p>I will append all texts and labels in lists that later I will feed to the tokenizer and to the label ids to transform everything into numbers.</p><p>There are three main parts of this PyTorch <strong>Dataset</strong> class:</p><ul><li><strong>init()</strong> where we read in the dataset and transform text and labels into numbers.</li><li><strong>__len__()</strong> where we need to return the number of examples we read in. This is used when calling <strong>len(MovieReviewsDataset())</strong> .</li><li><strong>__getitem__()</strong> always takes as an input an int valuet hat represents which example from our examples to return from our dataset. If a value of 3 is passed, we will return the example form our dataset at position 3. It needs to return an object with the format that can be fed to our model. Luckily our tokenizer does that for us and returns a dictionary of variables ready to be fed to the model in this way:model(**inputs).</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f5ec3e51c9647316cfc0aee1ebb2490/href">https://medium.com/media/0f5ec3e51c9647316cfc0aee1ebb2490/href</a></iframe><p><strong>train(dataloader, optimizer_, scheduler_, device_)</strong></p><p>I created this function to perform a full pass through the <strong>DataLoader</strong> object (the <strong>DataLoader</strong> object is created from our <strong>Dataset</strong> type object using the <strong>MovieReviewsDataset</strong> class). This is basically one epoch train through the entire dataset.</p><p>The <strong>dataloader </strong>is created from PyTorch <strong>DataLoader</strong> which takes the object created from <strong>MovieReviewsDataset</strong> class and puts each example in batches. <em>This way we can feed our model batches of data!</em></p><p>The <strong>optimizer_</strong> and <strong>scheduler_</strong> are very common in PyTorch. They are required to update the parameters of our model and update our learning rate during training. There is a lot more than that but I won’t go into details. <em>This can actually be a huge rabbit hole since A LOT happens behind these functions that we don’t need to worry. Thank you PyTorch!</em></p><p>In the process we keep track of the actual labels and the predicted labels along with the loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9344e858d82ace7096ff6fbfbe00fd81/href">https://medium.com/media/9344e858d82ace7096ff6fbfbe00fd81/href</a></iframe><p><strong>validation(dataloader, device_)</strong></p><p>I implemented this function in a very similar way as <strong>train</strong> but without the parameters update, backward pass and gradient decent part. We don’t need to do all of those VERY computationally intensive tasks because we only care about our model’s predictions.</p><p>I use the <strong>DataLoader</strong> in a similar way as in train to get out batches to feed to our model.</p><p>In the process I keep track of the actual labels and the predicted labels along with the loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e55d28c407534b832266f48f5d988144/href">https://medium.com/media/e55d28c407534b832266f48f5d988144/href</a></iframe><h4>Load Model and Tokenizer</h4><p>Loading the three essential parts of the pretrained transformers: <em>configuration</em>, <em>tokenizer</em> and <em>model</em>. I also need to load the model on the device I’m planning to use (GPU / CPU).</p><p>Since I use the <strong>AutoClass</strong> functionality from <strong>Hugging Face</strong> I only need to worry about the model’s name as input and the rest is handled by the transformers library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67199175689fe8d5d5f1725112a6f5d9/href">https://medium.com/media/67199175689fe8d5d5f1725112a6f5d9/href</a></iframe><h3>Dataset and DataLoader</h3><p>This is where I create the PyTorch <strong>Dataset</strong> and <strong>DataLoader</strong> objects that will be used to feed data into our model.</p><p>This is where I use the <strong>MovieReviewsDataset</strong> class and create the dataset variables. Since data is partitioned for both train and test I will create a PyTorch <strong>Dataset</strong> and PyTorch <strong>DataLoader</strong> object for train and test. <strong>ONLY for simplicity I will use the test as validation. In practice NEVER USE THE TEST DATA FOR VALIDATION!</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31e0e929ac0d882c8ce84011270a5b9d/href">https://medium.com/media/31e0e929ac0d882c8ce84011270a5b9d/href</a></iframe><h3>Train</h3><p>I create an optimizer and scheduler that will be used by PyTorch in training.</p><p>I loop through the number of defined epochs and call the <strong>train</strong> and <strong>validation</strong> functions.</p><p>I will output similar info after each epoch as in Keras: <em>train_loss: — val_loss: — train_acc: — valid_acc</em>.</p><p>After training, I plot the train and validation loss and accuracy curves to check how the training went.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3362fe4568f7a2965f54d9e717b8df13/href">https://medium.com/media/3362fe4568f7a2965f54d9e717b8df13/href</a></iframe><pre>Epoch <br>100%|████████████████████████████████|4/4[13:49&lt;00:00, 207.37s/it]  </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782[02:40&lt;00:00,4.86it/s]  Validation on batches... <br>100%|████████████████████████████████|782/782[00:46&lt;00:00,16.80it/s]    <br>train_loss: 0.44816 - val_loss: 0.38655 - train_acc: 0.78372 - valid_acc: 0.81892   </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782[02:40&lt;00:00,4.86it/s]  Validation on batches... <br>100%|████████████████████████████████|782/782 [02:13&lt;00:00,5.88it/s]    train_loss: 0.29504 - val_loss: 0.43493 - train_acc: 0.87352 -valid_acc: 0.82360   </pre><pre>Training on batches... <br>100%|████████████████████████████████|782/782[02:40&lt;00:00, 4.87it/s]  <br>Validation on batches... <br>100%|████████████████████████████████|782/782[01:43&lt;00:00,7.58it/s]    train_loss: 0.16901 - val_loss: 0.48433 - train_acc: 0.93544 -valid_acc: 0.82624 <br>  <br>Training on batches... <br>100%|████████████████████████████████|782/782[02:40&lt;00:00, 4.87it/s]  Validation on batches... <br>100%|████████████████████████████████|782/782[00:46&lt;00:00,16.79it/s]<br>train_loss: 0.09816 - val_loss: 0.73001 - train_acc: 0.96936 - valid_acc: 0.82144</pre><figure><img alt="train validation loss" src="https://cdn-images-1.medium.com/max/701/1*VGdA99VAk3PLd29xS2TMbQ.png" /></figure><figure><img alt="validation accuracy" src="https://cdn-images-1.medium.com/max/691/1*ZQn3A4bdSCVNcvxs0zVHMA.png" /></figure><p>It looks like a little over one epoch is enough training for this model and dataset.</p><h3>Evaluate</h3><p>When dealing with classification it’s useful to look at <em>precision,</em> <em>recall</em> and <em>f1 score</em>. Another good thing to look at when evaluating the model is the confusion matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29b1ed7ab3ed8302d3308c55b1cdf654/href">https://medium.com/media/29b1ed7ab3ed8302d3308c55b1cdf654/href</a></iframe><pre>Outputs:</pre><pre>100%|████████████████████████████████|782/782[00:46&lt;00:00,16.77it/s]</pre><pre>precision    recall  f1-score   support           <br>neg          0.83      0.81      0.82     12500          <br>pos          0.81      0.83      0.82     12500 <br>     <br>accuracy                         0.82     25000    <br>macro avg    0.82      0.82      0.82     25000 <br>weighted avg 0.82      0.82      0.82     25000  </pre><figure><img alt="confusion matrix" src="https://cdn-images-1.medium.com/max/910/1*T0g1dQqnA4ZLZlhlhBq62A.png" /></figure><p>Results are not great, but for this tutorial we are not interested in performance.</p><h3>Final Note</h3><p>If you made it this far <strong>Congrats! 🎊</strong> and <strong>Thank you! 🙏 for your interest in my tutorial!</strong></p><p>I’ve been using this code for a while now and I feel it got to a point where is nicely documented and easy to follow.</p><p>Of course is easy for me to follow because I built it. That is why any feedback is welcome and it helps me improve my future tutorials!</p><p>If you see something wrong please let me know by opening an <a href="https://github.com/gmihaila/ml_things/issues/new/choose"><strong>issue on my ml_things</strong></a> GitHub repository!</p><p>A lot of tutorials out there are mostly a one-time thing and are not being maintained. I plan on keeping my tutorials up to date as much as I can.</p><h3>Contact 🎣</h3><p>🦊 GitHub: <a href="https://github.com/gmihaila"><em>gmihaila</em></a></p><p>🌐 Website: <a href="https://gmihaila.github.io/"><em>gmihaila.github.io</em></a></p><p>👔 LinkedIn: <a href="https://www.linkedin.com/in/mihailageorge"><em>mihailageorge</em></a></p><p>📬 Email: <a href="/service/mailto:georgemihaila@my.unt.edu.com?subject=GitHub%20Website"><em>georgemihaila@my.unt.edu.com</em></a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href">https://medium.com/media/9eada0464b5b2e4dda19729e0c08cf26/href</a></iframe><p><em>Originally published on my GitHub website at </em><a href="https://gmihaila.github.io/tutorial_notebooks/finetune_transformers_pytorch/"><em>https://gmihaila.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57b40450635" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>