<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maximilien Roberti on Medium]]></title>
        <description><![CDATA[Stories by Maximilien Roberti on Medium]]></description>
        <link>https://medium.com/@maximilienroberti?source=rss-29c8f5cf1dc4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*8gGL5fR8GEykQq6zp9xQ4A.jpeg</url>
            <title>Stories by Maximilien Roberti on Medium</title>
            <link>https://medium.com/@maximilienroberti?source=rss-29c8f5cf1dc4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:09:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maximilienroberti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fastai with Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)]]></title>
            <link>https://medium.com/data-science/fastai-with-transformers-bert-roberta-xlnet-xlm-distilbert-4f41ee18ecb2?source=rss-29c8f5cf1dc4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f41ee18ecb2</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[fastai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Maximilien Roberti]]></dc:creator>
            <pubDate>Wed, 27 Nov 2019 15:27:44 GMT</pubDate>
            <atom:updated>2020-01-23T09:09:39.260Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>A tutorial to implement state-of-the-art NLP models with Fastai for Sentiment Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCE-OsWxx7NYivrxt69QOw.png" /></figure><p>In early 2018, <a href="https://medium.com/u/34ab754f8c5e">Jeremy Howard</a> (co-founder of fast.ai) and <a href="https://medium.com/u/e3999e445181">Sebastian Ruder</a> introduced the<a href="https://arxiv.org/pdf/1801.06146.pdf"> Universal Language Model Fine-tuning for Text Classification</a> (ULMFiT) method. ULMFiT was the first <strong>Transfer Learning</strong> method applied to NLP. As a result, besides significantly outperforming many state-of-the-art tasks, it allowed, with only 100 labeled examples, to match performances equivalent to models trained on 100× more data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HUhpxwRcyNFEXNNd" /><figcaption><em>ULMFiT requires less data than previous approaches.</em> (<a href="https://arxiv.org/abs/1801.06146">Howard and Ruder, ACL 2018</a>)</figcaption></figure><p>The first time I heard about ULMFiT was during a <a href="https://course.fast.ai/videos/?lesson=4">fast.ai course</a> given by Jeremy Howard. He demonstrated how it is easy — thanks to the fastai library — to implement the complete ULMFiT method with only a few lines of codes. In his demo, he used an AWD-LSTM neural network pre-trained on Wikitext-103 and get rapidly state-of-the-art results. He also explained key techniques — also demonstrated in ULMFiT — to fine-tune models like <strong>Discriminate Learning Rate</strong>, <strong>Gradual Unfreezing</strong> or <strong>Slanted Triangular Learning Rates</strong>.</p><p>Since the introduction of ULMFiT, <strong>Transfer Learning</strong> became very popular in NLP and yet Google (BERT, Transformer-XL, XLNet), Facebook (RoBERTa, XLM) and even OpenAI (GPT, GPT-2) begin to pre-train their own model on very large corpora. This time, instead of using the AWD-LSTM neural network, they all used a more powerful architecture based on the Transformer (cf. <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>).</p><p>Although these models are powerful, fastai do not integrate all of them. Fortunately, <a href="https://huggingface.co/">Hugging Face</a> 🤗 created the well know <a href="https://github.com/huggingface/transformers">transformers library</a>. Formerly known as pytorch-transformers or pytorch-pretrained-bert, this library brings together over 40 state-of-the-art pre-trained NLP models (BERT, GPT-2, RoBERTa, CTRL…). The implementation gives interesting additional utilities like tokenizer, optimizer or scheduler.</p><p><a href="https://github.com/huggingface/transformers">GitHub - huggingface/transformers: 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.</a></p><p>The transformers library can be self-sufficient but incorporating it within the fastai library provides simpler implementation compatible with powerful fastai tools like <strong>Discriminate Learning Rate</strong>, <strong>Gradual Unfreezing</strong> or <strong>Slanted Triangular Learning Rates</strong>. The point here is to allow anyone — expert or non-expert — to get easily state-of-the-art results and to “make NLP uncool again”.</p><p>It is worth noting that integrating the Hugging Face transformers library in fastai has already been demonstrated in:</p><ul><li><em>Keita Kurita</em>’s article <a href="https://mlexplained.com/2019/05/13/a-tutorial-to-fine-tuning-bert-with-fast-ai/">A Tutorial to Fine-Tuning BERT with Fast AI</a> which makes pytorch_pretrained_bert library compatible with fastai.</li><li><a href="https://medium.com/u/795cbf38e25d">Dev Sharma</a>’s article <a href="https://medium.com/analytics-vidhya/using-roberta-with-fastai-for-nlp-7ed3fed21f6c">Using RoBERTa with Fastai for NLP</a> which makes pytorch_transformers library compatible with fastai.</li></ul><p>Although these articles are of high quality, some part of their demonstration is not anymore compatible with the last version of transformers.</p><h3>🛠 Integrating transformers with fastai for multiclass classification</h3><p>Before beginning the implementation, note that integrating transformers within fastai can be done in multiple ways. For that reason, I brought — what I think are — the most generic and flexible solutions. More precisely, I tried to make the minimum modification in both libraries while making them compatible with the maximum amount of transformer architectures. However, if you find a clever way to make this implementation, please let us know in the comment section!</p><p>A Jupiter Notebook version of this tutorial is available on this <a href="https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta">Kaggle kernel</a>.</p><h4>Libraries Installation</h4><p>First, you will need to install the fastai and transformers libraries. To do so, just follow the instructions <a href="https://github.com/fastai/fastai/blob/master/README.md#installation">here</a> and <a href="https://github.com/huggingface/transformers#installation">here</a>.</p><p>For this demonstration, I used Kaggle which already has the fastai library installed. So I just installed transformers with the command :</p><pre>pip install transformers</pre><p>The versions of the libraries used for this demonstration are fastai 1.0.58 and transformers 2.1.1.</p><h4>🎬 The example task</h4><p>The chosen task is a multi-class text classification on <a href="https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/overview">Movie Reviews</a>.</p><p>The <a href="https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/data">dataset</a> and the respective <a href="https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta">Notebook</a> of this article can be found on Kaggle.</p><p><a href="https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/data">Sentiment Analysis on Movie Reviews</a></p><p>For each text movie review, the model has to predict a label for the sentiment. We evaluate the outputs of the model on classification <em>accuracy</em>. The sentiment labels are:</p><p>0 →Negative<br>1 →Somewhat negative<br>2 →Neutral<br>3 →Somewhat positive<br>4 →Positive</p><p>The data is loaded into a DataFrame using pandas.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b728f8c3d1d71536b2705995cb54c549/href">https://medium.com/media/b728f8c3d1d71536b2705995cb54c549/href</a></iframe><h4>Main transformers classes</h4><p>In transformers, each model architecture is associated with 3 main types of classes:</p><ul><li>A <strong>model class</strong> to load/store a particular pre-train model.</li><li>A <strong>tokenizer class</strong> to pre-process the data and make it compatible with a particular model.</li><li>A <strong>configuration class</strong> to load/store the configuration of a particular model.</li></ul><p>For example, if you want to use the BERT architecture for text classification, you would use <a href="https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification">BertForSequenceClassification</a> for the <strong>model class</strong>, <a href="https://huggingface.co/transformers/model_doc/bert.html#berttokenizer">BertTokenizer</a> for the <strong>tokenizer class</strong> and <a href="https://huggingface.co/transformers/model_doc/bert.html#bertconfig">BertConfig</a> for the <strong>configuration class</strong>.</p><p>Later, you will see that those classes share a common class method from_pretrained(pretrained_model_name, ...). In our case, the parameter pretrained_model_name is a string with the shortcut name of a pre-trained model/tokenizer/configuration to load, e.g bert-base-uncased. We can find all the shortcut names in the transformers documentation <a href="https://huggingface.co/transformers/pretrained_models.html#pretrained-models">here</a>.</p><p>In order to switch easily between classes — each related to a specific model type — I created a dictionary that allows loading the correct classes by just specifying the correct model type name.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0393118fcf01d72f2559e52952d3c7a/href">https://medium.com/media/f0393118fcf01d72f2559e52952d3c7a/href</a></iframe><p>It is worth noting that in this case, we use the transformers library only for a <em>multi-class text classification</em> task. For that reason, this tutorial integrates only the transformer architectures that have a model for sequence classification implemented. These model types are :</p><ul><li>BERT (from Google)</li><li>XLNet (from Google/CMU)</li><li>XLM (from Facebook)</li><li>RoBERTa (from Facebook)</li><li>DistilBERT (from Hugging Face)</li></ul><p>However, if you want to go further — by implementing another type of model or NLP task — this tutorial still an excellent starter.</p><h4>Data pre-processing</h4><p>To match pre-training, we have to format the model input sequence in a specific format. <br>To do so, you have to first <strong>tokenize</strong> and then <strong>numericalize</strong> the texts correctly.<br>The difficulty here is that each pre-trained model, that we will fine-tune, requires exactly the same specific pre-process — <strong>tokenization</strong> &amp; <strong>numericalization</strong> — than the pre-process used during the pre-train part.<br>Fortunately, the <strong>tokenizer class </strong>from<strong> </strong>transformers provides the correct pre-process tools that correspond to each pre-trained model.</p><p>In the fastai library, data pre-processing is done automatically during the creation of the DataBunch. <br>As you will see in the DataBunch implementation part, the<strong> tokenizer</strong> and the <strong>numericalizer</strong> are passed in the processor argument under the following format :</p><pre>processor = [TokenizeProcessor(tokenizer=tokenizer,…), NumericalizeProcessor(vocab=vocab,…)]</pre><p>Let’s first analyze how we can integrate the transformers tokenizer within the TokenizeProcessor function.</p><p><strong>Custom tokenizer</strong></p><p>This part can be a little confusing because a lot of classes are wrapped in each other and with similar names.<br>To resume, if we look attentively at the fastai implementation, we notice that :</p><ol><li>The <a href="https://docs.fast.ai/text.data.html#TokenizeProcessor">TokenizeProcessor object</a> takes as tokenizer argument a Tokenizer object.</li><li>The <a href="https://docs.fast.ai/text.transform.html#Tokenizer">Tokenizer object</a> takes as tok_func argument a BaseTokenizer object.</li><li>The <a href="https://docs.fast.ai/text.transform.html#BaseTokenizer">BaseTokenizer object</a> implement the function tokenizer(t:str) → List[str] that takes a text t and returns the list of its tokens.</li></ol><p>Therefore, we can simply create a new class TransformersBaseTokenizer that inherits from BaseTokenizer and overwrite a new tokenizer function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a198d6c1682e1f00c42cb321223f7bfa/href">https://medium.com/media/a198d6c1682e1f00c42cb321223f7bfa/href</a></iframe><p>In this implementation, be careful about 3 things:</p><ol><li>As we are not using RNN, we have to limit the <em>sequence length</em> to the model input size.</li><li>Most of the models require <em>special tokens</em> placed at the beginning and end of the sequences.</li><li>Some models like RoBERTa require a <em>space</em> to start the input string. For those models, the encoding methods should be called with add_prefix_space set to True.</li></ol><p>Below, you can find the resume of each pre-process requirement for the 5 model types used in this tutorial. You can also find this information on the <a href="https://huggingface.co/transformers/">Hugging Face documentation</a> in each model section.</p><blockquote>BERT: [CLS] + tokens + [SEP] + padding</blockquote><blockquote>DistilBERT: [CLS] + tokens + [SEP] + padding</blockquote><blockquote>RoBERTa: [CLS] + prefix_space + tokens + [SEP] + padding</blockquote><blockquote>XLM: [CLS] + tokens + [SEP] + padding</blockquote><blockquote>XLNet: padding + tokens + [SEP] + [CLS]</blockquote><p>It is worth noting that we don’t add padding in this part of the implementation. <br>As we will see later, fastai manage it automatically during the creation of the DataBunch.</p><p><strong>Custom Numericalizer</strong></p><p>In fastai, <a href="https://docs.fast.ai/text.data.html#NumericalizeProcessor">NumericalizeProcessor object</a> takes as vocab argument a <a href="https://docs.fast.ai/text.transform.html#Vocab">Vocab object</a>.<br>From this analyze, I suggest two ways to adapt the fastai <strong>numericalizer</strong>:</p><ol><li>You can like described in <a href="https://medium.com/analytics-vidhya/using-roberta-with-fastai-for-nlp-7ed3fed21f6c">Dev Sharma’s article</a> (Section 1. <em>Setting Up the Tokenizer</em>), retrieve the list of tokens and create a Vocab object.</li><li>Create a new class TransformersVocab that inherits from Vocab and overwrite numericalize and textify functions.</li></ol><p>Even if the first solution seems to be simpler, Transformers does not provide, for all models, a straightforward way to retrieve his list of tokens. <br>Therefore, I implemented the second solution, which runs for each model type. <br>It consists of using the functions convert_tokens_to_ids and convert_ids_to_tokens in respectively numericalize and textify.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f622055d96c6fca0d965e2fcc2124c68/href">https://medium.com/media/f622055d96c6fca0d965e2fcc2124c68/href</a></iframe><p>NB: The functions __gestate__ and __setstate__ allow the functions <a href="https://docs.fast.ai/basic_train.html#Learner.export">export</a> and <a href="https://docs.fast.ai/basic_train.html#load_learner">load_learner</a> to work correctly with TranformersVocab.</p><p><strong>Custom processor</strong></p><p>Now that we have our custom <strong>tokenizer</strong> and <strong>numericalizer</strong>, we can create the custom <strong>processor</strong>. Notice we are passing the include_bos = False and include_eos = False options. This is because fastai adds its own special tokens by default which interferes with the [CLS] and [SEP] tokens added by our custom tokenizer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42ad6c6b8cfb2ec92d3c3fbed8f5575a/href">https://medium.com/media/42ad6c6b8cfb2ec92d3c3fbed8f5575a/href</a></iframe><p><strong>Setting up the DataBunch</strong></p><p>For the DataBunch creation, you have to pay attention to set the processor argument to our new custom processor transformer_processor and manage correctly the padding.</p><p>As mentioned in the Hugging Face documentation, BERT, RoBERTa, XLM, and DistilBERT are models with absolute position embeddings, so it’s usually advised to pad the inputs on the right rather than the left. Regarding XLNET, it is a model with relative position embeddings, therefore, you can either pad the inputs on the right or on the left.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94465a3efbc8dc9e9626898bcf4a2e6d/href">https://medium.com/media/94465a3efbc8dc9e9626898bcf4a2e6d/href</a></iframe><h4>Custom model</h4><p>As mentioned <a href="https://github.com/huggingface/transformers#models-always-output-tuples">here</a>, every model&#39;s forward method always outputs a tuple with various elements depending on the model and the configuration parameters. In our case, we are interested to access only to the logits. <br>One way to access them is to create a custom model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03849baa5f99f03c02192addace559b9/href">https://medium.com/media/03849baa5f99f03c02192addace559b9/href</a></iframe><p>To make our transformers adapted to multiclass classification, before loading the pre-trained model, we need to precise the number of labels. To do so, you can modify the config instance or either modify like in <a href="https://mlexplained.com/2019/05/13/a-tutorial-to-fine-tuning-bert-with-fast-ai/"><em>Keita Kurita</em>’s article</a> (Section: <em>Initializing the Learner</em>) the num_labels argument.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdcea6fa7f2f625f64eb0bc3d56cab4d/href">https://medium.com/media/cdcea6fa7f2f625f64eb0bc3d56cab4d/href</a></iframe><h4>Learner : Custom Optimizer / Custom Metric</h4><p>In pytorch-transformers, Hugging Face had implemented two specific optimizers — BertAdam and OpenAIAdam — that have been replaced by a single AdamW optimizer.<br>This optimizer matches Pytorch Adam optimizer Api, therefore, it becomes straightforward to integrate it within fastai.<br>Note that for reproducing BertAdam specific behavior, you have to set correct_bias = False.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd28c7255eb5ee1fcb89f5b9d7350e31/href">https://medium.com/media/bd28c7255eb5ee1fcb89f5b9d7350e31/href</a></iframe><h4>Discriminative Fine-tuning and Gradual unfreezing</h4><p>To use <strong>Discriminative Learning Rate</strong> and <strong>G<em>radual Unfreezing</em></strong>, fastai provides one tool that allows to “split” the structure model into groups. An instruction to perform that “split” is described in the fastai documentation <a href="https://docs.fast.ai/basic_train.html#Discriminative-layer-training">here</a>.</p><p>Unfortunately, the model architectures are too different to create a unique generic function that can “split” all the model types in a convenient way. Thereby, you will have to implement a custom “split” for each different model architecture.</p><p>For example, if we use the DistilBERT model and that we observe his architecture by making print(learner.model). We can decide to divide the model in 8 blocks :</p><ul><li>1 Embedding</li><li>6 transformer</li><li>1 classifier</li></ul><p>In this case, we can split our model in this way:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22b8ac3c3ea37ae7c1c12323bc8b93e8/href">https://medium.com/media/22b8ac3c3ea37ae7c1c12323bc8b93e8/href</a></iframe><p>Note that I didn’t found any document that has studied the influence of <strong>Discriminative Learning Rate</strong> and <strong>Gradual Unfreezing</strong> or even <strong>Slanted Triangular Learning Rates </strong>with transformer architectures.<strong> </strong>Therefore, using these tools does not guarantee better results. If you found any interesting documents, please let us know in the comment.</p><h4>Train</h4><p>Now we can finally use all the fastai build-in features to train our model. Like the ULMFiT method, we will use <strong>Slanted Triangular Learning Rates, Discriminate Learning Rate</strong> and <strong>gradually unfreeze </strong>the model.</p><p>Therefore, we first freeze all the groups but the classifier with :</p><pre>learner.freeze_to(-1)</pre><p>For <strong>Slanted Triangular Learning Rates</strong> you have to use the function fit_one_cycle. For more information, please check the fastai documentation <a href="https://docs.fast.ai/callbacks.one_cycle.html">here</a>.</p><p>To use our fit_one_cycle we will need an optimum learning rate. We can find this learning rate by using a learning rate finder, which can be called by using <a href="https://docs.fast.ai/callbacks.lr_finder.html#callbacks.lr_finder">lr_find</a>. Our graph would look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*p0EmMlM1i5AG-OQ3uJoAog.png" /></figure><p>We will pick a value a bit before the minimum, where the loss still improves. Here 2x10–3 seems to be a good value.</p><pre>learner.fit_one_cycle(1,max_lr=2e-03,moms=(0.8,0.7))</pre><p>The graph of the loss would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*dJyzyjsaMaVUhea9beSkCA.png" /></figure><p>We then unfreeze the second group and repeat the operations until all the groups are unfrozen. If you want to use <strong>Discriminative Learning Rate</strong> you can use slice as follow :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/deb0695c2ed4ad5f70721b38c182178d/href">https://medium.com/media/deb0695c2ed4ad5f70721b38c182178d/href</a></iframe><p>To unfreeze all the groups, use learner.unfreeze() .</p><h4>Creating prediction</h4><p>Now that we have trained the model, we want to generate predictions from the test dataset.</p><p>As specified in <em>Keita Kurita</em>’s <a href="https://mlexplained.com/2019/05/13/a-tutorial-to-fine-tuning-bert-with-fast-ai/">article</a>, as the function get_preds does not return elements in order by default, you will have to resort the elements into their correct order.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e3d204274b352460cab28c7e93b3d5c/href">https://medium.com/media/6e3d204274b352460cab28c7e93b3d5c/href</a></iframe><p>In the <a href="https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta">Kaggle example</a>, without playing too much with the parameters, we get a Public Score of 0.70059, which leads us to the 5th position on the leaderboard!</p><h3>📋Conclusion</h3><p>In this article, I explain how to combine the transformers library with the beloved fastai library. It aims to make you understand where to look and modify both libraries to make them work together. Likely, it allows you to use <strong>Slanted Triangular Learning Rates</strong>, <strong>Discriminate Learning Rate</strong> and<strong> </strong>even<strong> Gradual Unfreezing</strong>. As a result, without even tunning the parameters, you can obtain rapidly state-of-the-art results.</p><p>This year, the transformers became an essential tool for NLP. Because of that, I think that pre-trained transformers architectures will be integrated soon to future versions of fastai. Meanwhile, this tutorial is a good starter.</p><p>I hope you enjoyed this first article and found it useful. <br>Thanks for reading and don’t hesitate in leaving questions or suggestions.</p><p>I will keep writing articles on NLP so stay tuned!</p><h3>📑 References</h3><p>[1] Hugging Face, Transformers GitHub (Nov 2019), <a href="https://github.com/huggingface/transformers">https://github.com/huggingface/transformers</a></p><p>[2] Fast.ai, Fastai documentation (Nov 2019), <a href="https://docs.fast.ai/text.html">https://docs.fast.ai/text.html</a></p><p>[3] Jeremy Howard &amp; Sebastian Ruder, Universal Language Model Fine-tuning for Text Classification (May 2018), <a href="https://arxiv.org/abs/1801.06146">https://arxiv.org/abs/1801.06146</a></p><p>[4] Keita Kurita , <a href="https://mlexplained.com/2019/05/13/a-tutorial-to-fine-tuning-bert-with-fast-ai/">A Tutorial to Fine-Tuning BERT with Fast AI</a> (May 2019)</p><p>[5] <a href="https://medium.com/u/795cbf38e25d">Dev Sharma</a>, <a href="https://medium.com/analytics-vidhya/using-roberta-with-fastai-for-nlp-7ed3fed21f6c">Using RoBERTa with Fastai for NLP</a> (Sep 2019)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f41ee18ecb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fastai-with-transformers-bert-roberta-xlnet-xlm-distilbert-4f41ee18ecb2">Fastai with 🤗Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>