<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amale Elhamri on Medium]]></title>
        <description><![CDATA[Stories by Amale Elhamri on Medium]]></description>
        <link>https://medium.com/@amale.elhamri?source=rss-e6548c4b11df------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jR6aytgu0jTw0q-J</url>
            <title>Stories by Amale Elhamri on Medium</title>
            <link>https://medium.com/@amale.elhamri?source=rss-e6548c4b11df------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:12:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amale.elhamri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serving FastAI models with Google Cloud AI Platform]]></title>
            <link>https://medium.com/artefact-engineering-and-data-science/serving-fastai-models-with-google-cloud-ai-platform-d2ef1b497231?source=rss-e6548c4b11df------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2ef1b497231</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Amale Elhamri]]></dc:creator>
            <pubDate>Wed, 31 Mar 2021 10:40:30 GMT</pubDate>
            <atom:updated>2021-03-31T10:40:30.241Z</atom:updated>
            <content:encoded><![CDATA[<h3>How we handled the training and the deployment of our FastAI models using AI Platform — Part II</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/0*E3hFAms2Xy4DFXFA" /></figure><h3>TL;DR</h3><p>In this second article of the series of two, I will dive into the deployment and the serving of our models at scale. If you missed the first one about training a fastai model at scale on AI Platform Training, here is the <a href="https://medium.com/artefact-engineering-and-data-science/automating-the-training-of-ml-models-with-google-cloud-ai-platform-318712629974">link</a>.</p><p>Serving a deep learning model can reveal several challenges among which:</p><ul><li>scaling resources on instances with or without accelerators (NVIDIA GPUs)</li><li>cost efficiency</li></ul><p>In this article, I will explain how I served a deep learning text classifier trained with the FastAI library following 2 main steps:</p><ul><li>Deploy fastai model using TorchServe</li><li>Host serving on GCP AI Platform Prediction</li></ul><p>All materials can be found in the <a href="https://github.com/artefactory/deploy-fastai-torchserve-aiplatform">github repository</a>. This repository was inspired by another project that aimed to deploy a fastai image classifier on AWS SageMaker Inference Endpoint <a href="https://github.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve">here</a>.</p><h3>1- Deploy fastai model using TorchServe</h3><p>TorchServe makes it easy to deploy PyTorch models at scale in production environments. It removes the heavy lifting of developing your own client server architecture. The FastAI library is based on the PyTorch framework. It makes it possible to use this technology to serve fastai models by loading your fastai model as a pure pytorch object (remove fastai abstraction).</p><h4>1–1 Export Model Weights from FastAI</h4><p>To do that, you need to restore the FastAI learner from the export pickle from the last post, and save its model weights with PyTorch.</p><pre>import torch</pre><pre>from fastai.text import load_learner<br>from fastai.text.learner import get_c, get_text_vocab</pre><pre>learn = load_learner(&quot;fastai_cls.pkl&quot;)</pre><pre>vocab_sz = len(_get_text_vocab(dls)) #dls is the dataloader you used for training</pre><pre>n_class = get_c(dls)</pre><pre>config = awd_lstm_clas_config.copy()</pre><pre>torch.save(learn.model.state_dict(), &quot;fastai_cls_weights.pth&quot;)</pre><h4>1–2 PyTorch Model from FastAI</h4><p>Once you’ve exported your pytorch weights, you need to rebuild the model structure to be able to load your weights into it. You might have to dig a little bit in fastai source code to find your implementation but luckily, In <a href="https://jupyter.org/">Jupyter</a> notebook, one can investigate the source code by adding <strong><em>??</em></strong> in front of a function name.</p><p>For text classifier, you can load a pure pytorch object by using the fastai <strong><em>get_text_classifier</em></strong> function</p><pre>from fastai.text.learner import get_text_classifier<br>from fastai.text.all import AWD_LSTM</pre><pre>torch_pure_model = get_text_classifier(AWD_LSTM, vocab_sz, n_class, config=config)</pre><h4>1–3 Reproduce fastai preprocessing steps</h4><p>Once you have obtained your pytorch pure model, you need to apply the same preprocessing that was used for training. FastAI has a very handy method .predict that can be applied to a text (simple string object), that naturally reproduces training preprocessing and therefore removes risk of training serving skew.</p><pre>text = &quot;This was a very good movie&quot;<br>pred_fastai = learn.predict(text)</pre><pre>pred_fastai<br>&gt;&gt;(Category tensor(1), tensor(1), tensor([0.0036, 0.9964]))</pre><p>In our case, we have to take this responsibility ourselves, since we need to get rid of fastai abstraction and work directly with PyTorch objects.</p><p>In my example, I used a spacy tokenizer so I reproduced fastai preprocessing as shown below:</p><pre>import torch</pre><pre>from fastai.text.core import Tokenizer, SpacyTokenizerfrom fastai.text.data import Numericalize</pre><pre>example = &quot;Hello, this is a test.&quot;</pre><pre>tokenizer = Tokenizer(<br>    tok=SpacyTokenizer(&quot;en&quot;)<br>    )</pre><pre>numericalizer = Numericalize(vocab=vocab)</pre><pre>example_processed = numericalizer(tokenizer(example))<br>example_processed<br>&gt;&gt;&gt; tensor([ 4,  7, 26, 29, 16, 72, 69, 31])</pre><pre>inputs = example_processed.resize(1, len(example_processed))<br>outputs = model_torch.forward(inputs)[0]</pre><pre>preds = torch.softmax(outputs, dim=-1) #You can use any activation function you need</pre><pre>preds<br>&gt;&gt;&gt; tensor([[0.0036, 0.9964]], grad_fn=&lt;SoftmaxBackward&gt;)</pre><p>As you can notice, the results I get using <strong><em>torch</em></strong> functions and <strong><em>learn.predict </em></strong>are the same because I managed to preserve the same preprocessing steps.</p><h4>1–4 Deploy your model via torchserve</h4><p>In this section we deploy the PyTorch model to TorchServe. For installation, please refer to TorchServe <a href="https://github.com/pytorch/serve">Github</a> Repository.</p><p>Overall, there are mainly 3 steps to use TorchServe:</p><ol><li>Archive the model into *.mar.</li><li>Start the torchserve.</li><li>Call the API and get the response.</li></ol><p>In order to archive the model, at least 2 files are needed in our case:</p><ol><li>PyTorch model weights fastai_cls_weights.pth.</li><li>TorchServe custom handler.</li></ol><h4>Custom Handler</h4><p>As shown in /deployment/handler.py, the TorchServe handler accepts data and context. In our example, we define another helper Python class with 4 instance methods to implement: <strong><em>initialize</em></strong>, <strong><em>preprocess</em></strong>, <strong><em>inference</em></strong> and <strong><em>postprocess</em></strong>.</p><p>Now it’s ready to setup and launch TorchServe.</p><h4>TorchServe in Action</h4><h4>Step 1: Archive the model PyTorch</h4><pre>torch-model-archiver \<br>    —-model-name=fastai_model \<br>    --version=1.0 \<br>    --serialized-file=/home/model-server/fastai_cls_weights.pth \<br>    —- extra-files=/home/model-server/config.py,/home/model-server/vocab.json \<br>    --handler=/home/model-server/handler.py \<br>    —-export-path=/home/model-server/model-store/</pre><h4>Step 2: Serve the Model</h4><pre>torchserve --start --ncs --model-store model_store --models fastai_model.mar</pre><h4>Step 3: Call API and Get the Response (here we use curl).</h4><pre>curl -X POST -H “Content-Type: application/json” -d ‘[“this was a bad movie”]’ <a href="http://127.0.0.1:8080/predictions/fastai_model">http://127.0.0.1:8080/predictions/fastai_model</a></pre><pre>{<br>    &quot;Categories&quot;: &quot;1&quot;,</pre><pre>    &quot;Tensor&quot;: [0.0036, 0.9964]</pre><pre>}</pre><p>The first call would have longer latency due to model weights loading defined in initialize, but this will be mitigated from the second call onward.</p><h3>2- Deployment to AI Platform Prediction</h3><p>In this section we deploy the FastAI trained model with TorchServe in GCP AI Platform Prediction using a customized Docker image. For more details about GCP AI Platform Prediction routines using custom containers please refer to <a href="https://cloud.google.com/ai-platform/prediction/docs/use-custom-container">this article</a>. Note that this option is only available if you use AI Platform Prediction with regional endpoints.</p><p>Steps to deploy a fastai model on AI Platform Prediction:</p><p>First, create an AI Platform Prediction model on a regional endpoint:</p><pre>gcloud beta ai-platform models create <strong>MODEL_NAME</strong> \ #eg: fastai_text_clf</pre><pre>--region=<strong>REGION</strong> \ #eg: europe-west1</pre><pre>--enable-logging \</pre><pre>--enable-console-logging</pre><h4>2–1 Build your docker image that will be used by your version</h4><ul><li>Create a folder model/ in the root of the repository</li><li>Place your fastai model weights in model/text/ and name it fastai_cls_weights.pth</li><li>Create an artifact repository</li></ul><pre>gcloud beta artifacts repositories create <strong>ARTIFACT_REGISTRY_NAME</strong> \ #eg: getting-started-fastai</pre><pre>    --repository-format=docker \<br>    --location=<strong>REGION</strong> #eg: europe-west1</pre><ul><li>Build your docker image</li></ul><pre>docker build -f TextDockerfile -t <strong>REGION</strong>-docker.pkg.dev/<strong>PROJECT_ID</strong>/<strong>ARTIFACT_REGISTRY_NAME</strong>/fastai_text_cls:v0</pre><h4>2–2 (Optional) Check that your docker image runs fine</h4><ul><li>Run your docker image locally and test it</li></ul><pre>docker run -it -p 8080:8080 <strong>REGION</strong>-docker.pkg.dev/<strong>PROJECT_ID</strong>/<strong>ARTIFACT_REGISTRY_NAME</strong>/fastai_text_cls:v0</pre><pre>curl -X POST -H &quot;Content-Type: application/json&quot; -d &#39;[&quot;this was a bad movie&quot;]&#39; 127.0.0.1:8080/predictions/fastai_model</pre><pre>{<br>    &quot;Categories&quot;: &quot;1&quot;,<br>    &quot;Tensor&quot;: [0.0036, 0.9964]</pre><pre>   }</pre><h4>2–3 Push your docker image to a container registry in your GCP project</h4><p>You need to have the IAM credentials to do that. Once you’ve ensured you have them, run the following</p><pre>gcloud auth configure-docker</pre><pre>docker push <strong>REGION</strong>-docker.pkg.dev/<strong>PROJECT_ID</strong>/<strong>ARTIFACT_REGISTRY_NAME</strong>/fastai_text_cls:v0</pre><h4>2–4 Create a model version using your docker image</h4><pre>gcloud beta ai-platform versions create <strong>VERSION_NAME</strong> \<br>    --region=<strong>REGION</strong><br>    --model=<strong>MODEL_NAME</strong><br>    --image=<strong>REGION</strong>-docker.pkg.dev/<strong>PROJECT_ID</strong>/<strong>ARTIFACT_REGISTRY_NAME</strong>/fastai_text_cls:v0<br>   --ports=8080<br>   --health-route=/ping<br>   --predict-route=/predictions/fastai_model</pre><h4>2–5 Test your model version</h4><pre>curl -X POST \</pre><pre>-H “Authorization: Bearer $(gcloud auth print-access-token)” \</pre><pre>-H “Content-Type: application/json”</pre><pre>-d ‘[“this was a bad movie”]’</pre><pre><a href="https://region-ml.googleapis.com/v1/projects/PROJECT_ID/models/MODEL_NAME/versions/VERSION_NAME:predict">https://<strong>REGION</strong>-ml.googleapis.com/v1/projects/<strong>PROJECT_ID</strong>/models/<strong>MODEL_NAME</strong>/versions/<strong>VERSION_NAME</strong>:predict</a></pre><pre>{<br>    &quot;Categories&quot;: &quot;1&quot;,<br>    &quot;Tensor&quot;: [0.0036, 0.9964]</pre><pre>}</pre><p>Your fastai model is now deployed in a serverless architecture on AI Platform Prediction. You can make online predictions by sending requests to your model as a REST API. All methods to request predictions can be found in <a href="https://cloud.google.com/ai-platform/prediction/docs/online-predict#requesting_predictions">google documentation</a>.</p><h3>Conclusion</h3><p>Using AI Platform Prediction to serve any type of model can be very useful. This article was aimed to show an example of a deep learning model using a heavy framework (pytorch) and serve it in a cost effective way.</p><p>Some limitations are to keep in mind:</p><ul><li>Even with autoscaling, it is not possible to downscale to 0 instances when you use AI Platform models deployed on regional endpoints. Since that’s the only option to use custom containers, you’ll always have at least one instance up</li><li>Another explored option was to use custom routines rather than custom containers but you can only do so if your model and packaged code are below a limit size of 500 MB which in our case was not possible to achieve.</li></ul><p>Thank you for reading. You can find more about us and our projects on our blog <a href="https://medium.com/artefact-engineering-and-data-science">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2ef1b497231" width="1" height="1" alt=""><hr><p><a href="https://medium.com/artefact-engineering-and-data-science/serving-fastai-models-with-google-cloud-ai-platform-d2ef1b497231">Serving FastAI models with Google Cloud AI Platform</a> was originally published in <a href="https://medium.com/artefact-engineering-and-data-science">Artefact Engineering and Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing NLPretext]]></title>
            <link>https://medium.com/artefact-engineering-and-data-science/introducing-nlpretext-a8bb7c03df89?source=rss-e6548c4b11df------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8bb7c03df89</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[text-preprocessing]]></category>
            <dc:creator><![CDATA[Amale Elhamri]]></dc:creator>
            <pubDate>Mon, 22 Feb 2021 10:39:44 GMT</pubDate>
            <atom:updated>2021-02-22T10:39:44.973Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing NLPretext, a unified framework to facilitate text preprocessing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*h3Nxj128FFVjZTe5ef70rQ.png" /></figure><p>Working on NLP projects? Tired of always looking for the same silly preprocessing functions on the web, such as removing accents from French posts? Tired of spending hours on Regex to efficiently extract email addresses from a corpus? <a href="https://github.com/artefactory/NLPretext.git">NLPretext</a> got you covered!</p><h3>NLPretext overview</h3><p>NLPretext is composed of 4 modules: basic, social, token and augmentation. Each of them includes different functions to handle the most important text preprocessing tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CwdEYglhgUFAD9WnMQ0jwg.jpeg" /></figure><h4>Basic preprocessing</h4><p>The <strong>basic</strong> module is a catalogue of transversal functions that can be used in any use case. They allow you to handle:</p><ul><li>bad whitespaces in a text, end of line characters</li><li>encoding issues</li><li>special characters such as currency symbols, numbers, punctuation marks, latin and non-latin characters</li><li>emails and phone numbers</li></ul><pre>from nlpretext.basic.preprocess import replace_emails</pre><pre>example = &quot;I have forwarded this email to obama@whitehouse.gov&quot;<br>example = replace_emails(example, replace_with=&quot;*EMAIL*&quot;)</pre><pre>print(example)</pre><pre># &quot;I have forwarded this email to *EMAIL*&quot;</pre><h4>Social preprocessing</h4><p>The <strong>social</strong> module is a catalogue of handy functions that can be useful when processing social data, such as:</p><ul><li>hashtags extraction/removal</li><li>emojis extraction/removal</li><li>mentions extraction/removal</li><li>html tags cleaning</li></ul><pre>from nlpretext.social.preprocess import extract_emojis</pre><pre>example = &quot;I take care of my skin 😀&quot;<br>example = extract_emojis(example)</pre><pre>print(example)</pre><pre># [&#39;:grinning_face:&#39;]</pre><h4>Token preprocessing</h4><p>The <strong>token</strong> module helps you clean your text on a token level. First you can load a tokenizer to split your sentence into tokens. Then you can:</p><ul><li>remove stopwords</li><li>remove small words</li><li>remove tokens with special characters</li></ul><pre>from nlpretext.token.preprocess import remove_stopwords</pre><pre>example = [“I”, “like”, “when”, “you”, “move”, “your”, “body”]<br>example = remove_stopwords(example, lang=&quot;en&quot;)</pre><pre>print(example)</pre><pre># [&#39;I&#39;, &#39;move&#39;, &#39;body&#39;]</pre><h4>Text augmentation</h4><p>The <strong>augmentation</strong> module helps you to <strong>generate new texts</strong> based on your given examples by modifying some words in the initial ones and to <strong>keep associated entities unchanged</strong>, if any, in the case of NER tasks. If you want words other than entities to remain unchanged, you can specify it within the stopwords argument. Modifications depend on the chosen method, the ones currently supported by the module are substitutions with synonyms using Wordnet or BERT from the <a href="https://github.com/makcedward/nlpaug">nlpaug</a> library.</p><pre>from nlpretext.augmentation.text_augmentation import augment_text</pre><pre>example = &quot;I want to buy a small black handbag please.&quot;<br>entities = [{&#39;entity&#39;: &#39;Color&#39;, &#39;word&#39;: &#39;black&#39;, &#39;startCharIndex&#39;: 22, &#39;endCharIndex&#39;: 27}]<br>example = augment_text(example, method=”wordnet_synonym”, entities=entities)</pre><pre>print(example)</pre><pre># &quot;I need to buy a small black pocketbook please.&quot;</pre><h3>Create your end to end pipeline</h3><h4>Default pipeline</h4><p>Our library provides a Preprocessor object to efficiently pipe all preprocessing operations.</p><p>If you need to keep all elements of your text and perform minimum cleaning, use the default pipeline. It normalizes whitespaces and removes newlines characters, fixes unicode problems and removes recurrent artifacts from social data such as mentions, hashtags and HTML tags.</p><pre>from nlpretext import Preprocessor</pre><pre>text = &quot;I just got the best dinner in my life @latourdargent !!! I  recommend 😀 #food #paris \n&quot;<br>preprocessor = Preprocessor()<br>text = preprocessor.run(text)</pre><pre>print(text)</pre><pre># &quot;I just got the best dinner in my life !!! I recommend&quot;</pre><h4>Custom pipeline</h4><p>If you have a clear idea of what preprocessing functions you want to pipe in your preprocessing pipeline, you can add them in your own Preprocessor.</p><pre>from nlpretext import Preprocessor<br>from nlpretext.basic.preprocess import (normalize_whitespace, remove_punct, remove_eol_characters, remove_stopwords, lower_text)<br>from nlpretext.social.preprocess import remove_mentions, remove_hashtag, remove_emoji</pre><pre>text = &quot;I just got the best dinner in my life @latourdargent !!! I  recommend 😀 #food #paris \n&quot;</pre><pre>preprocessor = Preprocessor()<br>preprocessor.pipe(lower_text)<br>preprocessor.pipe(remove_mentions)<br>preprocessor.pipe(remove_hashtag)<br>preprocessor.pipe(remove_emoji)<br>preprocessor.pipe(remove_eol_characters)<br>preprocessor.pipe(remove_stopwords, args={&#39;lang&#39;: &#39;en&#39;})<br>preprocessor.pipe(remove_punct)<br>preprocessor.pipe(normalize_whitespace)</pre><pre>text = preprocessor.run(text)</pre><pre>print(text)</pre><pre># &quot;dinner life recommend&quot;</pre><h3>NLPretext installation</h3><p>To install the library please run</p><pre>pip install nlpretext</pre><p>You can find the github repository <a href="https://github.com/artefactory/NLPretext/tree/master">here</a> and the library documentation <a href="https://nlpretext.readthedocs.io/en/latest/">here</a></p><h4>Co-authors:</h4><p>Rafaëlle Aygalenq, Bruce Delattre, Amale El Hamri, Hugo Vasselin</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8bb7c03df89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/artefact-engineering-and-data-science/introducing-nlpretext-a8bb7c03df89">Introducing NLPretext</a> was originally published in <a href="https://medium.com/artefact-engineering-and-data-science">Artefact Engineering and Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to train a language model from scratch without any linguistic knowledge?]]></title>
            <link>https://medium.com/artefact-engineering-and-data-science/how-to-train-a-language-model-from-scratch-without-any-linguistic-knowledge-11acaa933e84?source=rss-e6548c4b11df------2</link>
            <guid isPermaLink="false">https://medium.com/p/11acaa933e84</guid>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[ulmfit]]></category>
            <dc:creator><![CDATA[Amale Elhamri]]></dc:creator>
            <pubDate>Tue, 17 Nov 2020 15:24:12 GMT</pubDate>
            <atom:updated>2020-11-17T19:01:26.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CelBsRiUdLM4QHQkJzdFNg.jpeg" /></figure><h3>TL;DR</h3><p>This article explains how I created my own made language model in Korean, a complex language with limited training data. Here you’ll be able to learn how to train a language model without having the luxury of understanding this language yourself. You’ll find tips on where to get training data from, how much data you need, how to preprocess your data and how to find an architecture and a set of hyperparameters that best suit your model.</p><p>My key learnings are:</p><p>Data collection:</p><ul><li>When wikipedia does not have enough volume or that it’s not enough used by native speakers of the language you want to train your language model from, a good thing to do is to combine wikipedia with other data sources such as CommonCrawl</li></ul><p>Data volume:</p><ul><li>Choose documents that best represent the korean language. Too many documents would not be useful since the marginal performance improvement would be too small compared to the huge training time.</li><li>Choose documents that contain the most used words in the korean language</li><li>Find an architecture that manages to modelize the complexity of the training data</li><li>Find the right combination of regularization parameters not to overfit</li></ul><h3>Introduction</h3><p>If you don’t know it already, NLP had a huge hype of transfer learning in these past 2 years. The main idea is to re-use pre-trained language models for another NLP task such as text classification. A language model is a deep learning model that given part of a sentence is able to predict the next word of the sentence. The intuition to understand from this is that this kind of model understands really well the language structure, grammar, vocabulary, and the goal is to “transfer” that knowledge to other downstream models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*maA5NXtXpMp8KerA" /></figure><p><em>Example: a simple recipe on how to improve a text classifier using fine-tuning</em></p><p>This figure summarizes the ULM Fit method that I used for training my language model and therefore fine tune it and transfer it into a text classifier.</p><ul><li>Step 1: Train a general language model on a large corpus of data in the target language. This model will be able to understand the language structure, grammar and main vocabulary</li><li>Step 2: Fine tune the general language model to the classification training data. Doing that, your model will better learn to represent vocabulary that is used in your training corpus</li><li>Step 3: Train a text classifier using your fine tuned pretrained language model. This method allows your model to understand the words in their context. Furthermore, using a pretrained language model allows you to train your classifier on very few training examples (as little as 400 texts per label would do the job).</li></ul><p>We already know that text classification works nicely on English, French, German, Spanish, Chinese… but what should we do on languages with very few off the shelf language models?</p><p>Before going into further details, you may be wondering why a french data scientist like me would want to have a text classifier in Korean ? The reason is that I am part of a project that develops a product to classify social media posts into different categories. After validating the methodology on English and French, we started scaling it to other languages (english, french, japanese, chinese and korean). Only there was a bigger challenge in Korean language because there was no pretrained language model to be found in open source so I had to do it myself with very few Korean linguistic resources.</p><p>This article will be focusing on korean text classification by using the multifit method explained in the following <a href="https://arxiv.org/abs/1909.04761">paper</a>.</p><p>A lot of languages are very represented in the web such as : english, chinese, spanish, portuguese, french … Korean language remains very poorly documented and not a lot of content is ready for reuse. So I thought about contributing myself by sharing my key learnings with you, while discovering Korean NLP.</p><p>In this article I will tell you about my journey to train a Korean language model without understanding a single word of Korean and how I used it for text classification.</p><p><strong>Disclaimer: </strong>Usually, we consider a language model as good when it reaches an accuracy of about 45–50%. As my goal is not to generate korean text, I don’t need to reach such performances: I only need a model that “understands” the grammar and structure of the Korean language so that I can use it to train a korean text classifier.</p><h3>1 — Data collection for language model training</h3><h3>1.1 — Data source</h3><p>Usually, when training a language model from scratch, ULM FiT tutorial suggestions are to download all Wikipedia content in the given language. These guidelines only work if native speakers of this language are used to publishing a lot on this channel.</p><p>In Korean, it appears that people are not used to it: not only Wikipedia korean context has not enough volume, and it is also not representative of native korean speaking.</p><p>Here is a comparison between number of articles in english and korean wikipedia to give some hints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*-Lj8fQGUHUKcgH-CYxd8Uw.png" /><figcaption>Wikipedia volume in different languages</figcaption></figure><p>My advice: I combined wikipedia articles with Common Crawl data that you can download from <a href="https://traces1.inria.fr/oscar/">here</a>.</p><h3>1.2 — Data volume</h3><p>Let’s remember that a language model is a model supposed to predict the next word in a text. To do that, our model should have seen a lot of examples to learn the language and be good at speaking it. That being said, it is not useful to go beyond 100 millions of tokens. It only adds complexity to your model as well as a huge training time.</p><p>So at first glance, once I had retrieved all Wikipedia and Common Crawl data, I found myself with much more than 100 millions of tokens so I had to pick and choose the most relevant documents to train my model with. The goal of my methodology is to keep the documents that represent in the best way the native korean language:</p><ol><li>I first performed a weak tokenization on my corpus to approximate the number of tokens I had by splitting the corpus on spaces.</li><li>I removed all numbers, emojis, punctuation and other symbols that are not specific to korean from my obtained tokens</li><li>I computed a counter of all tokens in my corpus and retrieved the top 70,000 mentioned tokens</li><li>Then I retrieved documents that mention most of the top used tokens such that my corpus would be built of 100 million tokens and there was my training corpus!</li></ol><p>Now that we have our raw corpus of training we can start real business!</p><h3>2 — Data tokenization</h3><p>I guess when I told you earlier that I tokenized with a split function, you started thinking that this article was really a joke but let’s reassure you, this was never my end game!</p><p>First let’s remind you that no further data preprocessing is required for training a language model. A lot of NLP tasks perform some text stripping of numbers, stopwords, lowercasing, stemming … All of those would strip your text from its context and our goal is to learn to speak korean so we must keep all our text as it was originally written.</p><p>To tokenize korean text I tried two tokenization models:</p><ul><li><a href="https://github.com/explosion/spaCy/tree/master/spacy/lang/ko">korean spacy model</a> that is a wrapper to korean mecab tokenizer.</li><li><a href="https://github.com/google/sentencepiece">sentencepiece</a> subwords tokenizer model trained on my corpus with 28000 maximum tokens</li></ul><p>As it’s recommended in the <a href="https://nlp.fast.ai/classification/2019/09/10/multifit.html">multifit article</a>, I went with the second option to have a subword granularity.</p><h3>3 — Training model</h3><p>When training a language model as well as training any model, the two things that you want to avoid are <strong>underfitting</strong> and <strong>overfitting</strong>.</p><p>A model <strong>under fits</strong> when it is too simple with regards to the data it is trying to model. You can detect that when you find that your model cannot learn on your training data and that your training loss does not converge to 0 at all.</p><p>On the opposite, a model <strong>over fits</strong> when it learns “too well” to model your training data but that performance remains low on the test data. That is a sign that your model is not likely to predict well data that it hasn’t seen.</p><p>When I started to train my language model, at the beginning I was really struggling to learn anything from my data. As you can see on the picture below, after 10 epochs of training my training loss was not decreasing by an inch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/0*IU-CQKAbqWW_heiB" /></figure><p>What it means is that my model was too simple to represent the complexity of Korean language.</p><p>Here is what I did to overcome this issue:</p><p>As you can imagine, debugging any deep learning model is not easy as there are so many degrees of liberty. You have to find the right network structure as well as the right set of hyperparameters.</p><p>To simplify the problem at the beginning, the right way to go is to try to overfit on a single batch of data. The idea here is to make sure that given some data, your model is able to interpret its complexity and perform well on the training set.</p><p>Here are all the things I tried :</p><ul><li>Increase embedding size</li><li>Increase number of hidden layers</li><li>Changing optimizer functions</li><li>Changing learning rate</li></ul><p>After lots of attempts, here is the structure and hyperparameters that allowed my model to start learning :</p><p>Neural network architecture:</p><ul><li>QRNN Structure</li><li>Number of hidden layers: 2500</li><li>Number of layers : 4</li><li>Embedding size : 768</li></ul><p>Once your model is able to predict correctly on your training set, the next thing you want to avoid is overfitting.</p><p>Here are some regularizations that I tried to make sure my model would not overfit.</p><ul><li>Add dropout</li><li>Add weight decay</li><li>Add gradient clipping</li></ul><p>Here are the regularizers that I used for training my model</p><ul><li>Learning rate: 0.0002</li><li>Weight decay: 1e-8</li><li>Gradient clipping: 0.25</li></ul><h3>Results</h3><p>After training my model for 15 epochs, I finally reached an accuracy of 25% and a perplexity of 100. As I said at the beginning, I never intended to use my language model for text generation so I was already satisfied to know that my model is able to predict correctly one word out of 4.</p><p>Then I re-used my pre-trained model for text classification. The dataset I used is a balanced dataset made of 10k social documents coming from Instagram, Facebook, Youtube and websites that were labelled as “label1” or not “not label1”. My goal was to predict that a new publication is about “label1” or not.</p><p>Here are the performances I get for all the languages we developed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*bb1BsBoyrAIyNfaHasu-Dg.png" /><figcaption>Performances of different languages text classifiers</figcaption></figure><p>So even without speaking the language and training the pretrained language model myself, the performances for the Korean text classifier reaches quite well the other languages performances.</p><p>I still have a lot of things that I should try to improve the performances I get but still, it was kind of a hail mary to learn to process documents of a complex language like Korean without understanding a word of it and without finding relevant information and advice on the web.</p><h3>Next steps</h3><p>I have just described how I could improve a Korean text classification model leveraging a simple language model made from scratch. The initial performance is already good but there is room for improvement. I think what I would like to work on in the short run would be:</p><ul><li>Proofread the tokenization: as I don’t speak a word of korean, it would be interesting to have a native korean speaker have a look at the tokenization and confirm that it makes sense.</li><li>Enhance my language model and compare classification performances by:</li><li>Transfer-learning a backward language model, as it appeared to have been more performant on English or French</li><li>Transfer-learning a bi-directional language model</li><li>Having dynamic learning rates during training to avoid being stuck in a local minimum</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11acaa933e84" width="1" height="1" alt=""><hr><p><a href="https://medium.com/artefact-engineering-and-data-science/how-to-train-a-language-model-from-scratch-without-any-linguistic-knowledge-11acaa933e84">How to train a language model from scratch without any linguistic knowledge?</a> was originally published in <a href="https://medium.com/artefact-engineering-and-data-science">Artefact Engineering and Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>