<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ganesh Lokare on Medium]]></title>
        <description><![CDATA[Stories by Ganesh Lokare on Medium]]></description>
        <link>https://medium.com/@lokaregns?source=rss-52da288e98e4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SUR31O9DNQ1VbBuY7DmUtw.jpeg</url>
            <title>Stories by Ganesh Lokare on Medium</title>
            <link>https://medium.com/@lokaregns?source=rss-52da288e98e4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:41:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lokaregns/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Positional Encoding in Transformers]]></title>
            <link>https://medium.com/@lokaregns/understanding-positional-encoding-in-transformers-38b21cbc1662?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/38b21cbc1662</guid>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Sun, 09 Feb 2025 16:47:51 GMT</pubDate>
            <atom:updated>2025-02-09T16:47:51.355Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>Transformers have revolutionized the field of Natural Language Processing (NLP) and are widely used in models like BERT, GPT, and T5. However, unlike traditional sequence models such as Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs), Transformers lack an inherent way to understand the order of words in a sequence. This is where <strong>positional encoding</strong> comes into play.</p><p>In this section, we will explore:</p><ul><li>Why positional encoding is needed in Transformers</li><li>How sequence information is handled in traditional models like RNNs and CNNs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*8OauG09PU_cUhJyleabI4Q.png" /></figure><h4>Why Do Transformers Need Positional Encoding?</h4><p>The Transformer model, introduced in the famous paper <strong>“Attention is All You Need” (Vaswani et al., 2017)</strong>, relies entirely on the <strong>self-attention mechanism</strong> to process input sequences. Unlike RNNs or CNNs, Transformers do <strong>not</strong> process inputs sequentially. Instead, they analyze an entire sentence (or input sequence) <strong>at once</strong>, which allows for parallel processing and better efficiency.</p><p>However, this parallelization comes with a drawback — <strong>Transformers do not inherently capture the order of words in a sequence</strong>. For example, consider the two sentences below:</p><ol><li><strong>“The dog chased the cat”</strong></li><li><strong>“The cat chased the dog”</strong></li></ol><p>A Transformer without positional encoding would treat both sentences similarly because the <strong>self-attention mechanism alone does not differentiate word positions</strong>. This is a major issue because sentence structure carries crucial meaning in language.</p><h4>How Traditional Models Handle Sequence Information</h4><p>Before Transformers, sequential models like <strong>RNNs</strong> and <strong>CNNs</strong> naturally captured word order in different ways:</p><h4>1. Recurrent Neural Networks (RNNs, LSTMs, GRUs)</h4><ul><li>RNNs process text word by word in a sequential manner.</li><li>Each word’s hidden state depends on the previous word, which inherently captures positional information.</li><li>However, RNNs suffer from <strong>slow training</strong> and <strong>vanishing gradient issues</strong> for long sequences.</li></ul><h4>2. Convolutional Neural Networks (CNNs) for NLP</h4><ul><li>CNNs use <strong>convolutional filters</strong> over local word embeddings to extract patterns.</li><li>By stacking layers, they can capture <strong>some</strong> order information, but <strong>only in a local context</strong> (i.e., limited receptive field).</li><li>Unlike RNNs, CNNs can process words in parallel, but they do not fully preserve the word order across longer distances.</li></ul><p><strong>Takeaway:</strong> Both RNNs and CNNs have built-in mechanisms to capture sequential information, but Transformers do not. Thus, <strong>we need a way to introduce positional information into Transformers — this is where positional encoding comes in!</strong></p><h3>2. What is Positional Encoding?</h3><p>Now that we understand why positional encoding is necessary, let’s explore what it is and how it helps Transformers capture word order in a sequence.</p><h4>Understanding Positional Encoding</h4><p>Since Transformers process words in parallel, they lack an inherent sense of order. <strong>Positional Encoding (PE)</strong> is a technique used to inject sequential information into input embeddings so that the model can differentiate words based on their position.</p><p>At a high level, positional encoding works by adding a <strong>unique vector to each word embedding</strong> based on its position in the sentence. These encodings help the Transformer distinguish between:</p><ul><li>“The dog chased the cat.”</li><li>“The cat chased the dog.”</li></ul><p>even though both sentences contain the same words.</p><h4>How Positional Encoding Works</h4><p>When a sentence is input into a Transformer, each word is first converted into an embedding vector. However, these embeddings alone do not carry any order information. To fix this, we add a <strong>positional encoding vector</strong> to each word embedding.</p><p>Mathematically, for each word at position <strong>pos</strong> in the sequence, its new representation is:</p><p><strong>Final Word Representation = Word Embedding + Positional</strong></p><p>This ensures that words at different positions have different final representations, even if their original embeddings are the same.</p><p>The most common method for computing positional encodings is the <strong>sinusoidal function approach</strong>, as introduced in the original Transformer paper, <em>Attention Is All You Need</em> (Vaswani et al., 2017).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*-qNpvIRcU6AMqhqR_PUl0Q.png" /></figure><p>Where:</p><ul><li>pos = position of the word in the sequence</li><li>i = dimension index</li><li>dmodel​ = embedding size</li></ul><p>These functions create a <strong>unique positional encoding</strong> for each position while maintaining <strong>smooth transitions</strong> between positions.</p><h4>Why Sinusoidal Functions?</h4><ol><li><strong>Relative Position Awareness</strong>: The sinusoidal functions ensure that nearby positions have similar encodings while distant positions have distinct ones.</li><li><strong>Generalization</strong>: Unlike learned positional embeddings (which require training), sinusoidal encodings allow the model to extrapolate to unseen sequence lengths.</li><li><strong>Mathematical Properties</strong>: The periodic nature of sine and cosine functions makes it easier for the model to recognize patterns in sequential data.</li></ol><h3>3. Example of Positional Encoding</h3><p>Let’s say we have a sentence:</p><p>👉 <strong>“I love Transformers!”</strong></p><p>If the embedding size is dmodel=4, the word embeddings might look like:</p><pre>&quot;I&quot;         →  [0.2, 0.1, 0.3, 0.7]  <br>&quot;love&quot;      →  [0.4, 0.3, 0.8, 0.5]  <br>&quot;Transformers&quot; →  [0.9, 0.6, 0.4, 0.2]</pre><p>Now, we <strong>add</strong> positional encodings (generated using sine/cosine functions):</p><pre>PE(0)  →  [0.0, 1.0, 0.0, 1.0]  <br>PE(1)  →  [0.84, 0.54, 0.75, 0.66]  <br>PE(2)  →  [0.91, 0.42, 0.90, 0.43]</pre><p>After <strong>adding positional encodings</strong> to embeddings:</p><pre>&quot;I&quot;         →  [0.2, 1.1, 0.3, 1.7]  <br>&quot;love&quot;      →  [1.24, 0.84, 1.55, 1.16]  <br>&quot;Transformers&quot; →  [1.81, 1.02, 1.30, 0.63]</pre><p>Now, the Transformer <strong>understands the order of words</strong>!</p><h3>4. Python Code for Positional Encoding</h3><p>Here’s a <strong>PyTorch</strong> implementation:</p><pre>import torch<br>import torch.nn as nn<br>import math<br><br>class PositionalEncoding(nn.Module):<br>    def __init__(self, d_model, max_len=100):<br>        super().__init__()<br>        <br>        # Create a matrix of shape (max_len, d_model)<br>        pe = torch.zeros(max_len, d_model)<br>        <br>        # Create a position vector (0, 1, 2, ..., max_len)<br>        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)<br>        <br>        # Compute the positional encodings using sine and cosine functions<br>        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))<br>        pe[:, 0::2] = torch.sin(position * div_term)<br>        pe[:, 1::2] = torch.cos(position * div_term)<br>        <br>        # Add batch dimension<br>        pe = pe.unsqueeze(0)<br>        <br>        # Register buffer so it&#39;s not trained<br>        self.register_buffer(&#39;pe&#39;, pe)<br><br>    def forward(self, x):<br>        return x + self.pe[:, :x.size(1), :]<br><br># Example usage<br>d_model = 512  # Embedding size<br>seq_len = 10   # Sequence length<br>batch_size = 2 # Batch size<br><br>x = torch.rand(batch_size, seq_len, d_model)  # Random input embeddings<br>pos_enc = PositionalEncoding(d_model)<br><br>output = pos_enc(x)  # Apply positional encoding<br>print(output.shape)  # Output shape: (batch_size, seq_len, d_model)</pre><h3>5. Alternative Approaches to Positional Encoding</h3><p>While sinusoidal positional encoding is widely used, alternative methods exist:</p><ol><li><strong>Learned Positional Embeddings</strong>: Instead of using fixed functions, models can learn positional embeddings as trainable parameters, similar to word embeddings.</li><li><strong>Relative Positional Encoding</strong>: Used in models like Transformer-XL, this method encodes relative distances between tokens rather than absolute positions.</li><li><strong>Rotary Positional Embeddings (RoPE)</strong>: Employed in models like GPT-4, RoPE improves how positional information interacts with self-attention mechanisms.</li></ol><h3>6. Conclusion</h3><p>Positional encoding is a crucial component of Transformers, allowing them to understand the order of words in a sequence. In this blog, we explored its importance, mathematical formulation, and implementation. We also compared <strong>Fixed (Sinusoidal) Positional Encoding</strong> and <strong>Learnable Positional Encoding</strong>, highlighting their strengths and limitations.</p><h4>Key Takeaways</h4><p>✔ <strong>Positional encoding is necessary</strong> because Transformers lack inherent sequential order understanding.<br>✔ <strong>Fixed (Sinusoidal) encoding</strong> is efficient, does not require training, and generalizes well to longer sequences.<br>✔ <strong>Learnable positional encoding</strong> adapts better to specific datasets but increases model complexity and memory usage.<br>✔ <strong>Modern Transformers</strong> (e.g., GPT-4, LLaMA, T5) use hybrid approaches like <strong>RoPE, ALiBi, and Relative Positional Encoding</strong> for efficiency.<br>✔ The choice of encoding depends on the <strong>task requirements</strong>, whether it needs generalization or task-specific adaptability.</p><h4>Final Thoughts</h4><p>Understanding positional encoding is essential for working with Transformers, whether for <strong>NLP tasks like translation, text generation, or information retrieval</strong>. As Transformer architectures evolve, new and more efficient encoding techniques continue to emerge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38b21cbc1662" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Large Language Models APIs with Python: A Comprehensive Guide]]></title>
            <link>https://medium.com/@lokaregns/using-large-language-models-apis-with-python-a-comprehensive-guide-0020a51bf5b6?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/0020a51bf5b6</guid>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[llm-api]]></category>
            <category><![CDATA[openai-api]]></category>
            <category><![CDATA[google-gemini]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Tue, 29 Oct 2024 02:53:28 GMT</pubDate>
            <atom:updated>2024-10-29T02:53:28.974Z</atom:updated>
            <content:encoded><![CDATA[<p>This guide will walk you through how to interact with popular LLM APIs using Python, including OpenAI’s GPT models, Anthropic’s Claude, and Google’s Gemini.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*PDQwGtDV85rojdL8DAeQ3g.jpeg" /></figure><h3>Setting up your keys</h3><p>If you haven’t done so already, you’ll need to create API keys from OpenAI, Anthropic and Google.</p><p>For OpenAI, visit <a href="https://openai.com/api/">https://openai.com/api/</a><br>For Anthropic, visit <a href="https://console.anthropic.com/">https://console.anthropic.com/</a><br>For Google, visit <a href="https://ai.google.dev/gemini-api">https://ai.google.dev/gemini-api</a></p><p>When you get your API keys, you need to set them as environment variables.</p><p>EITHER (recommended) create a file called .env in this project root directory, and set your keys there:</p><pre>OPENAI_API_KEY=xxxx<br>ANTHROPIC_API_KEY=xxxx<br>GOOGLE_API_KEY=xxxx</pre><p>OR enter the keys directly in the cells below.</p><h3>Prerequisites</h3><p>First, install the required packages:</p><pre>pip install openai anthropic google-generativeai requests python-dotenv</pre><p><strong>Import Required Libraries</strong></p><pre># imports<br>import os<br>from dotenv import load_dotenv<br>from openai import OpenAI<br>import google.generativeai<br>import anthropic<br>from IPython.display import Markdown, display, update_display</pre><pre># Load environment variables in a file called .env<br><br>load_dotenv()<br>os.environ[&#39;OPENAI_API_KEY&#39;] = os.getenv(&#39;OPENAI_API_KEY&#39;, &#39;your-key-if-not-using-env&#39;)<br>os.environ[&#39;ANTHROPIC_API_KEY&#39;] = os.getenv(&#39;ANTHROPIC_API_KEY&#39;, &#39;your-key-if-not-using-env&#39;)<br>os.environ[&#39;GOOGLE_API_KEY&#39;] = os.getenv(&#39;GOOGLE_API_KEY&#39;, &#39;your-key-if-not-using-env&#39;)</pre><p><strong>Connect to OpenAI, Anthropic and Google</strong></p><p>All 3 APIs are similar <br>Having problems with API files? You can use <br>openai = OpenAI(api_key=”your-key-here”) and same for claude</p><pre>openai = OpenAI()<br><br>claude = anthropic.Anthropic()<br><br>google.generativeai.configure()</pre><h3>What information is included in the API</h3><p>Typically we’ll pass to the API:</p><p>The name of the model that should be used.</p><p><strong>System Message:</strong><br>The system message sets the context for the assistant. This message informs the model about its role, tone, and any specific guidelines it should follow while responding.</p><p><strong>User Message:</strong><br>The user message represents the input from the user, such as questions or instructions, which the model responds to. It contains the primary prompt or query you want the model to answer.</p><p>There are other parameters that can be used, including <strong>temperature</strong> which is typically between 0 and 1; higher for more random output; lower for more focused and deterministic.</p><ol><li><strong>OpenAI GPT Models</strong></li></ol><p>Here’s how to use OpenAI’s GPT models:</p><pre>system_message = &quot;You are an assistant that is great at telling jokes&quot;<br>user_prompt = &quot;Tell a light-hearted joke for an audience of Data Scientists&quot;</pre><pre>prompts = [<br>    {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_message},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_prompt}<br>  ]<br></pre><pre># You can use different models as well.<br># GPT-3.5-Turbo, gpt-4o-mini, gpt-4o<br><br>completion = openai.chat.completions.create(model=&#39;gpt-40-mini&#39;, <br>             messages=prompts,<br>             temperature=0.7)<br>print(completion.choices[0].message.content)</pre><p><strong>2. Anthropic’s Claude</strong></p><p>This API needs system message provided separately from user prompt.</p><pre>message = claude.messages.create(<br>    model=&quot;claude-3-5-sonnet-20240620&quot;,<br>    max_tokens=200,<br>    temperature=0.7,<br>    system=system_message,<br>    messages=[<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_prompt},<br>    ],<br>)<br><br>print(message.content[0].text)</pre><p>max_tokens is a parameter that defines the maximum number of tokens the model will generate in its response.</p><p><strong>3. Google’s Gemini</strong></p><p>The API for Gemini has a slightly different structure</p><pre>gemini = google.generativeai.GenerativeModel(<br>    model_name=&#39;gemini-1.5-flash&#39;,<br>    system_instruction=system_message<br>)<br>response = gemini.generate_content(user_prompt)<br>print(response.text)</pre><h3>Important Considerations</h3><ol><li><strong>API Costs</strong>: Be mindful that these services charge based on usage. Monitor your usage and implement proper rate limiting.</li><li><strong>Error Handling</strong>: Always implement proper error handling to deal with API limits, network issues, and other potential problems.</li><li><strong>Token Limits</strong>: Each model has different token limits. Consider implementing token counting and chunking for longer inputs.</li><li><strong>Security</strong>: Never expose your API keys. Use environment variables and proper security practices.</li><li><strong>Rate Limiting</strong>: Implement appropriate rate limiting to avoid hitting API limits and to manage costs.</li></ol><h3>Conclusion</h3><p>This guide provides a foundation for working with major LLM APIs in Python. Each provider offers unique features and capabilities, so choose the one that best fits your needs. Remember to review each provider’s documentation for the most up-to-date information and best practices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0020a51bf5b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning Transformers with custom dataset: Classification task]]></title>
            <link>https://medium.com/@lokaregns/fine-tuning-transformers-with-custom-dataset-classification-task-f261579ae068?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f261579ae068</guid>
            <category><![CDATA[fine-tuning-transformer]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Sat, 11 Feb 2023 12:03:02 GMT</pubDate>
            <atom:updated>2023-02-11T12:03:02.203Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qqsUvXR8TV97STw4VT7pg.png" /></figure><p>While pre-trained transformer models have many benefits, there are also several drawbacks to using them on a custom dataset compared to using fine-tuned models. They may not always perform as well as fine-tuned models on a custom dataset. Fine-tuning allows for greater control, adaptation, and optimization to the specific task and dataset, resulting in improved performance.</p><h3>What is fine-tuning in transformers?</h3><p>Fine-tuning is a process in which a pre-trained model is further trained on a new task using task-specific data. In the context of Transformer models, fine-tuning refers to the process of using a pre-trained Transformer model as the starting point for training on a new task.</p><p>The idea behind fine-tuning Transformer models is that they have already been trained on a large corpus of text data, and therefore have already learned many useful representations of language. By fine-tuning the model on a new task, the model can use these pre-learned representations as a good starting point, and learn task-specific information from the new task data.</p><p>The process of fine-tuning a Transformer model involves unfreezing some or all of the layers of the pre-trained model and training them on the new task data using a task-specific loss function. The remaining layers can be kept frozen, preserving the pre-learned representations and preventing overfitting on the small task-specific data.</p><p>In conclusion, fine-tuning is a powerful technique for leveraging pre-trained Transformer models for new NLP tasks, allowing practitioners to achieve state-of-the-art results with relatively small amounts of task-specific data. Fine-tuning has become a popular approach in NLP due to the high performance of Transformer models and the availability of large pre-trained models.</p><h3>How we fine-tune transformers models for a specific task?</h3><p>Fine-tuning a Transformer model for a specific task typically involves the following steps:</p><ol><li>Prepare the task-specific data</li><li>Tokenize the data</li><li>Choose a pre-trained model</li><li>Define a fine-tuning architecture</li><li>Compile the model</li><li>Train the model</li><li>Evaluate the model</li></ol><p>By following these steps, you can fine-tune a Transformer model for a specific task, leveraging the pre-learned representations of the model to achieve high performance with limited task-specific data.</p><h3>Approaches for fine-tuning architecture</h3><ol><li><strong>Chop off the final layer and add a new one:</strong></li></ol><p>This approach is often used when the task for which the transformer model is being fine-tuned is different from the task for which it was pre-trained. In this approach, the final layer of the pre-trained transformer model is removed, and a new layer is added to match the specific requirements of the target task. The new layer is then trained from scratch on the target task’s data, while the rest of the model is kept frozen. The idea behind this approach is to preserve the learned representations of the pre-trained model, which can be useful for the target task, and only update the final layer to make predictions for the new task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*hot2Dv9234TpJZaZHYAgqQ.png" /></figure><p>2.<strong> Fine-tune everything:</strong></p><p>In this approach, all the parameters of the pre-trained transformer model are updated during training on the target task’s data. This is usually done when the target task is similar to the task for which the model was pre-trained, and the pre-trained model’s learned representations can be fine-tuned for the target task. During fine-tuning, a smaller learning rate is often used to avoid undoing the learned representations from the pre-training step. This approach can lead to better performance than the previous one as the entire model is optimized for the target task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*6DmVzzfd4kJpEIP79yjd_Q.png" /></figure><p>Both of these approaches have their own advantages and disadvantages, and the choice between them depends on the specific use case and the similarities between the target task and the pre-training task.</p><p><strong>In this task we are using</strong> <strong>Fine-tune everything approach.</strong></p><p>In this task, we are fine-tuning a model for sentiment analysis, but the same procedure can be used for any classification task.</p><h3>Fine-tuning for sentiment analysis on a custom dataset</h3><p>You will get the all code here. <a href="https://github.com/GaneshLokare/Transformers">https://github.com/GaneshLokare/Transformers</a></p><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including sentiment analysis.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects</p><h3>1. Prepare the task-specific data</h3><p>Gather and prepare the annotated data for the specific task, such as text classification, sentiment analysis, or named entity recognition. This data will be used for fine-tuning the pre-trained model.</p><p><strong>Download dataset</strong></p><pre># download data from provided link<br>!wget -nc https://www.dropbox.com/s/lkd0eklmi64m9xm/AirlineTweets.csv?dl=0</pre><p><strong>Import required libraries</strong></p><pre># import required libraries<br>import pandas as pd<br>import numpy as np<br>import seaborn as sns<br><br>import torch<br><br>from sklearn.metrics import f1_score, accuracy_score, confusion_matrix<br>from sklearn.model_selection import train_test_split</pre><p><strong>Load dataset</strong></p><pre>df = pd.read_csv(&#39;AirlineTweets.csv?dl=0&#39;)</pre><p><strong>check how data looks like</strong></p><pre>df.head()</pre><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*oLAZkmiX4R5rVZxQvSDGHQ.png" /></figure><p><strong>Keep required columns only</strong></p><pre>df = df[[&#39;airline_sentiment&#39;,&#39;text&#39;]]</pre><p><strong>check data again</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*4KdqoheVkF2ksAAimNMmaA.png" /></figure><p>As we can see, it has the 2 columns which we have selected.</p><p><strong>Check the distribution of classes</strong></p><pre>df[&#39;airline_sentiment&#39;].hist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*jdNnJdq1mbsavbqTgpPG9w.png" /></figure><p>As we can see this is an imbalanced distribusion of classes. We will see if our model can handle imbalanced dataset or it will biased towards majority class.</p><p>Map classes to the integers</p><pre>target_map = { &#39;positive&#39;: 1, &#39;negative&#39;: 0, &#39;neutral&#39;: 2}<br>df[&#39;target&#39;] = df[&#39;airline_sentiment&#39;].map(target_map)</pre><p>The first line defines a dictionary target_map that maps the original categorical target variable &#39;sentiment&#39; to a numerical representation, with &#39;positive&#39; mapped to 1, &#39;negative&#39; mapped to 0 and ‘neutral’ mapped to 2.</p><p>The second line applies this mapping to the ‘airline_sentiment’ column of the DataFrame df using the map method and saves the result as a new column &#39;target&#39; in the same DataFrame. This can be useful for training machine learning models, which often require numerical input variables.</p><p>Save data to new csv file. Because transformers required special format of dataset to perform operations on it, which we will give using load_dataset class. We will see next how dataset required for the transformers.</p><pre>df1 = df[[&#39;text&#39;,&#39;target&#39;]]<br>df1.columns = [&#39;sentence&#39;,&#39;label&#39;]<br>df1.to_csv(&#39;data.csv&#39;, index = False)</pre><ol><li>df1 = df[[‘text’, ‘target’]]: This line selects the “text” and “target” columns from the data frame df and assigns them to a new data frame df1.</li><li>df1.columns = [‘sentence’, ‘label’]: This line renames the columns in df1 to “sentence” and “label”. As transformers must required ‘label’ as a target column name. Otherwise it will raise an error.</li><li>df1.to_csv(‘data.csv’, index=False): This line saves the data frame df1 as a CSV file named “data.csv”. The “index” argument is set to False, which means that the index of the data frame will not be saved to the CSV file.</li></ol><p>The resulting “data.csv” file will contain two columns, “sentence” and “label”, which are the pre-processed features for the text sequence and target label, respectively.</p><pre>!pip install datasets</pre><p>The “!pip install datasets” command installs the “datasets” library, which provides a unified API for accessing a variety of publicly available datasets for natural language processing tasks such as sentiment analysis, machine translation, and summarization.</p><ol><li>“from datasets import load_dataset” imports the “load_dataset” function from the “datasets” library.</li><li>“raw_dataset = load_dataset(‘csv’, data_files = ‘data.csv’)” uses the “load_dataset” function to load a dataset stored in a CSV file named “data.csv”, which we have stored above.</li></ol><p>Check how loaded dataset looks like</p><pre>raw_dataset</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*oiDdLMtoR3zGp3OaEQ-d-Q.png" /></figure><p>The “DatasetDict” is a dictionary-like object that contains one dataset named “train”. We can one or more datasets.</p><p>The “Dataset” object represents a single dataset and provides information about the features and structure of the data. The “features” attribute is a list of strings that specifies the names of the features in the dataset. In this case, the dataset has two features: “sentence” and “label”.</p><p>The “num_rows” attribute specifies the number of rows (examples) in the dataset. In this case, the “train” dataset has 14640 rows.</p><p>Split dataset into train and test</p><pre>split = raw_dataset[&#39;train&#39;].train_test_split(test_size=0.3, seed=42)</pre><ol><li>“raw_dataset[‘train’]” accesses the “train” dataset from the “raw_dataset” object.</li><li>“.train_test_split(test_size=0.3, seed=42)” uses the “train_test_split” method of the “Dataset” class to split the “train” dataset into training and test sets.</li></ol><p>The “test_size” argument is a float that specifies the proportion of the dataset to be used for testing. In this case, 0.3 means that 30% of the data will be used for testing, and 70% will be used for training.</p><p>The “seed” argument is an integer that sets the random seed for the split. This ensures that the split is deterministic and reproducible.</p><p><strong>Check what we have got back</strong></p><pre>split</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*iaaNq_LATyG-GR_7_iTB_A.png" /></figure><p>We have got test set.</p><p><strong>How to handle multiple files</strong></p><p>Below 2 codes are only for muliple files and if we have both train and test sets.</p><pre># if we have multiple csv files<br>raw_dataset = load_dataset(&#39;csv&#39;, data_files = [&#39;file1.csv&#39;,&#39;file2.csv&#39;])</pre><p><strong>If we already have train test split</strong></p><pre>raw_dataset = load_dataset(&#39;csv&#39;, <br>              data_files = { &#39;train&#39;: [&#39;train1.csv&#39;,&#39;train2.csv&#39;],<br>                             &#39;test&#39;: &#39;test.csv&#39;})</pre><h3>2. Tokenize the data</h3><p>Convert the task-specific data into a numerical representation suitable for input into the Transformer model. This typically involves tokenizing the text into subwords or words, mapping the tokens to integers, and encoding the input as a tensor.</p><p>To get more information about Tokenization, please read this</p><pre># Import AutoTokenizer and create tokenizer object<br>from transformers import AutoTokenizer<br>checkpoint = &#39;bert-base-cased&#39;<br>tokernizer = AutoTokenizer.from_pretrained(checkpoint)</pre><p>The code is using the AutoTokenizer class from the transformers library to load a pre-trained tokenizer for the BERT model with the &quot;base&quot; architecture and the &quot;cased&quot; version. The pre-trained tokenizer will be used to convert input sequences of text into numerical representations (tokens) that can be fed into the model. The checkpoint variable specifies the name of the pre-trained tokenizer to use, and the from_pretrained method is used to load the tokenizer from the transformers library&#39;s pre-trained models.</p><p><strong>Define tokenizer function</strong></p><pre>def tokenize_fn(batch):<br>  return tokernizer(batch[&#39;sentence&#39;], truncation = True)</pre><ol><li>“def tokenize_fn(batch):” defines the function “tokenize_fn”, which takes a single argument “batch”. The “batch” argument is expected to be a dictionary-like object that contains the text data to be tokenized.</li><li>“return tokernizer(batch[‘sentence’], truncation = True)” returns the result of applying a tokenization function, “tokernizer”, to the “sentence” feature of the “batch” data. The “truncation” argument is set to “True”, which means that the tokenization function will truncate sequences that are longer than the maximum length specified by the model.</li></ol><pre>tokenized_dataset = split.map(tokenize_fn, batched = True)</pre><ol><li>“split.map(tokenize_fn, batched = True)” applies the “tokenize_fn” function to each example in the “split” dataset, which was obtained by splitting the “raw_dataset” into training and test sets.</li><li>“batched = True” specifies that the tokenization function should be applied to batches of data, rather than to individual examples. This can improve performance by allowing the tokenization to be parallelized.</li><li>After that we will get tokenized data which we can directly feed to our model.</li></ol><p>for more details on Tokenization, please check below link.</p><p><a href="https://medium.com/@lokaregns/preparing-text-data-for-transformers-tokenization-mapping-and-padding-9fbfbce28028">Preparing Text Data for Transformers: Tokenization, Mapping and Padding</a></p><h3>3. Choose a pre-trained model</h3><p>Select a pre-trained Transformer model that is well-suited for the task. There are many pre-trained models available in popular NLP libraries such as Hugging Face Transformers, TensorFlow Hub, or the AllenNLP library. Choose a model with a good balance between the size of the model and the task complexity.</p><p><strong>Import classification model</strong></p><pre>from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments</pre><p>The code imports three classes from the transformers library: AutoModelForSequenceClassification, Trainer, and TrainingArguments.</p><p>AutoModelForSequenceClassification is a class from the transformers library that implements a sequence classification model, a type of model that is used to predict the class of a sequence of inputs (e.g., a sentence). It uses the AutoModel architecture, which automatically selects the most suitable model architecture for the given task and data.</p><p>Trainer is a class that provides a high-level API for training a machine learning model. It can be used to train a model using any torch.nn.Module instance, including models implemented using the transformers library.</p><p>TrainingArguments is a class that defines the arguments used to configure a training run. It includes arguments such as the number of training steps, the learning rate, the batch size, and many others. When using the Trainer class, an instance of TrainingArguments is passed to the constructor to specify the configuration for a training run.</p><h3>4. Define a fine-tuning architecture</h3><p>Decide which layers of the pre-trained model to fine-tune, and add additional layers as needed to perform the specific task. For example, for a text classification task, a dense layer with a softmax activation function may be added on top of the pre-trained model to produce class predictions.</p><p><strong>Load model</strong></p><pre>model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels = 3)</pre><p>If you have a binary classification or sentiment analysis problem then use “num_labels” = 2.</p><p><strong>Install the </strong><strong>torchinfo library.</strong></p><pre>!pip install torchinfo</pre><p>torchinfo is a Python library for getting information about PyTorch models and tensors. It provides a convenient way to inspect the architecture of a PyTorch model, including the shape and size of the tensors that are passed between the layers, as well as the number of parameters and memory usage of each layer.</p><p><strong>print the model summary</strong></p><pre>from torchinfo import summary<br>summary(model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*dWmM9VJCXoxPFlSgZwqcgQ.png" /></figure><p>Model has more than 108 M parameters and all they are trainable.</p><h3>5. Compile the model</h3><p>Choose a loss function and an optimizer suitable for the task, and compile the model by defining the training and evaluation procedures.</p><pre>training_args = TrainingArguments(output_dir=&#39;training_dir&#39;,<br>                                  evaluation_strategy=&#39;epoch&#39;,<br>                                  save_strategy=&#39;epoch&#39;,<br>                                  num_train_epochs=3,<br>                                  per_device_train_batch_size=16,<br>                                  per_device_eval_batch_size=64,<br>                                  )</pre><p>The above code creates an object of the TrainingArguments class with specified arguments for training a model. The output_dir argument sets the directory where the model and training-related files will be saved. The evaluation_strategy argument sets how often evaluation should be done, and in this case, it&#39;s set to be done every epoch.</p><p>The save_strategy argument sets when the model should be saved, and it&#39;s set to be saved every epoch. The num_train_epochs argument sets the number of training epochs, and it&#39;s set to 3.</p><p>The per_device_train_batch_size argument sets the batch size for training, and it&#39;s set to 16. The per_device_eval_batch_size argument sets the batch size for evaluation, and it&#39;s set to 64.</p><p><strong>Define evaluation metrics, which we will pass during training</strong></p><pre>def compute_metrics(logits_and_labels):<br>  logits, labels = logits_and_labels<br>  predictions = np.argmax(logits, axis=-1)<br>  acc = np.mean(predictions == labels)<br>  f1 = f1_score(labels, predictions, average = &#39;micro&#39;)<br>  return {&#39;accuracy&#39;: acc, &#39;f1_score&#39;: f1}</pre><p>The above code defines a function compute_metrics that takes a tuple of logits_and_labels as input and computes two evaluation metrics: accuracy and F1 score.</p><p>The function first unpacks the tuple into logits and labels. Then it calculates the predictions using np.argmax along the last axis of logits. The accuracy is calculated as the mean of the equality of predictions and labels. The F1 score is calculated using the f1_score function from scikit-learn with average=&#39;micro&#39;. The accuracy and F1 score are returned as a dictionary.</p><h3><strong>6. Train the model</strong></h3><p>Train the model on the task-specific data using a suitable number of epochs, and monitor the performance of the model on a validation set. If necessary, adjust the model architecture or training procedure, and repeat the training process until satisfactory performance is achieved.</p><pre>trainer = Trainer(model,<br>                  training_args,<br>                  train_dataset = tokenized_dataset[&quot;train&quot;],<br>                  eval_dataset = tokenized_dataset[&quot;test&quot;],<br>                  tokenizer=tokernizer,<br>                  compute_metrics=compute_metrics)</pre><p>The above code creates an object of the Trainer class and assigns it to the variable trainer. This class is used for training and evaluating a machine learning model.</p><p>The Trainer class takes several arguments:</p><ul><li>model is the model to be trained.</li><li>training_args is an instance of the TrainingArguments class that contains the arguments for training the model.</li><li>train_dataset is the training dataset, which is assigned tokenized_dataset[&quot;train&quot;] in this case.</li><li>eval_dataset is the evaluation dataset, which is assigned tokenized_dataset[&quot;test&quot;] in this case.</li><li>tokenizer is the tokenizer used for the input data, and it&#39;s assigned tokernizer in this case.</li><li>compute_metrics is the function used to compute the evaluation metrics, which is compute_metrics in this case.</li></ul><p>With these arguments, the Trainer object is configured to train and evaluate the specified model using the specified datasets, tokenizer, and evaluation metrics.</p><p><strong>Call trainer object and train the model</strong></p><pre>trainer.train()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*9_0_JGLqaaCKcLjlwRkEJw.png" /></figure><p>As we have set 3 epochs, we have got 3 evaluation metrics. Notice our model is overfitting as validation loss is increases for each epochs. So one epoch will be sufficient for this dataset as we are getting 83% accuracy as well as f1 score. Next I will explain how to do more fine-tuning to improve accuracy.</p><p>After training , training data will be saved at training_dir</p><pre>! ls training_dir</pre><p>Output: checkpoint-1282, checkpoint-1923, checkpoint-641 runs</p><p>As we can see there are 3 checkpoints, we will select first model as it has highest performance.</p><h3>7. Evaluate the model</h3><p>Evaluate the performance of the fine-tuned model on a test set, and compare it to other models or baselines.</p><p><strong>Import pipeline</strong></p><pre>from transformers import pipeline</pre><p>The above code imports the pipeline class from the transformers library. The pipeline class is a high-level API for using pre-trained models for a variety of tasks, such as text classification, sequence labeling, and generation.</p><pre>saved_model = pipeline(&#39;text-classification&#39;,<br>                       model = &#39;training_dir/checkpoint-1282&#39;)</pre><p>The above code creates an instance of the pipeline class and assigns it to the variable saved_model. The pipeline class is used to perform text classification tasks.</p><p>The pipeline class takes two arguments:</p><ul><li>&#39;text-classification&#39; is the task being performed, in this case text classification.</li><li>model is the model to be used, which is &#39;training_dir/checkpoint-1282&#39; in this case.</li></ul><p>This code creates a text classification pipeline that uses the specified pre-trained model stored in the directory &#39;training_dir/checkpoint-1282&#39;. With this pipeline, it&#39;s possible to perform text classification tasks using the pre-trained model with just a few lines of code.</p><p><strong>Get test set</strong></p><pre>split[&#39;test&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*XbOq6Jemy0wlyoRiibH8tw.png" /></figure><p><strong>Get predictions</strong></p><pre>predictions = saved_model(split[&#39;test&#39;][&#39;sentence&#39;])</pre><p><strong>printout few predictions</strong></p><pre>predictions[:10]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*c72ymdmeFs1PihIsc4NM6w.png" /></figure><p>we are getting list of dictionaries. Next we will write a function to get labels only.</p><pre>def get_label(d):<br>  return int(d[&#39;label&#39;].split(&#39;_&#39;)[1])<br>predictions = [get_label(d) for d in predictions]</pre><p>The function returns the integer value that is obtained by splitting the string value stored under the key ‘label’ in the dictionary, using the split method. The returned integer is the second element of the resulting list obtained after splitting the string, which is actually a predicted label.</p><p>The second line uses a list comprehension to apply the get_label function to each element in the list predictions and returns a new list with the results.</p><p>Once we got the labels, Calculate accuracy using accuracy_score function.</p><pre>print(&quot;acc:&quot;,accuracy_score(split[&#39;test&#39;][&#39;label&#39;], predictions))</pre><p><em>acc: 0.8376593806921676</em></p><p>On test set we have got 83.76% accuracy.</p><p><strong>Calculate f1 score</strong></p><pre>print(&quot;f1:&quot;,f1_score(split[&#39;test&#39;][&#39;label&#39;], predictions, average = &#39;macro&#39;))</pre><p><em>f1: 0.7837583256590012</em></p><p>On test set we have got 0.78 f1 score.</p><p><strong>Plot confusion matrix</strong></p><pre># create function for plotting confusion matrix<br>def plot_cm(cm):<br>  classes = [&#39;negative&#39;,&#39;positive&#39;,&#39;neutral&#39;]<br>  df_cm = pd.DataFrame(cm, index=classes, columns=classes)<br>  ax = sns.heatmap(df_cm, annot = True, fmt=&#39;g&#39;)<br>  ax.set_xlabel(&#39;Predicted&#39;)<br>  ax.set_ylabel(&#39;Actual&#39;)<br><br>cm = confusion_matrix(split[&#39;test&#39;][&#39;label&#39;],predictions, normalize = &#39;true&#39;)<br>plot_cm(cm)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*DtNTg6B4l2D4d943x5Lhyw.png" /><figcaption>Confusion Matrix</figcaption></figure><p>As we know, our dataset has a higher proportion of negative classes, and according to the confusion matrix, it appears that the model has a slight bias towards the negative class.</p><p><strong>Same procedure can be used for any classification task.</strong></p><p>Next, I will demonstrate how to train and fine-tune transformer models for other tasks on custom datasets.</p><p>If you enjoyed this post, please follow me and be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><p><a href="https://medium.com/@lokaregns">Ganesh Lokare - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f261579ae068" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preparing Text Data for Transformers: Tokenization, Mapping and Padding]]></title>
            <link>https://medium.com/@lokaregns/preparing-text-data-for-transformers-tokenization-mapping-and-padding-9fbfbce28028?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fbfbce28028</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Fri, 10 Feb 2023 07:13:53 GMT</pubDate>
            <atom:updated>2023-02-10T07:13:53.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*N_DGewxYutuK94hy8QXRkA.png" /></figure><p>In this blog we will be introducing our next section of the series which is how to fine tune a transformer models on a custom dataset.</p><p>Why this is different from previous section, previous section was easy because it only involved using models that someone else trained i.e. we only used the model for inference / prediction. The code was very simple because the library we were using provides a nice universal interface for different types of models. So we have the same interface for different kinds of task.</p><p>From now, we won’t just be making predictions but training our own models on our datasets as well.</p><h3>What We Will Cover in This Blog</h3><ol><li>Review text preprocessing for transformers</li><li>Tokenization, token to integer mapping, padding</li></ol><h3>Text Preprocessing</h3><p>The steps:</p><ol><li>Tokenization</li><li>Map tokens to integers (i.e. “I like Dogs” : [312, 820, 123]</li><li>Padding / Truncation (to process bathes of different length sequences)</li></ol><h3>1. Tokenization</h3><p>If you’ve already studied ML / DL for NLP, you’ve probably tokenized documents into words.</p><p>Tokenization is the process of dividing text into smaller units called tokens, which can be words, phrases, subwords, or characters. In the context of Transformer models, tokenization is a crucial step in preprocessing text data for use in natural language processing tasks.</p><p>Tokenization helps the model to identify the underlying structure of the text and process it more efficiently. It enables the model to handle variations in language and handle words with multiple meanings or forms. The choice of tokenization method depends on the task and language being processed and can be a major factor in determining the effectiveness of the model.</p><p>Tokenization is an essential step in the text preprocessing pipeline for Transformer models and plays a crucial role in the overall performance of the model.</p><h4><strong>Types of tokenization:</strong></h4><h4>a) word-level tokenization</h4><p>Word-level tokenization is a method of dividing text into smaller units called tokens, where each token is a single word.</p><p>For example, consider the following sentence:</p><pre>&quot;The quick brown fox jumps over the lazy dog.&quot;</pre><p>In word-level tokenization, this sentence would be split into the following tokens:</p><pre>&quot;The&quot;<br>&quot;quick&quot;<br>&quot;brown&quot;<br>&quot;fox&quot;<br>&quot;jumps&quot;<br>&quot;over&quot;<br>&quot;the&quot;<br>&quot;lazy&quot;<br>&quot;dog&quot;</pre><p>Each token is treated as a separate entity and can be processed individually. Word-level tokenization is simple and straightforward, but it can result in difficulties in handling words with multiple meanings or forms. It is also limited by the vocabulary size and cannot handle out-of-vocabulary words.</p><h4>b) Character-level tokenization</h4><p>Character-level tokenization is a method of dividing text into smaller units called tokens, where each token is a single character.</p><p>For example, consider the following sentence:</p><pre>&quot;I like Dogs.&quot;</pre><p>In character-level tokenization, this sentence would be split into the following tokens:</p><pre>&quot;I&quot;<br>&quot;l&quot;<br>&quot;i&quot;<br>&quot;k&quot;<br>&quot;e&quot;<br>&quot;D&quot;<br>&quot;o&quot;<br>&quot;g&quot;<br>&quot;s&quot;<br>&quot;.&quot;</pre><p>Each token is treated as a separate entity and can be processed individually.</p><h4>c) Subword-level tokenization</h4><p>Subword-level tokenization is a method of dividing text into smaller units called tokens, where each token is a subword unit, typically a sequence of characters. This method of tokenization is used when processing natural language text and aims to capture both the meaning of individual words and the relationships between subword units within words.</p><p>For example, consider the following sentence:</p><pre>&quot;The quick brown fox jumps over the lazy dog.&quot;</pre><p>In subword-level tokenization, this sentence would be split into the following tokens:</p><pre>&quot;The&quot;<br>&quot;quic&quot;<br>&quot;k&quot;<br>&quot;brown&quot;<br>&quot;fox&quot;<br>&quot;jump&quot;<br>&quot;s&quot;<br>&quot;over&quot;<br>&quot;the&quot;<br>&quot;lazy&quot;<br>&quot;dog&quot;</pre><p>In this example, subword-level tokenization has divided words like “quick” and “jump” into subword units, capturing both the meaning of individual words and the relationships between subword units within words.</p><p>Subword-level tokenization is a popular method of tokenization in NLP and is used in many state-of-the-art NLP models, including Transformer models. It provides a trade-off between the fine-grained representation of character-level tokenization and the simplicity of word-level tokenization, and can lead to improved performance in many NLP tasks.</p><h3>2. Map tokens to integers</h3><p>Mapping tokens to integers in transformers refers to the process of encoding text into numerical representations that can be processed by machine learning models. In NLP, tokens are usually words, subwords, or characters, and mapping them to integers involves assigning a unique integer value to each token. This integer representation is used to represent the text data in a numerical format that can be used by deep learning models, such as transformers.</p><p>The mapping is typically performed by creating a vocabulary of the tokens in the text corpus and assigning each token an integer value based on its frequency of occurrence in the corpus. The most common tokens are assigned lower integer values, while less frequent tokens are assigned higher values. This mapping process allows the model to process the text data efficiently and make predictions based on the underlying patterns in the data.</p><p>Here’s a simple example of mapping tokens to integers in transformers:</p><p>Suppose we have a text corpus consisting of the following sentences:</p><pre>sentence_1 : &quot;The cat sat on the mat&quot;<br>sentence_2 : &quot;The dog chased the cat&quot;<br>sentence_3 : &quot;The mouse ran away from the cat&quot;</pre><p>We can start by creating a vocabulary of all the unique tokens in the text corpus. The vocabulary for this example would be:</p><pre>[&quot;The&quot;, &quot;cat&quot;, &quot;sat&quot;, &quot;on&quot;, &quot;the&quot;, &quot;mat&quot;, &quot;dog&quot;, &quot;chased&quot;, &quot;mouse&quot;, &quot;ran&quot;, &quot;away&quot;, &quot;from&quot;]</pre><p>Next, we can assign integer values to each token in the vocabulary based on their frequency of occurrence. For example, the most frequent token, “the”, could be assigned the integer value 1, the next most frequent token, “cat”, could be assigned the integer value 2, and so on. The final mapping would look like this:</p><pre>{<br> &quot;The&quot;: 1,<br> &quot;cat&quot;: 2,<br> &quot;sat&quot;: 3,<br> &quot;on&quot;: 4,<br> &quot;the&quot;: 5,<br> &quot;mat&quot;: 6,<br> &quot;dog&quot;: 7,<br> &quot;chased&quot;: 8,<br> &quot;mouse&quot;: 9,<br> &quot;ran&quot;: 10,<br> &quot;away&quot;: 11,<br> &quot;from&quot;: 12<br>}</pre><p>Finally, we can use this mapping to encode the text data into numerical representations. For example, the first sentence “The cat sat on the mat” would be encoded as:</p><pre>[1, 2, 3, 4, 1, 6]</pre><p>This encoded representation of the text data can then be fed into a transformer model for processing and prediction.</p><h3><strong>3. Padding / Truncation</strong></h3><p>Padding and truncation are preprocessing techniques used in transformers to ensure that all input sequences have the same length.</p><p>Padding refers to the process of adding extra tokens (usually a special token such as [PAD]) to the end of short sequences so that they all have the same length. This is done so that the model can process all the sequences in a batch simultaneously. The padded tokens do not carry any semantic meaning and are just used to fill up the extra space in the shorter sequences.</p><p>Truncation, on the other hand, refers to the process of cutting off the end of longer sequences so that they are all the same length. This is done to ensure that the model is not overwhelmed by very long sequences and to reduce the computational overhead of processing large sequences.</p><p>In transformers, padding and truncation are usually performed before feeding the input sequences into the model, and the maximum length for the sequences is set based on the specific task and the available computational resources. The choice of the maximum length is a trade-off between keeping enough information from the original sequence and reducing computational overhead.</p><p>Here’s a simple example of padding and truncation in transformers:</p><p>Suppose we have a batch of sequences with varying lengths:</p><pre>Sequence 1: &quot;The cat sat on the mat&quot;<br>Sequence 2: &quot;The dog chased the cat&quot;<br>Sequence 3: &quot;The mouse ran away from the cat and the dog&quot;</pre><p>And let’s say that we want to set the maximum length for all the sequences to be 10 tokens.</p><p>For padding, we would add the special token [PAD] to the end of each sequence until it reaches the maximum length of 10 tokens. The padded sequences would look like this:</p><pre>Sequence 1: &quot;The cat sat on the mat [PAD] [PAD] [PAD] [PAD]&quot;<br>Sequence 2: &quot;The dog chased the cat [PAD] [PAD] [PAD] [PAD] [PAD]&quot;<br>Sequence 3: &quot;The mouse ran away from the cat and the dog&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*I0JbAgArgFzMCWiOB1o9CQ.png" /><figcaption>Padding</figcaption></figure><p>For truncation, we would cut off the end of each sequence so that it fits within the maximum length of 5 tokens. The truncated sequences would look like this:</p><pre>Sequence 1: &quot;The cat sat on the&quot;<br>Sequence 2: &quot;The dog chased the cat&quot;<br>Sequence 3: &quot;The mouse ran away from&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*97eDjgwFVO_QATNYIJBzIA.png" /><figcaption>Truncation</figcaption></figure><p>Note that the choice of padding or truncation, or a combination of both, will depend on the specific task and the desired trade-off between preserving information from the original sequence and reducing computational overhead.</p><h3>Enough theory!!! Let’s code…</h3><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects.</p><p>Import AutoTokenizer class from transformers and build tokenizer object. There are different types of tokenizers but AutoTokenizer is a generic tokenizer that can handle various types of pre-trained models, including BERT, GPT-2, RoBERTa, XLNet, etc.</p><pre># Import AutoTokenizer and create tokenizer object<br>from transformers import AutoTokenizer<br>checkpoint = &#39;bert-base-cased&#39;<br>tokernizer = AutoTokenizer.from_pretrained(checkpoint)</pre><p>The code is using the AutoTokenizer class from the transformers library to load a pre-trained tokenizer for the BERT model with the &quot;base&quot; architecture and the &quot;cased&quot; version. The pre-trained tokenizer will be used to convert input sequences of text into numerical representations (tokens) that can be fed into the model. The checkpoint variable specifies the name of the pre-trained tokenizer to use, and the from_pretrained method is used to load the tokenizer from the transformers library&#39;s pre-trained models.</p><p><strong>Print tokenizer object</strong></p><pre>tokenizer</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thckulpzyHKzkcNK_RW-jQ.png" /><figcaption>tokenizer object</figcaption></figure><p>The BertTokenizerFast class is a tokenizer that is specifically designed for BERT-based models and provides tokenization functionalities for text data. The vocab_size attribute of the BertTokenizerFast instance is set to 28996, which means the vocabulary of the tokenizer consists of 28996 unique tokens.</p><p>The model_max_length attribute is set to 512, which indicates that the maximum length of the input sequence that the BERT model can handle is 512 tokens. This means that if the input sequence is longer than 512 tokens, it will be truncated to fit the maximum length.</p><p>The is_fast attribute is set to True, which means that the tokenization process is optimized for speed and is faster than other tokenization methods.</p><p>The padding_side attribute is set to &quot;right&quot;, which means that the tokenizer will add padding tokens to the right side of the input sequence if it is shorter than the maximum length. The truncation_side attribute is set to &quot;right&quot;, which means that the tokenizer will truncate the input sequence from the right side if it is longer than the maximum length.</p><p>The special_tokens attribute is a dictionary that contains the special tokens used in the BERT model, such as the unknown token ([UNK]), the separator token ([SEP]), the padding token ([PAD]), the class token ([CLS]), and the mask token ([MASK]). These special tokens play a crucial role in the BERT model&#39;s input and output encoding.</p><p><strong>Test our tokenizer with simple string</strong></p><pre>tokernizer(&#39;hello world&#39;)</pre><p>Output: <em>{</em></p><p><em>input_ids’: [101, 19082, 1362, 102],</em></p><p><em>‘token_type_ids’: [0, 0, 0, 0],</em></p><p><em>‘attention_mask’: [1, 1, 1, 1] }</em></p><p>The code is applying the tokernizer object, which is an instance of the AutoTokenizer class, to the input text &#39;hello world&#39;. The tokernizer object will convert the input text into a numerical representation that can be fed into the pre-trained model.</p><p>The output of the code is a dictionary with three keys:</p><ol><li>input_ids: This is a list of integers that represent the numerical representation of the input text. Each integer corresponds to a token in the vocabulary of the pre-trained model.</li><li>token_type_ids: This is a list of integers that indicate the type of each token in the input sequence. For example, in a sequence classification task, the first token of the input sequence could be marked as type 0, and the second token as type 1.</li><li>attention_mask: This is a list of 1&#39;s and 0&#39;s that indicate which tokens should be attended to by the pre-trained model and which should be ignored. A 1 indicates that the token should be attended to, while a 0 indicates that the token should be ignored.</li></ol><p>‘input_ids’: [101, 19082, 1362, 102] Note that there are 4 input ids because the input has been converted to: [CLS] hello world [SEP]</p><p><strong>As we can see tokenizer’s output is a dictionary of lists, So we can not feed lists directly to the model. If we pass lists directly model will raise an error. It required tesnsor.</strong></p><h3>Proper model input</h3><pre>tokernizer(&#39;hello world&#39;, return_tensors = &#39;pt&#39;)</pre><p>Output: {</p><p>‘input_ids’: tensor([[ 101, 19082, 1362, 102]]),</p><p>‘token_type_ids’: tensor([[0, 0, 0, 0]]),</p><p>‘attention_mask’: tensor([[1, 1, 1, 1]])}</p><p>The return_tensors argument is set to &#39;pt&#39;, which indicates that the output should be in PyTorch tensor format.</p><p>The available options for return_tensors are:</p><ol><li>&#39;pt&#39;: Returns PyTorch tensors.</li><li>&#39;tf&#39;: Returns TensorFlow tensors.</li><li>None: Returns Python lists. This is the default option if return_tensors is not specified.</li></ol><p>You can choose the appropriate option based on the deep learning framework you are using for your NLP model. If you are using PyTorch, you can set return_tensors to &#39;pt&#39;. If you are using TensorFlow, you can set return_tensors to &#39;tf&#39;. If you are not using a deep learning framework, you can use the default option, which returns Python lists.</p><h3>Multiple Inputs</h3><pre>data = [&quot;I like cat&quot;,<br>        &quot;Do you like cat too?&quot;]<br><br>tokernizer(data,padding = True, truncation=True,return_tensors=&#39;pt&#39;)</pre><p>Output: {</p><p>‘input_ids’: tensor([[ 101, 146, 1176, 5855, 102, 0, 0, 0],</p><p>[ 101, 2091, 1128, 1176, 5855, 1315, 136, 102]]),</p><p>‘token_type_ids’: tensor([[0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0]]),</p><p>‘attention_mask’: tensor([[1, 1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1]])}</p><p>As we can see, we have inputs of different lengths, so it is necessary to apply padding or truncation. If we haven’t applied padding and truncation model will not accept input of different lengths.</p><p>By setting padding to True, the tokenized inputs will be padded with a special padding token (usually represented by 0 in the vocabulary) so that all inputs have the same length. This is often necessary in NLP tasks, as many models expect a fixed-length input.</p><p>By setting truncation to True, if any of the input texts exceed the maximum length of the pre-trained model, they will be truncated to that maximum length. This is necessary to ensure that the inputs can be processed by the pre-trained model.</p><p>In Transformers, we only require the above text preprocessing steps, which is called Tokenization. After that, we can directly feed this preprocessed data to the model.</p><h3>That’s it for the text preprocessing.</h3><p>Next, I will demonstrate how to train and fine-tune transformer models on custom datasets.</p><p>If you enjoyed this post, please follow me and be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><p><a href="https://medium.com/@lokaregns">Ganesh Lokare - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fbfbce28028" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Question Answering with Hugging Face Transformers: A Beginner’s Guide]]></title>
            <link>https://medium.com/@lokaregns/question-answering-with-hugging-face-transformers-a-beginners-guide-487ae1a91b9a?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/487ae1a91b9a</guid>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Thu, 09 Feb 2023 09:05:31 GMT</pubDate>
            <atom:updated>2023-02-09T09:05:31.227Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EpCJ7T1xlteNTYIkFTpWCQ.jpeg" /></figure><h3>What We Will Cover in This Blog</h3><ol><li>Review of what Question Answering is and where we use it.</li><li>How to use Hugging face Transformers for Question Answering with just a few lines of code.</li></ol><p>The main focus of this blog, using a very high level interface for transformers which is the Hugging face pipeline. Using this interface you will see that we can generate answers from given question and corpus of text with just 1 or 2 lines of code.</p><h3><strong>What is Question Answering in NLP?</strong></h3><p>Question Answering (QA) in Natural Language Processing (NLP) is a task that involves using computational methods to automatically answer questions posed in natural language. This task involves identifying the relevant information within a large corpus of text and extracting the answer that best addresses the question. It involves a combination of various NLP techniques, such as information retrieval, information extraction, and text understanding, to accurately answer questions based on the context and meaning of the input text. QA systems are widely used in applications such as customer service, knowledge management, and education, providing quick and accurate answers to a wide range of questions.</p><h3>How Question Answering works in NLP?</h3><p>Question Answering (QA) in NLP involves identifying the answer to a question from a large corpus of text, such as a document or a collection of documents. The process typically involves the following steps:</p><ol><li>Question Analysis: The question is analyzed to identify its type (e.g., factual, definition, comparison, etc.) and the information required to find the answer (e.g., named entities, keywords, etc.).</li><li>Information Retrieval: The corpus is searched to retrieve the relevant documents or text snippets that contain the information needed to answer the question.</li><li>Candidate Answer Generation: Possible answers are extracted from the retrieved text snippets and candidate answers are generated.</li><li>Answer Selection: The candidate answers are evaluated against the question and the best answer is selected based on various criteria, such as relevance, accuracy, and confidence.</li></ol><p>The process can be aided by techniques such as natural language processing (NLP), information retrieval (IR), and machine learning (ML), among others. The performance of QA systems can be improved by using large pre-trained language models, such as BERT or GPT, to encode the context of the question and candidate answers.</p><h3>Use of the Question Answering</h3><p>Question Answering (QA) systems have a wide range of applications, including:</p><ol><li>Customer Service: QA systems can be used to automate customer service by providing instant answers to common questions.</li><li>Knowledge Management: QA systems can be used to manage and retrieve information from large knowledge bases, such as company FAQs and product manuals.</li><li>Education: QA systems can be used to provide students with instant answers to their questions and help them learn more effectively.</li><li>Healthcare: QA systems can be used to provide healthcare professionals with instant answers to clinical questions, improving patient outcomes and reducing diagnostic errors.</li><li>News and Media: QA systems can be used to provide quick answers to factual questions about current events, politics, sports, and more.</li><li>Legal: QA systems can be used to provide lawyers and legal researchers with instant answers to legal questions, improving the speed and efficiency of their work.</li></ol><p>Overall, QA systems aim to improve decision-making, increase productivity, and provide fast and accurate answers to questions, making it a valuable tool in a wide range of industries and domains.</p><h3>Advantages of using Transformers for Question Answering</h3><p>Transformers are widely used for Question Answering (QA) for several reasons:</p><ol><li>Contextual Embeddings: Transformers can capture the context of a question and candidate answer, allowing them to better understand the relationship between the two and make more accurate predictions.</li><li>Pre-training: Transformers can be pre-trained on large amounts of text data, giving them a strong understanding of language and enabling them to perform well on a wide range of tasks, including QA.</li><li>Attention Mechanisms: Transformers use attention mechanisms, which allow them to focus on specific parts of the input when making predictions, leading to improved performance on QA tasks.</li><li>Transfer Learning: Transformers can be fine-tuned for specific QA tasks, allowing for transfer learning from a pre-trained model and reducing the amount of labeled data needed for training.</li><li>Performance: Transformers have demonstrated state-of-the-art performance on a wide range of QA tasks, outperforming traditional methods in terms of accuracy and speed.</li></ol><p>Overall, the use of transformers for QA allows for more accurate, fast, and scalable solutions compared to traditional methods, making it a popular choice in many applications.</p><h3>Enough theory!!! Let’s code…</h3><p>Let’s begin with installing transformers as we regularly do.</p><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including QA.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects.</p><pre>from transformers import pipeline<br><br># create pieline for QA<br>qa = pipeline(&#39;question-answering&#39;)</pre><p>The line “from transformers import pipeline” imports the pipeline module from the Transformers library, which provides an easy-to-use interface for common NLP tasks, including QA. As we have not explicitly supplied any model, by default it will select <strong>distilbert.</strong></p><p>The “pipeline” function is then used to load a pre-trained QA pipeline by passing in the argument “question-answering”. This pipeline is a combination of a pre-trained language model, such as BERT or GPT, and a classifier that predicts the likelihood of a candidate answer being the correct one.</p><p>Once the QA pipeline is loaded, it can be used to answer questions by passing a question and a context (e.g., a document or a paragraph) to the pipeline, and it will return the answer with the highest predicted probability.</p><p>This code provides a simple and convenient way to get started with Question Answering using pre-trained models, without the need to implement and train the models from scratch.</p><p>Let’s try with simple context and question.</p><pre>ctx = &quot;My name is Ganesh and I am studying Data Science&quot;<br>que = &quot;What is Ganesh studying?&quot;<br>qa(context = ctx, question = que)</pre><p>The code above is using the Question Answering (QA) pipeline loaded in the previous code to answer a question.</p><ol><li>Context: The variable “ctx” stores the context in which the question will be answered. In this case, it is a sentence stating “My name is Ganesh and I am studying Data Science”.</li><li>Question: The variable “que” stores the question to be answered. In this case, it is “What is Ganesh studying?”</li><li>QA Pipeline: The QA pipeline stored in the variable “qa” is called with the context and question as input. The “context” argument is set to “ctx” and the “question” argument is set to “que”.</li><li>Output: The QA pipeline will return the answer with the highest predicted probability, along with the start and end position of the answer in the context, and the confidence score.</li></ol><p>The code demonstrates how to use the pre-trained QA pipeline to answer questions in a given context. The pipeline uses the context and question to generate candidate answers, and then uses the pre-trained model to predict the likelihood of each candidate answer being the correct one. The answer with the highest predicted probability is returned as the final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*M0mN-x90yokwpN3ApdyjzQ.png" /></figure><p>The output is the result of using the Question Answering (QA) pipeline to answer the question.</p><ol><li>Score: The ‘score’ field represents the confidence score of the predicted answer, with a value between 0 and 1. A higher score indicates a higher level of confidence in the answer. In this case, the score is 0.99887, which is close to 1, indicating a high level of confidence in the answer.</li><li>Start: The ‘start’ field represents the starting position of the answer in the context. In this case, the start position is 36, which is the first character of the word “Data”.</li><li>End: The ‘end’ field represents the ending position of the answer in the context. In this case, the end position is 48, which is the last character of the word “Science”.</li><li>Answer: The ‘answer’ field represents the final answer to the question. In this case, the answer is “Data Science”, which is the correct answer to the question “What is Ganesh studying?”</li></ol><p>The output provides the answer to the question, along with the confidence score, start and end positions in the context, allowing the user to assess the reliability of the answer and use it in further applications as needed.</p><p>For the next portion of the script we are going to use a snippet from Wikipedia&#39;s page of Elon Musk.</p><pre>context = &#39;&#39;&#39;Elon Reeve Musk FRS (/ˈiːlɒn/ EE-lon; born June 28, 1971) is a business magnate and investor. He is the founder, CEO and chief engineer of SpaceX; <br>            angel investor, CEO and product architect of Tesla, Inc.; owner and CEO of Twitter, Inc.; founder of The Boring Company; co-founder of Neuralink <br>            and OpenAI; and president of the philanthropic Musk Foundation. With an estimated net worth of around $175 billion as of February 3, 2023, primarily <br>            from his ownership stakes in Tesla and SpaceX,[4][5] Musk is the second-wealthiest person in the world, according to both the Bloomberg Billionaires <br>            Index and Forbes&#39;s real-time billionaires list.[6][7]<br>            Musk was born in Pretoria, South Africa, and briefly attended at the University of Pretoria before moving to Canada at age 18, acquiring citizenship <br>            through his Canadian-born mother. Two years later, he matriculated at Queen&#39;s University and transferred to the University of Pennsylvania, <br>            where he received bachelor&#39;s degrees in economics and physics. He moved to California in 1995 to attend Stanford University. After two days, he <br>            dropped out and with his brother Kimbal, co-founded the online city guide software company Zip2. In 1999, Zip2 was acquired by Compaq for $307 million <br>            and Musk co-founded X.com, a direct bank. X.com merged with Confinity in 2000 to form PayPal, which eBay acquired for $1.5 billion in 2002.<br>            With $175.8 million, Musk founded SpaceX in 2002, a spaceflight services company. In 2004, he was an early investor in the electric vehicle manufacturer <br>            Tesla Motors, Inc. (now Tesla, Inc.). He became its chairman and product architect, assuming the position of CEO in 2008. In 2006, he helped create <br>            SolarCity, a solar energy company that was later acquired by Tesla and became Tesla Energy. In 2015, he co-founded OpenAI, a nonprofit artificial <br>            intelligence research company. The following year, he co-founded Neuralink—a neurotechnology company developing brain–computer interfaces—and The Boring <br>            Company, a tunnel construction company. Musk has also proposed a hyperloop high-speed vactrain transportation system. In 2022, his acquisition of Twitter <br>            for $44 billion was completed.&#39;&#39;&#39;<br><br></pre><pre>question = &quot;Who is Elon Musk?&quot;<br>qa(context = context, question = question)</pre><p>{‘score’: 0.4764349162578583, ‘start’: 63, ‘end’: 92, ‘answer’: ‘business magnate and investor’}</p><p>You can see that, the answer is correct. But, the QA model is not a generative model, it only selects an answer from the context and returns it. Additionally, it can only handle one continuous string as the context, so if the answer is present in multiple locations in the context, the model will only return the answer from one location. These limitations should be kept in mind when using the QA model.</p><pre>question = &quot;When Elon Musk was born?&quot;<br>qa(context = context, question = question)</pre><p>{‘score’: 0.9790981411933899, ‘start’: 43, ‘end’: 56, ‘answer’: ‘June 28, 1971’}</p><p>Here answer is correct.</p><pre>question = &quot;What was the name of Elon Musk&#39;s first company?&quot;<br>qa(context = context, question = question)</pre><p>{‘score’: 0.8074138164520264, ‘start’: 1364, ‘end’: 1369, ‘answer’: ‘X.com’}</p><p>Here also answer is correct.</p><pre>question = &quot;What companies has Elon Musk invested in?&quot;<br>qa(context = context, question = question)</pre><p>{‘score’: 0.6982435584068298, ‘start’: 509, ‘end’: 525, ‘answer’: ‘Tesla and SpaceX’}</p><p>As we know Elon Musk has invested in multiple companies, but our model only lists two because the answer is located in multiple contexts.</p><p>Let’s ask a question for which the answer is not present in the context.</p><pre>question = &quot;What is the capital city of Japan?&quot;<br>qa(context = context, question = question)</pre><p>{‘score’: 0.3223932087421417, ‘start’: 725, ‘end’: 747, ‘answer’: ‘Pretoria, South Africa’}</p><p>As we can observe, the model provided an incorrect answer because the information was not present in the context. It’s important to note that even if the answer is not available in the context, the model may still provide an answer. However, it may not be accurate. One can verify the accuracy of the model’s response by checking its confidence score. If the score is low, it’s advisable to verify the answer before accepting it.</p><h3>That’s it for now.</h3><p>Beginner&#39;s section is completed. Next I plan to write additional blogs on Transformers where I will demonstrate how to fine-tune transformer models on custom datasets, as well as how to create transformer models from scratch and train them on custom datasets.</p><p>If you enjoyed this post, be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><p><a href="https://medium.com/@lokaregns">Ganesh Lokare - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=487ae1a91b9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Summarization with Hugging Face Transformers: A Beginner’s Guide]]></title>
            <link>https://medium.com/@lokaregns/text-summarization-with-hugging-face-transformers-a-beginners-guide-9e6c319bb5ed?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e6c319bb5ed</guid>
            <category><![CDATA[text-summarization]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Wed, 08 Feb 2023 04:03:57 GMT</pubDate>
            <atom:updated>2023-02-08T04:03:57.975Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*9gevP3n2IHGc9a28matEog.png" /></figure><h3>What We Will Cover in This Blog</h3><ol><li>Review of what Text Summarization is and where we use it.</li><li>How to perform Text Summarization using Hugging face Transformers (just a few lines of code)</li></ol><p>The main focus of this blog, using a very high level interface for transformers which is the Hugging face pipeline. Using this interface you will see that we can summarize texts with just 1 or 2 lines of code.</p><h3>What is Text Summarization?</h3><p>Text summarization is the process of condensing a large text document into a shorter version while preserving its key information and meaning. The goal of text summarization is to extract the most important information from a text document and present it in a concise and comprehensible form. This can be done through techniques such as keyword extraction, sentence extraction, or abstractive summarization. Text summarization has various applications including news aggregation, content analysis, and information retrieval.</p><h3>How text summarization performed using Transformers?</h3><p>Text summarization using Transformers can be performed in two ways: extractive summarization and abstractive summarization.</p><ol><li>Extractive summarization: In this approach, the most important sentences or phrases from the original text are selected and combined to form a summary. This can be done using algorithms such as TextRank, which uses graph-based algorithms to rank sentences based on their relevance and importance. Transformers can be used to process the text, extract features, and perform sentence ranking.</li><li>Abstractive summarization: In this approach, a new summary is generated by understanding the context of the original text and generating new phrases and sentences that summarize its content. This can be done using techniques such as encoder-decoder models, where the encoder processes the input text to extract its features and the decoder generates the summary. Transformers can be used as the encoder or decoder in this architecture.</li></ol><p>In both extractive and abstractive summarization, Transformers can be trained on large amounts of text data to learn the patterns and relationships between words, sentences, and documents, making them well-suited for text summarization tasks.</p><h3>Use of Text summarization</h3><p>Text summarization has various uses, including:</p><ol><li>News aggregation: Summarizing news articles to provide a quick overview of the most important information.</li><li>Content analysis: Reducing large volumes of text data to identify patterns, trends, and insights.</li><li>Information retrieval: Summarizing search results to provide users with a concise and easy-to-understand overview of relevant information.</li><li>Document management: Summarizing long documents to make them easier to manage and search through.</li><li>Meeting minutes: Summarizing the key points discussed in a meeting to provide a concise and organized record.</li><li>Customer feedback: Summarizing customer feedback to identify common themes and issues.</li><li>Legal contracts: Summarizing legal contracts to provide a quick overview of their key terms and conditions.</li><li>Customer service: Summarizing customer support requests to quickly identify the root cause of an issue.</li></ol><p>By condensing large amounts of text into a more manageable form, text summarization can help users quickly understand the content of a document and make more informed decisions.</p><h3>Why use Transformers for text summarization?</h3><p>Transformers are used for text summarization because they are highly effective in processing and understanding large amounts of text data. There are several reasons why Transformers are a popular choice for text summarization:</p><ol><li>Contextual understanding: Transformers use attention mechanisms to understand the context of words, sentences, and documents, which is crucial for text summarization tasks. This allows Transformers to accurately identify the most important information in a text document.</li><li>Large language model: Transformers have been trained on vast amounts of text data, which enables them to have a deep understanding of language patterns and relationships. This makes them well-suited for text summarization tasks where a comprehensive understanding of language is required.</li><li>Scalability: Transformers can process large amounts of text data in parallel, making them well-suited for summarizing long documents or large volumes of text data.</li><li>End-to-end training: Transformers can be trained end-to-end on text summarization tasks, which allows them to optimize their performance for the specific task at hand.</li><li>State-of-the-art results: Transformers have achieved state-of-the-art results on a wide range of natural language processing tasks, including text summarization.</li></ol><p>By using Transformers for text summarization, organizations can benefit from their ability to understand and process large amounts of text data, leading to more accurate and reliable summarization results.</p><h3>Enough theory!!! Let’s code…</h3><p>We will begin by downloading dataset which is the BBC news dataset. This contains nice long articles, there would be nice candidates for summarization task.</p><pre>!wget -nc https://www.dropbox.com/s/7hb8bwbtjmxovlc/bbc_text_cls.csv?dl=0</pre><p>The command will download the file from the URL .</p><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including Text Summarization.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects.</p><pre># import required libraries<br>from transformers import pipeline<br>import textwrap<br>import numpy as np<br>import pandas as pd<br>from pprint import pprint</pre><p>The code imports the “pipeline” module from the “transformers” library, which is a high-level API for performing natural language processing tasks such as text generation and sentiment analysis. The code also imports the “textwrap” and “numpy” libraries, which are used for text formatting and numerical operations, respectively. Finally, the code imports the “pandas” library and the “pprint” function from the “pprint” library, which are used for data manipulation and pretty-printing, respectively.</p><p>Load dataset into dataframe</p><pre>df = pd.read_csv(&#39;bbc_text_cls.csv?dl=0&#39;)</pre><p>print first few rows of data</p><pre>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*wKnc1Djvo6bHGHNb7HRreg.png" /></figure><p>Choose the random news article</p><pre>doc = df[df.labels == &#39;business&#39;][&#39;text&#39;].sample(random_state=42)</pre><pre># text wrapping function<br>def wrap(x):<br>  return textwrap.fill(x, replace_whitespace = False, fix_sentence_endings = True)</pre><p>above function returns a copy of x where line breaks are inserted such that each line in the returned string is no longer than a specified number of characters (the default is 70).</p><p>The parameters used in the function call are:</p><ul><li>replace_whitespace - a boolean that specifies whether to replace consecutive whitespaces in x with a single space before wrapping. In this case, it&#39;s set to False, meaning that whitespaces will be preserved.</li><li>fix_sentence_endings - a boolean that specifies whether to ensure that wrapped lines end with a complete sentence. In this case, it&#39;s set to True, meaning that the function will try to wrap sentences at a period (.) if possible.</li></ul><p>Print the news article we have selected</p><pre>print(wrap(doc.iloc[0]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nP-1wTPweGCCwV8TEoVZIA.png" /></figure><p>Create summarization pipeline object</p><pre>summarizer = pipeline(&#39;summarization&#39;)</pre><p>The code creates a summarization pipeline from the “transformers” library using the “pipeline” function. The argument “summarization” specifies that the created pipeline will be used for summarizing texts. This pipeline will allow for inputting a text and generating a summarized version of the text as its output. As we haven’t specified any model, by default it uses <strong>distilbart-cnn-12–6 model</strong>. This model uses <strong>Abstractive summarization technique.</strong></p><p>Next pass selected article into the summarizer object</p><pre>summarizer(doc.iloc[0].split(&#39;\n&#39;,1)[1])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28AbHxZiDtOf8NpmS_78IA.png" /><figcaption>summarized text</figcaption></figure><p>The code applies the summarization pipeline “summarizer” to the first element of the “doc”.</p><p>The first element is extracted from the dataframe using the iloc method, which is used for indexing and selecting data from a dataframe based on its position.</p><p>The extracted first element is then split into two parts using the split method, where the first argument &#39;\n&#39; specifies the separator, and the second argument 1 specifies the maximum number of splits. The resulting list only contains two parts, and the second part is selected using indexing [1]. Here we are doing this to ignore title of the article.</p><p>Finally, the summarized text is generated by passing the selected text as an argument to the summarizer pipeline.</p><p>Let’s try on another article. This time we will select entertainment article.</p><pre>doc = df[df.labels == &#39;entertainment&#39;][&#39;text&#39;].sample(random_state=50)</pre><p>Print selected article</p><pre>doc = df[df.labels == &#39;entertainment&#39;][&#39;text&#39;].sample(random_state=50)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzvTFM2qeSB9aKAvAPAAdw.png" /></figure><p>Summarize selected article</p><pre>summarizer(doc.iloc[0].split(&#39;\n&#39;,1)[1])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8AERJnCwfWdvoP0-kNh3pQ.png" /></figure><p>This is the summarized texts of 2 lines from given article of 14 lines. This summarization using transformers effectively captures the essence of the original article, providing a comprehensive understanding of its main points and ideas. The use of advanced language processing technology has enabled the creation of a succinct summary that accurately represents the content of the original article. This approach to summarization is a testament to the effectiveness of transformers in transforming complex information into easily digestible formats.</p><p>Next I plan to write additional blogs on Transformers where I will demonstrate how to fine-tune transformer models on custom datasets, as well as how to create transformer models from scratch and train them on custom datasets.</p><p>If you enjoyed this post, be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><ul><li><a href="https://medium.com/@lokaregns/effortless-sentiment-analysis-with-hugging-face-transformers-a-beginners-guide-359b0c8a1787">Effortless Sentiment Analysis with Hugging Face Transformers: A Beginner’s Guide</a></li><li><a href="https://medium.com/@lokaregns/text-generation-with-hugging-face-transformers-a-beginners-guide-6b0b4b957379">Text Generation with Hugging Face Transformers: A Beginner’s Guide</a></li><li><a href="https://medium.com/@lokaregns/masked-language-modeling-with-hugging-face-transformers-a-beginners-guide-b36af74a7560">Masked Language Modeling with Hugging Face Transformers: A Beginner’s Guide</a></li><li><a href="https://medium.com/@lokaregns/named-entity-recognition-with-hugging-face-transformers-a-beginners-guide-e1ac6085fb3c">Named Entity Recognition with Hugging Face Transformers: A Beginner’s Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e6c319bb5ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Named Entity Recognition with Hugging Face Transformers: A Beginner’s Guide]]></title>
            <link>https://medium.com/@lokaregns/named-entity-recognition-with-hugging-face-transformers-a-beginners-guide-e1ac6085fb3c?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1ac6085fb3c</guid>
            <category><![CDATA[ner]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Tue, 07 Feb 2023 05:07:39 GMT</pubDate>
            <atom:updated>2023-02-07T05:07:39.203Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ya2liJfeYMrZH5zf0bB8yg.png" /></figure><h3>What We Will Cover in This Blog</h3><ol><li>Review of what Named Entity Recognition is and where we use it.</li><li>How to recognize named entities with Hugging face Transformers (just a few lines of code)</li></ol><p>The main focus of this blog, using a very high level interface for transformers which is the Hugging face pipeline. Using this interface you will see that we can recognize named entities from given text with just 1 or 2 lines of code.</p><h3>What is Named Entity Recognition</h3><p>Named Entity Recognition (NER) is a subtask of Natural Language Processing (NLP) that identifies and classifies named entities in a text into predefined categories such as person names, organizations and locations. The goal of NER is to extract structured information from unstructured text data and turn it into a machine-readable format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*Qh2okoGXjA_UFvd13gtJ-Q.png" /></figure><h3>How named entities recognized using Transformers</h3><p>Named Entity Recognition (NER) in transformers is typically achieved using a multi-stage process that involves the following steps:</p><ol><li>Text pre-processing: Clean and pre-process the text to remove noise, such as HTML tags, special characters, and punctuation marks.</li><li>Tokenization: Break the text into smaller units, called tokens, such as words, phrases, or sentences.</li><li>Embedding: Represent each token as a dense vector, also known as an embedding, using pre-trained transformer-based models such as BERT or RoBERTa. These embeddings capture the context and semantic information of each token in the text.</li><li>Named entity recognition: Use the information captured in the token embeddings to identify named entities and categorize them into predefined categories. This is typically done by training a classifier, such as a linear layer or a CRF layer, on top of the token embeddings.</li><li>Validation and correction: Validate and correct the named entity recognition results using additional techniques, such as entity linking and coreference resolution, to ensure high accuracy and reliability.</li></ol><p>During the named entity recognition stage, the token embeddings are passed through the classifier, which uses them to make predictions about the named entities in the text. The classifier can be trained on annotated training data, which includes labeled named entities, to learn to recognize named entities based on the token embeddings and the context they provide.</p><p>Transformers-based NER models have shown to achieve state-of-the-art performance on a variety of NER benchmarks and have been widely adopted in many NLP applications due to their ability to capture the context and relationships between tokens in a text effectively.</p><h3>Use of Named Entity Recognition</h3><p>Named Entity Recognition (NER) has a wide range of applications in various fields and industries, including:</p><ol><li>Information extraction: NER can be used to extract structured information from unstructured text data, such as news articles, scientific papers, or social media posts, to create machine-readable knowledge graphs or databases.</li><li>Question answering: NER can be used in question answering systems to identify named entities in a question and use them to retrieve relevant information from a database or a knowledge base.</li><li>Sentiment analysis: NER can be used in sentiment analysis to identify named entities, such as products or brands, in a text and determine the sentiment expressed towards them.</li><li>Event extraction: NER can be used to extract information about events, such as date, time, location, and participants, from a text.</li><li>Financial analysis: NER can be used in financial analysis to extract information about companies, stock prices, monetary values, and financial ratios from news articles, financial reports, and other sources.</li><li>Chatbots and customer service: NER can be used in chatbots and customer service to extract information about customer requests and provide relevant information or assistance.</li></ol><p>These are just a few examples of the many applications of NER. The ability to extract structured information from unstructured text data can provide valuable insights and support decision-making in many industries and fields.</p><h3>Why use Transformers for Named Entity Recognition</h3><p>Transformers are used for Named Entity Recognition (NER) due to the following reasons:</p><ol><li>Contextual information: Transformers are capable of capturing the context and relationships between words in a sentence, which is essential for accurate NER. By considering the context, transformers can disambiguate words that have multiple meanings and identify named entities more accurately.</li><li>Pre-training: Transformers are pre-trained on large corpus of text data, which provides them with a vast knowledge of the language and the ability to generate high-quality token representations that can be fine-tuned for specific NER tasks.</li><li>Transfer learning: Transformers can be fine-tuned for NER using a small annotated dataset, which reduces the amount of labeled data required to train NER models and makes it possible to adapt to new domains and languages easily.</li><li>High accuracy: Transformers have shown to achieve state-of-the-art performance on various NER benchmarks and have been widely adopted in many NLP applications.</li><li>Multi-lingual support: Transformers can be trained on multilingual text data, which makes them well-suited for NER tasks that require multilingual support, such as cross-lingual NER or multilingual NER.</li></ol><p>In summary, transformers provide a powerful and flexible framework for NER, making it possible to extract structured information from unstructured text data effectively and efficiently.</p><h3>Enough theory!!! Let’s code…</h3><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including NER.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects.</p><pre>from transformers import pipeline<br><br># create pipeline for NER<br>ner = pipeline(&#39;ner&#39;, aggregation_strategy = &#39;simple&#39;)</pre><p>The code above is using the pipeline function from the transformers library to create a Named Entity Recognition (NER) pipeline. The pipeline is a pre-configured, ready-to-use model for performing NER on text data.</p><p>The pipeline function takes two arguments:</p><ol><li>&#39;ner&#39;: The type of task for the pipeline, in this case Named Entity Recognition.</li><li>aggregation_strategy=&#39;simple&#39;: The argument to specify the aggregation strategy for the pipeline. The aggregation strategy defines how the pipeline will combine the results of multiple models when multiple models are used in the pipeline. The &#39;simple&#39; strategy is a simple aggregation method that concatenates the output of all models in the pipeline.</li></ol><p>The ner variable in the code will store the NER pipeline, which can be used to perform NER on text data by calling the predict method on the pipeline and passing it the text to process.</p><p>The pipeline function uses the latest state-of-the-art models and provides a convenient and efficient way to perform NER without having to write complex code. It is especially useful for quickly testing and evaluating NER models and for using NER in production applications.</p><pre>ner(&quot;Hi, my name is Ganesh Lokare. I am from Pune. I want to work with Google.&quot;)</pre><p>The code above is using the ner pipeline created earlier to perform Named Entity Recognition (NER) on a sample sentence.</p><p>The ner function takes a string argument, which is the text to process, in this case &quot;Hi, my name is Ganesh Lokare. I am from Pune. I want to work with Google.&quot;.</p><p>The NER pipeline will analyze the text and identify named entities in the sentence, such as person names, location names, and organization names. The output will be a list of dictionaries, where each dictionary represents a named entity and contains information such as the entity text, entity type, and entity start and end position in the text.</p><p>For example, the output for the sample sentence look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*hQzE1CJIE8wX4N0ctYmVFg.png" /></figure><p>This information can be used for various purposes, such as information extraction, sentiment analysis, or event extraction. The NER pipeline provides a convenient and efficient way to perform NER on text data and is especially useful for quickly testing and evaluating NER models and for using NER in production applications.</p><p>Next you can use it directly for your dataset.</p><p>In this blog , we covered the basics of Named Entity Recognition (NER), its importance in various applications and how to use pre-trained NER model from Hugging Face library. In next part, we will delve deeper into NER and explore more advanced topics.</p><p>First, we will discuss how to fine-tune NER models on custom datasets. This is crucial for those who have specific requirements that cannot be fulfilled by existing pre-trained models. We will go through the process of how to prepare your own dataset, fine-tune existing models, and evaluate their performance.</p><p>Next, we will move on to building NER models from scratch. We will discuss the various algorithms and models that can be used to perform NER and how to implement them. You will learn how to train, validate, and test NER models and how to optimize their performance.</p><p>If you enjoyed this post, be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><ul><li><a href="https://medium.com/@lokaregns/effortless-sentiment-analysis-with-hugging-face-transformers-a-beginners-guide-359b0c8a1787">Effortless Sentiment Analysis with Hugging Face Transformers: A Beginner’s Guide</a></li><li><a href="https://medium.com/@lokaregns/text-generation-with-hugging-face-transformers-a-beginners-guide-6b0b4b957379">Text Generation with Hugging Face Transformers: A Beginner’s Guide</a></li><li><a href="https://medium.com/@lokaregns/masked-language-modeling-with-hugging-face-transformers-a-beginners-guide-b36af74a7560">Masked Language Modeling with Hugging Face Transformers: A Beginner’s Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1ac6085fb3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Masked Language Modeling with Hugging Face Transformers: A Beginner’s Guide]]></title>
            <link>https://medium.com/@lokaregns/masked-language-modeling-with-hugging-face-transformers-a-beginners-guide-b36af74a7560?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b36af74a7560</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[masked-language-model]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Mon, 06 Feb 2023 10:12:47 GMT</pubDate>
            <atom:updated>2023-02-06T10:12:47.831Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*78ewPwqwDNo0irt1yuyKjg.png" /></figure><h3>What We Will Cover in This Blog</h3><ol><li>Review of what Masked Language Modeling is and where we use it.</li><li>How to use Masked Language Modeling with Hugging face Transformers (just a few lines of code)</li></ol><p>The main focus of this blog, using a very high level interface for transformers which is the Hugging face pipeline. Using this interface you will see that we can generate new words for masked words from given text with just 1 or 2 lines of code.</p><h3>What is Masked Language Modeling</h3><p>Masked Language Modeling (MLM) is a popular deep learning technique used in Natural Language Processing (NLP) tasks, particularly in the training of Transformer models such as BERT, GPT-2, and RoBERTa.</p><p>In MLM, a portion of the input text is “masked” or randomly replaced with a special token (usually [MASK]) and the model is trained to predict the original token based on the context surrounding it. The idea behind this is to train the model to understand the context of words and their relationships with other words in a sentence.</p><p>MLM is a self-supervised learning technique, meaning that the model learns to generate text without the need for explicit annotations or labels, but instead using the input text itself as supervision. This makes it a versatile and powerful tool for a wide range of NLP tasks, including text classification, question answering, and text generation.</p><h3>How Masked Language Modeling works?</h3><p>Masked Language Modeling (MLM) is a pre-training technique for deep learning models in NLP. It works by randomly masking a portion of the input tokens in a sentence and asking the model to predict the masked tokens. The model is trained on large amounts of text data, so that it can learn to understand the context of words and predict masked tokens based on their surrounding context.</p><p><strong>Here is a simple example: Given a sentence, “The cat [MASK] on the roof”, the model would predict the word “sat” as the masked token.</strong></p><p>During the training process, the model is updated based on the difference between its predictions and the actual words in the sentence. This pre-training stage helps the model to learn useful contextual representations of words, which can then be fine-tuned for specific NLP tasks. The idea behind MLM is to leverage the large amounts of text data available to learn a general-purpose language model that can be applied to different NLP problems.</p><h3>Use of Masked Language Modeling</h3><p>Masked Language Modeling (MLM) has several applications in the field of Natural Language Processing (NLP). Some of the most common applications include:</p><ol><li>Question answering: MLM can be used to pre-train the model for question answering tasks, where the model must identify the answer to a question given a context.</li><li>Named entity recognition: MLM can be used to pre-train the model for named entity recognition tasks, where the model must identify and categorize named entities in a text, such as people, organizations, and locations.</li><li>Text generation: MLM can be used to pre-train the model for text generation tasks, where the model must generate text based on a prompt or seed text.</li><li>Machine translation: MLM can be used to pre-train the model for machine translation tasks, where the model must translate text from one language to another.</li><li>Article spinning: Article spinning is a technique used to create new variations of existing articles by changing words, phrases, or sentences to create new content that is similar in meaning to the original. This technique is often used for SEO purposes, with the goal of creating multiple versions of an article that can be used to target different keywords or phrases.</li></ol><p>Overall, MLM has proven to be a powerful technique for improving the performance of NLP models on a wide range of tasks. By pre-training the model on large amounts of text data, MLM can help the model to learn useful contextual representations of words, which can then be fine-tuned for specific NLP tasks.</p><h3>Why use Transformers for Masked Language Modeling</h3><p>Transformers are well-suited for Masked Language Modeling (MLM) pre-training because they have the ability to effectively capture the context of words in a sentence. The Transformer architecture, introduced in 2017, is based on self-attention mechanisms that allow the model to weight the importance of different tokens in a sentence. This makes it possible for the Transformer to capture long-range dependencies between words, which is crucial for understanding the context of a sentence.</p><p>When used for MLM pre-training, the Transformer can be trained to predict the masked tokens in a sentence based on the surrounding context. This way, the model can learn to understand the relationships between words in a sentence, which can be useful for a wide range of NLP tasks.</p><p>In addition to its ability to capture context, Transformers are also highly parallelizable, making it possible to train large models on vast amounts of text data. This is important for MLM pre-training, as the model needs to be trained on a large corpus of text in order to learn useful representations of language.</p><p>Overall, the combination of the Transformer architecture and MLM pre-training has proven to be very effective for NLP tasks, and has become a standard pre-training technique in the field.</p><h3>Enough theory!!! Let’s code…</h3><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including sentiment analysis.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects</p><pre>from transformers import pipeline<br># create pieline for MLM<br>mlm = pipeline(&#39;fill-mask&#39;)</pre><p>The line “from transformers import pipeline” imports the pipeline module from the Transformers library, which provides an easy-to-use interface for common NLP tasks, including MLM. As we have not explicitly supplied any model, by default it will select <strong>distilroberta-base</strong>.</p><p>The line “mlm = pipeline(‘fill-mask’)” creates an instance of the MLM pipeline, which can be used to generate predictions for masked tokens in a sentence. The “fill-mask” argument specifies that the pipeline should be created for the task of MLM.</p><p>Once the MLM pipeline has been created, you can use it to generate predictions for masked tokens in a sentence.</p><pre>mlm(&quot;The cat &lt;mask&gt; on the roof&quot;)</pre><p>The above code is calling the instance of the Masked Language Modeling (MLM) pipeline created in the previous code snippet. The argument to the function, “The cat &lt;mask&gt; on the roof”, is a sentence with a masked token, indicated by the “&lt;mask&gt;” placeholder.</p><p>When this code is executed, the MLM pipeline will generate predictions for the masked token based on the surrounding context of the sentence. The pipeline will use the pre-trained language model to predict the most likely word to fill the mask based on the context of the sentence.</p><p>The function will return a list of predicted tokens and their probabilities, sorted by the model’s confidence in each prediction. The output will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*MOG4tTq8dtYRQ_li48ZlCA.png" /></figure><p>In this example, the model predicts that the most likely word to fill the mask is “sleeping” with a high score of 0.2022217. The other predictions, such as “sleeps”, “perched”, “sits”, and “sitting” have lower scores, indicating lower confidence from the model.</p><h3>Let’s Try it on a custom dataset</h3><pre>!wget -nc https://www.dropbox.com/s/7hb8bwbtjmxovlc/bbc_text_cls.csv?dl=0</pre><p>The command will download the file from the URL .</p><pre># import required libraries<br>import textwrap<br>import numpy as np<br>import pandas as pd<br>from pprint import pprint</pre><p>load data using pandas library</p><pre>df = pd.read_csv(&#39;bbc_text_cls.csv?dl=0&#39;)</pre><p>print first few rows of data</p><pre>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*wKnc1Djvo6bHGHNb7HRreg.png" /></figure><p>check the number of unique labels</p><pre>labels = set(df[&#39;labels&#39;])<br>labels</pre><p>{ ‘business’, ‘entertainment’, ‘politics’, ‘sport’, ‘ tech’}</p><p>As you can see we have 5 unique labels, for our task we will keep data which has ‘business’ label.</p><pre>texts = df[df[&#39;labels&#39;] == &#39;business&#39;][&#39;text&#39;]<br>texts.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*zrs_5Vwu7GMRlDVwd9kKGg.png" /></figure><p>Select random article</p><pre>i = np.random.choice(texts.shape[0])<br>doc = texts.iloc[i]</pre><p>The code uses numpy&#39;s random.choice method to select a random index from the number of rows in the texts pandas DataFrame. The resulting index is stored in the variable i. Then, the code selects the corresponding row in the texts DataFrame using .iloc[i] and stores it in the variable doc for further use.</p><p>Print the chosen article which is stored in “doc” variable</p><pre>print(textwrap.fill(doc, replace_whitespace = False, fix_sentence_endings = True))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WZW7AM0O3U20driXm14Yug.png" /></figure><p>Next step is test our model by passing title of chosen article.</p><p>Article title is “BBC poll indicates economic gloom”, we will mask the last word and check that will model predict correct or not.</p><pre>mlm(&quot;Trade gap narrows as exports &lt;mask&gt;&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*qiDLbsLQwodgXEPfMS7WOA.png" /><figcaption>Predictions</figcaption></figure><p>As you can see this returns multiple substitutions, all suggestions are equivalent to main word.</p><p>Next we will try on first line of the article</p><pre>text = &quot;Citizens in a majority of nations surveyed in a &lt;mask&gt; World Service poll believe the&quot; + \<br>        &quot;world economy is worsening&quot;<br>mlm(text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OC1ViPS9zgKXAd7b89e-Uw.png" /><figcaption>Predictions</figcaption></figure><p>Nice!! Model has predicted correctly ‘BBC’ word with 64.49% and other suggestions also related to main word.</p><p>Next you can try with other articles or other sentences, as well as you can write a function that automatically masks and replaces words in a whole document. You might choose which words to replace based on some statistics, e.g. TF-IDF.</p><p>Next I plan to write additional blogs on Transformers where I will demonstrate how to fine-tune transformer models on custom datasets, as well as how to create transformer models from scratch and train them on custom datasets.</p><p>If you enjoyed this post, be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><ul><li><a href="https://medium.com/@lokaregns/text-generation-with-hugging-face-transformers-a-beginners-guide-6b0b4b957379">Text Generation with Hugging Face Transformers: A Beginner’s Guide</a></li><li><a href="https://medium.com/@lokaregns/effortless-sentiment-analysis-with-hugging-face-transformers-a-beginners-guide-359b0c8a1787">Effortless Sentiment Analysis with Hugging Face Transformers: A Beginner’s Guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b36af74a7560" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Generation with Hugging Face Transformers: A Beginner’s Guide]]></title>
            <link>https://medium.com/@lokaregns/text-generation-with-hugging-face-transformers-a-beginners-guide-6b0b4b957379?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b0b4b957379</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[text-generation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Sun, 05 Feb 2023 11:32:28 GMT</pubDate>
            <atom:updated>2023-02-05T11:32:28.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gzG4q5GBnXVkD6HyolCq_A.jpeg" /></figure><h3>What is Text Generation</h3><p>Text generation in deep learning refers to the process of automatically generating natural language text using artificial neural networks. This is typically achieved through the use of language models, which are trained on large amounts of text data to predict the next word in a sequence based on the previous words. The generated text can be used for various applications, such as content creation, language translation, and chatbots.</p><h3>What We Will Cover in This Blog</h3><ol><li>Review of what text generation is and where we use it.</li><li>How to generate texts with Hugging face Transformers (just a few lines of code)</li></ol><p>The main focus of this blog, using a very high level interface for transformers which is the Hugging face pipeline. Using this interface you will see that we can generate texts from given prompts with just 1 or 2 lines of code.</p><h3>How texts are generated using deep learning models</h3><p>Texts are generated using deep learning models through a process called language modeling. The goal of language modeling is to train a model to predict the probability distribution of the next word in a sequence given the previous words. This is usually accomplished by feeding the model a large corpus of text data and training it to predict the next word in the sequence based on the context provided by the previous words. The model is then fine-tuned through a process called backpropagation, which adjusts the weights of the model to minimize the prediction error.</p><p>Once the model is trained, text generation is performed by starting with a seed sequence of words, then repeatedly sampling the next word from the predicted probability distribution, and appending it to the sequence. The generated text can then be post-processed to fine-tune its quality, such as by adjusting its grammar, spelling, or style.</p><p>There are several deep learning models that can be used for text generation, including Recurrent Neural Networks (RNNs), Transformers, and Generative Adversarial Networks (GANs). The choice of model depends on the specific requirements of the task and the resources available.</p><h3>Use of Text Generation</h3><p>Text generation using deep learning models has several applications, including:</p><ol><li>Content creation: Text generation models can be used to generate articles, summaries, headlines, and other types of content, allowing organizations to increase the efficiency and consistency of their content production.</li><li>Chatbots: Text generation models can be used to create conversational agents that can respond to questions and engage in natural language conversations with users.</li><li>Language translation: Text generation models can be used to translate text from one language to another, making it possible to reach a global audience.</li><li>Summarization: Text generation models can be used to summarize long articles or documents, making it easier for users to quickly understand the key points.</li><li>Creative writing: Text generation models can be used to generate fiction, poetry, and other types of creative writing, allowing artists to explore new forms of expression.</li><li>Question answering: Text generation models can be used to generate answers to questions, making it possible to automate the process of answering frequently asked questions.</li></ol><p>These are just a few examples of the many possible uses of text generation using deep learning models. The versatility of these models makes them a valuable tool for a wide range of applications in the fields of natural language processing and artificial intelligence.</p><h3>Why use Transformers for Text Generation</h3><p>Transformers are widely used for text generation tasks due to several key advantages:</p><ol><li>Attention mechanism: The attention mechanism in Transformers allows the model to focus on different parts of the input sequence and weigh their contribution to the output prediction, making it possible to handle longer sequences of text and capture long-range dependencies.</li><li>Parallel computation: Transformers can be trained in parallel, which speeds up the training process and makes it possible to train on large amounts of data.</li><li>Transfer learning: Transformers have been pre-trained on massive amounts of text data, allowing them to be fine-tuned for specific tasks with relatively small amounts of additional data. This makes it possible to leverage existing knowledge and improve performance.</li><li>High accuracy: Transformers have achieved state-of-the-art performance on a variety of text generation tasks, including language translation, text summarization, and text completion.</li><li>Contextualization: Transformers allow the model to contextualize the input sequence, which is critical for tasks such as text generation where the model must generate text that is coherent and contextually appropriate.</li></ol><p>In summary, the attention mechanism, parallel computation, transfer learning capabilities, high accuracy, and contextualization of Transformers make them a powerful tool for text generation tasks.</p><h3>Enough theory!!! Let’s code…</h3><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including sentiment analysis.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects</p><pre>from transformers import pipeline<br><br># create pieline for generating text<br>gen = pipeline(&#39;text-generation&#39;) # uses GPT-2</pre><p>The code above is using the pipeline function from the transformers library to create a text generation pipeline. The text-generation argument specifies that the pipeline should be created for text generation.</p><p>The pipeline function creates a high-level interface for working with pre-trained models from the Hugging Face Transformers library. It allows you to perform various NLP tasks, including text generation, with a few lines of code, without having to write the underlying model architecture.</p><p>Note that the pipeline function will automatically download the pre-trained model specified in the argument, if it has not been previously downloaded to your system. By default it downloads GPT-2 model.</p><h3>Let’s Generate Texts</h3><pre>gen(&quot;I am using transformers text-generation pipeline from Hugging Face library to generate&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2C5YUPqsiUehJCULCAZuQ.png" /><figcaption>Generated output</figcaption></figure><p>We can use other arguments also</p><pre>prompt = &quot;I am using transformers text-generation pipeline from Hugging Face library to generate&quot;<br>pprint(gen(prompt,num_return_sequences = 3, max_length = 30))</pre><p>The num_return_sequences argument is set to 3, meaning the code will generate 3 separate sequences of text. The max_length argument is set to 30, meaning the generated text will be limited to a maximum of 30 tokens (i.e., words or punctuation). The generated text will be a continuation of the given prompt based on the model&#39;s training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*PYYs51PKDOgND4SxF-TNrw.png" /><figcaption>Generated output</figcaption></figure><h3>Next we will try to generate poems using Robert Frost poems data</h3><pre># download data from provided link<br>!wget https://www.dropbox.com/s/0xylltp1vxf31vj/robert_frost.txt?dl=0</pre><p>The command will download the file from the URL .</p><pre># import required libraries<br>import textwrap<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from pprint import pprint</pre><p>Display the content from file</p><pre>!cat /content/robert_frost.txt?dl=0</pre><p>The code you provided is a command in the Unix/Linux shell. The command cat is used to concatenate and display the contents of one or more text files. In this case, the command is attempting to display the contents of a file named &quot;robert_frost.txt&quot;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*x3izrkZLGwzSZgQmPg6RRw.png" /><figcaption>file content</figcaption></figure><p>Note that some lines are empty, which we need to remove.</p><pre>lines = [line.rstrip() for line in open(&quot;robert_frost.txt?dl=0&quot;)]<br>lines = [line for line in lines if len(line) &gt; 0]</pre><ol><li>code creates a list called lines by using a list comprehension to iterate over the lines of a file named &quot;robert_frost.txt?dl=0&quot;. The rstrip() method is used to remove any trailing whitespace from each line.</li><li>Another list called lines by using a list comprehension to iterate over the lines of the previous lines list. The line is included in the new list only if its length is greater than 0. This effectively removes any empty lines from the lines list.</li></ol><p>Let’s print the first line</p><pre>pprint(lines[0])</pre><p><em>‘Two roads diverged in a yellow wood,’</em></p><p>Generate texts using first line</p><pre>pprint(gen(lines[0],num_return_sequences = 3, max_length = 20))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*HmfCxolsuBdoRoYb0KyTWw.png" /></figure><p>Basically these texts are not looks like poetry which makes sense because this model was trained on text from the internet rather than text from poetry. So the model is not a model of poetry but just generic text.</p><p>Note that this one might expect from language model, the results are grammatically correct but it obviously doesn&#39;t make a lot of sense.</p><p>You can try out generating texts using other data also.</p><p>With the next example I want to show you more typical way that GPT-2 is used. We have already seen that it not too great for a poetry but this is simply because it wasn&#39;t trained on poetry. We can use GPT more efficient for which it has build. The more common way to use GPT is to pass a prompt about some subject and then to observe that is it able to generate very coherent text with very long range dependencies.</p><p>Basically we can generate a whole document with providing some starting texts.</p><pre>def wrap(x):<br>  return textwrap.fill(x, replace_whitespace = False, fix_sentence_endings = True)</pre><p>above function returns a copy of x where line breaks are inserted such that each line in the returned string is no longer than a specified number of characters (the default is 70).</p><p>The parameters used in the function call are:</p><ul><li>replace_whitespace - a boolean that specifies whether to replace consecutive whitespaces in x with a single space before wrapping. In this case, it&#39;s set to False, meaning that whitespaces will be preserved.</li><li>fix_sentence_endings - a boolean that specifies whether to ensure that wrapped lines end with a complete sentence. In this case, it&#39;s set to True, meaning that the function will try to wrap sentences at a period (.) if possible.</li></ul><p>Create an article of 300 words on Deep Learning by providing a line of texts</p><pre>prompt  = &quot;Current state of deep learning and its potential impact on various industries&quot;<br>out = gen(prompt, max_length = 300)<br>print(wrap(out[0][&#39;generated_text&#39;]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*1rOg1ufsfZgs4bR0UNdpMA.png" /></figure><p>This is the generated output from given prompt. The text doesn&#39;t perfectly make sense some times but grammatically everything seems to be perfect. It’s also far better than anything we have done in the past. Which is why this was so exciting when it first came out. It’s also pretty impressive that the model understand the structure of the abstracts. Also note that the whole thing is coherent from start to end, without conducting any training, showcasing the strength of Transformers. This was a high-level overview of the use of Transformers for Text Generation.</p><p>Next I plan to write additional blogs on Transformers where I will demonstrate how to fine-tune transformer models on custom datasets, as well as how to create transformer models from scratch and train them on custom datasets.</p><p>If you enjoyed this post, be sure to check out some of my other blog posts for more insights and information. I’m constantly exploring new topics and writing about my findings, so there’s always something new and interesting to discover.</p><p><a href="https://medium.com/@lokaregns/effortless-sentiment-analysis-with-hugging-face-transformers-a-beginners-guide-359b0c8a1787">Effortless Sentiment Analysis with Hugging Face Transformers: A Beginner’s Guide</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b0b4b957379" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effortless Sentiment Analysis with Hugging Face Transformers: A Beginner’s Guide]]></title>
            <link>https://medium.com/@lokaregns/effortless-sentiment-analysis-with-hugging-face-transformers-a-beginners-guide-359b0c8a1787?source=rss-52da288e98e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/359b0c8a1787</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Ganesh Lokare]]></dc:creator>
            <pubDate>Sat, 04 Feb 2023 14:49:06 GMT</pubDate>
            <atom:updated>2023-02-04T14:49:06.674Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*78ewPwqwDNo0irt1yuyKjg.png" /></figure><h3>Sentiment Analysis</h3><p>Sentiment analysis is the process of determining the emotion or opinion expressed in a piece of text, such as a tweet or a review. It is a subfield of natural language processing (NLP) that involves using machine learning and NLP techniques to classify the sentiment of a piece of text as positive, negative, or neutral. The goal of sentiment analysis is to extract insights from large amounts of unstructured text data and understand the overall sentiment of a population towards a particular topic.</p><h3>What We Will Cover in This Blog</h3><ol><li>Review of what sentiment analysis is.</li><li>How to perform sentiment analysis with Hugging face Transformers (just a few lines of code)</li></ol><p>The main focus of this blog, using a very high level interface for transformers which is the Hugging face pipeline. Using this interface you will see that we can perform sentiment analysis with just 1 or 2 lines of code.</p><h3>Sentiment Analysis as a classification task</h3><ul><li>Binary classification (positive, negative)</li><li>Multiclass: positive, negative or neutral</li><li>Fine-grain: very positive, positive, neutral, negative, very negative</li><li>Dependent on the dataset you’re using or how you made your dataset</li><li>If we use prebuild classifier (e.g. Hugging Face) we are stuck with whatever the model was trained to do (in this case binary classification)</li></ul><h3>Use of Sentiment Analysis</h3><ol><li>Improve customer experience: Sentiment analysis can be used to understand customer opinions and feedback on products and services, allowing companies to improve the customer experience and build stronger customer relationships.</li><li>Market research: Sentiment analysis can be used to monitor public sentiment towards a particular topic or brand, providing valuable insights into market trends and customer preferences.</li><li>Improve product design: Sentiment analysis can be used to understand customer preferences and opinions on product features, allowing companies to design products that better meet customer needs.</li><li>Improve social media monitoring: Sentiment analysis can be used to monitor social media conversations and understand the tone and sentiment behind them, providing valuable insights into public perception and sentiment.</li><li>Improve decision making: Sentiment analysis can provide a broad overview of the sentiment of a population towards a particular topic or brand, allowing decision-makers to make informed decisions based on data-driven insights.</li></ol><h3>History of Sentiment Analysis</h3><p>Traditionally, sentiment analysis was performed using shallow learning techniques such as Support Vector Machines (SVMs) and Naive Bayes Classifiers. However, these techniques had several limitations, such as the inability to capture context and the need for a large annotated dataset for training.</p><h3>Why use Transformers for Sentiment Analysis</h3><p>Enter Transformers, a breakthrough in NLP that has revolutionized the field of sentiment analysis. Transformers are deep learning models that were introduced in 2017 by Vaswani et al. in the paper “Attention is All You Need”. They are based on the Transformer architecture and use self-attention mechanisms to process the input sequence, allowing the model to capture the context and dependencies between words.</p><ol><li>Improve context representation: Transformers use self-attention mechanisms to process the input sequence, allowing the model to capture the context and dependencies between words, leading to improved representation of the input text.</li><li>Handling long sequences: Transformers are able to handle long sequences of text, making them well-suited for sentiment analysis where the context and dependencies between words play an important role.</li><li>Large number of parameters: Transformers have a large number of parameters, allowing them to learn complex representations of the input text, resulting in improved performance compared to shallow learning methods.</li><li>Pre-training: Many Transformer-based models, such as BERT, are pre-trained on large annotated datasets, providing a strong base for fine-tuning on specific NLP tasks, including sentiment analysis.</li><li>State-of-the-art performance: Transformer-based models, such as BERT, have achieved state-of-the-art results on a variety of NLP tasks, including sentiment analysis, making them a popular choice for the task.</li></ol><h3>Enough theory!!! Let’s code…</h3><pre># install transformers<br>!pip install transformers</pre><p>The command pip install transformers is used to install the transformers package, which provides access to state-of-the-art Transformer-based models for NLP tasks, including sentiment analysis.</p><p>Once the transformers package is installed, you can import and use the Transformer-based models in your own projects</p><pre>from transformers import pipeline<br><br># create pipeline for sentiment analysis<br>classification = pipeline(&#39;sentiment-analysis&#39;)</pre><p>The above code is a demonstration of how to use the transformers package for sentiment analysis in Python.</p><ol><li>Importing the pipeline class from the transformers package: The first line from transformers import pipeline imports the pipeline class from the transformers package. The pipeline class provides a simple and convenient interface to perform NLP tasks, including sentiment analysis, using pre-trained Transformer-based models.</li><li>Creating a sentiment analysis pipeline: The second line classification = pipeline(&#39;sentiment-analysis&#39;) creates a sentiment analysis pipeline using the pipeline class. The argument &#39;sentiment-analysis&#39; specifies the task to perform, which in this case is sentiment analysis.</li></ol><p>With these two lines of code, you have created a sentiment analysis pipeline that can be used to classify the sentiment of a given text as positive, negative, or neutral. You can now use the classification object to perform sentiment analysis on any text by calling its predict method and passing the text as an argument.</p><h3>Perform Sentiment Analysis on custom text</h3><p>No need to convert input into pyTorch tensor, Numpy array or TensorFlow tensor. In this case we can simply pass raw text.</p><pre>#let&#39;s try on simple movie reviews <br>classification(&quot;I thoroughly enjoyed this movie!&quot;)<br>classification(&quot;I did not understand anything in this movie.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*C1ElVl9lV7V0JhkaAydByg.png" /><figcaption>Results on simple movie review</figcaption></figure><pre># let&#39;s try on difficult movie reviews<br>classification(&#39;&#39;&#39;Although the movie had its flaws and some scenes were slow-paced, <br>                I still found myself captivated by the strong performances and unique storyline. <br>                The cinematography was stunning and added another level of depth to the film. Overall, <br>                I would still recommend it for those looking for a thought-provoking movie experience.&#39;&#39;&#39;)<br><br>classification(&#39;&#39;&#39;At first, I was intrigued by the premise and was excited to see where the story would go. <br>                However, as the movie progressed, I became disappointed by the lackluster execution and <br>                underdeveloped characters. Despite a few moments of promise, the movie ultimately fell <br>                flat for me and failed to deliver on its potential. Not recommended.&#39;&#39;&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*qWE2d3wyJQ5hJJw4xSkoqg.png" /></figure><p>This pipeline is very flexible, we can pass list of reviews at a time and will get multiple outputs.</p><pre>classification([&quot;I thoroughly enjoyed this movie!&quot;,<br>                &quot;I did not understand anything in this movie.&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*0v_WYB6VLRl7jKOnisysUw.png" /><figcaption>Multiple results</figcaption></figure><h3>Sentiment Analysis on a custom dataset</h3><pre># download dataset from given link for sentiment analysis<br>! wget https://www.dropbox.com/s/h5n6nl55bngk8rx/twitter.zip?dl=0<br><br># Unzipping the data<br>!unzip -q &quot;twitter.zip?dl=0&quot;</pre><p>The command is downloading the file twitter_training.csv from the URL ! wget <a href="https://www.dropbox.com/s/h5n6nl55bngk8rx/twitter.zip?dl=0">https://www.dropbox.com/s/h5n6nl55bngk8rx/twitter.zip?dl=0</a> and unzip it. The file is a CSV (Comma-Separated Values) file containing training data for sentiment analysis on Twitter data.</p><pre># import required liabraries<br>from transformers import pipeline<br>import numpy as np<br>import pandas as pd<br>import seaborn as sns<br>import matplotlib.pyplot as plt<br>from sklearn.metrics import roc_auc_score, f1_score, confusion_matrix<br>from sklearn.model_selection import train_test_split</pre><pre># create pipeline for sentiment analysis<br>classification = pipeline(&#39;sentiment-analysis&#39;)<br>type(classification)</pre><p><em>transformers.pipelines.text_classification.TextClassificationPipeline</em></p><p>TextClassificationPipeline is a pre-defined pipeline in the transformers library for text classification tasks. It consists of a sequence of pre-defined processing steps for natural language text data, such as tokenization, converting text into numerical representations (i.e., embeddings), and using these embeddings as input to a classifier. The purpose of this pipeline is to simplify the process of building and training a text classification model, allowing you to quickly train a model with good performance on your specific task.</p><h3>Load twitter data</h3><pre># read data into DataFrame<br>df = pd.read_csv(&quot;twitter_training.csv&quot;)<br># set the columns<br>df.columns = [&#39;id&#39;,&#39;entity&#39;,&#39;sentiment&#39;,&#39;Tweet content&#39;]<br># drop null values<br>df = df.dropna()<br># print first 5 rows<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*Qt3kl-wpBPs6j2dCxEdf2Q.png" /></figure><pre># select only required columns<br>df = df[[&#39;sentiment&#39;,&#39;Tweet content&#39;]]</pre><p>As this is the sentiment analysis, we require only text and label columns. Here they are sentiment and Tweet content.</p><p>Let’s check the number of classes available in dataset and their distribution.</p><pre>df[&#39;sentiment&#39;].hist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Q5jFSeE36TKXHRVmJQBHew.png" /><figcaption>Histogram</figcaption></figure><p>Here we have 4 number of classes in our dataset. As we are using pre-trained sentiment analysis model from Hugging Face which is trained on binary classes e.g. positive and negative. We are filtering positive and negative labeled data only.</p><pre># filter positive and negative sentiments only<br>df = df[(df.sentiment == &#39;Positive&#39;) | (df.sentiment == &#39;Negative&#39;)]</pre><p>Now, we are only keeping positive and negative classes. Let’s check distribution now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*B_JOvygLMYG5eWYbdkJjkg.png" /></figure><p>Now we have only 2 classes and almost balanced distribution of the classes.</p><pre>target_map = { &#39;Positive&#39;: 1, &#39;Negative&#39;: 0}<br>df[&#39;target&#39;] = df[&#39;sentiment&#39;].map(target_map)</pre><p>The first line defines a dictionary target_map that maps the original categorical target variable &#39;sentiment&#39; to a numerical representation, with &#39;Positive&#39; mapped to 1 and &#39;Negative&#39; mapped to 0.</p><p>The second line applies this mapping to the ‘sentiment’ column of the DataFrame df using the map method and saves the result as a new column &#39;target&#39; in the same DataFrame. This can be useful for training machine learning models, which often require numerical input variables.</p><pre>texts = df[&#39;Tweet content&#39;].to_list()</pre><p>The code above is extracting the ‘Tweet content’ column from a Pandas DataFrame df and converting it to a list of text values.</p><p>The line df[&#39;Tweet content&#39;] is selecting the &#39;Tweet content&#39; column from the DataFrame df and returns it as a Pandas Series object. The to_list() method is then used to convert the Series object to a list of text values. The resulting list texts contains all the text values in the &#39;Tweet content&#39; column of the DataFrame.</p><p>This is a common preprocessing step when working with text data in machine learning, as many models expect text inputs to be represented as lists of strings.</p><h3>Predictions</h3><pre>predictions = classification(texts)</pre><p>Here we are directly making predictions using pre-trained transformer. It will take a while.</p><p>printout few predictions</p><pre>predictions[:10]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*4dNJVsqYZSNy1W1nKRkoBg.png" /></figure><p>We got predictions in the form of list of dictionaries with label and score. See score for positive as well as negative classes showing near to 1. By using this score it will be difficult to evaluate performance. We will convert it into probabilities ranging from 0–1 with 0 tends to negative class ad 1 tends to positive class.</p><pre>probs = [d[&#39;score&#39;] if d[&#39;label&#39;].startswith(&#39;P&#39;) else 1 - d[&#39;score&#39;] for d in predictions ]</pre><p>The code above is creating a list of probabilities from a list of dictionaries named “predictions”. Each dictionary in the list “predictions” has two keys: “label” and “score”.</p><p>The code iterates over each dictionary in the “predictions” list and applies the following logic for each dictionary:</p><ol><li>If the value of the “label” key starts with the letter “P”, it takes the value of the “score” key as the probability.</li><li>If the value of the “label” key does not start with the letter “P”, it subtracts the value of the “score” key from 1 and takes that as the probability.</li></ol><p>The result of this logic is a new list “probs” that contains the calculated probabilities.</p><pre># print few probabilities<br>probs[:10]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*X9yZSD7yduiNUiuoLeZc2A.png" /></figure><pre>preds = [1 if d[&#39;label&#39;].startswith(&#39;P&#39;) else 0 for d in predictions]</pre><p>Extracting labels from prediction and creating new list. Here we are performing similar operation as we preformed above.</p><pre># convert into numpy array <br>preds = np.array(preds)</pre><pre>print(&quot;acc:&quot;, np.mean(df[&#39;target&#39;] == preds))</pre><p><strong>acc: 0.7869896772993583</strong></p><p>The code calculates the accuracy of binary predictions stored in the list “preds” by comparing them with the true values stored in the “target” column of the dataframe “df”. The accuracy is calculated as the mean of a boolean array, which is created by element-wise comparison of the “target” and “preds”.</p><p><strong>We got 78.69% accuracy. </strong>We are considering here accuracy as a evaluation matrix because we have almost balanced dataset.</p><h3>Plot Confusion matrix</h3><pre># calculate confusion matrix<br>cm = confusion_matrix(df[&#39;target&#39;], preds)<br># create function for plotting confusion matrix<br>def plot_cm(cm):<br>  classes = [&#39;negative&#39;,&#39;positive&#39;]<br>  df_cm = pd.DataFrame(cm, index=classes, columns=classes)<br>  ax = sns.heatmap(df_cm, annot = True, fmt=&#39;g&#39;)<br>  ax.set_xlabel(&#39;Predicted&#39;)<br>  ax.set_ylabel(&#39;Actual&#39;)<br><br>plot_cm(cm)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*MW8cBNfXAVxGTVJaU6i7kA.png" /><figcaption>Confusion matrix</figcaption></figure><p>We have achieved 78% accuracy without conducting any training, showcasing the strength of Transformers. This was a high-level overview of the use of Transformers for Sentiment Analysis.</p><p>Next I plan to write additional blogs on Transformers where I will demonstrate how to fine-tune transformer models on custom datasets, as well as how to create transformer models from scratch and train them on custom datasets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=359b0c8a1787" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>