<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sonal Prasad on Medium]]></title>
        <description><![CDATA[Stories by Sonal Prasad on Medium]]></description>
        <link>https://medium.com/@sonalprasad.sp?source=rss-255ce29b8ae3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5x8r-2hmK_zTUeD3.</url>
            <title>Stories by Sonal Prasad on Medium</title>
            <link>https://medium.com/@sonalprasad.sp?source=rss-255ce29b8ae3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:12:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sonalprasad.sp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Transformer Models Part 2: Transformer Block & Transformer Architecture]]></title>
            <link>https://medium.com/@sonalprasad.sp/transformer-models-part-2-transformer-block-transformer-architecture-dbc20408aaed?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbc20408aaed</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[encoder-decoder]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 12 May 2026 23:44:31 GMT</pubDate>
            <atom:updated>2026-05-12T23:44:31.097Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Feel free to read the previous article in this series on Self-Attention: </em><a href="https://www.linkedin.com/pulse/transformer-models-part-1-self-attention-multi-head-attention-prasad-vb2lc"><em>https://www.linkedin.com/pulse/transformer-models-part-1-self-attention-multi-head-attention-prasad-vb2lc</em></a></p><h3>What You’ll Learn</h3><ul><li>Basis of Transformers Architecture</li><li>Transformer Block</li><li>3 Transformer Architectures</li></ul><h3>Basis of Transformers Architecture</h3><p>As we noticed in the previous article (link above), the self-attention mechanism is a powerful core feature of the Transformer architecture. To be more precise, self-attention is at the core of a Transformer block. This is important because the attention mechanism makes way for the benefits we get of handling long-term dependencies and running in parallel to decrease computation time.</p><h3>Transformer Block</h3><p>The transformer block consists of the following layers: self-attention, a feed-forward layer, residual connections, and normalizing layers (e.g. layer norm). The diagram below shows the architecture of a transformer block. Note: this is the ‘prenorm’ version where the Layer Norm comes before the attention and feed-forward layers, as opposed to after it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIdPpH9Mn8vaX-EtOGdpMA.png" /><figcaption>The above image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>“In the residual stream viewpoint, we consider the processing of an individual token i through the transformer block as a single stream of d-dimensional representations for token position i. This residual stream starts with the original input vector, and the various components read their input from the residual stream and add their output back into the stream.” [1] For example, the input vector gets passed through a layer norm and and an attention layer, from which we take the result and add it back to the residual stream.</p><p>The feed-forward layer is a fully connected network with 2 layers. There is 1 hidden layer and 2 matrices.</p><p>Layer norm involves normalization that can improve the performance when training a deep neural network. The Layer norm is applied to an embedding vector of a single token.</p><p>Below is the step-by-step computation of the Transformer block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*9_BSSlMbljKCMfsvf1I29w.png" /><figcaption>The above image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>The multi-head attention is the only layer that factors in the neighboring tokens in the context. After stacking multiple Transformer blocks, we must add an extra Layer norm component at the end of the last Transformer block.</p><h3>The Input</h3><p>The Transformer block takes in an input that is composed of an embedding for each word in the context. The Transformer computes an input token embedding and an input positional embedding. The token embedding represents the input embedding. “As we pass vectors up through the transformer layers in the residual stream, this embedding representation will change and grow, incorporating context and playing a different role depending on the kind of language model we are building.” [1] The token embeddings are combined with the positional embedding which contains the position of an input. The embedding is of dimensions [Nxd], where N is the number of tokens, and d is the length of position embeddings. The positional embeddings can be based on absolute position or relative position, where the latter is more complex.</p><h3>Transformer Architecture</h3><p>Now we expand from the Transformer block to the full Transformer architecture. There are 3 variants of the Transformers models: encoder-decoder, encoder-only, and decoder-only. You need to choose the correct model for your given task.</p><h4>Encoder-Decoder</h4><p>Encoder-decoder models perform well on tasks that involve summarization, translation, and other tasks related to sentence generation. Encoders and decoders are each transformers.</p><p>Encoders take input tokens and transform them into contextualized representation. The encoder model in the original paper by Vaswani has 6 identical layers based on the Transformer block described above. The main component in the encoder model is the self-attention mechanism.</p><p>“The decoder is essentially a conditional language model that attends to the encoder representation and generates the target words one by one, at each timestep conditioning on the source sentence and the previously generated target language words to generate a token.” [1] The decoder’s architecture is slightly different than that of an encoder, since the decoder model also has a cross-attention layer. “Cross-attention has the same form as the multi-head attention in a normal transformer block, except that while the queries as usual come from the previous layer of the decoder, the keys and values come from the output of the encoder.” [1] The equation for cross-attention is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*skw69cXCy5sd0piYM85cIQ.png" /><figcaption>The above image is credit to Speech and Language Processing by Jurafsky.</figcaption></figure><p>The decoder has 6 identical layers, where each layer contains: masked self-attention mechanism, multi-head attention or cross-attention, and feed-forward neural network. In the masked self-attention mechanism, there are some tokens that are masked and so the predictions for a particular position depend on the known outputs before that position.</p><p>The encoder-decoder model is a sequence-to-sequence model. The output of the encoder model, also known as a context vector, is then brought in as the input for the decoder model. Below is a diagram of the encoder-decoder model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7noKeb-KB9ZfZXKh1m46pQ.png" /><figcaption>The above image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>An example of an encoder-decoder model is BART, which does translation.</p><h4>Encoder-Only</h4><p>The encoder-only model performs well on sentence classification, such as Named Entity Recognition (NER) or word classification. The encoder model understands text but can’t generate text. The encoder model is based on the transformer block described above, with 6 such blocks. At a given word, the attention mechanism can access input that’s both prior to and after the current word, making it bi-directional. An example of an encoder-only model is BERT.</p><h4>Decoder-Only</h4><p>The decoder-only model performs well on text generation tasks. At a given word, the attention mechanism can only access input prior to the current word. The decoder has 6 identical layers, where each layer contains: masked self-attention mechanism, multi-head attention or cross-attention, and feed-forward neural network. An example of a decoder-only model is GPT.</p><h3>Conclusion</h3><p>In conclusion, the self-attention mechanism feeds into the Transformer block. The Transformer block is the main component of an encoder model. The Transformer block with cross-attention becomes the basis of a decoder model. There are 3 variants of the Transformer model and you can pick one based on the task that’s being performed.</p><h3>Reference</h3><p>[1] Speech and Language Processing by Jurafsky</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbc20408aaed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformer Models Part 1: Self-Attention & Multi-Head Attention]]></title>
            <link>https://medium.com/@sonalprasad.sp/transformer-models-part-1-self-attention-multi-head-attention-21feb8738194?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/21feb8738194</guid>
            <category><![CDATA[self-attention]]></category>
            <category><![CDATA[multi-head-attention]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 02:27:35 GMT</pubDate>
            <atom:updated>2026-05-12T18:57:31.420Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Feel free to read about RNNs here: </em><a href="https://www.linkedin.com/pulse/recurrent-neural-network-rnn-part-1-sonal-prasad-1kcjc"><em>https://www.linkedin.com/pulse/recurrent-neural-network-rnn-part-1-sonal-prasad-1kcjc</em></a></p><h3>What You’ll Learn</h3><ul><li>Issues with RNNs</li><li>Transformer Model</li><li>Self-Attention</li><li>Multi-Head Self-Attention</li></ul><h3>Issues with RNNs</h3><p>Although RNNs are theoretically supposed to carry ‘memory’ in their hidden states, in reality they suffer in doing so. RNNs have trouble with long-range dependencies, since they can’t remember information from many steps ago. Additionally, RNNs perform sequential computation which prevents parallelization, making it computationally slow.</p><h3>Transformer Model</h3><p>Luckily, transformer models solve both of the issues found in RNNs by having long-range dependencies and parallelization. Similar to RNNs, transformer models process sequential data, and are based on neural network architecture. The transformer model was first explained in the paper ‘Attention is All You Need’ by Vaswani et al. There are also variations of transformers, such as vision transformers or speech transformers that perform well for object detection and speech recognition, respectively.</p><p>The core feature of a transformer model is its self-attention mechanism, which brings into fruition the benefits of a transformer model. This article will focus on attention mechanisms in Part 1 of the Transformers series.</p><h4>Attention</h4><p>Let’s take a look at how attention is similar to performing a fuzzy look-up in a key-value dictionary. In a dictionary, we have keys that map to values. We can take a query, match it with a key, and return a value. In the attention mechanism, the query matches all keys softly, where each match is associated with a weight between 0 and 1. The keys’ values are multiplied by the weights and summed. The diagrams below illustrate these concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LF2O2E1NMHWhpen4eY6zpQ.png" /><figcaption>The diagram on the left side depicts a dictionary lookup. The diagram on the right side depicts the weighted sum in attention. The above image is credited to Chelian from the Deep Learning course (CMPE 258) at San Jose State University.</figcaption></figure><p>With attention, we have access to all information, but we focus on the most relevant information. An attention mechanism calculates attention weights where the weights represent the importance of each part of an input sequence.</p><h4>Self-Attention</h4><p>Self-attention plays a major role in in Large Language Models (LLMs) by enabling parallelization, which allows for processing attention weights simultaneously. Let’s set some definitions first. Query refers to the current element. Key refers to a preceding element that is being compared to the current element. The value is the output of the current element.</p><p>We start by taking a sequence of words, such as a sentence, and we embed these words using an embedding model. The embeddings help us calculate the attention weights that give us the relative importance of the relevant input data. Next, we transform each word embedding into weighted matrices: Q, K, and V.</p><pre>q_i = Qx_i (queries)<br>k_j = Kx_j (keys)<br>v_j = Vx_j (values)</pre><p>We take the dot product between the current element’s query vector q_i and the preceding element’s key vector k_j. To avoid dealing with very large computations as a result from the dot product, we scale the dot product by the size of the embeddings. Here is the full equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*jH92iC0BXkls1YyLGOIEAw.png" /><figcaption>The above equations show how attention is computed. The image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><h4>Multi-Head Self-Attention</h4><p>Self-attention can be expanded into multi-head self-attention, in which we attend to different elements within the input simultaneously. We replace self-attention with multi-head attention. We do so by defining multiple “heads” through multiple Q, K, V matrices. Each attention head independently performs attention, and then the outputs of all of the heads are combined. Each head looks at different things and construct value vectors differently. Multi-head self-attention is computationally efficient. [1]</p><h3>Conclusion</h3><p>The attention mechanism is the key feature in the Transformer model. We expand on self-attention by using multi-head self-attention. Due to attention, we have the ability to have long-range dependencies and perform parallelization. Part 2 of this series will look into the Transformer architecture.</p><h3>Reference</h3><p>[1] Chelian from the Deep Learning course (CMPE 258) at San Jose State University.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21feb8738194" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recurrent Neural Network (RNN) Part 2 — Long Short Term Memory (LSTM)]]></title>
            <link>https://medium.com/@sonalprasad.sp/recurrent-neural-network-rnn-part-2-long-short-term-memory-lstm-d6f7c878a29b?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6f7c878a29b</guid>
            <category><![CDATA[vanishing-gradient]]></category>
            <category><![CDATA[rnn]]></category>
            <category><![CDATA[long-short-term-memory]]></category>
            <category><![CDATA[lstm]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 07 Apr 2026 03:35:41 GMT</pubDate>
            <atom:updated>2026-04-07T03:35:41.324Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Feel free to read RNN Part 1 here: </em><a href="https://www.linkedin.com/pulse/recurrent-neural-network-rnn-part-1-sonal-prasad-1kcjc*"><em>https://www.linkedin.com/pulse/recurrent-neural-network-rnn-part-1-sonal-prasad-1kcjc</em></a></p><p>Why do vanilla RNNs struggle with long-term dependencies, and how do LSTMs solve this problem?</p><h3>What You’ll Learn</h3><ul><li>Issue with vanilla RNNs</li><li>Applications of LSTMs</li><li>LSTM architecture</li><li>Advantages and disadvantages of LSTMs</li></ul><h3>What Was Missing in RNNs that Led to LSTMs?</h3><p>Vanilla RNNs face the issue of vanishing gradients, which means that over time as the algorithm backpropagates multiple times, at some point the value of the derivative reaches 0, or in other words, the signal vanishes. The vanishing gradient results in significant learning difficulties in vanilla RNNs. RNNs especially have a hard time with long sequences. Long Short Term Memory (LSTM) model introduces a gating mechanism that controls the flow of information through the network. The gates in the LSTM help determine whether to keep certain memories or to let some of them go.</p><p>You can think of the LSTM like the control center in the movie <em>Inside Out</em>, where different emotions decide what memories to keep, update, or act on. Each gate plays a similar role in deciding how information flows through the system.</p><h3>What Are LSTMs?</h3><p>LSTMs are a type of RNN that take in sequences, such as time series, speech, and text, where the context from earlier in the input matters. As described above, the gating mechanism significantly supports keeping track of memory across hundreds of timesteps, making it possible for LSTMs to perform various applications.</p><h4>Applications of LSTMs</h4><p>One application of LSTMs is sequence labeling, such as Named Entity Recognition, Part-of-Speech tagging, and text chunking. A second application is language modeling, where the model predicts the next word in a sentence. A third application is sequence classification in which the model provides a classification (e.g. sentiment analysis, identifying emotions in speech, and identifying human activities like walking or sitting). A fourth application is Visual Question Answering (VQA) where you can ask a question about an image (e.g. what is the dog doing?).</p><h4>LSTM Architecture</h4><p>LSTMs are able to handle long-term dependencies, which vanilla RNNs can’t, by introducing a memory cell. The memory cells holds onto important information that it has learned over time and can be carried over into many timesteps. At a given timestep, the memory cell adjusts its container with new information that passes through 3 different phases — forget gate, input gate, and output gate. The input gate handles what information enters the memory cell. The forget gate handles what information gets removed from the memory cell. The output gate outputs the information required for the current hidden state (and is not about a future decision). As a result, LSTMs can keep or discard information, making it possible to learn long-term dependencies.</p><p><strong>Forget Gate</strong></p><p>The forget gate deletes information from the memory cell that is no longer needed. It performs a weighted sum of the previous state’s hidden layer with the current input and passes this sum through the Sigmoid activation function. The output is then multiplied by the memory cell, element-wise, to determine what content to remove. If the value is 0, then the piece of information is deleted, and if the value is 1, then the piece of information stays in the memory cell. The forget gate decides what information is no longer useful and should be erased from memory.</p><p>The equation for the forget gate is:</p><p>f_t = σ(W_f · [h_t-1, x_t] + b_f)</p><ul><li>x_t: input at a particular time</li><li>h_t — 1: output from the previous cell</li><li>W_f: weighted matrix</li><li>b_f: bias with the forget gate</li><li>σ: sigmoid activation function</li></ul><p>This equation computes how much of the previous memory should be kept or forgotten. The sigmoid function outputs values between 0 and 1 for each element, where values closer to 0 mean “forget this information” and values closer to 1 mean “keep this information.” In other words, the forget gate acts like a filter that removes information that is no longer relevant.</p><p><strong>Input Gate</strong></p><p>The input gate controls important information being added to the memory cell. It has 2 steps. First, it uses a sigmoid activation function to decide which data is important and which should be ignored. Second, it creates a list of potential new memories using the tanh activation function. By multiplying these two together, the gate ensures that only the most relevant updates are added to the system’s internal state, effectively keeping the memory gate fresh and focused on the task at hand. The input gate decides what new information is important enough to remember. To help visualize this, the process follows these core equations:</p><p>i_t = σ(W_i · [h_t-1, x_t] + b_i)</p><p>Ĉ_t = tanh(W_c · [h_t-1, x_t] + b_c)</p><p>C_t = f_t ⊙ C_t-1 + i_t ⊙ Ĉ_t</p><p>Note: ⊙ denotes element-wise multiplication</p><p>These equations determine what new information should be added to the memory. The first equation (with the sigmoid function) decides which parts of the new information are important. The second equation (with the tanh function) generates candidate values that could be added to memory. By combining these two, the model ensures that only relevant new information is written into the memory cell.</p><p><strong>Output Gate</strong></p><p>The output gate is not about preserving memory for a future decision. It is about the present state. The output gate extracts useful information from the current state. First, we produce a vector by passing the memory cell into the tanh activation function. Second, the information is passed through sigmoid to control what information stays. Finally, the output is generated by multiplying the vector from step 1 with the regulated values from step 2. The output gate decides what information to expose at the current timestep.</p><p>The equation for the output gate is: o_t = σ(W_o · [h_t-1, x_t] + b_o)</p><p>This equation determines what information from the current memory should be used to produce the output at this timestep. The sigmoid function acts as a filter, selecting which parts of the memory are important right now. This filtered information is then combined with the memory (after passing through tanh) to produce the hidden state output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wz1PaOXPijNSHnAbAG3a7g.png" /><figcaption>The above diagram shows the full picture of the 3 gates in the LSTM model. Credit for this image goes to: <a href="https://www.geeksforgeeks.org/deep-learning/deep-learning-introduction-to-long-short-term-memory/">https://www.geeksforgeeks.org/deep-learning/deep-learning-introduction-to-long-short-term-memory/</a></figcaption></figure><h3>Advantages and Disadvantages of LSTMs</h3><p>The main advantage of LSTMs over RNNs is that LSTMs can preserve information over many time steps; it’s harder for vanilla RNNs to do so. In order words, LSTMs can learn long-distance dependencies.</p><p>One disadvantage is that LSTMs can still face vanishing/exploding gradients. Second, LSTMs (and RNNs) can’t handle long context. Three, it’s not possible to make the computations run in parallel.</p><h3>Conclusion</h3><p>Overall, LSTMs were a major step forward in sequence modeling, enabling models to better capture long-term dependencies — something that vanilla RNNs struggled with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6f7c878a29b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recurrent Neural Network (RNN) Part 1]]></title>
            <link>https://medium.com/@sonalprasad.sp/recurrent-neural-network-rnn-part-1-bbd1bb1110ff?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd1bb1110ff</guid>
            <category><![CDATA[vanishing-gradient]]></category>
            <category><![CDATA[rnn]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[time-series-data]]></category>
            <category><![CDATA[state]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 14:12:43 GMT</pubDate>
            <atom:updated>2026-03-24T14:12:43.827Z</atom:updated>
            <content:encoded><![CDATA[<p>How do we give a neural network memory of what it has seen so far?</p><h3>What You’ll Learn</h3><ul><li>Characteristics of RNNs</li><li>Recurrence Formula</li><li>Challenges of vanishing gradient</li><li>Advantages and Disadvantages of RNNs</li></ul><p>A Recurrent Neural Network (RNN) is a type of neural network that is trained on time series data. When RNNs process a time series, they keep track of an internal state. RNNs can be used for speech recognition, language modeling, and translation.</p><h3>Characteristics of RNNs</h3><p>RNNs have cycles and a notion of time. They process sequences of data and also output sequences of data. RNNs have “memory” when processing sequential data and can keep track of information from previous steps. RNNs are different from feedforward neural networks because RNNs have recurrent layers that keep track of a hidden state. The output of RNNs is based on prior elements within the sequences.</p><p><em>Note: “internal state,” “hidden state,” “state,” “memory,” and “context” are used interchangeably</em></p><p>We can take an idiom, such as “a blessing in disguise.” The idiom makes sense only in that order. RNNs are sensitive to the position of each word and they use the memory of the order of words to predict the next word. The order matters for each word in “a blessing in disguise.” RNNs store a hidden state at each time step that keeps track of the memory or context. The hidden state is passed from one time step to the next time step. At a given time step, the RNN takes in the current input and the hidden state from the previous time step. As a result, RNNs “remember” context when predicting the output.</p><p>Another characteristic of RNNs is that the parameters of the nodes across layers are the same. In RNNs, we reuse the same weights at every time step (parameter sharing through time). However, feedforward neural networks have different weights across each node. RNNs follow the backpropagation process and perform gradient descent. Training is done by unrolling the RNN across time steps and applying backpropagation through time (BPTT). By unrolling the network, RNNs perform prediction and training the same way as for a feedforward neural network. In the diagram below, we can see an unrolled network where the weights U, V, and W are shared across all time steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Booc3sAoRR731DRBWNnBJQ.png" /><figcaption>The above image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>RNNs have a unique type of neural layer, known as recurrent layers, that help the network track memory. As shown in the diagram below, the number of inbound connections is the same as the number of neurons in the previous layer, and the number of outgoing connections is the same as the number of neurons in the following layer. A recurrent layer has r² connections, where r is the number of neurons in the current layer. This is one reason RNNs can get computationally heavy. The diagram below shows a recurrent layer with its connections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W07SmVsjKpuvLCyvplrYCA.png" /><figcaption>The above image is credited to Fundamentals of Deep Learning by Buduma.</figcaption></figure><p>RNNs use activation functions that introduce nonlinearity allowing for the network to learn complex patterns. Learning nonlinear patterns of data is very important for tasks in NLP and sequential data prediction. At each time step, RNNs apply activation functions, making way to update the memory. Some activation functions are Sigmoid, Tanh, and ReLU.</p><p>*Note: Feel free to read more about activation functions at the bottom of a previous article: <a href="https://www.linkedin.com/pulse/neural-networks-part-1-sonal-prasad-72klc/?trackingId=XZzuHhWMQ3ieBoQRhV0Hhw%3D%3D*">https://www.linkedin.com/pulse/neural-networks-part-1-sonal-prasad-72klc/?trackingId=XZzuHhWMQ3ieBoQRhV0Hhw%3D%3D*</a></p><h4>Mini Example</h4><ul><li>Input words: “I”, “love”, “pizza”</li><li>At each step: take current word embedding + previous hidden state → new hidden state</li><li>Output: predict next word (or classify sentiment at the end)</li></ul><h3>Recurrence Formula</h3><p>At each time step, we apply a recurrence formula to a sequence of vectors x, that predicts another vector. The hidden state is a single vector h. The diagram below shows how we pass the hidden state, h, and an input vector, x, to produce the next state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-f6j59X52roDUjXcjAGflQ.png" /><figcaption>The above image is credited to Chelian from the Deep Learning course (CMPE 258) at San Jose State University.</figcaption></figure><p>Here are the values in the formula:</p><ul><li>xₜ = input at time t (e.g. the word at position t)</li><li>hₜ = “memory” after seeing inputs up to time t; a vector (an array of floats) that acts like a running summary of what we’ve seen so far</li><li>W = shared weights reused at every time step</li></ul><p>After we get the new state, we pass it through an activation function, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*Fjg8eX0Zjmvo_URV8rr5eA.png" /><figcaption>The above image is credited to Chelian from the Deep Learning course (CMPE 258) at San Jose State University.</figcaption></figure><h3>Challenges of Vanishing Gradient</h3><p>The weights in the model are initialized to small values at the start of training. As a result “the value of this derivative approaches zero as <em>k</em> increases. In other words, the gradient quickly diminishes when it’s computed with respect to inputs several time steps into the past, severely limiting our model’s ability to learn long-term dependencies” [1]. This is known as vanishing gradients and it affects vanilla RNNs’ ability to learn. Because the learning signal fades as we go back in time, vanilla RNNs often struggle to learn long-range dependencies (e.g. connecting a subject at the start of a long sentence to a verb later).</p><h3>Advantages and Disadvantages of RNNs</h3><p><strong>Advantages</strong></p><ol><li>Input can be of any length</li><li>In theory, can use information from many steps back</li><li>Model size doesn’t change for longer input</li><li>The weights are the same at every time step</li></ol><p><strong>Disadvantages</strong></p><ol><li>Computation is slow</li><li>In reality, it’s hard to retrieve information from memory that was many steps ago</li></ol><h3>Part 2</h3><p>In Part 2, we will see how LSTMs help with the above issues in vanilla RNNs.</p><h3>Bibliography</h3><p>[1] Fundamentals of Deep Learning by Buduma</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd1bb1110ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Modeling with Neural Networks]]></title>
            <link>https://medium.com/@sonalprasad.sp/language-modeling-with-neural-networks-fdba55c1c3e3?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdba55c1c3e3</guid>
            <category><![CDATA[one-hot-encoding]]></category>
            <category><![CDATA[feedforward-neural]]></category>
            <category><![CDATA[ngrams]]></category>
            <category><![CDATA[language-model]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 04:52:07 GMT</pubDate>
            <atom:updated>2026-03-17T04:52:07.072Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Feel free to take a look at my previous article on basic (n-gram) language models here: </em><a href="https://www.linkedin.com/pulse/language-models-sonal-prasad-2qf0c/?trackingId=47MDLmhdRtG9n52kVDJUCQ%3D%3D"><em>https://www.linkedin.com/pulse/language-models-sonal-prasad-2qf0c/?trackingId=47MDLmhdRtG9n52kVDJUCQ%3D%3D</em></a></p><p>How do language models predict the next word from previous words?</p><h3>What You’ll Learn</h3><ul><li>Compare n-gram vs. neural language models</li><li>Understand the forward computation in a feedforward neural LM</li><li>Learn what the model trains and what it produces (next-word probability distribution)</li></ul><h3>A Quick Mental Model</h3><p>Before we get into the math, it helps to have a simple “pipeline” in mind. A feedforward neural language model is basically a function that takes a fixed-size window of previous words and returns a probability distribution for the next word.</p><p>At a high level, each prediction looks like this:</p><ol><li><strong>Choose a context window: </strong>The model looks at the last <em>N</em> words (for example, <em>N = 3</em>): “the dog gets”.</li><li><strong>Turn words into IDs (indices): </strong>Each word is mapped to an integer ID using a vocabulary (a dictionary of all known words).</li><li><strong>Convert IDs into vectors (embeddings): </strong>You can think of this step as a lookup: each word ID maps to a dense vector (its embedding) from an embedding matrix <strong>E</strong>.</li><li><strong>Combine the context vectors into one input: </strong>The model concatenates the embeddings for the <em>N</em> previous words into a single long vector.</li><li><strong>Run that vector through a small neural network: </strong>A hidden layer applies weights and a nonlinearity to produce a representation of the context.</li><li><strong>Output next-word probabilities (softmax): </strong>The final layer produces a score for every word in the vocabulary, and softmax converts those scores into probabilities that sum to 1. The word with the highest probability is the model’s best guess for the next word.</li></ol><p>During training, the model repeatedly does this prediction on real text and updates its parameters so that the true next word becomes more likely over time.</p><h3>Language Modeling with Neural Networks</h3><p>According to Jurafsky [1], “A language model is a machine learning model that predicts upcoming words. More formally, a language model assigns a probability to each possible next word, or equivalently gives a probability distribution over possible next words.” A language model is an application of Natural Language Processing (NLP). Modern NLP is based on the transformer architecture, which will be covered in a future article. For now, we will take a look at a simpler version of neural language models known as feedforward networks (or feedforward neural language model), but still more advanced than n-gram language models.</p><h3>N-Grams vs. Neural Language Models</h3><p>Neural language models are an improvement from n-gram models. <em>Feel free to learn more about n-gram language models here: </em><a href="https://www.linkedin.com/pulse/language-models-sonal-prasad-2qf0c/?trackingId=47MDLmhdRtG9n52kVDJUCQ%3D%3D"><em>https://www.linkedin.com/pulse/language-models-sonal-prasad-2qf0c/?trackingId=47MDLmhdRtG9n52kVDJUCQ%3D%3D</em></a><em>.</em></p><p>Neural language models can generalize better, compared to n-grams, since they capture syntactic relationships and capture meaning. For example, let’s take the sentence “I have to make sure that the cat gets fed.” The context window is “the dog gets”, and the neural language model outputs the following probability distribution: fed (0.22), wet (0.05), etc. The neural language model can now reasonably predict ‘fed’ after ‘the dog gets’ even if it never saw that exact phrase in training. It can predict “fed” after “the dog gets” because “cat” and “dog” have similar embeddings, but the n-gram language model cannot do that. The n-gram language model will predict “fed” after seeing “that the cat gets,” but not if the phrase is “that the dog gets.” Neural language models can also handle larger context and have more accurate predictions. On the other hand, n-grams are simpler, faster to train, have smaller computational complexity, are more interpretable, and are preferred for smaller tasks.</p><h3>Feedforward Neural Language Model (LM)</h3><p>The feedforward neural LM estimates the next word based on the prior context, or previous words. It generates a probability distribution over possible next words. The context, or previous words, are represented by their embeddings, which allows for better generalization to unseen data.</p><h3>Forward Inference In Feedforward Neural Language Model</h3><p>The inference generates a probability distribution over the next possible words. Here is how it does that. The feedforward neural LM keeps track of the past N words in a window. These N words are each represented by a one-hot vector.</p><p><strong>One-Hot Vector</strong></p><p>“A one-hot vector is a vector [or array] that has one element equal to 1 — in the dimension corresponding to that word’s index in the vocabulary — while all the other elements are set to zero” [1]. The below shows how the one-hot vector has a “1” marked at index 5, and the index 5 maps to a specific word in the vocabulary; all of the other indices are marked as “0.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*jHh1B2F-mFDcxMQ1Omsgfg.png" /></figure><p>One-hot vector is basically an index turned into a vector so matrix multiplication can “look up” the embedding. The index of the vocabulary maps to a specific word, and the dimension value of 1 tells us which index (aka word) is lighting up or being represented. Each one-hot vector is multiplied by the embedding matrix E, which has a size of d x |V|, where |V| is the length of the vocabulary and there are d dimensions. Each of the embedding vectors are concatenated to create the embedding layer. Afterward, there is an output layer and then a softmax layer that creates the probability distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*lXrTIFV6aoBQSKjj46VE4A.png" /><figcaption>The image above is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p><strong>Embedding</strong></p><p>“The input is a one-hot vector for each word which serves as an input index into an embedding matrix E. The embedding matrix E has a column (embedding vector) for each word in the vocabulary. Let’s say there’s a word with a 1-hot vector and it has an index number of 35 (which is the location in the vocabulary), we use this index to find its embedding vector in an embedding matrix. The embedding matrix may contain Word2Vec embeddings or the embedding may have random vectors which is how it was trained. This initial vector for each word in E is the initial embedding for the word that we have. When treated as a weight matrix (that can be updated), E contains the fine tuned word embeddings at the end of the training phase” [2].</p><p><em>Note: Read about Word2Vec here: </em><a href="https://www.linkedin.com/pulse/word-embeddings-sonal-prasad-7peec/?trackingId=OBoLGsSbQVmio38wqhvyXA%3D%3D"><em>https://www.linkedin.com/pulse/word-embeddings-sonal-prasad-7peec/?trackingId=OBoLGsSbQVmio38wqhvyXA%3D%3D</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FvacACeISjsapVaVKB98eg.png" /><figcaption>The image above is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p><strong>Algorithm Example</strong></p><p>Let’s say we have 3 previous words (shown in the diagram above), for which we create a total of 3 one-hot vectors. To create a one-hot vector, we first look up the index of the word from the vocabulary and mark the element at the index in the vector with “1” and the rest with “0.” We multiply the one-hot vector for “for” (index 35) with the embedding matrix E. Here is an example of the algorithm:</p><ol><li>The embedding layer for one word is: E*x_i = e_i E is the embedding matrix, x_i is the one-hot vector for one word, and e_i is the embedding layer for one word. We concatenate all of the N previous words’ one-hot vectors multiplied by E to get the embedding layer ‘e.’</li><li>We pass the following through an activation function (such as ReLU): e is multiplied by W and then added with b. The final result is ‘h,’ or hidden layer.</li><li>We multiply ‘h’ by ‘U.’</li><li>With softmax, we get the probability distribution.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*CROqDVRkJ8DiMI7p8UEkFw.png" /><figcaption>The semicolons refer to concatenation. The image above is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><h3>Training A Neural LM</h3><p>When training a neural LM, we have the ability to adjust parameters E, W, U, and b. Training helps adjust W and U. We perform gradient descent, and use backpropagation. We train the parameters to minimize loss. “As we’re predicting upcoming words, we’re learning the embeddings E for each word that best predict upcoming words” [1]. Training results in 1) having a language model, and 2) a new set of embeddings E that can be used elsewhere. Alternatively, neural language models can use pre-trained embeddings.</p><h3>Bibliography</h3><p>[1] Speech and Language Processing by Jurafsky</p><p>[2] Jetcheva, San Jose State University Natural Language Processing Course, Module 9 slides</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdba55c1c3e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks Part 2]]></title>
            <link>https://medium.com/@sonalprasad.sp/neural-networks-part-2-b9f822ca77d9?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9f822ca77d9</guid>
            <category><![CDATA[forward-pass]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[backpropagation]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 00:59:21 GMT</pubDate>
            <atom:updated>2026-03-10T00:59:21.479Z</atom:updated>
            <content:encoded><![CDATA[<h3>What You’ll Learn</h3><p>This Part 2 builds on the basics from Part 1 and focuses on how neural networks actually learn. <em>You may read Part 1 here: </em><a href="https://medium.com/@sonalprasad.sp/neural-networks-part-1-972dd8188077"><em>https://medium.com/@sonalprasad.sp/neural-networks-part-1-972dd8188077</em></a></p><ul><li>How neural networks learn from data (training)</li><li>Gradient descent + learning rate</li><li>Forward pass vs. backpropagation</li><li>Why models overfit and how dropout/L1/L2 reduce it</li></ul><h3>Advantages and Disadvantages</h3><p>Neural networks have multiple advantages. One, neural networks perform more complex tasks, such as taking images as input and outputting a specific category. Two, neural networks can work with disorganized data to ultimately find patterns. Three, neural networks are adaptive and will adjust their weights based on the goal.</p><p>Neural networks also come with disadvantages. One, although neural networks can perform complex tasks and are adaptive, they require heavy machinery and hardware to train the model properly. Two, neural networks tend to create incomplete results, due to not being trained for the whole process. Three, since the quality of neural networks depend on the data that they are trained on, when the dataset is small or the data is not easily interpreted, the algorithm can go wrong.</p><h3>Training</h3><p>The neural network has varying weights across its connections. How does it figure out the weights? When we train a neural network, it determines the optimal weights for its connections. During the training process, the neural network takes in training examples and then iteratively adjusts the weights to minimize the errors. After multiple loops, or iterations, the neural network should perform well for its task.</p><h4>Gradient Descent</h4><p>Gradient descent helps minimize the error across the training input. In the below diagram, for simplicity’s sake, our linear neuron has 2 inputs, with one weight for each input. When taking into account all errors across all possible weights, we get a 3D surface as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z19wHEyF7XKKg_IxYGhfNQ.png" /><figcaption>The image is credited to Fundamentals of Deep Learning by Buduma.</figcaption></figure><p>The minimum error in the diagram above is at the center, where the star icon is placed. The slope is steeper when the contours are closer to each other. “The direction of the steepest descent is always perpendicular to the contours. This direction is expressed as a vector known as the gradient” [1]. It is optimal to go in the direction of the steepest descent. The algorithm for reaching the point of minimum error is known as “gradient descent.”</p><h4>Hyperparameters</h4><p>The training process involves engineers choosing hyperparameters, as opposed to the model learning the weight parameters behind-the-scenes. One example of a hyperparameter is the learning rate, which controls the distance we move before recalculating our new direction. The learning rate is dependent on the steepness of the surface. For example, if we are fairly close to the minimum error spot, then we would want to move a shorter distance. When we are close to the minimum, the steepness is flatter. The way the learning rate factors into the algorithm is by multiplying the learning rate by the gradient. The learning rate is something that the engineer must choose and experiment with. A small learning rate results in a longer training time, while a large learning rate may not necessarily lead us to minimum error spot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RlqwG438P0SPtJbl31O4Pw.png" /><figcaption>The image above shows a learning rate that’s too big, and doesn’t lead to convergence. The image is credited to Fundamentals of Deep Learning by Buduma.</figcaption></figure><p>Other hyperparameters apart from the learning rate include the number of layers, the number of units per layer, activation functions, gradient descent variant, mini-batch size, and regularization.</p><h3>Propagation</h3><h4>Feedforward Propagation</h4><p>In the forward pass, we move from the input (all training examples), to the hidden layers, and then to the output value. The direction is moving from the input to the output. Every time we pass over the entire training set, it is known as a training epoch. After getting the output value, if the algorithm hasn’t converged, the algorithm does a backpropagation by propagating error signals back to the input, as described in the below section.</p><h4>Backpropagation</h4><p>In the backpropagation algorithm, it takes into account how fast the error changes when we change hidden activity. It picks up the relationship between the change in error and the change in the weight of an individual connection. The goal is to find the steepest descent. The backpropagation algorithm finds the partial derivative of the error in each hidden layer, one after another, where the previous layer’s derivative is factored into the current layer’s derivative; this inductive step takes partial derivatives. After we have all of the partial derivatives, we sum them up.</p><p>Note: the above is more precisely known as batch gradient descent, where we take our entire dataset upon which we compute the error to then find the gradient with the steepest descent.</p><p>The general steps for training a neural network in PyTorch are:</p><ol><li>Make a prediction (forward pass)</li><li>Compute the loss</li><li>Zero the stored gradients</li><li>Backpropagate the gradients</li><li>Update the weights by taking a step of the gradient descent</li></ol><p>The above steps are repeated in each epoch, until convergence is achieved.</p><p><strong>Backpropagation Challenges</strong></p><p>One backpropagation challenge is the vanishing gradient problem. When we multiply the partial derivatives in backpropagation, and each derivative is close to 0, then the final product of the multiplication is also close to 0. As a result, it’s hard to make much progress since the propagation is basically extinguished. One way to deal with vanishing gradients is to use batch normalization in stochastic gradient descent.</p><p>A second challenge is the exploding gradient problem. The gradients become too large, which leads to unstable and large weight updates. The loss function updates dramatically. The model overshoots the optimal solution. One approach to avoiding the exploding gradient problem is by using L1 and L2 regularization.</p><p>A third challenge is being stuck in a local minimum, instead of reaching the global minima. At a local minimum, the gradient is zero and the loss is higher than the global minimum. It makes the algorithm stuck in a suboptimal solution, it has slower convergence due to taking smaller steps, and it has poor generalization.</p><p>A fourth challenge is being stuck in a saddle point, which means the gradient is zero. The saddle point is neither a local minimum or the global minimum. It makes gradient descent slow down.</p><h3>Regularizers</h3><p>Regularization techniques help avoid overfitting. Overfitting occurs when a model is trained too well on the training data and performs poorly on new, unseen data. This is a common problem in deep learning, especially with large neural networks, and can be caused by a lack of data, a complex model, or a lack of regularization. Regularization adds constraints or noise during training to reduce overfitting. Below are examples of regularizers.</p><p><strong>Dropout</strong></p><ul><li>What it means: In training, some number of layer outputs are randomly ignored (dropped out) with probability p. ”Dropping values” means temporarily removing them from the network for the current forward pass. Dropout tends to make the training process noisy.</li><li>Pros: Prevents overfitting</li><li>Cons: Slows down training, doesn’t work well with small training datasets</li></ul><p><strong>Decay L1</strong></p><ul><li>What it means: L1 removes some features by forcing some weights to be zero.</li><li>Pros: Deeper feature analysis, feature selection, prevents overfitting</li><li>Cons: Empirically slower than L2, Robust to outliers</li></ul><p><strong>Decay L2</strong></p><ul><li>What it means: L2 is a common type of regularization. L2 forces the weights to be small but doesn’t make them zero.</li><li>Pros: Empirically performs better than L1, prevents overfitting without eliminating input features</li><li>Cons: Less effective for feature selection, sensitive to outliers</li></ul><p>Integrating the right regularizer(s) can help improve model performance, and avoid overfitting. Some other ways to avoid overfitting include having a smaller learning rate, and using normalization.</p><p>This concludes the series on neural networks.</p><h3>Bibliography</h3><p>[1] Fundamentals of Deep Learning, 2nd Edition by Buduma</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9f822ca77d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks Part 1]]></title>
            <link>https://medium.com/@sonalprasad.sp/neural-networks-part-1-972dd8188077?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/972dd8188077</guid>
            <category><![CDATA[feed-forward-networks]]></category>
            <category><![CDATA[activation-functions]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[perceptron]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 24 Feb 2026 16:48:20 GMT</pubDate>
            <atom:updated>2026-02-24T16:48:20.976Z</atom:updated>
            <content:encoded><![CDATA[<p>How do neural networks work?</p><h3>What You’ll Learn</h3><ul><li>Brain → neuron inspiration</li><li>Linear vs. non-linear</li><li>Neuron math (weights + bias)</li><li>Perceptron basics</li><li>Feed-forward neural networks (MLPs)</li><li>Activation functions (sigmoid, tanh, ReLu)</li></ul><p>We will start with the biological inspiration, then build up the math of a single neuron, and finally connect that to multi-layer networks and activation functions.</p><h3>Human Brain (Neuron)</h3><p>The human brain consists of neurons. A small piece of the brain has over 10,000 neurons. The entire brain has around 86 billion neurons. Each neuron is connected to thousands of other neurons. The biological network in the brain helps us experience the world. The structure of the neuron inspires the units or neurons in the artificial neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Fvmb_-IIVjTZBdLgl-3JA.png" /><figcaption>The image above is credited to Fundamentals of Deep Learning by Buduma.</figcaption></figure><p>The biological neuron receives information from other neurons, processes the information, and sends its results to other cells. The neuron’s front contains dendrites that take in input information from another cell. Different inputs have varying strength, and, thus, the stronger the strength, the stronger the connection. The connection’s strength is based on how often we use the information. Each piece of information has weights that are summed together in the cell body, or soma, and are transformed into a signal. Afterwards, the axon propagates the transformed signal to the synaptic terminal from where it’s sent to other neurons. However, the neuron only sends the signal to other neurons, if it fires an action potential. Action potentials are also known as “nerve impulses.” A neuron that emits an action potential is said to “fire.” An action potential’s spike is shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*F4ZkkDoZcN2Ot9Kfw-H52w.png" /><figcaption>The image above is credited to Fundamentals of Deep Learning by Buduma.</figcaption></figure><p>Now that we have the biological intuition for how information flows and gets amplified or weakened, we can translate the same idea into a simple mathematical unit.</p><h3>Linear Vs. Non-Linear Functions</h3><p>Models have a way of creating a decision boundary amongst the data from the dataset. Some decision boundaries are linear, while others are non-linear. Before we build a full neural network, it helps to see what a <em>linear</em> model can and cannot do, so we know why neural networks need non-linearities. Equations are used to model whether there is activated potential or not. Typically, in neural networks, activation functions are non-linear equations.</p><h4>Logistic Regression</h4><p>First of all, why are we talking about logistic regression in an article for neural networks? That is because, logistic regression illustrates a linear model. We will understand first what a linear model is before we move onto non-linear models used for activation functions.</p><p>Logistic regression helps with classification tasks, and is a linear model. For example, it can be used to train a model to predict whether to label an email as spam or not spam, which is a binary classification task. Logistic regression uses the training set to learn a vector of weights <em>w</em> and a bias term <em>b</em>. Each weight corresponds to the impact of a specific feature. The logistic regression function is as such:</p><pre>z = w*x + b</pre><p>In the above formula, <em>w</em> is the vector of weights, <em>x</em> is the vector of inputs and <em>b</em> is the bias term.</p><p>In binary logistic regression, the output is a probability across 2 possible outputs. The sum of the probabilities of each output adds up to 1. To convert <em>z</em> (from the function above) into a probability, we use the Sigmoid function, which is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*qhpJIcYHir-XHMOO_n3cXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*rK_NmRuLGCGEwwNO1ZGOag.png" /></figure><p>If the probability of a class is greater than 0.5, then that class will be the output for the binary logistic regression function. For example, if the probability of an email being ‘spam’ is 0.6, then the output will be ‘spam.’ Logistic regression forms a decision boundary as shown below. As we can see, the decision boundary is linear. The sigmoid makes the output non-linear, but the boundary is still linear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ENpM1Lgsb8ccw8XnpdTUJQ.png" /><figcaption>The above image is credited to Baeldung (<a href="https://www.baeldung.com/cs/draw-logistic-regression-decision-boundary">https://www.baeldung.com/cs/draw-logistic-regression-decision-boundary</a>)</figcaption></figure><p>I wanted to illustrate the difference between linear and non-linear functions. Logistic regression is an example of a linear function that does not utilize neural networks. Non-linear means that you can’t predict a label with a model of the form:</p><pre>z = w1*x1 + w2*x2 + b</pre><p>The decision surface for non-linear functions is not a line, such as the one shown for logistic regression above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*xDCGgsUYB_TFHG9cn1_eZA.png" /></figure><p>If the data is plotted as the above, there cannot be a linear decision boundary, since the data from one class overlaps with data from the other class. The data plot shown above would require a non-linear approach to modeling.</p><p>Logistic regression is a great baseline because it turns a weighted sum into a probability, but it still creates a linear decision boundary.</p><h3>Basic Unit (or Neuron)</h3><p><em>Note:</em> <em>x = inputs, w = weights, b = bias, z = weighted sum, f = activation function, a/y = output</em></p><p>The basic unit, or neuron, in a neural network is inspired by the biological neuron. A neural unit takes a weighted sum of its inputs and adds it to a bias term. The equation is similar to the one for logistic regression:</p><pre>z = w*x + b</pre><p>However, “instead of using <em>z</em>, a linear function of <em>x</em>, as the output, neural units apply a non-linear function <em>f</em> to <em>z</em>. We will refer to the output of this function as the activation value for the unit, <em>a</em>.” [1] In the equation below, <em>z</em> is the weighted sum, and <em>a</em> is the activation function.</p><pre>y = a = f(z)</pre><p>In the below image, it shows how each of the 2 inputs has its respective weight. The weighted inputs are summed together with the bias to get the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hTYP9wXYHKLMOiDQXufrw.png" /><figcaption>The image is credited to <a href="https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/">https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/</a></figcaption></figure><p>To reiterate, the neural unit has inputs, weights, a bias, and an output.</p><p>“Let’s walk through an example just to get an intuition. Let’s suppose we have a unit with the following weight vector and bias:</p><pre>w = [0.2,0.3,0.9]<br>b = 0.5</pre><p>What would this unit do with the following input vector:</p><pre>x = [0.5, 0.6, 0.1]</pre><p>The resulting output y would be:” [1]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KXQxXf1ebZfph59ak-7Nvg.png" /><figcaption>The equation above is taken from Speech and Language Processing by Jurafsky.</figcaption></figure><p>The activation function used above is known as the Sigmoid function. We will cover this as well as other activation functions in more depth in a later section.</p><p>A neural network neuron starts the same way as logistic regression with a weighted sum, but the key upgrade is applying a non-linear activation so the model can represent more complex patterns.</p><h3>Perceptron</h3><p>Let’s step back a little to understand a simple type of unit of a neural network. A perceptron contains a non-linear activation function that is based on a simple step function. The perceptron has a binary output of 0 or 1. Unlike neural networks, a perceptron has no hidden layers. The perceptron is made up of inputs, weights, a bias, a non-linear activation function, and the output. The perceptron is a linear classifier, which serves as a decision boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3dYzX1QS82xcm-l7Ka3Bg.png" /><figcaption>The above image is credited to <a href="https://medium.com/geekculture/neural-network-19b26dba9b1a">https://medium.com/geekculture/neural-network-19b26dba9b1a</a></figcaption></figure><p>Let’s say we are asking a binary question, such as, should we purchase this home? We can use a perceptron to answer this type of question. There are multiple factors that contribute to our decision. The factors for purchasing a house can include mortgage rates, climate, location, and job stability.</p><pre>(Buy / Not Buy Score) = Mortgage Rates + Climate + Location + Job Stability</pre><p>We can mark each factor or feature as x:</p><pre>y = w1x1 + w2x2 + w3x3 + w4x4</pre><p>We will purchase the home if the result passes the threshold. If the result does not pass the threshold, then we will not purchase the home.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*n99UKmh8w6mHVPIfBmUw5g.png" /><figcaption>The above image is taken from <a href="https://medium.com/datascienceray/perceptron-learning-algorithm-1-c01932a23fd3">https://medium.com/datascienceray/perceptron-learning-algorithm-1-c01932a23fd3</a>If the result falls on the left side, it doesn’t pass the threshold. If the result falls on the right side, it passes the threshold.</figcaption></figure><p>A perceptron learns the optimal line (or decision boundary) by the following steps:</p><ol><li>Identifies a mistake point</li><li>Adds or subtracts to correct the mistake</li><li>Do steps 1 and 2 until there are no mistakes</li><li>Returns the final version of weights that creates the correct separation</li></ol><p>The perceptron is a historical stepping stone that shows how a neuron uses a threshold.</p><h3>Feed-Forward Neural Network</h3><p>A single neuron is more powerful than linear perceptrons with step functions, but we need many neurons to build a complex model that can learn complicated problems. Our brain, of course, contains more than one neuron, and our cerebral cortex — responsible for human intelligence, contains 6 layers. The neurons in the human brain are organized into layers.</p><p>Feed-forward neural networks are sometimes known as Multi-Layer Perceptrons (MLPs) which is a slight misnomer because feed-forward neural networks typically don’t contain perceptrons. A feed-forward neural network consists of 3 types of nodes: input unit, hidden units, and output units. The bottom most layer of neurons takes in input data. The top layer of neurons computes the output. The middle layer(s) are also known as hidden layers.</p><p>The feed-forward neural network, as noted in the name, only moves forward from the bottom-most layer to the output layer. There is no loop from the output layer back to the bottom-most layer. Additionally, the neurons in the same layer don’t share connections. These are some properties that make feed-forward neural networks different and simpler than regular neural networks.</p><p>Typically, a feed-forward neural network has a non-linear activation function. We will learn more about types of activation functions below.</p><h3>Activation Functions</h3><p>Activation functions add non-linearity so stacks of neurons can model complex patterns.</p><p><strong>Sigmoid</strong></p><ul><li>Function: S(x) = 1 / [1 + e^-x]</li><li>Output: between 0 and 1</li><li>Pros: Historically popular, Gives an analog activation unlike step functions</li><li>Cons: Saturated neurons “kill” the gradient (known as vanishing gradients that’ll be covered in Part 2), Outputs are not zero-centered which can result in bias and make the gradient descent perform inefficiently, Computationally expensive due to the exponent</li></ul><p><strong>Tanh</strong></p><ul><li>Function: f(z) = tanh(z)</li><li>Output: between -1 to 1</li><li>Pros: Zero-centered, Good for S-shaped non-linearities, Stronger gradient</li><li>Cons: Vanishing gradient, Saturation, Computationally expensive due to the exponent</li></ul><p><strong>ReLu (Rectified Linear Unit)</strong></p><ul><li>Function: f(x) = max(0,x)</li><li>Output: 0 or x</li><li>Pros: Does not saturate, Very computationally efficient, Avoids vanishing gradient / exploding issues, Converges much faster than sigmoid or tanh, Yields 0 for negative numbers, Fast, not computationally expensive</li><li>Cons: Suffers from dying ReLu problem, where negative inputs lead to inactive neurons, Not zero-centered output</li></ul><p><strong>Leaky ReLu</strong></p><ul><li>Function: f(x) = max(0.01x, x)</li><li>Output: 0.01x or x</li><li>Pros: Does not saturate, Computationally efficient, Converges much faster than sigmoid or tanh, Has a small slope for negative numbers as opposed to being flat, Mitigates dying ReLu problem by adding a small gradient for negative numbers</li><li>Cons: Struggles with consistency for negative inputs, A bit more computationally expensive than ReLu</li></ul><p>Activation functions are the switch that makes deep networks powerful in practice, because they control non-linearity and strongly affect training stability and speed.</p><h3>Part 2</h3><p>With the building blocks in place, Part 2 will focus on how these neural networks actually learn from data through forward pass, backpropagation, and gradient descent.</p><h3>Bibliography</h3><p>[1] Speech and Language Processing by Jurafsky</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=972dd8188077" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Regular Expressions]]></title>
            <link>https://medium.com/@sonalprasad.sp/regular-expressions-9b4a5da65959?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b4a5da65959</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[regular-expressions]]></category>
            <category><![CDATA[pattern-matching]]></category>
            <category><![CDATA[eliza]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 16:10:45 GMT</pubDate>
            <atom:updated>2026-02-17T16:10:45.752Z</atom:updated>
            <content:encoded><![CDATA[<p>How were chatbots implemented before transformers came about?</p><h3>Things You’ll Learn</h3><ul><li>What regular expressions are, and why they matter in early NLP</li><li>Pattern matching</li><li>How early chatbots, such as ELIZA, used rules and patterns</li><li>Where regex still shows up today</li></ul><h3>Regular Expressions</h3><p>The very early premise of natural language processing was built upon regular expressions which allow for pattern matching. Regular expressions help you find patterns in text as opposed to finding exact keywords. Here are some examples of how to construct a regular expression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9T8mZL9knAfCqszMOl4VWA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2-5e13RZviMw5b3v4wpUQ.png" /><figcaption>The above images are credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>The following are some examples of regular expressions:</p><p>1. You may want to extract the date from a document:</p><ul><li>\d{2}-\d{2}-\d{4}</li><li>In the regular expression (regex) above, it looks for 2 digits (digits are referenced by “\d”), a dash, 2 digits, a dash, and then 4 digits.</li></ul><p>2. You may look for an email address in a document:</p><ul><li>^[a-zA-Z0–9._%+-]+@[a-zA-Z0–9.-]+\.[a-zA-Z]{2,}$</li><li>In the regex above, it is essentially as such: ____@___<strong>.</strong> There is an @ sign and there is a period before the last set of characters. The underlined regions refer to some or all of the following: letters, numbers, or special characters.</li></ul><p>3. You may search for a dollar value in a document:</p><ul><li>^\$?\d+(\.\d{2})?$</li><li>In the regex above, there is first a dollar sign ‘$’, then some digits, and then optionally afterward there can be a period and digits to indicate cents after the dollar value.</li></ul><h3>Chatbots</h3><p>In addition to regular expressions being used to search for strings with a certain pattern in a document, the chatbots from the early days also relied on regular expressions. Pattern matching can be used to recognize phrases from the user’s messages, and then provide relevant outputs.</p><p>In 1966, Weizenbaum created a chatbot called ELIZA that mimicked the responses of the psychologist Carl Rogers. The Rogerian style of mental health therapy involves reflecting the patient’s statements back at them. ELIZA can recognize a phrase like “I need X” and output “What would it mean to you if you got X?” ELIZA relies on rules, that are showcased below.</p><p>User Message 1:</p><p>You hate me</p><p>Rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*B8csS9zZEf9UFQbLWK8KJA.png" /></figure><p>ELIZA’s Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*svGTBCoNHjiITIvZuB0nIA.png" /></figure><p>Since ELIZA reflects dialogue back to the user, it takes the word that the user entered. “hate,” and reflects it back at the user. Note: Since ELIZA copies the words that the user says, the chatbot can reply with typos if that’s how the user typed.</p><p>User Message 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*3mXro9Gr9YyA1uhyujFbOA.png" /></figure><p>Rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*2b8Ooa11DGgecv-o_h49Xg.png" /></figure><p>ELIZA’s Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*IoQoD3TzL-w1nm1TVTCg0Q.png" /></figure><p>In the above example, ELIZA copies the phrase “know everybody laughed at” and substitutes it into the output response.</p><p><em>The above examples of user messages, rules, and outputs are taken from Speech and Language Processing by Jurafsky.</em></p><p>“An input sentence is scanned from left to right. Each word is looked up in a dictionary of keywords. If a word is identified as a keyword, then (apart from the issue of precedence of keywords) only decomposition rules containing that keyword need to be tried.” [1]</p><h3>Regular Expression Substitution</h3><p>In addition to regular expressions helping extract strings with a certain pattern, they also can manipulate or substitute strings based on rules. That is what ELIZA does in order to provide a response.</p><p>Here is a full conversation between a user and ELIZA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*5VlPlbUO5klJ5IJ-9vcyFw.png" /><figcaption>The above image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>ELIZA uses substitution to replace “I” with “you” or “my” with “your.” ELIZA repeats back to the user what the user is saying. The substitutions also involve ELIZA replacing the user’s message with another string based on a rule. Here are some examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wF_IcEtnQVOOFFXAYIt76g.png" /><figcaption>The above image is credited to Speech and Language Processing by Jurafsky.</figcaption></figure><p>In the above example, we can take a look at the first line. The user’s message is “He says I’m depressed much of the time.” Since the keyword is “depressed”, we go to the first rule in the above image, and we construct an output message by writing “I am sorry to hear you are depressed.”</p><h3>How Are Regular Expressions Used Today?</h3><p>Regular expressions are used today for guardrails and monitoring, such as identifying intense mental health situations. Regex can filter sensitive information, such as Personal Identifiable Information (e.g. credit card number, social security number). Regex can identify whether a topic outside of its scope is being discussed, such as a crisis, and indicate to the user to talk with healthcare professionals. Amazon Bedrock has guardrails that rely on regex to identify, block, or anonymize sensitive data.</p><p>Regular expressions can also be used for text processing, such as removing the formatting containing asterisks in a ChatGPT response, data validation, such as when making sure a user is registering an account with their username and password correctly, and for web scraping.</p><p>Even with the advent of Large Language Models, regular expressions are still being used regularly in the modern day. Regex was the basis for chatbots historically, but now we have shifted to more advanced architecture for chatbots and agents.</p><h3>Bibliography</h3><p>[1] <a href="https://courses.cs.umbc.edu/undergraduate/CMSC471/papers/weizenbaum.eliza.1966.pdf">https://courses.cs.umbc.edu/undergraduate/CMSC471/papers/weizenbaum.eliza.1966.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b4a5da65959" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Models]]></title>
            <link>https://medium.com/@sonalprasad.sp/language-models-3debed709737?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3debed709737</guid>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <category><![CDATA[markov-assumption]]></category>
            <category><![CDATA[laplace-smoothing]]></category>
            <category><![CDATA[ngrams]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 22:33:43 GMT</pubDate>
            <atom:updated>2026-02-10T22:33:43.569Z</atom:updated>
            <content:encoded><![CDATA[<p>How do language models generate words?</p><h3>What You’ll Learn</h3><ul><li>What is a language model</li><li>How language models assign probabilities to sequences</li><li>What are n-grams and the Markov assumption</li><li>Compute Maximum Likelihood Estimation (MLE)</li><li>Core fixes for sparsity (Laplace smoothing, interpolation, and backoff)</li></ul><h3>What are language models?</h3><p>Language models are used to predict the next word in a sequence. The language model guesses what the next word is most likely to be based on the previous sequence of words. For example,</p><ul><li>My dog is wagging his _____.</li></ul><p>You may guess that the next word is “tail.” The language model comes up with a probability distribution for the possible upcoming words.</p><p>Language models also assign probabilities to a sentence. For example,</p><ul><li>I love my dog → high probability</li><li>I my dog love → low probability</li></ul><p>Language models help generate text, can be used to correct spelling or grammatical errors, translate from one language to another, and can answer a question when it’s trained on question-answer pairs. This is the same basic problem behind autocomplete, speech-to-text, and modern LLMs, just with different modeling assumptions.</p><h3>How to Calculate the Probability of the Next Word</h3><p>One way to calculate the probability of the next word is to keep track of the frequency of words in a document. In practice, we do not just count how often a word appears overall. We count how often a word appears after a specific context. **For example, P(word | history), known as probability of word given history:</p><ul><li>P(field | My dog runs around in the green)</li></ul><p>Based on a corpus, we can count the number of times the sentence “My dog runs around in the park” appears. The word that has the highest frequency would be the selected word by the language model.</p><p>A second way to calculate the probability of the next word is to use the chain rule of probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*2rYa3DVhHY32qtMJxEm0Lw.png" /><figcaption>The formula above showcases the chain rule applied to words, or the joint probability. The image credit goes to Speech and Language Processing by Jurafsky.</figcaption></figure><p>We essentially compute the conditional probability of a word given a previous word in a sequence, and do that recursively for all words in the sequence. Multiplying the conditional probability of a word across a sequence of words is known as the joint probability. However, this approach is difficult because the sequence/sentence for which we are trying to calculate the probability might not exist in the corpus. Language is dynamic and creative, and not every sentence we may try to predict exists in a corpus or on the internet.</p><h3>N-Grams</h3><p>N-grams are the simplest type of language models. The term ‘n-gram’ refers to predicting the nth word based on a sequence of n-1 words.</p><ul><li>Unigrams (1-word) consist of one word, e.g. “the”, “dog”, “wants”, “to”, “play”</li><li>Bigrams (2-word) consists of two words, e.g. “the dog”, “dog wants”, “wants to”, “to play”</li><li>Trigrams (3-word) consists of three words, e.g. “the dog wants”, “dog wants to”, “wants to play”</li></ul><p>In bigrams, we predict the next word based on one word before it. This is based on the Markov assumption which states that the likelihood of the upcoming word only depends on the previous word in the sequence. Instead of using the chain rule applied to words for the following calculation:</p><ul><li>P(field | My dog runs around in the green)</li></ul><p>We approximate the full history with only the previous word (bigram):</p><ul><li>P(field | green)</li></ul><h4>Maximum Likelihood Estimation (MLE)</h4><p>We get the probabilities for bigrams, or n-grams, from the maximum likelihood estimation (MLE), which takes the counts from a corpus and normalizes it so that the probability is between 0 and 1. The formula is the following for bigrams (MLE):</p><ul><li>P(w_i | w_{i-1}) = count(w_{i-1}, w_i) / count(w_{i-1})</li></ul><p>For example, we have the following corpus:</p><p>&lt;s&gt; I like having a dog &lt;/s&gt;</p><p>&lt;s&gt; My dog is beautiful &lt;/s&gt;</p><p>&lt;s&gt; I like to play with my dog &lt;/s&gt;</p><p>We can notice the counts:</p><p>count(&lt;s&gt;, I) = 2</p><p>count(&lt;s&gt;, My) = 1</p><p>count(&lt;s&gt;) = 3</p><p>So, P(I | &lt;s&gt;) = count(&lt;s&gt;, I)/count(&lt;s&gt;) = 2/3 = 0.67</p><p>We can generate the MLE bigram estimation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u8bCLlr57206mW4FPVl3sA.png" /></figure><p>The probability of one of the above sentences can be computed as follows in a bigram model based on the chain rule of probability:</p><ul><li>P(I like having a dog) = P(I) * P(like | I) * P(having | like) * P(a | having) * P(dog | a)</li></ul><p>A trigram will try to predict the next likely word based on 2 words before it.</p><p>Real corpora are huge, but the number of possible word pairs is even larger, so many valid pairs never appear even in large datasets.</p><h4>Issues with MLE</h4><p>MLE doesn’t account for when sequences are missing from a corpus. Such sequences have a count of 0 in a corpus and, thus, a probability of 0. Additionally, having 0 occurrences of a sequence in a corpus underestimates the probability of what a word sequence could be. Thus, we introduce smoothing, interpolation, and backoff.</p><p><strong>Smoothing</strong></p><p>Laplace Smoothing adds 1 to all the counts in the n-gram before we get their probabilities. For example, here we have a frequency table for bigrams before applying Laplace Smoothing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uz2yTspzB3B6lLpHVWdNaA.png" /><figcaption>The image credit goes to Speech and Language Processing by Jurafsky.</figcaption></figure><p>Notice that many bigrams have a count of 0 in the above table.</p><p>In the following table, we apply Laplace Smoothing by adding 1 to all counts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mt7AFEYdUAiEssXdTrLE8w.png" /><figcaption>The image credit goes to Speech and Language Processing by Jurafsky.</figcaption></figure><p>In the above table, we can see all counts are greater than 0, and, thus have a probability greater than 0. However, Laplace Smoothing has performance issues in n-gram models, since it pushes too much probability mass to unseen events, which can make common events underweighted.</p><p><strong>Interpolation</strong></p><p>We use interpolation when we may not have all the words in, for example, a trigram. In that case, we can work with bigrams or unigrams. We estimate the trigram probability by summing the weighted unigram, bigram, and trigram probabilities. Here is the formula for interpolation of a trigram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*XQWeeQ43juf3GJCuWl-PMw.png" /><figcaption>The image credit goes to Speech and Language Processing by Jurafsky.</figcaption></figure><p>With interpolation, we can run into an issue of an n-gram, such as a trigram, having 0 counts. That’s where backoff comes in.</p><p><strong>Backoff</strong></p><p>We use backoff when the full context was never observed, so we fall back to shorter contexts. We backoff to the n-1-gram until we reach a sequence that has some counts. We apply a discount to the higher order n-grams so that the lower order n-grams have some probability mass. However, it can get tricky to work with discounting, and so, instead, we use “Stupid Backoff,” which doesn’t use discounting. If a trigram has a 0 count, then we shift to a bigram, for example. Stupid backoff uses a fixed weight. Stupid backoff stops once it reaches the unigram, if it gets to that point.</p><h3>Intrinsic Evaluation: Perplexity</h3><p>Perplexity evaluates a language model. Perplexity calculates the inverse probability of a test set, which is then normalized by the number of words. The formula is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*Okr0KJO5wGw8KDDkDJVO4g.png" /><figcaption>The image credit goes to Speech and Language Processing by Jurafsky.</figcaption></figure><p>The smaller the value of perplexity, the better the model is. You can think of perplexity as the effective number of choices the model is ‘confused between’ at each word.</p><p>N-grams capture local word patterns, but they struggle with meaning and long-range context, which shows up in broader NLP challenges.</p><h3>Why is Natural Language Processing (NLP) Challenging?</h3><p>Language is creative and there are syntactic/grammatical rules that make language more complex. Here are a few reasons NLP is challenging:</p><ol><li>Homonymy: A word has completely different meanings, e.g. “match” is a small stick used to start a fire, a competition or game, and a person who pairs well with another person.</li><li>Lexical Ambiguity: A word can have different roles in a sentence (noun, verb, adjective), e.g. “He saw the ghost hover” vs. “We saw a hover craft”</li><li>Coreference resolution: e.g. “Michelle told me Mary needed to go to the hospital so I took her to the ER on Stanford Avenue.” In this sentence, “her” references to “Mary”, which could be confusing to a computer.</li><li>Grammar: There are some disagreements on grammar where some say it’s grammatically correct and others say it’s not, e.g. “not to be invited is sad” vs “to be not invited is sad”</li><li>Vagueness: e.g. “That’s awesome!” The sentence doesn’t indicate what is awesome and precisely how awesome it is.</li></ol><h3>Final Thoughts</h3><p>Language models ultimately led to the creation of Large Language Models. Some of the challenges in NLP are addressed by Large Language Models. However, LLMs present their own challenges of hallucinations and stating harmful content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3debed709737" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Word Embeddings]]></title>
            <link>https://medium.com/@sonalprasad.sp/word-embeddings-0d7144948db6?source=rss-255ce29b8ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d7144948db6</guid>
            <category><![CDATA[embedding-model]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[word2vec]]></category>
            <dc:creator><![CDATA[Sonal Prasad]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 15:53:23 GMT</pubDate>
            <atom:updated>2026-02-10T15:53:23.051Z</atom:updated>
            <content:encoded><![CDATA[<p>How do we convert words into numbers during the embedding process?</p><h3>What You’ll Learn</h3><ul><li>What an embedding is and why we need it</li><li>Why and how we tokenize first</li><li>How embedding models converts tokens into vectors</li></ul><p>Computers can’t work directly with words. People instantly understand that “cat” and “dog” are related, but a computer sees them as unrelated symbols unless we do something. This is where embedding models come into the picture. Models need numbers as input, so we convert text into IDs and vectors. The vectors are known as an embedding. A model might represent the token ‘cat’ as an ID like 5312, then map it to a vector like [0.12, -0.03, …]. There are many available embedding models to embed text into vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*KZsTAoMNF-6R4GScl7NbTg.png" /><figcaption>This graph shows how the vectors for dog and cat are close to each other. Credit to Elastic for the image.</figcaption></figure><p>There are different ways embeddings are learned and used. Earlier approaches like Word2Vec learn word vectors directly. Modern transformers (like BERT) learn token embeddings as part of a larger model. Embedding models are used for search, clustering, recommendations, and semantic similarity.</p><h3>Semantic Similarity</h3><p>Advanced embedding models produce embeddings based on how similar words are to each other. An example of a distance metric is cosine similarity. The below diagram shows how words similar to each other are grouped based on cosine similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*5ktl4XFP4aBKjQCBgU1m5g.png" /><figcaption>The diagram above shows a hierarchy of words to visualize how words are embedded in space. Credit for the diagram goes to Speech and Language Processing by Jurafsky.</figcaption></figure><p>In order to find the similarity between words and produce embeddings, the embedding model starts by preprocessing the words to ultimately quantify them.</p><h3>Preprocessing Words Into Tokens</h3><h4>Text Normalization</h4><p>In text normalization we process the text in a standard way. For example, let’s take a look at these 2 sentences:</p><ul><li>“She likes to take her dog to the dog park!”</li><li>“she likes to take her dog to the dog park”</li></ul><p>Both of the sentences are the same but the second one is in all lowercase and doesn’t have punctuation. Some ways to perform text normalization include:</p><ul><li>Converting all characters to lowercase</li><li>Removing punctuation marks</li><li>Removing extra whitespace</li><li>Removing accent marks</li><li>Converting words to their stem/root word (lemmatization and stemming): “dancing”, “dances”, and “danced” can be converted to “dance”.</li></ul><h4>Tokenization</h4><p>In tokenization, we take a body of words and split them into smaller units called tokens. A token can be one word or part of a word. There are 3 different ways to split words into smaller units. Word tokenization converts the string of words (e.g. “It is sunny today.”) into individual words (e.g. [“It”, “is”, “sunny”, “today”]). Subword tokenization splits a word into common pieces (for example, ‘hyper’ + ‘parameter’). This helps the model handle new or rare words. Character tokenization converts text into a set of characters.</p><h4>Tokens to IDs</h4><p>Next, we take the tokens and convert them into IDs, or numerical values. There are two approaches for the conversion from tokens to IDs. One approach is working with a lookup table that contains a mapping of a token to an ID. Another approach is hashing where we input a token into a hashing function and output an ID.</p><h4>Bringing the Steps Together</h4><ol><li>Text: “She likes dogs.”</li><li>Normalized: “she likes dogs”</li><li>Tokens: [“she”, “likes”, “dogs”]</li><li>IDs: [1012, 3298, 3899, 2015]</li><li>Embeddings: each ID → a vector</li></ol><h3>Properties</h3><p>Once we have IDs, an embedding layer turns each ID into a dense vector the model can learn. In order to do so the embedding models produce properties or dimensions. For illustration purposes, people can come up with properties, as shown in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iltWlOC9VI_i52KvmQxfWA.png" /></figure><p>We can compare words just by their 1 or 0 list using distance metrics. If we were to plot these points in a 3D space, we could imagine how some words would be near other words, while others would be apart. We will later cover how similarity between words are calculated in embedding models. Note: the computer does not come up with properties in the fashion shown above; the table was made for your conceptual understanding.</p><h3>Embedding Models</h3><h4>Word2Vec</h4><p>Word2Vec is a dense embedding with 50–1000 dimensions. Sparse vectors (10s of thousands of dimensions) are mostly zeros, such as one-hot vectors. Dense vectors pack information into many non-zero values. Dense embeddings are better than sparse embeddings because having fewer weights is less likely to result in overfitting, and distance metrics work better with smaller dimensions. More dimensions can capture more nuance, but cost more compute and can overfit on small data.</p><h4>Word2Vec Using Skip-Gram With Negative Sampling</h4><p>Word2Vec with skip-gram and negative sampling learns a single, context-independent vector for each vocabulary token, so a token’s embedding stays the same regardless of the sentence it’s being trained on. During training, the model picks a target token and treats nearby tokens as positive examples. It then draws negative examples by sampling other tokens from the vocabulary. Doing so helps bring context to the embedding which is helpful for retrieval later on. By using logistic regression, we can classify between positive or negative samples. We, finally, use the learned weights as embeddings. Word2Vec is a simple and efficient embedding model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*5yUc2Hdf0rWs884Y-zNgBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*l94lDflC2998f9DllbM7wQ.png" /><figcaption>In the table above, we classify the positive examples (neighbors of target words) and negative examples from the sentence above. Credit to Speech and Language Processing by Jurafsky for the diagrams.</figcaption></figure><p>Word2Vec gives one vector per word. Transformers extend this by producing vectors that depend on surrounding words; there is a unique vector for a token each time it appears in a new sentence.</p><h4>BERT</h4><p>BERT uses a tokenizer with a fixed vocabulary (a list of tokens it knows). When text is tokenized, each token is matched to an ID from that vocabulary. A common version of BERT has 30,522 tokens in its vocabulary. BERT creates contextual embeddings, where positioning and context matter. For example, let’s take a look at these 2 sentences:</p><ul><li>I would like to purchase a bag</li><li>Can you bag the groceries?</li></ul><p>The word “bag” is used in two different ways. The same word means 2 completely different things. The model needs to have associations with the words. The vector for a token depends on the surrounding words, so the embedding for ‘bag’ will differ in the two sentences. Transformer-based models, like BERT, can assign different vector representations to the same word based on the surrounding context.</p><p>BERT (Bidirectional Encoder Representations from Transformers) has a transformer-based architecture. The base model of BERT represents each token with 768 properties. This means, for each token, there are 768 dimensions of information that capture the position and meaning of the token. These numbers are learned during training and together capture patterns about meaning and usage.</p><p>There were 2 tests that were used in BERT. In the Masked Language Modeling (MLM) Test, we provide BERT with a list of tokens and randomly remove one token giving it the stage to guess that token. In the Next Sentence Prediction Test (NST), BERT is given 2 sentences and must decide whether the second sentence should come after the first one. The base model of BERT uses 768-dimensional vectors internally. It also has an embedding lookup table with one vector per vocabulary token (30,522 tokens). During training, BERT learns these representations using MLM (and NSP in the original version).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0d7144948db6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>