<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sainiharreddy Palem on Medium]]></title>
        <description><![CDATA[Stories by Sainiharreddy Palem on Medium]]></description>
        <link>https://medium.com/@nihar-palem?source=rss-77905d842883------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7vBqf4W06I7yni-SdpoGJQ@2x.jpeg</url>
            <title>Stories by Sainiharreddy Palem on Medium</title>
            <link>https://medium.com/@nihar-palem?source=rss-77905d842883------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:46:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nihar-palem/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“Linear Attention” Breaking the O(N²) Curse]]></title>
            <link>https://nihar-palem.medium.com/linear-attention-breaking-the-o-n%C2%B2-curse-fa307ef37d42?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa307ef37d42</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Sun, 25 Jan 2026 05:09:18 GMT</pubDate>
            <atom:updated>2026-01-25T05:09:18.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bJp5v-8FCp8Taz5y6Ala2A.jpeg" /><figcaption>Image Generated by Gemini</figcaption></figure><p>In the world of Transformers, the “Attention” mechanism is both the secret sauce and the biggest bottleneck. As we move toward processing massive high-resolution images or entire books, the standard <strong>Softmax Attention</strong> hits a wall: memory usage grows quadratically with the sequence length.</p><p><strong>Linear Attention</strong> provides a mathematical escape hatch by changing how we handle matrix multiplication.</p><blockquote>“Linear Attention doesn’t make the model ‘smaller’ in terms of file size; it makes the model <strong>‘lighter’</strong> in terms of execution. It swaps a massive, sequence-dependent bottleneck for a tiny, constant-sized one.”</blockquote><h4>1. The Core Problem: The O(N²) Softmax Bottleneck</h4><p>In standard attention, we compute a similarity score between every pair of tokens. If you have a sequence of 10,000 tokens (or pixels), the model creates a <strong>10,000 × 10,000</strong> matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*7NNDh64vmqBWKxq--nvCPw.png" /></figure><p>The <strong>Softmax</strong> is the culprit. Because it’s a non-linear operation applied to the QK^T product, it “binds” the queries and keys together. You <strong>must</strong> materialize that giant N *N matrix before you can multiply it by the Values (V).</p><h4>2. The Linear Trick: Associative Property</h4><p>If we replace Softmax with a linear feature map <strong><em>phi</em></strong>, we can use the <strong>associative property</strong> of matrix multiplication:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*IhVoO0OaHVfVJBGO9kGrGg.png" /></figure><ul><li><strong>Standard Attention:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*DOb9Z1UiqqeOOkXSya70tw.png" /></figure><ul><li><strong>Linear Attention:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*MgxJz3afOrRAfOx4Dj8i_Q.png" /></figure><p>In almost all modern models, the feature dimension <strong>d</strong> (e.g., 64 or 128) is much smaller than the sequence length <strong>N</strong> (e.g., 10,000). By simply moving the parentheses, we transform the complexity from <strong>quadratic</strong> to <strong>linear</strong>.</p><h4>3. The “Reality Check”: Pros, Cons, and Fixes</h4><p>While computationally elegant, Linear Attention isn’t a free lunch. Here is why we don’t use it for everything:</p><h4>The Stability Issue (No Softmax)</h4><p>Softmax ensures all scores are positive and sum to 1. Without it, the output can explode.</p><ul><li><strong>The Fix:</strong> We use a feature map like <strong>ReLU</strong> or <strong>ELU+1</strong> to ensure positivity, and we manually normalize the output. We add a small <strong><em>epsilon</em></strong> (like <strong>1e-6</strong>) to avoid division by zero.</li></ul><h4>4. The “Blurring” Problem</h4><p>Softmax is “sharp” — it can focus on one single word. Linear kernels are “blurry” — they tend to average information.</p><ul><li><strong>The Fix:</strong> Modern architectures like <strong>LiT (Linear Diffusion Transformer)</strong> often add <strong>Depth-wise Convolutions</strong> to help the model capture local spatial details that the “global” linear attention might miss.</li></ul><h4>5. The Implementation</h4><p>Both mechanisms use the <strong>exact same weight matrices</strong>. The difference lies entirely in the forward pass logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQZJasLqO_MfQ7L-cSPtLw.png" /></figure><h4>6. PyTorch Code Comparison</h4><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import time<br><br>class StandardSoftmaxAttention(nn.Module):<br>    &quot;&quot;&quot;Standard O(N²) Softmax Attention with explicit shape tracking.&quot;&quot;&quot;<br>    <br>    def __init__(self, dim, n_heads=1):<br>        super().__init__()<br>        self.dim = dim<br>        self.n_heads = n_heads<br>        self.head_dim = dim // n_heads<br>        self.scale = self.head_dim ** -0.5<br>        <br>        self.qkv = nn.Linear(dim, dim * 3, bias=False)<br>        self.proj = nn.Linear(dim, dim, bias=False)<br>        <br>    def forward(self, x, verbose=False):<br>        &quot;&quot;&quot;<br>        Args:<br>            x: [B, N, dim] - input features<br>        Returns:<br>            out: [B, N, dim] - attention output<br>        &quot;&quot;&quot;<br>        B, N, C = x.shape<br>        <br>        # Step 1: Project to Q, K, V<br>        qkv = self.qkv(x)  # [B, N, 3*dim]<br>        q, k, v = qkv.chunk(3, dim=-1)  # Each: [B, N, dim]<br>        <br>        if verbose:<br>            print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>            print(&quot;STANDARD SOFTMAX ATTENTION&quot;)<br>            print(f&quot;{&#39;=&#39;*60}&quot;)<br>            print(f&quot;Input shape: {list(x.shape)}&quot;)<br>            print(f&quot;Q, K, V shapes: {list(q.shape)}&quot;)<br>        <br>        # Step 2: Split into multiple heads (if n_heads &gt; 1)<br>        if self.n_heads &gt; 1:<br>            q = q.reshape(B, N, self.n_heads, self.head_dim).transpose(1, 2)<br>            k = k.reshape(B, N, self.n_heads, self.head_dim).transpose(1, 2)<br>            v = v.reshape(B, N, self.n_heads, self.head_dim).transpose(1, 2)<br>            # Now: [B, n_heads, N, head_dim]<br>            <br>            if verbose:<br>                print(f&quot;After head split: {list(q.shape)}&quot;)<br>        <br>        # Step 3: Compute attention scores (THE O(N²) BOTTLENECK!)<br>        attn_scores = q @ k.transpose(-2, -1)  # [B, n_heads, N, N] ⚠️<br>        <br>        if verbose:<br>            print(f&quot;\n&gt;&gt;&gt; BOTTLENECK: Attention Matrix&quot;)<br>            print(f&quot;Q @ K.T shape: {list(attn_scores.shape)}&quot;)<br>            print(f&quot;Memory for attention matrix: {attn_scores.numel() * 4 / 1024:.2f} KB&quot;)<br>            print(f&quot;Elements per head: {N} × {N} = {N*N:,}&quot;)<br>        <br>        # Step 4: Scale and softmax<br>        attn_scores = attn_scores * self.scale<br>        attn_weights = torch.softmax(attn_scores, dim=-1)  # [B, n_heads, N, N]<br>        <br>        # Step 5: Apply attention to values<br>        out = attn_weights @ v  # [B, n_heads, N, head_dim]<br>        <br>        # Step 6: Concatenate heads and project<br>        if self.n_heads &gt; 1:<br>            out = out.transpose(1, 2).reshape(B, N, C)<br>        <br>        out = self.proj(out)<br>        <br>        if verbose:<br>            print(f&quot;Output shape: {list(out.shape)}&quot;)<br>            print(f&quot;{&#39;=&#39;*60}\n&quot;)<br>        <br>        return out<br><br><br>class LinearAttention(nn.Module):<br>    &quot;&quot;&quot;Linear O(N) Attention using kernel trick.&quot;&quot;&quot;<br>    <br>    def __init__(self, dim, n_heads=1, eps=1e-6):<br>        super().__init__()<br>        self.dim = dim<br>        self.n_heads = n_heads<br>        self.head_dim = dim // n_heads<br>        self.eps = eps<br>        <br>        self.qkv = nn.Linear(dim, dim * 3, bias=False)<br>        self.proj = nn.Linear(dim, dim, bias=False)<br>        <br>    def forward(self, x, verbose=False):<br>        &quot;&quot;&quot;<br>        Args:<br>            x: [B, N, dim] - input features<br>        Returns:<br>            out: [B, N, dim] - attention output<br>        &quot;&quot;&quot;<br>        B, N, C = x.shape<br>        <br>        # Step 1: Project to Q, K, V<br>        qkv = self.qkv(x)  # [B, N, 3*dim]<br>        q, k, v = qkv.chunk(3, dim=-1)  # Each: [B, N, dim]<br>        <br>        if verbose:<br>            print(f&quot;\n{&#39;=&#39;*60}&quot;)<br>            print(&quot;LINEAR ATTENTION&quot;)<br>            print(f&quot;{&#39;=&#39;*60}&quot;)<br>            print(f&quot;Input shape: {list(x.shape)}&quot;)<br>            print(f&quot;Q, K, V shapes: {list(q.shape)}&quot;)<br>        <br>        # Step 2: Apply kernel function (make positive for stability)<br>        q = F.elu(q) + 1  # or F.relu(q)<br>        k = F.elu(k) + 1<br>        <br>        if verbose:<br>            print(f&quot;After kernel function (ELU+1): positive values ensured&quot;)<br>        <br>        # Step 3: Split into multiple heads (if n_heads &gt; 1)<br>        if self.n_heads &gt; 1:<br>            q = q.reshape(B, N, self.n_heads, self.head_dim).transpose(1, 2)<br>            k = k.reshape(B, N, self.n_heads, self.head_dim).transpose(1, 2)<br>            v = v.reshape(B, N, self.n_heads, self.head_dim).transpose(1, 2)<br>            # Now: [B, n_heads, N, head_dim]<br>            <br>            if verbose:<br>                print(f&quot;After head split: {list(q.shape)}&quot;)<br>        <br>        # Step 4: THE KEY TRICK - Compute K.T @ V first!<br>        k_t = k.transpose(-2, -1)  # [B, n_heads, head_dim, N]<br>        <br>        context = k_t @ v  # [B, n_heads, head_dim, head_dim] ✅ SMALL!<br>        <br>        if verbose:<br>            print(f&quot;\n&gt;&gt;&gt; LINEAR TRICK: K.T @ V&quot;)<br>            print(f&quot;K.T shape: {list(k_t.shape)}&quot;)<br>            print(f&quot;V shape: {list(v.shape)}&quot;)<br>            print(f&quot;Context (K.T @ V) shape: {list(context.shape)}&quot;)<br>            print(f&quot;Memory for context: {context.numel() * 4 / 1024:.2f} KB&quot;)<br>            print(f&quot;Elements per head: {self.head_dim} × {self.head_dim} = {self.head_dim**2:,}&quot;)<br>        <br>        # Step 5: Q @ (K.T @ V)<br>        out = q @ context  # [B, n_heads, N, head_dim]<br>        <br>        # Step 6: Normalization (prevent unbounded outputs)<br>        # Compute denominator: sum of attention weights per query<br>        k_sum = k.sum(dim=-2, keepdim=True)  # [B, n_heads, 1, head_dim]<br>        normalizer = (q @ k_sum.transpose(-2, -1)) + self.eps  # [B, n_heads, N, 1]<br>        out = out / normalizer  # Normalize<br>        <br>        if verbose:<br>            print(f&quot;After normalization shape: {list(out.shape)}&quot;)<br>        <br>        # Step 7: Concatenate heads and project<br>        if self.n_heads &gt; 1:<br>            out = out.transpose(1, 2).reshape(B, N, C)<br>        <br>        out = self.proj(out)<br>        <br>        if verbose:<br>            print(f&quot;Output shape: {list(out.shape)}&quot;)<br>            print(f&quot;{&#39;=&#39;*60}\n&quot;)<br>        <br>        return out<br><br><br>def benchmark_attention(seq_lengths=[64, 128, 256, 512, 1024], dim=256, n_heads=4):<br>    &quot;&quot;&quot;Benchmark standard vs linear attention at different sequence lengths.&quot;&quot;&quot;<br>    <br>    print(f&quot;\n{&#39;=&#39;*80}&quot;)<br>    print(f&quot;ATTENTION BENCHMARK (dim={dim}, n_heads={n_heads})&quot;)<br>    print(f&quot;{&#39;=&#39;*80}&quot;)<br>    print(f&quot;{&#39;Seq Length&#39;:&lt;12} {&#39;Standard (ms)&#39;:&lt;15} {&#39;Linear (ms)&#39;:&lt;15} {&#39;Speedup&#39;:&lt;10} {&#39;Mem Ratio&#39;:&lt;10}&quot;)<br>    print(f&quot;{&#39;-&#39;*80}&quot;)<br>    <br>    device = &#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;<br>    <br>    for N in seq_lengths:<br>        batch_size = 16<br>        x = torch.randn(batch_size, N, dim, device=device)<br>        <br>        std_attn = StandardSoftmaxAttention(dim, n_heads).to(device)<br>        lin_attn = LinearAttention(dim, n_heads).to(device)<br>        <br>        # Warmup<br>        with torch.no_grad():<br>            _ = std_attn(x)<br>            _ = lin_attn(x)<br>        <br>        # Benchmark standard attention<br>        torch.cuda.synchronize() if device == &#39;cuda&#39; else None<br>        start = time.time()<br>        with torch.no_grad():<br>            for _ in range(100):<br>                _ = std_attn(x)<br>        torch.cuda.synchronize() if device == &#39;cuda&#39; else None<br>        std_time = (time.time() - start) / 100 * 1000  # ms<br>        <br>        # Benchmark linear attention<br>        torch.cuda.synchronize() if device == &#39;cuda&#39; else None<br>        start = time.time()<br>        with torch.no_grad():<br>            for _ in range(100):<br>                _ = lin_attn(x)<br>        torch.cuda.synchronize() if device == &#39;cuda&#39; else None<br>        lin_time = (time.time() - start) / 100 * 1000  # ms<br>        <br>        # Memory comparison (attention matrix size ratio)<br>        head_dim = dim // n_heads<br>        std_mem = N * N  # Standard attention matrix size<br>        lin_mem = head_dim * head_dim  # Linear attention context size<br>        mem_ratio = std_mem / lin_mem<br>        <br>        speedup = std_time / lin_time<br>        <br>        print(f&quot;{N:&lt;12} {std_time:&lt;15.3f} {lin_time:&lt;15.3f} {speedup:&lt;10.2f}x {mem_ratio:&lt;10.1f}x&quot;)<br>    <br>    print(f&quot;{&#39;=&#39;*80}\n&quot;)<br><br><br>def visualize_complexity():<br>    &quot;&quot;&quot;Show complexity growth for different sequence lengths.&quot;&quot;&quot;<br>    <br>    print(f&quot;\n{&#39;=&#39;*80}&quot;)<br>    print(&quot;COMPLEXITY COMPARISON: Standard O(N²) vs Linear O(N×d)&quot;)<br>    print(f&quot;{&#39;=&#39;*80}&quot;)<br>    print(f&quot;{&#39;N&#39;:&lt;10} {&#39;Standard O(N²)&#39;:&lt;20} {&#39;Linear O(N×d)&#39;:&lt;20} {&#39;Ratio&#39;:&lt;10}&quot;)<br>    print(f&quot;{&#39;-&#39;*80}&quot;)<br>    <br>    d = 64  # head_dim<br>    <br>    for N in [64, 128, 256, 512, 1024, 2048, 4096]:<br>        std_ops = N * N * d<br>        lin_ops = N * d * d<br>        ratio = std_ops / lin_ops<br>        <br>        print(f&quot;{N:&lt;10} {std_ops:&lt;20,} {lin_ops:&lt;20,} {ratio:&lt;10.1f}x&quot;)<br>    <br>    print(f&quot;{&#39;=&#39;*80}\n&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    # Example 1: Detailed walkthrough with shapes<br>    print(&quot;\n&quot; + &quot;=&quot;*80)<br>    print(&quot;EXAMPLE 1: Detailed Shape Tracking&quot;)<br>    print(&quot;=&quot;*80)<br>    <br>    batch_size = 2<br>    seq_len = 256<br>    dim = 256<br>    n_heads = 4<br>    <br>    x = torch.randn(batch_size, seq_len, dim)<br>    <br>    std_attn = StandardSoftmaxAttention(dim, n_heads)<br>    lin_attn = LinearAttention(dim, n_heads)<br>    <br>    # Forward pass with verbose output<br>    _ = std_attn(x, verbose=True)<br>    _ = lin_attn(x, verbose=True)<br>    <br>    # Example 2: Parameter count<br>    print(&quot;\n&quot; + &quot;=&quot;*80)<br>    print(&quot;EXAMPLE 2: Parameter Comparison&quot;)<br>    print(&quot;=&quot;*80)<br>    std_params = sum(p.numel() for p in std_attn.parameters())<br>    lin_params = sum(p.numel() for p in lin_attn.parameters())<br>    <br>    print(f&quot;Standard Attention Parameters: {std_params:,}&quot;)<br>    print(f&quot;Linear Attention Parameters:   {lin_params:,}&quot;)<br>    print(f&quot;Same parameters! ✓ (Both use same projections)&quot;)<br>    <br>    # Example 3: Complexity visualization<br>    visualize_complexity()<br>    <br>    # Example 4: Runtime benchmark<br>    if torch.cuda.is_available():<br>        benchmark_attention()<br>    else:<br>        print(&quot;\n⚠️  CUDA not available - skipping benchmark&quot;)<br>        print(&quot;For meaningful benchmarks, run on GPU&quot;)</pre><ul><li>Standard: matrix size grows with <strong>sequence length</strong> (N)</li><li>Linear: matrix size grows with <strong>feature dimension</strong> (head_dim)</li></ul><h3>Further Reading</h3><ul><li><a href="https://arxiv.org/abs/2006.16236"><strong>Transformers are RNNs (2020)</strong></a><strong>:</strong> The foundational paper by Katharopoulos et al.</li><li><a href="https://arxiv.org/abs/2501.12976"><strong><em>LiT: Linear Diffusion Transformer (2024):</em></strong></a><em> High-res image generation using these principles.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa307ef37d42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM + LSTM = LMM (Large Memory models)]]></title>
            <link>https://nihar-palem.medium.com/llm-lstm-lmm-large-memory-models-f4325a4f562d?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4325a4f562d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 23:07:52 GMT</pubDate>
            <atom:updated>2025-02-28T23:07:52.764Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y6IkKRT9Qsz4QpqL.jpeg" /><figcaption>Image generated by: GEMINI</figcaption></figure><p>I was curious as current LLM’s are doing great doing in understanding sequence of texts for predicting or answering the questions, But <strong>what if</strong> we give it a big book of say 10000 pages to read considering an average of tokens 100,000+, Given this large context LLM can hallucinate because of its limit on context window which is basically the size of the Q,K,V a model can take. So, curiously digging on how to handle long reasoning i found a paper called “Large Memory Models”.</p><p>This paper talks about handling the long context reasoning by adding a memory bank near attention layer and performing the same way how an LSTM handles sequence of data (gating mechanism)</p><blockquote><strong><em>PS: They never mentioned LSTM in their paper but as the attention and memory bank arch looks similar to LSTM drawing comparison with it (LSTM)</em></strong></blockquote><p>for example, refer the arch of LSTM below and LMM (Large Memory models)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u061crMgQEEohpUA.png" /><figcaption>LSTM arch</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BXJAKkDjhJBWG-Wj.png" /><figcaption>LMM arch</figcaption></figure><h3>Architecture</h3><p>The architecture of the LMM near a attention layer looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*S89AgTIDtY-_3ZAW.png" /></figure><blockquote><em>As we know that the attention is between 3 matrices Q (query) ,K (key) ,V (value) and we apply them from the same input data and we update accordingly</em></blockquote><p>For understanding of what is about the Q,K,V (<a href="https://niharpalem.substack.com/p/understanding-attention-mechanisms?r=37q16z">check this article</a>)</p><p>Coming back to the LMM, here we we Create/ Update the Query (Q) matrix from the data (<strong>training data</strong>) and the Key (K) and Value (V) matrices are taken/ Updated to the <strong>memory bank</strong></p><p><strong>Memory bank: </strong>Each memory bank has <strong>N slots</strong> and each slot has a <strong>hidden dimension</strong> (identity matrix) details of these are mentioned later in the article</p><h3>Working of Memory module</h3><h4>The memory module works in 2 stages:</h4><p><strong>Stage 1: Memory information flow</strong></p><ul><li>As said earlier the Q,K,V matrices are taken from data (Q) and memory bank (K,V)</li><li>The below equation shows how the matrices are being calculated</li><li>Q (query): It takes embeddings from tokens (data) and does matrix multiplication with the weights matrix</li><li>K (key): It takes the embeddings or values for the Memory bank and does the same matrix multiplication with its weight matrix</li><li>V (value): It takes the embeddings or values for the Memory bank and does the same matrix multiplication with its weight matrix</li></ul><p><strong>Simple analogy: </strong><em>Think of like for the question (query) raised based on data search for information from the memory book (key and value)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*tu26EtrFMMCTbSrP.png" /></figure><ul><li>from the below diagram if we focus on selected area</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*b28tG_R5NKjaiEyC.png" /></figure><ul><li>So, focusing of the attention is being updated between heads, The below equation shows the mathematical calculation of it</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*gRJa_4WpD2zqIbBl.png" /></figure><ul><li>The Attention scores to the next layer (attention) will be current attention scores + gated attention scores, The below 2 equations explains how the Attention scores of gated are being calculated.</li><li>Here the <strong>E_attn</strong> is the regular attention calculation and <strong>E_gated</strong> is the new addition of attention scores of the gated</li><li>The E_attention is passed as we have a skip through connection (check the below diagram) (highlighted part)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*Q90xUT3d9spNKsL2.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*Kr9KHweZrmxCUIMH.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/0*F7VulWdYgp0_UWX5.png" /></figure><p><strong>Stage 2: Memory Update</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9bQ85yJnY1KhicbT.png" /></figure><p><strong>The memory update has 3 gate:</strong></p><p>Output, input and forget</p><p><strong><em>This is where we draw comparison with the LSTM as LSTM has similar arch (forget, input and output gates and memory updates)</em></strong></p><blockquote><em>Here M_t is the Memory and E_t is the attention scores</em></blockquote><p>The below equations are how gating works and memory gets updated</p><p><strong>Input gate:</strong></p><p>The model decides how much of the newly computed embeddings (E_mem) to incorporate into the memory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*czzczgOp7xa3W2nf.png" /></figure><p><strong>Forgetting Phase:</strong> Once new information is made available during the input phase, the memory must also decide which parts of its existing content to discard. This is governed by the forget gate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/0*PaEg9zDBoi-MOrAw.png" /></figure><p><strong>Memory Update:</strong> Combining these two gating mechanisms leads to the updated memory state</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*Mhp6sL_Bq1Lf7ua-.png" /></figure><h3>Incorporation of Memory to Transformer Arch</h3><ul><li>They’ve used decoder part from the transformer and</li><li>having 16 decoder blocks each with a dimension of 2048</li><li>Feed forward network dimension size being 8192</li><li>Their Model has 32 attention heads</li></ul><p>Memory bank:</p><ul><li>each memory bank has 2048 slots and each slot is of dimension (2048*2048)</li><li>Each memory bank is added to each of the 16 decoder blocks (16 memory banks)</li></ul><h3>Performance results</h3><ul><li>As already mentioned this model arch is good at handling long context windows</li><li>Also performed better in: <strong>Multi Hop Inference</strong></li></ul><p><strong>Multi Hop Inference</strong></p><p>think of multi hop inference as drawing to final answer in steps rather directly</p><pre>Multi-hop inference is like solving a puzzle where you need multiple clues. <br>Example: To answer &#39;What&#39;s John&#39;s dog&#39;s name?&#39;, <br>you might need to know:  <br>1. John has a pet <br>2. John&#39;s pet is a dog <br>3. The dog&#39;s name is Spot  <br>You combine these facts to reach the answer: Spot.</pre><h3>Previous works</h3><p>Apart from LMM before this the other way of handling long context reasoning were use of Recurrent Prompts</p><h4>Recurrent prompts</h4><p>This is like giving the AI a short note to remember important things. For example, imagine reading a long story and writing down key points on sticky notes. These notes (prompts) are passed from one part of the AI to the next, helping it remember important information.</p><pre>Example: While reading a story about a family vacation, <br>the AI might create prompts like &quot;Family of 4&quot; or &quot;Visiting Hawaii&quot; to <br>help it remember key details.<br><br>1. Summarizing previous answers:This means the AI takes its previous <br>responses and makes them shorter, using these summaries as new prompts. <br>However, this method doesn&#39;t fully handle very long texts.<br>    <br>Example: Imagine answering questions about a 100-page book. <br>After each question, the AI might summarize its answer in a sentence. <br>For the next question, it uses this summary instead of rereading the whole <br>book. This works for some questions but might miss details from the original <br>long text.</pre><p>But, this suffers from maintaining long context reasoning.</p><blockquote><strong>Thanks for reading till end :)</strong></blockquote><h3>References:</h3><p>paper link and (formula screenshots): <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">https://arxiv.org/pdf/2502.06049</a></p><p>LSTM arch image taken from: <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">https://en.wikipedia.org/wiki/Long_short-term_memory</a></p><p><a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4325a4f562d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Job Easz]]></title>
            <link>https://nihar-palem.medium.com/job-easz-9d47ef093ec1?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d47ef093ec1</guid>
            <category><![CDATA[parallel-processing]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Sun, 23 Feb 2025 00:40:49 GMT</pubDate>
            <atom:updated>2025-02-23T00:40:49.385Z</atom:updated>
            <content:encoded><![CDATA[<h3>data engineering breakdown</h3><p>This article provides a comprehensive explanation of the job scraping workflow designed in Apache Airflow, running within a Docker ecosystem. The workflow is structured to efficiently scrape job postings, process the data, and upload it to a Hugging Face dataset. It integrates PostgreSQL for managing metadata and logging, ensuring robust task tracking and management. Here’s a step-by-step breakdown of the entire process, from environment setup to the final upload and cleanup.</p><figure><img alt="High level overview of the data pipeline" src="https://cdn-images-1.medium.com/max/1024/0*GD3esOhMlRMpmPs6.png" /><figcaption>High level overview of the data pipeline</figcaption></figure><h3>1. Docker Environment and Container Setup</h3><p>The Airflow job scraper workflow operates within a Docker environment. A dedicated container, hosts the Airflow instance and is configured to interact with a PostgreSQL container, which plays a key role in storing metadata and logs. The Docker container is equipped with specific settings that facilitate the execution of the Airflow tasks while maintaining efficient resource management.</p><h3>Key Container Configuration:</h3><ul><li><strong>Resource Management:</strong> The container is limited to a maximum of 2 CPU cores and 4GB of memory, with resource reservations in place to ensure stable performance.</li><li><strong>Volumes:</strong> The container mounts essential volumes, including directories for storing DAG Python scripts, execution logs, plugins, and temporary files. Temporary files are used to store intermediate data, such as job listings scraped during the execution of the tasks.</li><li><strong>Environment Variables:</strong> The container loads environment variables from a configuration file, which contains sensitive information like API tokens and usernames, along with settings specific to Airflow, such as parallelism limits and task execution timeouts.</li><li><strong>PostgreSQL Database:</strong> Airflow communicates with the PostgreSQL container for storing internal metadata, logging task results, and tracking execution states.</li></ul><p>The Docker configuration ensures that the system runs in a controlled and scalable environment while allowing Airflow and PostgreSQL to function in tandem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lYm3D5_yo9qC05nc.png" /><figcaption><em>Diagram illustrating Docker container configuration, mounted volumes, resource management, and connection to PostgreSQL</em></figcaption></figure><h3>2. DAG Workflow Execution</h3><p>The workflow orchestrates the scraping, processing, and uploading of job data using Apache Airflow. The core of the workflow is a Directed Acyclic Graph (DAG) that defines a series of tasks to be executed in a specific order. The tasks are responsible for scraping job listings from various job boards, processing the collected data, and uploading it to a dataset on Hugging Face. Each stage in the workflow is carefully designed to ensure efficient parallel execution, data integrity, and retry mechanisms.</p><h3>a. Setting Up the Environment</h3><p>Before any tasks are executed, the system prepares the runtime environment. It first loads essential configurations, such as job titles and geographical locations. These configurations serve as input for generating scraping tasks, ensuring that the job scraping process is both diverse and comprehensive.</p><p>In addition, the environment checks for the presence of required credentials, such as API tokens, ensuring that the system has the necessary access to external services like the Hugging Face API. The system also verifies the structure of directories for temporary file storage, confirming that the necessary paths exist for data processing.</p><h3>b. Defining the Scraping Process</h3><p>Once the environment is prepared, the workflow initiates the scraping process. The system divides the scraping task into smaller, parallelizable units, allowing it to scrape job listings for different job titles and geographical locations concurrently.</p><p>The scraping task involves collecting job postings from multiple job boards, such as Indeed and LinkedIn. To ensure efficient use of resources, the system employs multiple threads that work in parallel, each dedicated to scraping data from a specific location. This approach reduces the overall time required for the scraping process by leveraging parallel processing.</p><p>During the scraping process, the system collects data asynchronously to handle rate limits imposed by external job boards. A brief pause is introduced between scraping attempts to prevent overwhelming the target sites.</p><blockquote><strong><em>Difference Between Threads and Cores</em></strong></blockquote><blockquote><strong><em>Cores</em></strong><em>: Physical hardware components within a CPU that can execute instructions independently.</em></blockquote><blockquote><strong><em>Threads</em></strong><em>: Software constructs that allow concurrent execution of code.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h1y1AammDhIFQibv.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NBMr9rMlRi_N27Sd.png" /><figcaption><em>Diagram showing parallel scraping tasks for each location, executed concurrently.</em></figcaption></figure><h3>c. Data Processing and Cleaning</h3><p>After scraping the job data, the next stage in the workflow is data processing. This phase involves cleaning and organizing the raw data to make it suitable for uploading to the Hugging Face dataset.</p><p>First, the system filters and structures the data, focusing on essential fields such as job title, company, location, and posting date. Additional information, such as the timestamp of when the data was scraped, is also added to each record. This timestamp helps track the freshness of the data and ensures that each entry is associated with its corresponding scraping session.</p><p>The cleaned data is then saved to temporary files in a lightweight, efficient format, such as Feather. These files are stored in a temporary directory, ready for the next stage in the workflow: uploading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/0*ijxya1GaOgefJa1S.png" /><figcaption><em>Flowchart illustrating data cleaning, timestamping, and saving to temporary files.</em></figcaption></figure><h3>d. Uploading Data to Hugging Face</h3><p>The next task in the workflow involves uploading the processed job data to a dataset on Hugging Face. This is done in a sequential manner, where each temporary file containing job data is uploaded one at a time.</p><p>The upload task is designed with resilience in mind. If an upload fails, the system employs a retry mechanism that waits for increasing intervals before attempting the upload again. The first retry occurs after 120 seconds, followed by subsequent retries at 240 seconds and 480 seconds. If all retry attempts fail, the system logs the error and skips the file, ensuring that the workflow continues with the remaining data.</p><p>Once a file is successfully uploaded, the system logs the successful upload event and moves on to the next file. After all files have been processed, the system cleans up by deleting the temporary files, ensuring that there is no leftover data that could clutter the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/0*rQgRyVSHAN_aAORP.png" /><figcaption><em>Diagram showing the sequential upload process, retry mechanism, and cleanup of temporary files.</em></figcaption></figure><h3>e. DAG Task Dependencies</h3><p>The Airflow DAG defines a clear structure of task dependencies. The workflow starts with an initial task that sets up the environment. Each job title has corresponding tasks for scraping and uploading data. These tasks are arranged so that the upload task depends on the successful completion of the scraping task for each job title. The DAG also includes a dummy task at the end to mark the completion of the workflow.</p><p>The DAG is scheduled to run daily, ensuring that the job scraping process is automated and runs at a fixed time each day. Task dependencies ensure that tasks are executed in the correct order, and retry policies are applied where necessary.</p><h3>3. Role of PostgreSQL in Airflow</h3><p>PostgreSQL plays a crucial role in supporting the Airflow workflow by storing metadata, logs, and configuration details. The database serves as the backend for Airflow, managing the state of tasks, logging execution details, and ensuring that the system operates smoothly.</p><h3>Key Uses of PostgreSQL in the Workflow:</h3><ul><li><strong>Metadata Storage:</strong> Airflow uses PostgreSQL to store details about DAGs, task instances, and their execution history. This allows Airflow to track the progress of each task and the overall workflow.</li><li><strong>Logging and Monitoring:</strong> The system stores logs related to task execution, errors, and performance in PostgreSQL. These logs provide valuable insights into the operation of the workflow, helping with troubleshooting and optimization.</li><li><strong>User Management and Security:</strong> PostgreSQL handles user authentication and authorization, ensuring that only authorized individuals can access and modify the workflow.</li><li><strong>Persistent Configuration:</strong> The database also stores configuration settings, ensuring that the web server, scheduler, and workers are synchronized and operate according to the defined parameters.</li></ul><p>PostgreSQL’s role is integral to the overall functionality of Airflow, providing a reliable and persistent backend for task management and system monitoring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HfL8h3rE8vUc7WXb.png" /></figure><h3>Conclusion</h3><p>This job scraping workflow in Airflow demonstrates a highly efficient, parallelized process for scraping job data, processing it, and uploading it to Hugging Face. By leveraging Docker and PostgreSQL, the system ensures that each stage of the workflow is executed reliably and at scale. The use of parallel scraping, data validation, and robust upload mechanisms guarantees both efficiency and data integrity. The final cleanup steps ensure that the environment remains clean, preventing unnecessary clutter and maintaining system performance.</p><p>The integration of Docker, Airflow, and PostgreSQL results in a scalable and fault-tolerant system capable of automating job data scraping on a daily basis, ensuring the continuous growth and accuracy of the Hugging Face dataset.</p><p><a href="https://huggingface.co/spaces/Niharmahesh/job_easz">Jobeasz application link</a></p><p><a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">Connect with me on LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d47ef093ec1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Attention Mechanisms]]></title>
            <link>https://nihar-palem.medium.com/understanding-attention-mechanisms-3e170de3eae8?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e170de3eae8</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Sat, 15 Feb 2025 20:40:06 GMT</pubDate>
            <atom:updated>2025-02-15T20:40:06.022Z</atom:updated>
            <content:encoded><![CDATA[<p>A Deep Dive into Query, Key, and Value Vectors</p><h4>Introduction</h4><p>The attention mechanism, introduced in the landmark paper “Attention Is All You Need,” revolutionized natural language processing and laid the foundation for modern Large Language Models (LLMs). This article explores the key concepts behind attention mechanisms and their implementation.</p><h4>Background</h4><p>Before transformers, neural networks like RNNs and LSTMs struggled with understanding long-range dependencies in text. The attention mechanism solved this limitation by introducing a novel approach to process sequential data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VyVlcCKXAjzOSZA7.png" /><figcaption>Mermaid chart generated by: Claude AI</figcaption></figure><p>Imagine you’re reading a long news article. To understand the end, you need to remember what was discussed at the beginning. RNNs have a hard time with this because they process the text sequentially, and information from the beginning can get ‘lost’ along the way. So Transformers, and specifically the attention mechanism, solve this by allowing the model to directly look at “<strong><em>all”</em></strong> parts of the sentence at once, paying attention to the most important words.</p><p>For any model to parse text it needs to be converted as tokens, the following example of byte pair encoding:</p><p>Consider this example sentence:</p><p>“I am nihar palem, i am a graduate student from san jose state university”</p><p>The processing of text involves multiple steps:</p><ol><li>Converting strings to bits</li><li>Bits to byte</li><li>Byte-pair encoding</li><li>Generating embedding vectors</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CI5E5C3Hq6NAp10B.png" /><figcaption>Mermaid chart generated by: Claude AI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/0*Lm1sauesh8ogRPfK.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><p>To try how tokenizer works check this: <a href="https://tiktokenizer.vercel.app/">Tiktokenizer</a></p><p>Next after generate tokens we send them to attention block in which,</p><h3>Core Components of Attention</h3><blockquote>Q: “What information am I looking for?”</blockquote><blockquote>K: “What information do I <em>have</em> in this word?”</blockquote><blockquote>V: “The actual content of this word.”</blockquote><p><em>“We use the Query vector of one word and the Key vector of another word to calculate their attention score. The Value vector is then used to extract the actual content from the important words.”</em></p><h4>Query Vector (Q)</h4><p>The Query vector represents the <em>question</em> being asked by a particular word. It’s like asking, ‘What information am I, this word, looking for from other words in the sentence?</p><p>The query vector is generated by multiplying the embedding vector with a learned matrix. It essentially asks questions about the relationships between different parts of the text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MsU0F3nvVOxDi9r-.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VlTox2Rm5CufMGRc.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><h4>Key Vector (K)</h4><p>The Key vector represents the <em>information</em> that a particular word <em>contains</em>. It’s like saying, ‘Here’s what I, this word, have to offer to other words in the sentence.’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c1y_DPBHSFs2tIpi.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><p>The key vector works in conjunction with the query vector. Both Q and K are trained in the same pattern and align in the same dimensional space (typically 128D). Their interaction is measured through dot products:</p><p>- Positive values indicate similarity</p><p>- Negative values indicate difference</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BwzWQK9Xjrl35Vox.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><p>We then apply softmax to them the make fall in range between 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j-9snemraXfiS_4p.png" /></figure><p>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></p><h4>Value Vector (V)</h4><p>The Value vector represents the <em>actual content</em> of the word. It’s the information that will be used if another word finds this word relevant.</p><p>They are generated from the embeddings through raw embeddings with matrix multiplication. Later they are sent to QK (masked attention)+ this v vector gives the context aware outputs.</p><p>The value vector is derived through the formula: V = X · W^V</p><p>Where:</p><p>- X is the input embedding</p><p>- W^V is a trainable matrix</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iBYs7Q-e2GgCgh39.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><h4>Masked Attention Pattern</h4><p>In tasks like generating text (e.g., writing a story), we want the model to predict the next word <strong><em>without</em></strong> peeking at the words that come later in the sequence. This is like trying to guess the next word in a sentence without seeing it. To prevent the model from ‘cheating’ and looking ahead, we use masking.</p><p>1. Values after the current position are set to negative infinity</p><p>2. When softmax is applied, these values become zero</p><p>3. This creates a causal attention pattern</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2Ktz1UggbDpHlMWC.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><h4>Attention Computation Process</h4><p>1. Computing Attention Scores</p><p>The attention score is calculated using:</p><p><strong>Attention Weight = softmax(QK^T / sqrt(d_k))</strong></p><ul><li><strong>QK^T:</strong> “This is the dot product of the Query and Key vectors. It measures how similar they are. A higher score means the words are more related.”</li><li><strong>sqrt(d_k):</strong> “We divide by the square root of d_k (the dimension of the Key vector) to stabilize the training process.”</li><li><strong>softmax:</strong> “Softmax converts these scores into probabilities between 0 and 1. This makes it easier to compare the importance of different words.”</li></ul><p>2. Weighted Sum Calculation</p><p>Finally, we take a weighted sum of the Value vectors, using the attention scores as weights. This gives us a context-aware representation of the input text</p><p><strong>Output = ∑(Attention Weight_i · v_i)</strong></p><p>This formula means that each Value vector is multiplied by its corresponding attention weight, and then all the weighted Value vectors are added together. This creates a final output that represents the meaning of the input text, taking into account the relationships between the different words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m9XpqR2y7PARBB6u.png" /><figcaption>Image taken from: <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></figcaption></figure><p><strong>References:</strong> <a href="https://youtu.be/KJtZARuO3JY?si=eRNyOBlQxanhmw7Z">3 blue 1 brown (decoding transformer)</a></p><blockquote><a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">Connect with me on LinkedIn</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e170de3eae8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Agent powered Job Search and matching]]></title>
            <link>https://nihar-palem.medium.com/multi-agent-powered-job-search-and-matching-9cf2bb67f3e3?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cf2bb67f3e3</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[job-search]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Sat, 25 Jan 2025 21:05:37 GMT</pubDate>
            <atom:updated>2025-01-25T21:05:37.767Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Job hunting can often feel overwhelming, with countless listings, resumes to tailor, and the challenge of finding roles that align with your skills. The <strong>Multi-Agent powered Job Search and matching</strong> app transforms this process by combining <strong>prompt engineering</strong>, <strong>web scraping</strong>, and <strong>LLM-based matching</strong> to deliver a personalized, intelligent, and efficient job search experience. By automating key tasks like parsing user inputs, fetching job listings, and matching roles to resumes, this tool helps users focus on landing their dream jobs instead of navigating endless searches.</p><p>This app uses a <strong>multi-agent approach</strong> to divide tasks into specialized agents, ensuring each step — from extracting parameters to analyzing job matches — is handled with precision. The result is a robust system that seamlessly integrates various technologies for a superior job search experience.</p><p><strong><em>You can try the app here: </em></strong><a href="https://huggingface.co/spaces/Niharmahesh/AI-Job-Search-and-match/blob/main/app.py">application</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*Z1AqJfffdTZokvgx.png" /></figure><h3>How It Works</h3><p>The app streamlines job searching through five key stages, utilizing advanced technologies to ensure precision and efficiency:</p><h4>1. User Input and Prompt Parsing (AGENT-1)</h4><p><strong>Step 1: Collecting User Input</strong></p><p>The user provides the following:</p><ul><li>A brief job description and desired location (e.g., “DA in CA”).</li><li>Their resume, either uploaded as a file or entered as plain text.</li></ul><p><strong>Step 2: Refining Input with LLMs</strong></p><ul><li>Using <strong>LLM-powered prompt engineering</strong>, the app processes user inputs to extract structured parameters:</li><li>search_term: Extracted job role (e.g., &quot;Data Analyst&quot;).</li><li>location: Extracted location (e.g., &quot;California&quot;), expanding abbreviations like &quot;DA&quot; and &quot;CA&quot; to ensure clarity.</li></ul><p><strong>Example System Prompt</strong></p><pre>You are a language model designed to extract job search parameters. Parse the following input to determine:<br>- Job role (expand abbreviations, e.g., &quot;DA&quot; becomes &quot;Data Analyst&quot;).<br>- Location (expand abbreviations, e.g., &quot;CA&quot; becomes &quot;California&quot;).<br>#Return the output as a structured JSON object.</pre><p><strong>Step 3: Output Structured Parameters</strong></p><ul><li>The result is a JSON object for precise job searches:</li></ul><pre>{&quot;search_term&quot;: &quot;Data Analyst&quot;, &quot;location&quot;: &quot;California&quot;}</pre><p>This ensures clean and accurate inputs for the next steps.</p><h4>2. Web Scraping for Job Listings</h4><p>The app leverages the jobspy library to fetch job listings from multiple platforms in real-time:</p><p><strong>Supported Job Platforms:</strong></p><ul><li><strong>Indeed</strong></li><li><strong>LinkedIn</strong></li><li><strong>Glassdoor</strong></li><li><strong>ZipRecruiter</strong></li><li><strong>Google</strong></li></ul><p><strong>Scraping Parameters:</strong></p><ul><li>Based on refined inputs (search_term and location).</li><li>Configurations include:</li><li>results_wanted=60: Fetch up to 60 job listings.</li><li>hours_old=24: Limit to jobs posted in the last 24 hours.</li><li>country_indeed=&#39;USA&#39;: Specify geographic region.</li></ul><p><strong>Output:</strong></p><ul><li>A Pandas DataFrame containing detailed job information:</li><li><strong>Columns:</strong> Job Title, Company, Location, Description, Application Links, and Posting Date.</li></ul><p>The scraped data forms the basis for further analysis.</p><h4>3. Resume Analysis (AGENT-1)</h4><p>To enhance matching accuracy, the app performs in-depth resume analysis using advanced LLM models:</p><p><strong>Step 1: AI-Driven Resume Summarization</strong></p><p>The user’s resume is sent to the LLM for structured analysis.</p><p>Key sections summarized include:</p><ul><li><strong>Professional Summary</strong>: Concise overview of the candidate’s background.</li><li><strong>Skills</strong>: Highlighted technical and soft skills.</li><li><strong>Education</strong>: Academic achievements.</li><li><strong>Work Experience</strong>: Notable roles, responsibilities, and accomplishments.</li><li><strong>Achievements</strong>: Key milestones and measurable results.</li></ul><p><strong>Example System Prompt</strong></p><pre>You are an AI model tasked with summarizing resumes. Extract key sections and format them into:<br>- Professional Summary<br>- Skills<br>- Education<br>- Work Experience<br>- Achievements</pre><p><strong>Output: </strong>A structured summary that highlights strengths and areas for improvement.</p><p>Example:</p><pre>- Professional Summary: Experienced Data Analyst with expertise in SQL and Python.<br>- Skills: SQL, Python, Power BI, Tableau.<br>- Education: MS in Data Analytics.<br>- Work Experience: 3 years in data analysis and reporting.<br>- Achievements: Improved query performance by 40%.</pre><h4>4. Job Compatibility Matching (AGENT-2)</h4><p>Using the refined resume summary and scraped job data, the app determines compatibility:</p><p><strong>Step 1: Batch Processing</strong></p><ul><li>Jobs are analyzed in batches of 3 to optimize performance and ensure timely responses.</li></ul><p><strong>Step 2: LLM-Powered Matching</strong></p><p>Each batch includes:</p><ul><li>The <strong>resume summary</strong> from Step 3.</li><li>Job details scraped in Step 2.</li></ul><p><strong>Evaluation Metrics:</strong></p><ul><li><strong>Match Score (0–100):</strong> Overall relevance of the job to the user’s qualifications.</li><li><strong>Strengths:</strong> Key reasons for compatibility (e.g., “Strong SQL skills”).</li><li><strong>Improvement Suggestions:</strong> Actionable advice (e.g., “Add advanced Excel skills”).</li></ul><p><strong>Example System Prompt</strong></p><pre>You are an AI model matching resumes to job descriptions. For each job, provide:<br>- Match Score (0-100).<br>- Key strengths.<br>- Suggestions for improvement.</pre><p><strong>Output:</strong></p><p>A ranked DataFrame of job matches with:</p><ul><li>Match Scores.</li><li>Reasons for compatibility.</li><li>Improvement suggestions.</li></ul><h4>5. Visualization and Insights</h4><p>The app presents results through an interactive interface built with <strong>Streamlit</strong>:</p><p><strong>Tabs for Results:</strong></p><ul><li><strong>Job Results Tab: </strong>Interactive table displaying job listings, descriptions, and clickable application links.</li><li><strong>Resume Insights Tab: </strong>Highlights summarized strengths and areas for improvement.</li><li><strong>Match Insights Tab: </strong>Displays job compatibility scores with reasons and suggestions.</li></ul><h3>Technology Stack</h3><ul><li><strong>Backend: Groq API</strong>: Powers LLM-based prompt processing and resume-job matching and <strong>jobspy Library</strong>: Fetches real-time job data.</li><li><strong>Frontend: Streamlit</strong>: Provides an intuitive user interface for input and results.</li><li><strong>Processing: Pandas</strong>: Organizes and processes job listings and match data.</li></ul><h4>WHY Multi-Agent Approach</h4><p>The app employs a multi-agent architecture where each agent is responsible for a specific task:</p><ul><li><strong>Input Agent:</strong> Handles user input and extracts structured parameters.</li><li><strong>Scraping Agent:</strong> Fetches job data from multiple platforms in real-time.</li><li><strong>Analysis Agent:</strong> Summarizes resumes and evaluates compatibility.</li><li><strong>Matching Agent:</strong> Scores jobs against resumes and ranks results.</li></ul><h4>Optimized Model Usage</h4><p>To balance efficiency and cost, the app uses different models for specific tasks:</p><ul><li><strong>Lighter Models:</strong> For basic tasks like resume summarization, a smaller (<strong>8-billion</strong> parameter model), faster model is employed to limit API calls and token usage.</li><li><strong>High-Parameter Models:</strong> For core tasks like job-resume matching, the app utilizes a <strong>70-billion</strong> parameter model for nuanced analysis and higher accuracy.</li></ul><h4>Tested Alternatives:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5ccFa6Jb0NpC516S.png" /></figure><blockquote><strong>Conclusion</strong></blockquote><blockquote>The <strong>Multi Agent Job Search with Matching</strong> app redefines job hunting with cutting-edge technologies like prompt engineering, LLMs, and web scraping. By employing a multi-agent architecture, it divides the job search pipeline into specialized tasks, ensuring accuracy and efficiency. With optimized model usage and detailed job-resume matching, the app saves time, delivers personalized recommendations, and empowers users to target roles that truly align with their skills and aspirations.</blockquote><blockquote>Thanks for reading till end,</blockquote><blockquote>You can connect me on <a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">LinkedIN</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cf2bb67f3e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Temperature in Language Models (LLMs)]]></title>
            <link>https://nihar-palem.medium.com/understanding-temperature-in-language-models-llms-67079f1d6193?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/67079f1d6193</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Mon, 20 Jan 2025 03:29:13 GMT</pubDate>
            <atom:updated>2025-01-20T03:30:14.203Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Control the creativity and randomness of text generation.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/0*lSAyrmIVv99nzaAW.png" /></figure><h3>What is Temperature in LLMs?</h3><p>Temperature is a parameter that determines the probability distribution of the next word in text generation. It acts as a “creativity knob,” controlling the randomness of the output:</p><ul><li><strong>Low Temperature (e.g., 0.0–0.4):</strong> Results are deterministic, precise, and repetitive.</li><li><strong>Moderate Temperature (e.g., 0.5–1.0):</strong> Outputs become more diverse while maintaining coherence.</li><li><strong>High Temperature (e.g., 1.1–2.0):</strong> Text becomes creative and experimental, often at the cost of relevance and focus.</li></ul><p>Here’s a Python example that demonstrates this:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>def softmax_with_temperature(logits, temperature):<br>    logits = np.array(logits)<br>    exp_logits = np.exp(logits / temperature)<br>    return exp_logits / np.sum(exp_logits)<br># Raw scores for 5 tokens<br>logits = [2.0, 1.0, 0.1, -1.0, -2.0]<br>temperatures = [0.2, 0.5, 1.0, 2.0]<br># Calculate probabilities for each temperature<br>probabilities = {temp: softmax_with_temperature(logits, temp) for temp in temperatures}<br># Plotting<br>for temp, probs in probabilities.items():<br>    plt.plot(range(1, len(probs) + 1), probs, label=f&#39;Temperature = {temp}&#39;)<br>    <br>plt.title(&quot;Effect of Temperature on Softmax Probability Distribution&quot;)<br>plt.xlabel(&quot;Token Index&quot;)<br>plt.ylabel(&quot;Probability&quot;)<br>plt.legend()<br>plt.grid()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*pbQI-MXmgCvqFewg.png" /><figcaption>the output of the above code</figcaption></figure><h3>What The Numbers Mean</h3><p>Let’s look at the probabilities for different temperatures:</p><h4>Temperature = 0.2 (Very Cold)</h4><ul><li>Token 1: 99.3%</li><li>Token 2: 0.67%</li><li>Other tokens: nearly 0% This makes the model very focused on the highest-scoring token.</li></ul><h3>Temperature = 2.0 (Very Hot)</h3><ul><li>Token 1: 42.5%</li><li>Token 2: 25.8%</li><li>Token 3: 16.4%</li><li>Token 4: 9.5%</li><li>Token 5: 5.8% This spreads out the probabilities, making the model more random.</li></ul><h3>Visual Explanation</h3><p>The plot shows how temperature affects token selection:</p><ul><li><strong>Blue line (T=0.2)</strong>: Very steep, almost all probability goes to the highest-scoring token</li><li><strong>Orange line (T=0.5)</strong>: Still favors high-scoring tokens but less extremely</li><li><strong>Green line (T=1.0)</strong>: Moderate distribution</li><li><strong>Red line (T=2.0)</strong>: Flatter distribution, giving more chances to lower-scoring tokens</li></ul><h3>Practical Examples</h3><p>Think of it like this:</p><ul><li><strong>Low temperature (0.2)</strong>: “The sky is blue” → Always picks the most common completion</li><li><strong>Medium temperature (1.0)</strong>: “The sky is…” → Might pick “blue,” “cloudy,” or “clear”</li><li><strong>High temperature (2.0)</strong>: “The sky is…” → Might pick more creative options like “dancing,” “infinite,” or “a canvas”</li></ul><blockquote>The higher the temperature, the more willing the model is to take “creative risks” with its responses.</blockquote><h3>Example of how temperature works in text generation (Using GROQ API)</h3><p>Below is a simple python code which uses GROQ API for accessing LLAMA 3.3 70B parameter model.</p><pre>import numpy as np<br>from groq import Groq<br>def generate_completion(client, model, prompt, temperature):<br>    completion = client.chat.completions.create(<br>        model=model,<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        temperature=temperature,<br>        max_tokens=100,<br>        top_p=1,<br>        stream=False,<br>        stop=None,<br>    )<br>    return completion.choices[0].message.content<br><br>def main():<br>    client = Groq(api_key=&quot;your api key&quot;)<br>    model = &quot;llama-3.3-70b-versatile&quot;<br>    prompt = &quot;Explain AI in exactly 50 words. Do not exceed this limit.&quot;<br># Generate temperatures from 0 to 2 with a step of 0.1<br>    temperatures = np.arange(0, 2.1, 0.1)  <br>    for temp in temperatures:<br>        print(f&quot;\nTemperature: {temp:.1f}&quot;)<br>        result = generate_completion(client, model, prompt, temp)<br>        print(result)<br>#divider line <br>        print(&quot;-&quot; * 50)<br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Comparing Outputs at Different Temperatures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*i1fPp1kRTfWv2Giw.png" /></figure><p>Practical Use Cases of Temperature are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GtV9xK5tPpgM0Y2_n6T_7w.png" /></figure><h3>Conclusion</h3><p>Temperature in LLMs plays a crucial role in tailoring the tone, creativity, and coherence of generated text. By adjusting this parameter, users can achieve outputs ranging from precise technical content to highly creative narratives. Understanding and leveraging temperature effectively can optimize text generation for specific applications, enhancing the value of tools like Groq API.</p><p>Thank you for reading to the end! Here’s a refined version of your statement: I recently developed an application for <strong>resume analysis and generation</strong> using an advanced <strong>LLM (LLAMA-3.3, 70B versatile)</strong>. This tool leverages <strong>prompt engineering</strong> and the <strong>GROQ API</strong> (with the temperature parameter) to analyze resumes and job descriptions. It generates detailed analysis reports and enhances resumes, providing tailored suggestions to align with job requirements effectively.</p><blockquote>You can check out the app at: <a href="https://resume-easz.streamlit.app/"><strong>Resume_easz</strong></a></blockquote><blockquote>— — — You can connect me on <a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">LinkedIn</a> — — —</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67079f1d6193" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NVIDIA’s — CES 2025 announcements]]></title>
            <link>https://nihar-palem.medium.com/nvidias-ces-2025-announcements-ae9c9463d4c6?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae9c9463d4c6</guid>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[ces-2025]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Thu, 09 Jan 2025 00:11:36 GMT</pubDate>
            <atom:updated>2025-01-09T00:11:36.521Z</atom:updated>
            <content:encoded><![CDATA[<h3>NVIDIA’s — CES 2025 announcements</h3><p>Revolutionizing Gaming, AI, and Autonomous Systems with Cutting-Edge Innovations</p><p><em>NVIDIA’s CES 2025 announcements showcase a range of cutting-edge technologies across multiple domains. In gaming and graphics, DLSS and Multi-Frame Generation promise enhanced visuals and smoother gameplay. For AI, NVIDIA introduces scaling laws and ecosystems like NIM, NEMO, and AI Blueprints, along with the LLAMA Nemotron language models. Physical AI and simulation see advancements with NVIDIA COSMOS and integrated computer solutions. Autonomous vehicle technology progresses with the THOR processor and Drive OS. In robotics, the ISAAC Groot platform offers comprehensive development tools. Additional innovations include Project Digits supercomputer and improvements to Windows Subsystem for Linux 2. These developments reinforce NVIDIA’s leadership in gaming, AI, autonomous systems, robotics, and cloud computing.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kbVFMf6Y0_WDA5nS.png" /><figcaption>Image generated using: Note-GPT</figcaption></figure><h3>Gaming and Graphics Technologies</h3><h4>Deep Learning Super Sampling (DLSS)</h4><p>DLSS is an AI-powered rendering technology that transforms gaming visuals by enhancing performance and image quality.</p><p><strong>Process</strong>:</p><ol><li>Renders games at lower resolutions.</li><li>Uses a deep neural network to upscale images.</li><li>Produces high-quality, 4K-like output.</li></ol><p><strong>Features</strong>:</p><ul><li>Trained on thousands of high-resolution images.</li><li>Delivers higher frame rates and improved image clarity.</li><li><strong>Example</strong>: A game running at 1080p appears as though it’s rendered in 4K.</li></ul><h4>Multi-Frame Generation</h4><p>This technology enhances frame rates for smoother gameplay using NVIDIA’s neural networks.</p><p><strong>Process</strong>:</p><ol><li>Analyzes motion data and previous frames.</li><li>Predicts and generates up to three intermediate frames.</li></ol><p><strong>Benefits</strong>:</p><ul><li>Smoother gameplay.</li><li>Improved responsiveness, especially in fast-paced scenarios.</li></ul><h3>Evolution of AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vrqFu10bW_Rk2Nyv.png" /><figcaption>image credits: <strong>NVIDIA CEO Jensen Huang Keynote at CES 2025</strong></figcaption></figure><h3>AI Scaling and Ecosystems</h3><h4>Scaling Laws</h4><p>NVIDIA’s research has identified three scaling laws that optimize AI systems:</p><ol><li><strong>Pre-training Scaling</strong>: Enhances initial model training.</li><li><strong>Post-training Scaling</strong>: Boosts model performance after training.</li><li><strong>Test-time Scaling</strong>: Refines reasoning and long-term thinking.</li></ol><h4>Agentic AI Ecosystems</h4><p>NVIDIA’s advanced ecosystems streamline AI development and deployment:</p><p><strong>NIM (NVIDIA Inference Microservices)</strong>:</p><ul><li>Pre-trained models functioning as microservices.</li><li>Accelerates AI capability integration.</li></ul><p><strong>NEMO (NVIDIA Enterprise Managed Online)</strong>:</p><ul><li>A digital employee for model management and evaluation.</li><li>Simplifies deployment.</li></ul><p><strong>AI Blueprints</strong>: Open-source resources designed to foster AI adoption.</p><h4>NVIDIA LLAMA Nemotron</h4><p>A suite of advanced language models available in three versions:</p><ol><li><strong>Nano</strong>: Lightweight and efficient.</li><li><strong>Super</strong>: Balanced for versatility.</li><li><strong>Ultra</strong>: High-performance for intensive applications.</li></ol><h3>Physical AI and Simulation</h3><h4>NVIDIA COSMOS</h4><p>COSMOS, paired with the Omniverse platform, creates ultra-realistic simulations for diverse applications.</p><p><strong>Key Features</strong>:</p><ul><li>Trained on 20 million hours of video data.</li><li>Available in Nano, Super, and Ultra versions.</li><li>Applications: Gaming, robotics, autonomous vehicles.</li><li><strong>Integration</strong>: Omniverse provides a digital skeleton; COSMOS adds realism.</li></ul><h3>Integrated Computer Solutions</h3><p>NVIDIA offers three specialized systems for AI:</p><ol><li><strong>DGX</strong>: High-performance computing for complex AI model training.</li><li><strong>AGX</strong>: Edge computing solution for real-world AI deployments.</li><li><strong>THOR</strong>: A next-gen processor for autonomous systems, capable of processing diverse inputs (image, LIDAR, SONAR) and predicting optimal navigation paths.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z6gr7GJwIblhdl5d.png" /><figcaption>image credits: <strong>NVIDIA CEO Jensen Huang Keynote at CES 2025</strong></figcaption></figure><h3>Autonomous Vehicles (AV)</h3><h4>THOR Processor</h4><p>Designed specifically for AVs, THOR processes multiple sensor inputs simultaneously.</p><p><strong>Features</strong>:</p><ul><li>Tokenizes data for efficient handling.</li><li>Employs predictive algorithms for path planning.</li><li>Applications extend to humanoid robots.</li></ul><h4>Drive OS</h4><p>This operating system integrates with NVIDIA’s hardware to provide a reliable platform for AV development.</p><h4>Data Generation for AV Training</h4><p>NVIDIA leverages COSMOS and Omniverse to create realistic training scenarios, supported by DGX, AGX, and THOR systems for:</p><ol><li>Data processing.</li><li>Model training.</li><li>Real-world deployment.</li></ol><h3>Robotics</h3><h4>ISAAC Groot Platform</h4><p>A comprehensive ecosystem for robotics development:</p><ol><li><strong>Foundation Models</strong>: Pre-trained for common tasks.</li><li><strong>Data Pipelines</strong>: Efficient data processing.</li><li><strong>Simulation Capabilities</strong>: Virtual testing environments.</li><li><strong>Robotic Systems</strong>: Optimized hardware solutions.</li></ol><h4>Simulation Workflow</h4><p>A three-stage blueprint for robotic system development:</p><ol><li><strong>Teleop</strong>: Captures real-world data for digital twin creation.</li><li><strong>Mimic</strong>: Expands and diversifies data for training.</li><li><strong>GEN</strong>: Generates realistic simulations using COSMOS and Omniverse.</li></ol><h3>Additional Technologies</h3><h4>Project Digits</h4><p>NVIDIA’s latest supercomputer integrates the GB110 chip to:</p><ul><li>Enhance cloud computing.</li><li>Push AI and high-performance computing boundaries.</li></ul><h4>Windows Subsystem for Linux 2 (WSL-2)</h4><p>This feature enables seamless cross-platform development, benefiting NVIDIA users by enhancing software flexibility.</p><h3>Conclusion</h3><p>NVIDIA’s integrated technologies bridge gaming, AI, robotics, autonomous systems, and cloud computing. With innovations like DLSS, COSMOS, and THOR, NVIDIA continues to redefine industry standards, driving progress and transforming possibilities across hardware and software domains.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><blockquote><em>Thanks for reading till end</em></blockquote><blockquote><em>You can connect with me on </em><a href="https://www.linkedin.com/in/nihar-palem-1b955a183/"><em>LinkedIn</em></a></blockquote><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae9c9463d4c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Predicting Tokens to Sentences: Meta’s LCM]]></title>
            <link>https://nihar-palem.medium.com/from-predicting-tokens-to-sentences-metas-lcm-cfa0850a6fe3?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfa0850a6fe3</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Mon, 06 Jan 2025 07:03:36 GMT</pubDate>
            <atom:updated>2025-01-06T07:03:36.042Z</atom:updated>
            <content:encoded><![CDATA[<h3>The New Era of Language Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/0*dxCZmdKIVnLg3kGb.png" /><figcaption>Image source: <a href="https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/">https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/</a></figcaption></figure><p>Large Concept Models (LCMs) represent a paradigm shift in natural language processing, moving away from token-level predictions to a more human-like approach of reasoning with higher-level concepts. This article explores the architecture, training approaches, and evaluation of LCMs, highlighting their potential to revolutionize language understanding and generation.</p><h3>The Concept-Based Approach of LCM</h3><p>LCMs introduce a novel processing pipeline that elevates language understanding to a more abstract level:</p><ol><li>Input text is segmented into sentences</li><li>Sentences are encoded into embeddings using <strong>SONAR</strong> (<strong>S</strong>entence-level multim<strong>O</strong>dal and la<strong>N</strong>guage-<strong>A</strong>gnostic <strong>R</strong>epresentations)</li><li>LCM processes these embeddings, generating new sequences of concepts</li><li>The resulting concepts are decoded back into words using SONAR</li></ol><p>This approach allows LCMs to reason about language at a higher level of abstraction, more akin to human thought processes.</p><h3>Key Advantages of LCMs</h3><ol><li><strong>Language Agnostic Processing</strong>: By maintaining the sequence of concepts, the output of the decoder can be directly converted into different languages without the need for explicit translation.</li><li><strong>Efficient Summarization</strong>: LCMs can generate summaries by understanding the context at a conceptual level, without needing to process each word individually.</li><li><strong>Improved Handling of Long Context Windows</strong>: LCMs can process and reason over much longer sequences of text without the computational overhead typically associated with extending context windows in token-based models.</li></ol><h3>LCM Architecture and Variants</h3><h3>Base LCM</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lSHQ_4QneVL_z8-g.png" /><figcaption>Base LCM arch (image source: official paper)</figcaption></figure><p>The Base LCM utilizes a decoder-only Transformer architecture designed for next-concept prediction. Key components include:</p><ol><li>PreNet: Normalizes input SONAR embeddings and maps them to the model’s hidden dimension.</li><li>PostNet: Denormalizes the output and maps it back to the SONAR embedding dimension.</li><li>Transformer Decoder: Processes the sequence of preceding concepts.</li></ol><p>The model is trained to minimize Mean Squared Error (MSE) between predicted and ground truth next concepts.</p><h3>Diffusion-based LCM</h3><p>Diffusion-based LCMs incorporate principles from diffusion models, offering a more nuanced approach to concept generation:</p><ol><li>Forward Process: Gradually adds Gaussian noise to original data.</li><li>Reverse Process: Learns to denoise data, predicting the original noiseless state.</li></ol><p>This approach allows for better capturing of the diversity in possible sentence continuations.</p><h3>Classifier-free Guidance LCM</h3><p>This variant combines conditional and unconditional models to improve output quality and controllability:</p><ol><li>Conditional Model: Generates outputs based on given inputs.</li><li>Unconditional Model: Generates outputs without specific conditioning.</li></ol><p>The guidance scale (γ) controls the balance between these models during inference.</p><p>CFG involves training two models simultaneously: (example arch show below)</p><ol><li>A conditional model that generates outputs based on given inputs</li><li>An unconditional model that generates outputs without specific conditioning</li></ol><p>At inference time, these models are combined to create a balance between quality and diversity.</p><pre>[Input Conditioning (y)]<br>         |<br>         v<br>[Conditional Model] [Unconditional Model]<br>         |                 |<br>         v                 v<br>[Conditional Score]  [Unconditional Score]<br>         |                 |<br>         v                 v<br>    [Score Combination (CFG)]<br>         |<br>         v<br>[Iterative Denoising Process]<br>         |<br>         v<br>[Generated Output (x)]</pre><h3>One-Tower vs. Two-Tower Diffusion LCMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KAHs3lmEaPizIaxg.png" /><figcaption>one-tower (image source: official paper)</figcaption></figure><ol><li><strong>One-Tower:</strong> A single model handles both context understanding and denoising.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IpTG8V37bHGky71t.png" /><figcaption>two-tower (image source: official paper)</figcaption></figure><ol><li>Two-Tower: Separates the contextualizer and denoiser into distinct components.</li><li>The Two-Tower approach allows for more specialized processing of context and noise removal.</li></ol><h3>Training Approaches and Techniques</h3><ol><li><strong>Pre-training</strong>: Models are initially trained on large-scale data to learn general language patterns.</li><li><strong>Instruction-tuning</strong>: After pre-training, models undergo fine-tuning on specific tasks, such as following instructions to generate consistent stories.</li><li><strong>Gradient Clipping</strong>: Prevents exploding gradients by scaling down large updates.</li><li><strong>Cosine Learning Rate Schedule</strong>: Smoothly decays the learning rate following a cosine curve.</li><li><strong>Warm-up Period</strong>: Gradually increases the learning rate at the start of training.</li><li><strong>Weight Decay</strong>: Adds a small penalty for larger weights to prevent overfitting.</li></ol><h3>SONAR and Data Preparation</h3><p>SONAR plays a crucial role in the LCM architecture, creating universal representations of meaning across multiple languages. To ensure high-quality input for SONAR embeddings, robust automatic text segmentation techniques are employed:</p><ol><li>SpaCy Segmenter: A rule-based approach using punctuation and language-specific rules.</li><li>Segment any Text (SaT): A machine learning-based approach for predicting sentence boundaries.</li></ol><p>These methods are refined with a sentence length cap of 200 characters and evaluated using AUTOBLEU.</p><h3>Quantization of SONAR Embeddings</h3><p><strong>SONAR Embedding Quantization:</strong> SONAR embedding quantization is a process of converting high-dimensional, continuous sentence embeddings into a series of discrete indices. This is done using multiple codebooks, each containing a set of centroids (representative vectors).</p><p><strong>How it works:</strong></p><ol><li><strong>Input</strong>: A high-dimensional SONAR embedding (e.g., 1024 dimensions).</li><li><strong>Process</strong>: For each codebook (typically 64 codebooks): a. Find the closest centroid to the current vector (starting with the original embedding). b. Store the index of this centroid. c. Calculate the residual (difference between the vector and the chosen centroid). d. Use this residual as input for the next codebook.</li><li><strong>Output</strong>: A series of indices (e.g., 64 indices, one per codebook).</li></ol><p><strong>Why Quantize SONAR Embeddings?</strong></p><ol><li><strong>Efficiency</strong>: Reduces storage and computational requirements.</li><li><strong>Discretization</strong>: Allows use of techniques designed for discrete tokens.</li><li><strong>Controllability</strong>: Enables fine-grained control over generation.</li><li><strong>Compression</strong>: 1024-dimensional vectors become simple indices (e.g., “Cluster 1”).</li><li><strong>Faster Operations</strong>: Searching or generating with discrete categories is faster than with continuous vectors.</li></ol><p><strong>Quantized LCM (Large Concept Model): </strong>A language model designed to work with quantized SONAR embeddings.</p><p>Variants:</p><ol><li><strong>Quant-LCM-d</strong>: Predicts discrete centroid indices.</li><li><strong>Quant-LCM-c</strong>: Predicts continuous vectors, then maps them to centroids.</li></ol><p><strong>Why use Quantized LCM?</strong></p><ol><li><strong>Bridging Approaches</strong>: Combines benefits of continuous embeddings with discrete modeling techniques.</li><li><strong>Efficiency</strong>: More computationally efficient than working with full continuous embeddings.</li><li><strong>Flexibility</strong>: Allows for controlled generation while retaining semantic richness.</li><li><strong>Scalability</strong>: Potentially more scalable for very large language models.</li></ol><h3>Evaluation Metrics and Results</h3><p><strong>LCMs are evaluated using various metrics:</strong></p><ol><li><strong>L2 distance (ℓ2):</strong> Measures closeness of predicted embedding to actual next sentence embedding.</li><li><strong>Round-trip L2 distance (ℓ2-r):</strong> L2 distance after decoding and re-encoding the prediction.</li><li><strong>Contrastive Accuracy (CA):</strong> How often the prediction is closer to the true next sentence than other sentences.</li><li><strong>Paraphrasing (PAR):</strong> Similarity of prediction to sentences in the context.</li><li><strong>Mutual Information (MI):</strong> Information shared between prediction and previous sentences.</li></ol><p>(AS per paper) Results showed that diffusion-based models (One-Tower and Two-Tower) generally performed better than the Base-LCM across these metrics, with improvements in contrastive accuracy, paraphrasing, and mutual information. The quantized models (Quant-LCM-d and Quant-LCM-c) also showed competitive performance while offering efficiency benefits.</p><h3>SONAR Fragility and L2 Limitations</h3><p>While L2 distance is commonly used to evaluate embedding quality, it has limitations in capturing semantic changes, out-of-distribution issues, and context sensitivity. To address these limitations, fragility metrics are introduced to measure the semantic stability of embeddings under perturbation.</p><h3>Measuring Fragility</h3><p>Fragility in SONAR embeddings is defined as the sensitivity of embeddings to small perturbations. Two main methods are used to score fragility:</p><ol><li>Auto-Encoding BLEU: Measures how well the perturbed text matches the original using BLEU score.</li><li>External Cosine Similarity: Uses an external encoder (mGTE) to compare the semantic similarity of the original and perturbed texts.</li></ol><p>The fragility of a text fragment w with SONAR code x = encode(w) is calculated as:</p><pre>fragility(w)=−E[score(w,wα,ϵ)]</pre><p>Where W_α,ϵ is a perturbed version of w, created by adding noise to its embedding and then decoding it back to text.</p><h3>Key Findings from Fragility Studies</h3><ol><li>Decoder Dependence: Fragility scores are sensitive to the choice of decoder. A fine-tuned decoder showed better robustness to noise.</li><li>Text Length Sensitivity: Longer texts (&gt;250 characters) are more challenging for SONAR and LCM models.</li><li>Noise Sensitivity: BLEU scores decrease faster than cosine similarity as noise increases.</li><li>Variability in Fragility: There’s a wide distribution of fragility scores across SONAR samples.</li><li>Most Fragile Embeddings: The 5% most fragile embeddings often correspond to hyperlinks, references, unique IDs, and code-switched or numerical entries.</li></ol><h3>Implications of Fragility Scores</h3><ol><li>Data Filtering: Fragility scores can be used to filter out problematic samples from training data.</li><li>Model Improvement: Understanding fragility can guide improvements in embedding models and decoders.</li><li>Quality Assessment: Fragility provides a more nuanced view of embedding quality than L2 distance alone.</li><li>Task-Specific Considerations: Different applications might have different tolerances for fragility.</li></ol><h3>Additional LCM Training Techniques</h3><h3>Gradient Clipping</h3><p>Gradient clipping is used to prevent the “exploding gradient” problem:</p><ol><li>Calculate the gradient for all model parameters.</li><li>Compute the L2 norm of the gradient.</li><li>If the norm exceeds a threshold (for example 10 ), scale down the gradient to match the threshold.</li></ol><p>This technique helps maintain a stable learning process and allows for higher learning rates without risking divergence.</p><h3>Cosine Learning Rate Schedule</h3><p>The cosine learning rate schedule is an annealing technique where the learning rate follows a cosine curve:</p><ol><li>Starts at a high value</li><li>Decreases following a cosine function</li><li>Reaches its minimum at the end of training</li></ol><p>This approach often leads to better convergence compared to step or linear decay and helps the model settle into a good minimum at the end of training.</p><h3>Warm-up Period and Weight Decay</h3><ol><li>Warm-up Period: The learning rate gradually increases from a very small value to the initial learning rate over 10,000 steps. This helps stabilize early training and allows the model to adjust its parameters slowly at first.</li><li>Weight Decay: A small penalty (0.1 in this case) is added for larger weights in the model. This helps prevent overfitting and encourages the model to learn simpler, more generalizable features.</li></ol><h3>Importance of Diffusion Inference Hyper-parameters</h3><p>The quality of generated text is significantly impacted by diffusion inference hyper-parameters:</p><ol><li>Guidance Scale: Controls how much attention the model pays to the conditioning context.</li><li>Initial Noise Scale: Determines the level of noise added to the input at the start of inference.</li><li>Number of Inference Steps: The number of steps used to iteratively denoise and generate the output.</li></ol><p>These parameters affect key metrics such as Mutual Information (MI) and L2 Distance (ℓ2), influencing the coherence and accuracy of generated content.</p><p>Conclusion, Large Concept Models represent a significant advancement in natural language processing, offering improved coherence, language-agnostic processing, and potential for more human-like reasoning. The incorporation of fragility metrics, advanced training techniques, and careful consideration of diffusion inference hyper-parameters further enhances their capabilities and robustness, paving the way for more advanced and nuanced language understanding and generation capabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfa0850a6fe3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Embeddings and Absolute Positional Embeddings in NLP]]></title>
            <link>https://nihar-palem.medium.com/understanding-embeddings-and-absolute-positional-embeddings-in-nlp-42cc73ecc567?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/42cc73ecc567</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[positional-encoding]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Mon, 30 Dec 2024 06:39:25 GMT</pubDate>
            <atom:updated>2024-12-30T06:39:25.416Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*As2iBLXCucdraOrH.png" /></figure><h3>1. Introduction to Embeddings</h3><p>Embeddings are a crucial concept in Natural Language Processing (NLP), acting as a bridge between human language and machine understanding. They transform discrete objects, such as words or tokens, into continuous vectors of numbers within a high-dimensional space.</p><p><strong>Why Do We Need Embeddings?</strong></p><ul><li><strong>Machine Comprehension:</strong> Computers cannot directly process words or symbols; embeddings convert human language into a machine-readable format.</li><li><strong>Efficient Processing:</strong> Vector representations facilitate fast and efficient computations on large text datasets.</li><li><strong>Semantic Relationships:</strong> Embeddings capture the meanings and relationships between words mathematically.</li></ul><p><strong>Types of Embeddings:</strong></p><ul><li><strong>Word Embeddings:</strong> Represent individual words or tokens.</li><li><strong>Sentence Embeddings:</strong> Capture the meaning of entire sentences.</li><li><strong>Document Embeddings:</strong> Represent whole documents or paragraphs.</li><li><strong>Cross-modal Embeddings:</strong> Represent data from different modalities (e.g., text and images) in a shared space.</li></ul><h3>2. Word Embeddings</h3><p>Word embeddings encode the meanings of words based on their contexts. They enable models to understand semantic similarities and relationships effectively.</p><p><strong>Dimensionality of Word Embeddings:</strong> The dimensionality (embedding size) of word embeddings is typically set based on the model’s architecture and can vary widely. Common sizes include:</p><ul><li><strong>50 dimensions:</strong> Often used in simpler models.</li><li><strong>100 dimensions:</strong> A balance between complexity and performance.</li><li><strong>300 dimensions:</strong> Commonly used in advanced embeddings like Word2Vec and GloVe for richer semantic representation.</li></ul><p><strong>Training Methods:</strong> Word embeddings are typically trained using unsupervised methods on large text corpora. Popular algorithms include:</p><ul><li><strong>Word2Vec:</strong> Uses skip-gram or continuous bag-of-words (CBOW) approaches to predict context words from target words or vice versa.</li><li><strong>GloVe (Global Vectors for Word Representation):</strong> Utilizes word co-occurrence statistics to learn embeddings.</li><li><strong>FastText:</strong> Extends Word2Vec by representing words as bags of character n-grams, allowing it to generate embeddings for out-of-vocabulary words.</li></ul><p><strong>Example: Visualizing Word Embeddings</strong></p><p>Here’s a Python code snippet to visualize word embeddings using t-SNE:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.manifold import TSNE<br># Sample text<br>sample_text = &quot;The cat sat on the mat. The dog barked at the moon.&quot;<br>words = sample_text.lower().replace(&#39;.&#39;, &#39;&#39;).split()<br>vocab = list(set(words))  # Unique words in the text<br># Create one-hot encodings for each word<br>one_hot_encodings = np.eye(len(vocab))<br># Simulate word embeddings by projecting one-hot vectors into a random 50D space<br>np.random.seed(42)<br>embedding_dim = 50<br>word_embeddings = np.dot(one_hot_encodings, np.random.rand(len(vocab), embedding_dim))<br># Reduce dimensionality to 2D using t-SNE<br>tsne = TSNE(n_components=2, random_state=42, perplexity=5)<br>reduced_embeddings = tsne.fit_transform(word_embeddings)<br># Plot the word embeddings<br>plt.figure(figsize=(8, 6))<br>plt.scatter(reduced_embeddings[:, 0], reduced_embeddings[:, 1], color=&#39;blue&#39;)<br>for i, word in enumerate(vocab):<br>    plt.text(reduced_embeddings[i, 0], reduced_embeddings[i, 1], word, fontsize=12)<br>plt.title(&quot;Word Embeddings Visualization (t-SNE)&quot;)<br>plt.xlabel(&quot;Dimension 1&quot;)<br>plt.ylabel(&quot;Dimension 2&quot;)<br>plt.grid(True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/0*t4_UuuluKDIFDw-7.png" /><figcaption>Output of the above code</figcaption></figure><blockquote>The embeddings are random in this example and that meaningful relationships would emerge with trained embeddings.</blockquote><p><strong>Word Embeddings Visualization (t-SNE plot):</strong> This plot shows how the words from your sample sentence “The cat sat on the mat. The dog barked at the moon.” are arranged in a 2D space after dimensionality reduction using t-SNE. Here’s what we can observe:</p><ul><li>Related words are positioned closer together. For example:</li><li>“cat” and “dog” are somewhat close to each other since they’re both animals</li><li>“sat” and “at” share similar positions, possibly due to their usage as position/action words</li><li>“mat” and “moon” are far apart, which makes sense as they’re semantically different</li><li>Words like “the” and “on” appear in distinct locations, reflecting their unique roles as function words</li></ul><p>The distances between points represent the model’s understanding of semantic relationships between words, though in this simple example with random embeddings, these relationships are not as meaningful as they would be in a properly trained word embedding model like Word2Vec.</p><p><strong>Key Characteristics of Word Embeddings:</strong></p><ul><li>Capture semantic relationships between words.</li><li>Do not encode sequential information; for example, “The cat sat on the mat” and “Mat on sat cat the” would have identical embeddings.</li></ul><h3>3. Absolute Positional Embeddings</h3><p>Absolute positional embeddings are essential in Transformer-based models and it was first introduced in the paper <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>, allowing them to understand the sequential nature of language.</p><p><strong>The Formula</strong></p><p>The formula for absolute positional embeddings uses sine and cosine functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FM-Mn_lM5IHGhV_MXzu1qQ.png" /></figure><p>Where:</p><ul><li><strong>pos</strong> is the position of the token in the sequence.</li><li><strong>i</strong> is the dimension index.</li><li><strong>d_model</strong> is the dimensionality of the model.</li></ul><p><strong>Handling Position Awareness:</strong> In Transformer models, which lack recurrence (as seen in RNNs), there is no inherent way to capture the order of tokens. Absolute positional embeddings provide explicit positional information that allows models to differentiate between tokens based on their positions within a sequence.</p><p><strong>Properties of Absolute Positional Embeddings:</strong></p><ul><li>Handle variable-length sequences.</li><li>Allow models to attend to relative positions easily.</li><li>Values are bounded between -1 and 1, preventing issues with large numbers.</li></ul><p><strong>Methods of Implementation:</strong></p><ol><li><strong>Fixed (Sinusoidal) Approach: </strong>Generates fixed positional encodings using the above formula. Advantages include no additional parameters to learn and handling longer sequences than seen during training.</li></ol><pre>def get_sinusoidal_positional_encoding(max_seq_len, d_model):<br>    position = np.arange(max_seq_len)[:, np.newaxis]<br>    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))<br>    pe = np.zeros((max_seq_len, d_model))<br>    pe[:, 0::2] = np.sin(position * div_term)<br>    pe[:, 1::2] = np.cos(position * div_term)<br>    return pe</pre><ol><li><strong>Limitation of Sinusoidal Positional Embeddings:</strong> While sinusoidal embeddings have advantages like interpolation capabilities, they are limited in their ability to generalize to sequences longer than those seen during training. This limitation has led some models (e.g., T5) to adopt learned positional embeddings instead.</li><li><strong>Learned Approach:</strong>Positional embeddings are learned during training, similar to word embeddings.</li><li>Advantages include capturing more complex positional relationships tailored to specific datasets.</li></ol><pre>import torch.nn as nn<br>class LearnedPositionalEncoding(nn.Module):<br>    def __init__(self, d_model, max_len=512):<br>        super().__init__()<br>        self.positional_embedding = nn.Embedding(max_len, d_model)<br>    def forward(self, x):<br>        positions = torch.arange(x.size(1), device=x.device).unsqueeze(0)<br>        return self.positional_embedding(positions)</pre><p><strong>Visualization: Heatmap of Positional Embeddings</strong></p><pre>import seaborn as sns<br># Define a function to generate sinusoidal positional embeddings<br>def get_positional_encoding(max_seq_len, d_model):<br>    position = np.arange(max_seq_len)[:, np.newaxis]  # Sequence positions (rows)<br>    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))  # Scaling factor<br>    pos_encoding = np.zeros((max_seq_len, d_model))<br>    pos_encoding[:, 0::2] = np.sin(position * div_term)  # Sine for even indices<br>    pos_encoding[:, 1::2] = np.cos(position * div_term)  # Cosine for odd indices<br>    return pos_encoding<br># Generate positional embeddings for the sample text<br>max_seq_len = len(words)  # Number of words in the sample text<br>d_model = 16  # Embedding dimensions<br>positional_embeddings = get_positional_encoding(max_seq_len, d_model)<br># Plot positional embeddings as a heatmap<br>plt.figure(figsize=(10, 6))<br>sns.heatmap(positional_embeddings, cmap=&quot;viridis&quot;, cbar=True)<br>plt.title(&quot;Positional Embeddings Heatmap&quot;)<br>plt.xlabel(&quot;Embedding Dimensions&quot;)<br>plt.ylabel(&quot;Sequence Positions&quot;)<br>plt.show()</pre><figure><img alt="Output of above code" src="https://cdn-images-1.medium.com/max/797/0*4JVu6qzm51A-GChP.png" /><figcaption>output of above code</figcaption></figure><p><strong>Positional Embeddings Heatmap:</strong> This visualization shows how positional information is encoded across different dimensions:</p><ul><li>The y-axis shows sequence positions (0–11) for the sample sequence taken (same as in word embeddings plot)</li><li>The x-axis shows the embedding dimensions (0–15)</li></ul><p>Colors represent the values of the positional embeddings:</p><ul><li>Yellow areas represent higher positive values (closer to 1.0)</li><li>Purple areas represent lower negative values (closer to -1.0)</li><li>Green areas represent values around 0</li></ul><p><em>The alternating pattern you see (waves of different colors) is characteristic of sinusoidal positional embeddings because:</em></p><ul><li>Each row represents a position in the sequence</li><li>The pattern changes more slowly in earlier dimensions (left side)</li><li>The pattern changes more rapidly in later dimensions (right side)</li><li>This varying frequency helps the model distinguish between different positions in a sequence</li></ul><p><strong>This sinusoidal pattern is intentionally designed so that:</strong></p><ul><li>Each position gets a unique encoding</li><li>The relative position of tokens can be easily computed</li><li>The pattern can generalize to sequences of different lengths</li></ul><p><strong>Key Characteristics of Positional Embeddings:</strong></p><ul><li>Encode sequential order rather than semantic meaning.</li><li>Provide unique representations for each word position.</li></ul><h3>Alternatives to Absolute Positional Embeddings</h3><p>In addition to absolute positional embeddings, there are alternatives such as relative positional encodings used in models like Transformer-XL. These approaches allow models to consider relative distances between tokens rather than their absolute positions, which can be beneficial for tasks requiring longer context handling.</p><h3>4. Combining Word and Positional Embedding</h3><p>Transformer models utilize both types of embeddings:</p><ul><li><strong>Word Embeddings:</strong> Capture semantic meanings.</li><li><strong>Absolute Positional Embeddings:</strong> Capture word order in sequences.</li></ul><p><strong>How They Are Combined:</strong> In most cases, word embeddings and positional embeddings are summed element-wise before being fed into the model. This integration is crucial because it allows the model to leverage both types of information effectively.</p><p>For example: Ecombined= Eword+Epositional</p><p>Additionally, some models like BERT introduce special tokens for segment or sentence embeddings that further enrich contextual understanding.</p><h3>Conclusion</h3><p>The integration of word embeddings with absolute positional embeddings enables Transformer models to process language effectively by understanding both semantic content and structural information. This dual representation is critical for modern NLP models’ success across various language understanding and generation tasks. By exploring alternatives like relative positional encodings and discussing limitations of sinusoidal approaches, we gain a more comprehensive view of how these techniques contribute to advancements in NLP.</p><p>Thanks for reading till end :)</p><p>You can connect me on <a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42cc73ecc567" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Easz]]></title>
            <link>https://nihar-palem.medium.com/prompt-easz-330166c5abc7?source=rss-77905d842883------2</link>
            <guid isPermaLink="false">https://medium.com/p/330166c5abc7</guid>
            <dc:creator><![CDATA[Sainiharreddy Palem]]></dc:creator>
            <pubDate>Sun, 07 Jul 2024 22:48:52 GMT</pubDate>
            <atom:updated>2024-07-07T23:16:40.533Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Prompt Converter for Tailored Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8If-rCvO1Fh0tcIcFnCMQ.jpeg" /></figure><p>Prompt Easz is a prompt converter built based on sentence transformers. It helps in tuning prompts based on defined styles such as ELI5, chain of thoughts, role play, etc. This tool is designed to make learning new topics less overwhelming by providing tailored prompts.</p><p><a href="https://prompteaz.streamlit.app/">tap here to try the app</a></p><h3>What It Does</h3><p>Though we can use ChatGPT to get responses, sometimes the generated responses can be overwhelming, especially when learning a new topic. Prompt easz helps by:</p><ul><li>Converting user prompts to vector embeddings using a sentence transformer.</li><li>Finding cosine similarity between the user’s prompt and a set of predefined prompts.</li><li>Suggesting an updated prompt based on this similarity.</li><li>Allowing users to pick the prompt strength (low to high).</li><li>Providing predefined categories and identifying user question keywords accordingly.</li><li>Offering flexibility for users to pick their own style, making the prompt generator customizable.</li></ul><h3>Prompt Types and Categories:</h3><ul><li>10 main prompt types (e.g., ELI5, Chain-of-Thought, Role Play)</li><li>Each prompt type has 3–4 variations</li><li>10 predefined categories (e.g., Programming Platforms, Data Science)</li><li>Approximately 32 total prompt variations</li><li>Around 320 possible combinations of prompt styles and categories</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AinF_PNtI54ZO7RT8L1L4g.png" /><figcaption>List of prompt styles</figcaption></figure><h3>Theory</h3><p>Let’s dig into the theory of sentence transformers in three steps:</p><ul><li><strong>What are they? : </strong>Sentence transformers are alternatives built to replace RNNs (Recurrent Neural Networks).</li><li><strong>Architecture: </strong>They use the encoder part of the transformer architecture to convert input into embeddings (context vectors). For example, BART.</li><li>The model used in Prompt easz is developed by Google: <strong>all-MiniLM-L6-v2.</strong></li><li><strong>Use Cases: </strong>Used for Q&amp;A, sentence translation, semantic search, clustering, and semantic textual similarity tasks.</li></ul><h3>Pros and Cons</h3><p><strong>Pros:</strong></p><ul><li>Generates prompt styles based on predefined categories.</li><li>Customizable by the user.</li></ul><p><strong>Cons:</strong></p><ul><li>Limited to a few categories.</li><li>Does not auto-correct lexicon and grammar mistakes.</li><li>Potential for more diverse prompt styles.</li><li>Can be further extended and retrained based on user feedback. (which can be <a href="https://prompteaz.streamlit.app/About">given here</a>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*49eyy-4njLcrYBT8" /><figcaption>Photo by <a href="https://unsplash.com/@kevin_xue_pic?utm_source=medium&amp;utm_medium=referral">kevin Xue</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Thank you for reading. You can connect with me on <a href="https://www.linkedin.com/in/nihar-palem-1b955a183/">LinkedIn</a> for more insights and discussions on AI and language technologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=330166c5abc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>