<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anjilakshetri on Medium]]></title>
        <description><![CDATA[Stories by Anjilakshetri on Medium]]></description>
        <link>https://medium.com/@anjilakshetri?source=rss-5a2b01018d9e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*abMRZynqZrMu8ArW</url>
            <title>Stories by Anjilakshetri on Medium</title>
            <link>https://medium.com/@anjilakshetri?source=rss-5a2b01018d9e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:10:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anjilakshetri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implement a RAG system with Ollama, ChromaDB and Streamlit]]></title>
            <link>https://medium.com/@anjilakshetri/implement-a-rag-system-with-ollama-chromadb-and-streamlit-7ffc9d1ba586?source=rss-5a2b01018d9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ffc9d1ba586</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[rag-system]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[Anjilakshetri]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 07:10:46 GMT</pubDate>
            <atom:updated>2025-06-09T07:10:46.191Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Retrieval-Augmented Generation (RAG) has revolutionized how we interact with documents by combining the power of vector search with large language models. In this comprehensive guide, I’ll walk you through building a production-ready RAG application that can process PDF documents and answer questions based on their content using local models through Ollama.</p><p>Our application implements a complete RAG pipeline with robust error handling, persistent storage, and an intuitive user interface. By the end of this article, you’ll have a deep understanding of each component and be able to build your own document Q&amp;A system.</p><h3>Architecture Overview</h3><p>Our RAG application follows a five-step pipeline:</p><ol><li><strong>PDF Upload and Chunking</strong> — Document ingestion and text splitting</li><li><strong>Vector Database and Embeddings</strong> — Converting text to searchable vectors</li><li><strong>Query Collection</strong> — Retrieving relevant document chunks</li><li><strong>LLM Integration</strong> — Generating contextual responses</li><li><strong>Streamlit Interface</strong> — User-friendly web application</li></ol><figure><img alt="RAG architecture" src="https://cdn-images-1.medium.com/max/1024/1*GOMJ80-dd4ZL3LG6U7pijw.png" /></figure><h3>Import the necessary libraries</h3><pre>import streamlit as st<br>import tempfile<br>import os<br><br># Langchain dependencies<br>from langchain_community.document_loaders import PyMuPDFLoader<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from langchain_ollama import OllamaEmbeddings<br>import chromadb<br>from langchain_ollama import ChatOllama<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser</pre><h3>Step 1: PDF Upload and Chunking</h3><p>The foundation of any RAG system is effective document processing. Our application handles PDF uploads and converts them into manageable chunks for processing.</p><h4>Why Document Chunking Matters</h4><p>Large documents can’t be processed as single units due to:</p><ul><li><strong>Token limitations</strong> in language models</li><li><strong>Context window constraints</strong> that affect retrieval accuracy</li><li><strong>Memory efficiency</strong> requirements for vector storage</li></ul><h4>Implementation Details</h4><pre># Step 1: PDF Processing<br>def process_pdf(uploaded_file):<br>    &quot;&quot;&quot;Process PDF and return document chunks&quot;&quot;&quot;<br>    with tempfile.NamedTemporaryFile(delete=False, suffix=&#39;.pdf&#39;) as temp_file:<br>        temp_file.write(uploaded_file.getvalue())<br>        temp_path = temp_file.name<br><br>    try:<br>        loader = PyMuPDFLoader(temp_path)<br>        docs = loader.load()<br>        <br>        if not docs:<br>            st.error(&quot;No content found in the PDF file.&quot;)<br>            return None<br>            <br>        text_splitter = RecursiveCharacterTextSplitter(<br>            chunk_size=400,<br>            chunk_overlap=50,<br>            separators=[&quot;\n\n&quot;, &quot;\n&quot;, &quot;.&quot;, &quot;?&quot;, &quot;!&quot;, &quot; &quot;, &quot;&quot;]<br>        )<br>        all_splits = text_splitter.split_documents(docs)<br>        return all_splits<br>    except Exception as e:<br>        st.error(f&quot;Error processing PDF: {str(e)}&quot;)<br>        return None<br>    finally:<br>        try:<br>            if os.path.exists(temp_path):<br>                os.unlink(temp_path)<br>        except Exception as e:<br>            st.warning(f&quot;Could not delete temporary file: {e}&quot;)</pre><h4>Key Parameters used in RecursiveCharacterTextSplitter :</h4><p><strong>Chunk Size </strong>: Small enough to maintain semantic coherence while large enough to provide meaningful context. This size works well with our embedding model’s context window.</p><p><strong>Overlap </strong>: Prevents important information from being split across chunks, ensuring continuity in retrieved content.</p><p><strong>Hierarchical Separators</strong>: The text splitter tries to split on paragraphs first, then sentences, then smaller units, preserving document structure.</p><p><strong>Temporary File Handling</strong>: We use temporary files for security and cleanliness, ensuring uploaded files don’t persist on the server.</p><h3>Step 2: Vector Database and Embeddings</h3><p>This is where the magic of semantic search happens. We convert text chunks into high-dimensional vectors that capture their meaning.</p><h4>Why ChromaDB as Our Vector Store?</h4><p>ChromaDB offers several advantages for our use case:</p><ul><li><strong>Persistent storage</strong> — Data survives application restarts</li><li><strong>Efficient similarity search</strong> — Optimized for vector operations</li><li><strong>Local deployment</strong> — No external dependencies</li><li><strong>Easy integration</strong> — Simple Python API</li></ul><h4>Initializing the ChromaDB and Embeddings</h4><p>Decorator to cache functions that return global resources (e.g. database connections, ML models).</p><p>Cached objects are shared across all users, sessions, and reruns. They must be thread-safe because they can be accessed from multiple threads concurrently. If thread safety is an issue, consider using st.session_state to store resources per session instead.</p><p><strong>Why PersistentClient?</strong></p><ul><li><strong>Data Persistence</strong>: Your processed documents remain available between sessions</li><li><strong>Performance</strong>: Avoids re-processing documents on every restart</li><li><strong>Scalability</strong>: Can handle growing document collections efficiently</li><li><strong>Cost Efficiency</strong>: No need to re-embed documents repeatedly</li></ul><h4>Handling Embedding Dimension Inconsistencies</h4><p>One critical challenge we address is embedding dimension consistency then the final code:</p><pre>@st.cache_resource<br>def initialize_components():<br>    &quot;&quot;&quot;Initialize embeddings and Chroma client&quot;&quot;&quot;<br>    try:<br>        embeddings = OllamaEmbeddings(model=&quot;mxbai-embed-large&quot;)<br>        client = chromadb.PersistentClient(path=&quot;chroma_store&quot;)<br>        <br>        # Test both embedding methods to detect inconsistency<br>        test_single = embeddings.embed_query(&quot;test&quot;)<br>        test_batch = embeddings.embed_documents([&quot;test&quot;])<br>        <br>        single_dim = len(test_single)<br>        batch_dim = len(test_batch[0]) if test_batch else 0<br>        <br>        if single_dim != batch_dim:<br>            st.warning(f&quot;⚠️ Embedding dimension inconsistency detected!&quot;)<br>            st.warning(f&quot;Single query: {single_dim}d, Batch: {batch_dim}d&quot;)<br>            st.info(&quot;Using batch embedding dimension for consistency.&quot;)<br>            embedding_dim = batch_dim<br>        else:<br>            embedding_dim = single_dim<br>        <br>        return embeddings, client, embedding_dim<br>    except Exception as e:<br>        st.error(f&quot;Failed to initialize components: {str(e)}&quot;)<br>        st.error(&quot;Make sure Ollama is running and the mxbai-embed-large model is installed.&quot;)<br>        return None, None, None</pre><p>This addresses a common issue where different embedding methods return vectors of different dimensions, which would cause vector database errors.</p><h4>The Importance of @st.cache_resource</h4><p>The @st.cache_resource decorator is crucial for several reasons:</p><ol><li><strong>Expensive Operations</strong>: Initializing embeddings and database connections is computationally expensive</li><li><strong>Singleton Pattern</strong>: Ensures only one instance of resources across the application</li><li><strong>Memory Efficiency</strong>: Prevents resource duplication on Streamlit reruns</li><li><strong>Performance</strong>: Dramatically improves application responsiveness</li></ol><p>Without caching, every user interaction would reinitialize the entire embedding pipeline, making the app unusably slow.</p><pre>def vector_database(docs: list, file_name: str):<br>    &quot;&quot;&quot;Add documents to vector database with unique IDs&quot;&quot;&quot;<br>    if not st.session_state.collection_initialized:<br>        st.error(&quot;Vector database not initialized properly.&quot;)<br>        return False<br>    <br>    try:<br>        texts = [doc.page_content for doc in docs]<br>        embeddings_list = embeddings.embed_documents(texts)<br>        <br>        # Use filename + index for unique IDs to avoid conflicts<br>        for i, (text, emb) in enumerate(zip(texts, embeddings_list)):<br>            doc_id = f&quot;{file_name}_{i}&quot;<br>            collection.add(<br>                ids=[doc_id],<br>                documents=[text],<br>                embeddings=[emb]<br>            )<br>        return True<br>    except Exception as e:<br>        st.error(f&quot;Error adding documents to vector database: {str(e)}&quot;)<br>        return False</pre><h3>Step 3: Query Collection — The Heart of Retrieval</h3><p>When a user asks a question, we need to find the most relevant document chunks. This involves converting the query to a vector and performing similarity search.</p><h4>Robust Query Implementation</h4><pre># Step 3: Query function<br>def query_collection(prompt: str, n_results: int = 5):<br>    &quot;&quot;&quot;Query the collection for relevant documents&quot;&quot;&quot;<br>    if not st.session_state.collection_initialized:<br>        st.error(&quot;No documents have been processed yet.&quot;)<br>        return None<br>    <br>    try:<br>        query_embedding = embeddings.embed_documents([prompt])[0]<br><br>        # Use the embedding directly instead of query_texts<br>        results = collection.query(<br>            query_embeddings=[query_embedding], <br>            n_results=n_results<br>        )<br>        return results<br>    except Exception as e:<br>        st.error(f&quot;Error querying collection: {str(e)}&quot;)<br>        # Fallback: try with single query method<br>        try:<br>            st.info(&quot;Trying fallback query method...&quot;)<br>            query_embedding = embeddings.embed_query(prompt)<br>            st.write(f&quot;Fallback query embedding dimension: {len(query_embedding)}&quot;)<br>            <br>            results = collection.query(<br>                query_embeddings=[query_embedding], <br>                n_results=n_results<br>            )<br>            return results<br>        except Exception as e2:<br>            st.error(f&quot;Fallback also failed: {str(e2)}&quot;)<br>            return None</pre><h4>Why the Fallback Mechanism?</h4><p>Different embedding models may have inconsistent APIs between single and batch embedding methods. Our fallback ensures the application works regardless of these inconsistencies, providing a robust user experience.</p><h4>Optimizing Retrieval</h4><p><strong>Number of Results (5)</strong>: Balances between comprehensive context and token efficiency. Too few results miss important information; too many create noise and exceed token limits.</p><p><strong>Embedding Consistency</strong>: We use the same embedding method for both indexing and querying to ensure accurate similarity calculations.</p><h3>Step 4: LLM Integration — Generating Intelligent Responses</h3><p>The final step transforms retrieved chunks into coherent, contextual answers using a large language model.</p><h4>Crafting the Perfect System Prompt</h4><p>Our system prompt is carefully designed to ensure accurate, grounded responses:</p><pre>system_prompt = &quot;&quot;&quot;<br>You are an intelligent assistant that answers questions based on provided context from documents. Your role is to:<br><br>1. **Analyze the provided context carefully** and extract relevant information to answer the user&#39;s question<br>2. **Answer based ONLY on the information provided** in the context - do not use external knowledge<br>3. **Be accurate and precise** - if the context doesn&#39;t contain enough information to answer the question, clearly state this<br>4. **Quote directly from the context** when appropriate, using quotation marks<br>5. **Maintain the same tone and style** as the source material when possible<br><br>## Instructions:<br>- If the answer is clearly stated in the context, provide a direct answer<br>- If the context contains partial information, explain what you can determine and what is unclear<br>- If the context doesn&#39;t contain relevant information, respond with: &quot;The provided context doesn&#39;t contain enough information to answer this question.&quot;<br>- Always be honest about the limitations of the provided context<br><br>## Context:<br>{context}<br><br>## Question:<br>{question}<br><br>## Answer:<br>&quot;&quot;&quot;</pre><h3>Why This Approach Works</h3><p><strong>Explicit Instructions</strong>: Clear guidelines prevent hallucination and ensure grounded responses.</p><p><strong>Context Limitation</strong>: By instructing the model to only use provided context, we maintain accuracy and prevent fabricated information.</p><p><strong>Fallback Behavior</strong>: When information isn’t available, the model explicitly states this rather than guessing.</p><h4>LangChain Integration</h4><pre>@st.cache_resource<br>def initialize_llm():<br>    &quot;&quot;&quot;Initialize the LLM model&quot;&quot;&quot;<br>    try:<br>        llm = ChatOllama(model=&quot;llama3&quot;, temperature=0.7)<br>        return llm<br>    except Exception as e:<br>        st.error(f&quot;Error initializing LLM: {str(e)}&quot;)<br>        return None<br><br>def call_llm(context: str, prompt: str):<br>    &quot;&quot;&quot;Generate response using LLM with context and question&quot;&quot;&quot;<br>    llm = initialize_llm()<br>    if llm is None:<br>        return &quot;Sorry, I couldn&#39;t initialize the language model. Please check your Ollama installation.&quot;<br>    <br>    try:<br>        # Create the prompt template<br>        chat_prompt = ChatPromptTemplate.from_template(system_prompt)<br>        <br>        # Create the chain<br>        output_parser = StrOutputParser()<br>        chain = chat_prompt | llm | output_parser<br>            <br>        # Invoke the chain with context and question<br>        response = chain.invoke({<br>            &quot;context&quot;: context,<br>            &quot;question&quot;: prompt<br>        })<br>        <br>        return response<br>    except Exception as e:<br>        return f&quot;Error generating response: {str(e)}&quot;</pre><p><strong>Why LangChain?</strong></p><ul><li><strong>Standardized Interface</strong>: Consistent API across different LLM providers</li><li><strong>Chain Composition</strong>: Easy to modify and extend the pipeline</li><li><strong>Error Handling</strong>: Built-in robustness for production use</li><li><strong>Flexibility</strong>: Can swap models without changing core logic</li></ul><h3>Step 5: Streamlit Application — Bringing It All Together</h3><p>Streamlit transforms our RAG pipeline into an accessible web application with minimal code.</p><h4>Key UI Components</h4><p><strong>Sidebar for Document Management:</strong></p><ul><li>File upload interface</li><li>Processing controls</li><li>Database management tools</li></ul><p><strong>Main Chat Interface</strong>:</p><ul><li>Question input</li><li>Response display</li><li>Source context revelation</li></ul><h4>Session State Management</h4><pre>if &#39;processed_files&#39; not in st.session_state:<br>    st.session_state.processed_files = set()<br>if &#39;collection_initialized&#39; not in st.session_state:<br>    st.session_state.collection_initialized = False</pre><p>Session state ensures:</p><ul><li><strong>Persistence</strong>: User data survives page refreshes</li><li><strong>State Tracking</strong>: Application knows what files have been processed</li><li><strong>User Experience</strong>: Prevents redundant processing</li></ul><p>Now, The streamlit Implementation:</p><pre>with st.sidebar:<br>    st.header(&quot;📄 Document Upload&quot;)<br>    uploaded_file = st.file_uploader(&quot;**Upload your PDF files for QnA:**&quot;, type=[&#39;pdf&#39;])<br>    process = st.button(&quot;Process PDF&quot;)<br>    <br>    st.header(&quot;🗂️ Database Management&quot;)<br>    if st.button(&quot;Clear Current Collection&quot;):<br>        if st.session_state.collection_initialized and embeddings and client:<br>            try:<br>                collection_name = f&quot;docs_mxbai_{embedding_dim}d&quot;<br>                client.delete_collection(name=collection_name)<br>                collection = client.get_or_create_collection(name=collection_name)<br>                st.session_state.processed_files.clear()<br>                st.success(&quot;Current collection cleared successfully!&quot;)<br>            except Exception as e:<br>                st.error(f&quot;Error clearing collection: {str(e)}&quot;)<br>    <br>    if st.button(&quot;Clear All Old Collections&quot;):<br>        if client:<br>            try:<br>                # Get all collections<br>                collections = client.list_collections()<br>                cleared_count = 0<br>                <br>                for coll in collections:<br>                    # Clear old collections with different names<br>                    if coll.name.startswith(&quot;docs&quot;) and coll.name != f&quot;docs_mxbai_{embedding_dim}d&quot;:<br>                        client.delete_collection(name=coll.name)<br>                        cleared_count += 1<br>                <br>                st.success(f&quot;Cleared {cleared_count} old collection(s)!&quot;)<br>                st.session_state.processed_files.clear()<br>            except Exception as e:<br>                st.error(f&quot;Error clearing old collections: {str(e)}&quot;)<br><br># Main processing logic<br>if uploaded_file and process:<br>    if not st.session_state.collection_initialized:<br>        st.error(&quot;System not properly initialized. Please check Ollama connection.&quot;)<br>    elif uploaded_file.name in st.session_state.processed_files:<br>        st.warning(f&quot;File &#39;{uploaded_file.name}&#39; has already been processed.&quot;)<br>    else:<br>        with st.spinner(&quot;Processing PDF...&quot;):<br>            all_splits = process_pdf(uploaded_file)<br>            if all_splits:<br>                if vector_database(all_splits, uploaded_file.name):<br>                    st.session_state.processed_files.add(uploaded_file.name)<br>                    st.success(f&quot;✅ Document &#39;{uploaded_file.name}&#39; parsed and added to vector database.&quot;)<br>                    st.info(f&quot;Added {len(all_splits)} document chunks to the database.&quot;)<br><br># Chat interface<br>st.header(&quot;💬 Ask Questions&quot;)<br>query = st.chat_input(&quot;Enter your question about the uploaded file:&quot;)<br><br>if query:<br>    if not st.session_state.collection_initialized:<br>        st.error(&quot;Please upload and process a PDF file first.&quot;)<br>    else:<br>        with st.spinner(&quot;Searching for relevant information...&quot;):<br>            results = query_collection(query)<br>            <br>        if results:<br>            top_docs = results.get(&quot;documents&quot;, [[]])[0]<br>            if top_docs:<br>                # Combine top documents as context<br>                context = &quot;\n\n&quot;.join(top_docs[:3])  # Use top 3 documents<br>                <br>                with st.spinner(&quot;Generating response...&quot;):<br>                    response = call_llm(context, query)<br>                <br>                # Display the response prominently<br>                st.markdown(&quot;### 🤖 Assistant Response&quot;)<br>                st.markdown(response)<br>                <br>                # Show context in expandable sections<br>                st.markdown(&quot;### 🔍 Source Context&quot;)<br>                for i, doc in enumerate(top_docs, 1):<br>                    with st.expander(f&quot;Source {i}&quot;, expanded=False):<br>                        st.markdown(doc)<br>            else:<br>                st.warning(&quot;No relevant documents found for your query.&quot;)<br>        else:<br>            st.error(&quot;Unable to search the document database.&quot;)<br><br># Display processed files<br>if st.session_state.processed_files:<br>    with st.sidebar:<br>        st.header(&quot;📋 Processed Files&quot;)<br>        for file_name in st.session_state.processed_files:<br>            st.text(f&quot;✅ {file_name}&quot;)</pre><h4>Advanced Features</h4><p><strong>Collection Management</strong>: Users can clear old collections to manage storage and avoid dimension conflicts.</p><p><strong>Error Recovery</strong>: Comprehensive error handling with user-friendly messages and fallback mechanisms.</p><p><strong>Visual Feedback</strong>: Progress indicators and status messages keep users informed during processing.</p><h3>Production Considerations</h3><h3>Performance Optimizations</h3><ol><li><strong>Caching Strategy</strong>: @st.cache_resource for expensive operations</li><li><strong>Lazy Loading</strong>: Components initialize only when needed</li><li><strong>Efficient Chunking</strong>: Optimized chunk sizes for speed and accuracy</li><li><strong>Connection Pooling</strong>: Persistent database connections</li></ol><h3>Error Handling</h3><p>Our application implements multiple layers of error handling:</p><ul><li><strong>File Processing</strong>: Graceful handling of corrupted PDFs</li><li><strong>Embedding Failures</strong>: Fallback embedding methods</li><li><strong>Database Errors</strong>: Clear error messages and recovery suggestions</li><li><strong>LLM Issues</strong>: Informative error responses</li></ul><h3>Scalability Considerations</h3><p><strong>Local vs. Cloud Deployment</strong>:</p><ul><li>Local deployment offers privacy and cost control</li><li>Cloud deployment enables collaboration and resource scaling</li></ul><p><strong>Model Selection</strong>:</p><ul><li>mxbai-embed-large for high-quality embeddings</li><li>llama3 for balanced performance and accuracy</li><li>Easy model swapping for different use cases</li></ul><h3>Future Enhancements</h3><h4>Advanced Features to Consider</h4><ol><li><strong>Multi-Document Chat</strong>: Conversations spanning multiple documents</li><li><strong>Citation Tracking</strong>: Automatic source attribution in responses</li><li><strong>Advanced Chunking</strong>: Semantic chunking based on document structure</li><li><strong>User Authentication</strong>: Multi-user support with isolated document stores</li><li><strong>API Integration</strong>: RESTful API for programmatic access</li></ol><h4>Performance Improvements</h4><ol><li><strong>Async Processing</strong>: Non-blocking document processing</li><li><strong>Batch Operations</strong>: Efficient handling of multiple documents</li><li><strong>Caching Layers</strong>: Response caching for common queries</li><li><strong>Model Optimization</strong>: Fine-tuned models for specific domains</li></ol><h3>Conclusion</h3><p>Building a production-ready RAG application requires careful consideration of each component in the pipeline. Our implementation demonstrates several key principles:</p><ul><li><strong>Robustness</strong>: Comprehensive error handling and fallback mechanisms</li><li><strong>Performance</strong>: Strategic caching and resource management</li><li><strong>User Experience</strong>: Intuitive interface with clear feedback</li><li><strong>Maintainability</strong>: Clean, modular code structure</li></ul><p>The combination of Streamlit’s simplicity, ChromaDB’s persistence, and Ollama’s local model serving creates a powerful, privacy-focused document Q&amp;A system. Whether you’re building internal knowledge bases, research tools, or customer support systems, this architecture provides a solid foundation.</p><p>The code is production-ready, handles edge cases gracefully, and can be extended for more advanced use cases. Most importantly, it runs entirely locally, ensuring your sensitive documents remain private and secure.</p><p><strong>Key Takeaways:</strong></p><ul><li>RAG combines retrieval and generation for accurate, contextual responses</li><li>Proper chunking and embedding strategies are crucial for performance</li><li>Persistent storage and caching dramatically improve user experience</li><li>Robust error handling is essential for production deployment</li><li>Local deployment offers privacy and cost advantages over cloud solutions</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ffc9d1ba586" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own Transformer : A Complete Step-by-Step Implementation Guide]]></title>
            <link>https://medium.com/@anjilakshetri/build-your-own-transformer-a-complete-step-by-step-implementation-guide-4680443df83b?source=rss-5a2b01018d9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4680443df83b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[transformer-architecture]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Anjilakshetri]]></dc:creator>
            <pubDate>Tue, 27 May 2025 08:57:00 GMT</pubDate>
            <atom:updated>2025-05-27T08:57:00.620Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build Your Own Transformer : A Complete Step-by-Step Implementation Guide</h3><p><em>Understanding the architecture that revolutionized NLP by building it from scratch</em></p><p>The Transformer architecture, introduced in the groundbreaking paper “Attention Is All You Need,” has become the backbone of modern natural language processing. From GPT to BERT, from ChatGPT to Google’s LaMDA, transformers power the AI systems that are reshaping our world.</p><p>But how do transformers actually work under the hood? The best way to truly understand any complex system is to build it yourself. In this comprehensive guide, we’ll implement a transformer from scratch using PyTorch, breaking down each component step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZGON_NygaDgzacrWLOeXA.png" /><figcaption>Understanding Transformer Architecture</figcaption></figure><h3>What We’ll Build</h3><p>By the end of this article, you’ll have implemented:</p><ul><li>Input embeddings and positional encoding</li><li>Multi-head self-attention mechanism</li><li>Feed-forward networks</li><li>Layer normalization and residual connections</li><li>Complete encoder and decoder blocks</li><li>A full transformer model</li></ul><p>Let’s start our journey!</p><h3>Understanding the Big Picture</h3><p>Before diving into code, let’s understand what a transformer does at a high level. The original transformer was designed for sequence-to-sequence tasks like machine translation, where you input a sequence in one language and get a sequence in another language.</p><pre>import torch<br>import torch.nn as nn<br><br># PyTorch provides a built-in transformer<br>model = nn.Transformer(<br>    d_model=512,<br>    nhead=8,<br>    num_encoder_layers=6,<br>    num_decoder_layers=6<br>)<br><br>print(model)</pre><p>When you print this model, you’ll see the transformer contains encoder and decoder blocks, each with multi-head attention, feed-forward networks, layer normalization, and dropout layers. We’ll implement each of these components ourselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pnf8XFfm4wxYwXwDdIE59Q.png" /><figcaption>Full Architecture of the Transformer</figcaption></figure><p>Let’s go step by step starting from :</p><h3>Input Embeddings — Converting Words to Vectors</h3><p>The first challenge any neural network faces with text is that computers don’t understand words — they understand numbers. This is where embeddings come in.</p><h3>Why Do We Need Embeddings?</h3><p>Imagine you have the tokens: [“Cat”, “Dog”, “Fish”]</p><p>Each token has a unique ID in the model’s vocabulary:</p><ul><li>“Cat” → ID: 833</li><li>“Dog” → ID: 234</li><li>“Fish” → ID: 310</li></ul><p>But just using these IDs as-is doesn’t capture semantic meaning. The number 234 isn’t inherently more similar to 3 10 than to 833, but “Dog” and “Cat” (both animals) should be more similar to each other than to “Fish” in some contexts.</p><p>Embeddings solve this by mapping each token to a high-dimensional vector where semantically similar words are closer together in the vector space.</p><h3>Implementing Input Embeddings</h3><pre>import torch<br>import math<br>import torch.nn as nn<br><br>class InputEmbeddings(nn.Module):<br>    def __init__(self, vocab_size: int, d_model: int) -&gt; None:<br>        super().__init__()<br>        <br>        self.d_model = d_model<br>        self.vocab_size = vocab_size<br>        self.embedding = nn.Embedding(vocab_size, d_model)<br><br>    def forward(self, x):<br>        # Multiply by sqrt(d_model) for scaling as per the original paper<br>        return self.embedding(x) * math.sqrt(self.d_model)</pre><p><strong>Key Points:</strong></p><ul><li>vocab_size: The total number of unique tokens in our vocabulary</li><li>d_model: The dimensionality of our embeddings (typically 512 in the original paper)</li><li>We multiply by sqrt(d_model) to scale the embeddings as done in the original transformer paper</li></ul><p>Let’s test our embedding layer:</p><pre># Example usage<br>vocab_size = 10000  # Assuming we have 10,000 unique tokens<br>d_model = 512       # 512-dimensional embeddings<br><br>embedding_layer = InputEmbeddings(vocab_size, d_model)<br><br># Sample input: batch of token IDs<br>input_tokens = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]])  # Shape: (batch_size=2, seq_len=4)<br>embedded = embedding_layer(input_tokens)<br><br>print(f&quot;Input shape: {input_tokens.shape}&quot;)<br>print(f&quot;Embedded shape: {embedded.shape}&quot;)<br># Output: torch.Size([2, 4, 512])</pre><h3>Positional Encoding — Teaching Position Awareness</h3><p>Here’s a crucial insight: unlike RNNs that process sequences step by step, transformers process all tokens in parallel. This is great for speed, but it means the model has no inherent understanding of token positions.</p><p><strong>Positional Encoding</strong> is added to give the model information about the position of each word in a sequence.</p><h3>Why Do We Need Positional Encoding?</h3><p>The word “ate” in “The cat ate the fish” is different from “ate” in “Ate the cat the fish?” — the order matters! Without positional information, the transformer would treat both sentences identically, missing the crucial difference in meaning that comes from word order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ooRKa1PtiTdrUpp6AzOGQw.png" /></figure><h4>The Mathematics Behind Positional Encoding</h4><p>The positional encodings are generated using special encoding equations, where sin is used for even embedding values and cos is used for odd embedding values.</p><p>The positional encoding for position pos and dimension i is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*XTEJEetJHtdvhW_CbQJ-Zw.png" /></figure><p>Where:</p><ul><li>pos is the position in the sequence</li><li>i is the dimension index</li><li>d is the total embedding dimension (d_model)</li></ul><h4>Why Sine and Cosine Functions?</h4><p>Sin and Cosine are periodic functions with values between -1 and 1. But why do we use them specifically?</p><p><strong>1. Provide Unique Patterns for Each Position</strong></p><ul><li>The combination of sin and cos with different frequencies ensures that each position has a unique encoding vector</li><li>No two positions have the same encoding, and nearby positions have similar vectors, which helps the model recognize local context</li></ul><p><strong>2. Capture Relative Position Information</strong></p><ul><li>The sinusoidal form makes it easy for the model to learn relative positions between words</li><li>For example, PE(pos + k) can be expressed as a linear function of PE(pos), allowing the model to infer order differences like “word A is 2 steps ahead of word B”</li></ul><h4>Implementing Positional Encoding</h4><pre>class PositionalEncoding(nn.Module):<br>    def __init__(self, d_model: int, seq_len: int, dropout: float) -&gt; None:<br>        super().__init__()<br>        <br>        self.d_model = d_model<br>        self.seq_len = seq_len<br>        self.dropout = nn.Dropout(dropout)<br>        <br>        # Create a matrix to hold positional encodings<br>        pe = torch.zeros(seq_len, d_model)<br>        <br>        # Create position vector<br>        position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)<br>        <br>        # Create division term for the sinusoidal pattern<br>        div_term = torch.exp(torch.arange(0, d_model, 2).float() * <br>                           (-math.log(10000.0) / d_model))<br>        <br>        # Apply sin to even indices<br>        pe[:, 0::2] = torch.sin(position * div_term)<br>        <br>        # Apply cos to odd indices  <br>        pe[:, 1::2] = torch.cos(position * div_term)<br>        <br>        # Add batch dimension and register as buffer<br>        pe = pe.unsqueeze(0)  # Shape: (1, seq_len, d_model)<br>        <br>        # Register as buffer (not a parameter, but part of the model state)<br>        self.register_buffer(&#39;pe&#39;, pe)<br>    <br>    def forward(self, x):<br>        # Add positional encoding to embeddings<br>        x = x + (self.pe[:, :x.shape[1], :]).requires_grad_(False)<br>        return self.dropout(x)</pre><h4>Testing Our Positional Encoding</h4><pre># Create positional encoding<br>seq_len = 100<br>dropout = 0.1<br>pos_encoding = PositionalEncoding(d_model, seq_len, dropout)<br><br># Test with our embedded tokens<br>positioned_embeddings = pos_encoding(embedded)<br>print(f&quot;Shape after positional encoding: {positioned_embeddings.shape}&quot;)</pre><h3>MultiHead Attention</h3><p>efore diving into multi-head attention, let’s first understand what self-attention is.</p><h4>What is Self-Attention?</h4><p>Self-attention is what enables transformers to identify relationships between tokens and determine which ones to focus on. It allows a model to look at other positions in the same input sequence when encoding a word — hence the name “self” attention.</p><p>Self-attention determines: <strong>“Which other words in the sentence should I pay attention to when understanding this word?”</strong></p><h4>Example:</h4><p>Take the sentence: <em>“The cat sat on the mat because it was warm.”</em></p><p>To understand what “it” refers to, self-attention helps the model focus on “cat” or “mat” rather than every word equally. The model figures this out on its own during training.</p><h4>The Q, K, V Mechanism</h4><p>Each input word is converted into embeddings, and then each embedding is projected into three different matrices: <strong>Q</strong>, <strong>K</strong>, and <strong>V</strong>.</p><ul><li><strong>Q (Query)</strong>: Indicates what each token is looking for in other tokens</li><li><strong>K (Key)</strong>: Represents the content of each token that other tokens might find relevant</li><li><strong>V (Value)</strong>: The actual content to be aggregated or weighted</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24YoOCDH7Foiml-L4Pg-8w.png" /></figure><h4>🧠 Analogy: Job Search Example</h4><p>Imagine you’re trying to hire someone:</p><ul><li>Your <strong>Query (Q)</strong> is the job requirement</li><li>Each candidate has a <strong>Key (K)</strong> = their resume</li><li>The actual <strong>Value (V)</strong> is what you’d get if you hired them</li></ul><p>You compare your Query to all the Keys (resumes) to get scores, then use those scores to weigh the Values (candidates’ actual skills).</p><h4>The Self-Attention Process</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*_i-zGNNcZ0Mo1VLBMBeCxw.png" /></figure><p>Values are based on attention scores, which are computed by doing the dot-product of the Key and Query matrices:</p><p><strong>Attention scores = Q·K similarity (dot-product)</strong></p><p>From the attention scores, we apply softmax to get the attention weights.</p><h4>Example in Action:</h4><p>In <em>“orange is my favorite fruit,”</em> the tokens “favorite” and “fruit” receive the highest attention when processing “orange,” as they directly influence its context and meaning. The model interprets “orange” as a favored fruit rather than a color.</p><h4>⚙️ Step-by-step Self-Attention:</h4><h4>1. Compute Dot Products: Q·K^T</h4><p>This gives a score of how much attention word A should pay to word B.</p><h4>2. Scale and Apply Softmax and Multiply with V:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zFa9guHF8G5BhIAA3I-FQg.png" /></figure><p>This normalizes the scores into probabilities and Each word’s final output is a <strong>weighted sum of all the Value vectors</strong>, based on attention.</p><h3>Multi-Head Attention</h3><p>Multi-Head Attention is an advanced form of self-attention. Instead of calculating just one set of attention outputs (with one Q/K/V), it creates multiple “attention heads” — each learning different relationships or features in the input.</p><h4>⚙️ Why do we need Multi-Head Attention?</h4><p>A single self-attention layer may focus too narrowly. With multi-head attention:</p><ul><li>Each head looks at the sequence from a different perspective</li><li>Some heads may learn syntax (e.g., subject-verb links), others learn semantics (e.g., coreference, word meaning)</li><li>This makes the model much more expressive</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nIk-4ZHejAZkXhEwp_TVKA.png" /></figure><p>The resulting embeddings capture token meaning, positional encoding, and contextual relationships.</p><h4>Implementing Multi-Head Attention</h4><pre>import torch.nn as nn<br>import torch.nn.functional as F<br><br>class MultiHeadAttention(nn.Module):<br>    def __init__(self, d_model, num_heads):<br>        super().__init__()<br>        assert d_model % num_heads == 0, &quot;d_model must be divisible by 0&quot;<br><br>        self.num_heads = num_heads  <br>        self.d_model = d_model<br><br>        self.head_dim = d_model // num_heads<br><br>        #Three linear layers are defined for the attention inputs: query, key, and value, and one for the final concatenated output. <br>        #Using bias=False for query, key, and value layers eliminates the bias term, <br>        # reducing model parameters without impacting the ability to capture relationships.<br>        self.query_linear = nn.Linear(d_model, d_model, bias=False)<br>        self.key_linear = nn.Linear(d_model, d_model, bias=False)<br>        self.value_linear = nn.Linear(d_model, d_model, bias=False)<br>        self.output_linear = nn.Linear(d_model, d_model)<br><br>    def split_heads(self, x, batch_size):<br>        seq_length = x.size(1)<br>        x = x.reshape(batch_size, seq_length, self.num_heads, self.head_dim)<br>        return x.permute(0,2,1,3)<br>    <br>    def compute_attention(self, query, key, value, mask = None):<br>        scores = torch.matmul(query, key.transpose(-2, -1)) / (self.head_dim ** 0.5)<br>        if mask is not None:<br>            scores = scores.masked_fill(mask == 0, float(&#39;-inf&#39;))<br>        attention_weights = F.softmax(scores, dim = 1)<br>        return torch.matmul(attention_weights, value)<br>    <br>    def combine_heads(self, x, batch_size):<br>        x = x.permute(0,2,1,3).contiguous()<br>        return x.view(batch_size, 1, self.d_model)<br>    <br>    def forward(self, query, key, value, mask = None):<br>        batch_size = query.size(0)<br><br>        query = self.split_heads(self.query_linear(query), batch_size)<br>        key = self.split_heads(self.key_linear(key), batch_size)<br>        value = self.split_heads(self.value_linear(value), batch_size)<br><br>        attention_weights = self.compute_attention(query, key, value, mask)<br><br>        output = self.combine_heads(attention_weights, batch_size)<br><br>        return self.output_linear(output)</pre><pre># Define attention parameters<br>d_model = 512<br>num_heads = 8<br><br># Instantiate a MultiHeadAttention instance<br>multihead_attn = MultiHeadAttention(d_model, num_heads)<br><br># Pass the query, key, and value matrices through the mechanism<br># output = multihead_attn(query, key, value)<br># print(multihead_attn.shape)</pre><h3>FeedForward SubLayer</h3><p>After the Multi-Head Attention layer in a Transformer block, there’s a FeedForward Neural Network (FFN) layer, also known as the FeedForward SubLayer. It adds non-linearity and transformation to each token independently.</p><h4>📌 Why It’s Used</h4><p>While attention layers let tokens communicate, the FFN lets each token transform itself — enriching its internal representation after it has “heard” from others.</p><p>Our FeedForwardSubLayer class contains two fully connected linear layers separated by a ReLU activation. Notice we use a dimension d_ff between linear layers, typically different from the embedding dimension used throughout the model to further facilitate capturing complex patterns.</p><pre>class FeedForwardSubLayer(nn.Module):<br>    def __init__(self, d_model, d_ff):<br>        super().__init__()<br>        self.fc1 = nn.Linear(d_model, d_ff)<br>        self.fc2 = nn.Linear(d_ff, d_model)<br><br>        self.relu = nn.ReLU()<br><br>    def forward(self, x):<br>        return self.fc2(self.relu(self.fc1(x)))</pre><pre>feed_forward = FeedForwardSubLayer(512, 2048)<br># output = feed_forward(x)<br># print(f&quot;Input shape: {x.shape}&quot;)<br># print(f&quot;Output shape: {output.shape}&quot;)</pre><h3>Bringing MultiHead Attention and FeedForward Layer Together: Encoder Layer</h3><p>A Transformer Encoder Layer is a single block in the stack of encoder blocks used in models like BERT, GPT (decoder-only variant), and the original Transformer. Each layer processes a sequence of tokens to build richer, context-aware representations.</p><p>Encoder-only transformers simplify this architecture to place greater emphasis on understanding and representing the input data, such as text classification.</p><p>They have two main components:</p><p>- Each encoder layer incorporates a multi-head self-attention mechanism to capture relationships between tokens in the sequence</p><p>- followed by feed-forward sublayers to map this knowledge into abstract, nonlinear representations. Both elements are usually combined with other techniques like layer normalizations and dropouts to improve training.</p><pre>class EncoderLayer(nn.Module):<br>    def __int__(self, d_model, num_heads, d_ff, dropout):<br>        super().__init__()<br><br>        self.attn = MultiHeadAttention(d_model, num_heads)<br>        self.ff_sublayer = FeedForwardSubLayer(d_model, d_ff)<br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, x, src_mask):<br>        attn_output = self.attn(x,x,x,src_mask)<br>        x = self.norm1(x + self.dropout(attn_output))<br>        ff_output = self.ff_sublayer(x)<br>        x = self.norm2(x + self.dropout(ff_output))<br>        return x</pre><h4>Masking The Attention Process:</h4><p>Padding ensures equal length across sequences by appending padding tokens, or zeros in this case. These padded tokens are irrelevant for the language task so we exclude them from the attention mechanism. We do this by applying a padding mask to the attention scores, so scores linked to padded tokens are set to zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*_j6AMt7EpeSO1TOiOsbDjQ.png" /></figure><p>Now, Finally the Transformer Encoder:</p><pre>class TransformerEncoder(nn.Module):<br>    def __init__(self, vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length):<br>        super().__init__()<br>        # Define the embedding, positional encoding, and encoder layers<br>        self.embedding = InputEmbeddings(vocab_size, d_model)<br>        self.positional_encoding = PositionalEncoding(d_model, max_seq_length)<br>        self.layers = nn.ModuleList([EncoderLayer(d_model, num_heads, d_ff, dropout) for _ in range(num_layers)])<br><br>    def forward(self, x, src_mask):<br>        # Perform the forward pass through the layers<br>        x = self.embedding(x)<br>        x = self.positional_encoding(x)<br>        for layer in self.layers:<br>            x = layer(x, src_mask)<br>        return x</pre><h4>Classification Head</h4><p><strong>Tasks </strong>: Text Classification, Sentiment Analysis, NER Recognition, Extractive QA and more</p><p>It consists of a linear layer with softmax activation to map the resulting encoder hidden states into class probabilities.</p><pre># Complete the classification head<br>class ClassifierHead(nn.Module):<br>    def __init__(self, d_model, num_classes):<br>        super().__init__()<br>        self.fc = nn.Linear(d_model, num_classes)<br><br>    def forward(self, x):<br>        logits = self.fc(x)<br>        return F.log_softmax(logits, dim=-1)</pre><h4><strong>Regression Head</strong></h4><p><strong>Tasks</strong> : Estimate text readbility, language complexity and more</p><p>A regression head has a linear layer with an output dimension equal to the number of target regression outputs.</p><p>It’s suited for tasks like estimating text readability or language complexity.</p><pre># Complete the classification head<br>class RegressionHead(nn.Module):<br>    def __init__(self, d_model, num_classes):<br>        super().__init__()<br>        self.fc = nn.Linear(d_model, num_classes)<br><br>    def forward(self, x):<br>        return self.fc(x)</pre><h3>What is a Transformer Decoder?</h3><p>A Transformer Decoder Layer transforms previously generated tokens into the next token prediction, using both its own input sequence (called the target sequence) and the output from the encoder (called the memory). This dual-input design enables the decoder to maintain context from both what it has generated so far and the original input it’s processing.</p><h4>The Magic of Masked Multi-Head Self-Attention</h4><p>The decoder’s most crucial component is its masked multi-head self-attention mechanism. This is what gives our model its autoregressive or causal behavior, preventing the model from “cheating” by looking at future tokens during training.</p><h4>How Masking Works</h4><p>The masking mechanism ensures that each token can only attend to earlier tokens in the sequence. For instance, during training, if we have the sequence “orange is my favorite fruit,” the token <strong>“favorite”</strong> would only pay attention to the previous tokens: “orange,” “is,” “my,” and “favorite” itself.</p><p>This constraint is crucial because during inference, the model predicts the likelihood of the next token sequentially. In our example, after seeing “orange is my favorite,” the model should assign a relatively high probability to “fruit.”</p><h4>Creating the Causal Mask</h4><p>The implementation of this masking is elegant in its simplicity:</p><pre>seq_length = 4<br><br># Create a Boolean matrix to mask future tokens<br>tgt_mask = (1 - torch.triu(<br>    torch.ones(1, seq_length, seq_length), diagonal=1)<br>).bool()</pre><p>This creates an upper triangular matrix where True values indicate positions the model can attend to, and False values mask future positions.</p><h3>Building the Decoder Layer</h3><p>The decoder layer architecture follows a specific pattern with residual connections and layer normalization:</p><pre>class DecoderLayer(nn.Module):<br>    def __init__(self, d_model, num_heads, d_ff, dropout):<br>        super().__init__()<br>        self.attn = MultiHeadAttention(d_model, num_heads)<br>        self.ff_sublayer = FeedForwardSubLayer(d_model, d_ff)<br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, x, tgt_mask):<br>        attn_output = self.attn(x, x, x, tgt_mask)<br>        x = self.norm1(x + self.dropout(attn_output))<br>        ff_output = self.ff_sublayer(x)<br>        x = self.norm2(x + self.dropout(ff_output))<br>        return x</pre><p>Each layer applies masked self-attention, followed by a feed-forward network, with residual connections and layer normalization at each step.</p><h3>Transformer Decoder</h3><p>The full decoder stacks multiple decoder layers and includes embedding and positional encoding:</p><pre>class TransformerDecoder(nn.Module):<br>    def __init__(self, vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length):<br>        super(TransformerDecoder, self).__init__()<br>        self.embedding = InputEmbeddings(vocab_size, d_model)<br>        self.positional_encoding = PositionalEncoding(d_model, max_seq_length)<br>        self.layers = nn.ModuleList([DecoderLayer(d_model, num_heads, d_ff, dropout) <br>                                   for _ in range(num_layers)])<br>        self.fc = nn.Linear(d_model, vocab_size)<br>  <br>    def forward(self, x, tgt_mask):<br>        x = self.embedding(x)<br>        x = self.positional_encoding(x)<br>        for layer in self.layers:<br>            x = layer(x, tgt_mask)<br>        x = self.fc(x)<br>        return F.log_softmax(x, dim=-1)</pre><p>The final linear layer projects the hidden states to vocabulary-sized vectors, and log softmax converts these to token probabilities.</p><h4>Encoder-Decoder Attention: Building Bridges</h4><p>In encoder-decoder architectures like those used in translation models, the decoder needs to access information from the encoder. This is where encoder-decoder attention (also called cross-attention) comes into play.</p><h3>Cross Attention Mechanism</h3><p>Cross-attention occurs in each decoder layer after the masked self-attention, taking two distinct inputs:</p><ul><li><strong>Queries (Q)</strong> are derived from the decoder’s current state</li><li><strong>Keys (K) and Values (V)</strong> come from the encoder’s output</li></ul><p>This setup allows the decoder to attend to relevant parts of the input sequence while generating each token in the output.</p><h4>A Translation Example</h4><p>Let’s see this in action with a translation example:</p><p><strong>Source Sentence (English):</strong> “We should go.”</p><p><strong>Target Translation (Spanish in progress):</strong> “wir sollten“</p><p>Now the decoder is trying to predict the next word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*JQFpsiCcXSUT3G4eInUzrA.png" /></figure><h4>Step 1: Encoder Processing</h4><p>The encoder reads “We should go.” and stores contextual information for each word. This information becomes the Key (K) and Value (V) vectors.</p><h4>Step 2: Decoder State</h4><p>The decoder has generated “wir sollten” and must predict the next word, ideally “gehen” (Spanish for “go”).</p><h4>Step 3: Cross-Attention Magic</h4><p>The decoder forms a Query (Q) using its current hidden state, influenced by “we sollten” It performs attention over the encoder’s K &amp; V vectors, looking at which source word is most relevant. It attends most strongly to the English word “go,” and based on this, predicts “gehen.”</p><h3>Enhanced Decoder with Cross-Attention</h3><p>The encoder-decoder version of the decoder layer includes an additional cross-attention mechanism:</p><pre>class DecoderLayer(nn.Module):<br>    def __init__(self, d_model: int, num_heads: int, d_ff: int, dropout: float):<br>        super().__init__()<br>        self.self_attn = MultiHeadAttention(d_model, num_heads)<br>        self.cross_attn = MultiHeadAttention(d_model, num_heads)<br>        self.ff = FeedForwardSubLayer(d_model, d_ff)<br><br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.norm3 = nn.LayerNorm(d_model)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, x, enc_out, tgt_mask, cross_mask):<br>        x = self.norm1(x + self.dropout(self.self_attn(x, x, x, tgt_mask)))<br>        x = self.norm2(x + self.dropout(self.cross_attn(x, enc_out, enc_out, cross_mask)))<br>        x = self.norm3(x + self.dropout(self.ff(x)))<br>        return x</pre><p>Notice the three-step process: masked self-attention, cross-attention with encoder output, and feed-forward processing, each with its own normalization layer.</p><h3>Putting It All Together: The Complete Transformer</h3><p>Finally, the complete Transformer combines both encoder and decoder:</p><pre>class Transformer(nn.Module):<br>    def __init__(self, vocab_size, d_model, num_heads, num_layers, d_ff, max_seq_length, dropout):<br>        super().__init__()<br>        self.encoder = TransformerEncoder(vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length)<br>        self.decoder = TransformerDecoder(vocab_size, d_model, num_layers, num_heads, d_ff, dropout, max_seq_length)<br><br>    def forward(self, x, src_mask, tgt_mask, cross_mask):<br>        encoder_output = self.encoder(x, src_mask)<br>        decoder_output = self.decoder(x, encoder_output, tgt_mask, cross_mask)<br>        return decoder_output</pre><h3>Conclusion</h3><p>The Transformer architecture, introduced by Vaswani et al. in 2017, revolutionized natural language processing (NLP) and beyond by replacing recurrent and convolutional layers with self-attention mechanisms, enabling parallel processing and capturing long-range dependencies more effectively. Its key innovations — scaled dot-product attention, multi-head attention, and positional encoding — allow the model to weigh input tokens dynamically and process sequences without relying on sequential operations.</p><p>Transformers have become the foundation for state-of-the-art models like BERT, GPT, and T5, dominating tasks such as machine translation, text generation, and even vision applications through Vision Transformers (ViTs). Their scalability, flexibility, and superior performance solidify Transformers as a cornerstone of modern deep learning architectures.</p><p><strong>References :</strong></p><blockquote><a href="https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf">https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf</a></blockquote><blockquote><a href="https://app.datacamp.com/learn/courses/transformer-models-with-pytorch">Transformer Models with Pytorch — </a>Datacamp</blockquote><blockquote><a href="https://heidloff.net/article/foundation-models-transformers-bert-and-gpt/">https://heidloff.net/article/foundation-models-transformers-bert-and-gpt/</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4680443df83b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>