<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jinad Gamage on Medium]]></title>
        <description><![CDATA[Stories by Jinad Gamage on Medium]]></description>
        <link>https://medium.com/@jinadgamage?source=rss-5ca59f35dc04------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*98Wec5hmbhZU3L9r-D6zQA.jpeg</url>
            <title>Stories by Jinad Gamage on Medium</title>
            <link>https://medium.com/@jinadgamage?source=rss-5ca59f35dc04------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:15:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jinadgamage/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RAG Lesson 04 — Text Embeddings & Semantic Similarity]]></title>
            <link>https://jinadgamage.medium.com/rag-lesson-04-text-embeddings-semantic-similarity-ad2210fc2bab?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad2210fc2bab</guid>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[vector-embeddings]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 05:12:06 GMT</pubDate>
            <atom:updated>2026-11-18T05:12:06.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fMa-xALmzjs4BQhM_deW9A.png" /></figure><p>In Lesson 03, we learned how to split large documents into smaller chunks using different chunking strategies. We also built a Python application using LangChain’s RecursiveCharacterTextSplitter to generate chunks.json from our company policy document.</p><p>Now we have another challenge.</p><p>Our document has been divided into smaller chunks, but how can a computer understand which chunk is relevant to a user’s question?</p><p>For example, consider these two sentences:</p><ul><li>“Employees are entitled to 20 annual leave days.”</li><li>“How much paid vacation do employees receive?”</li></ul><p>The sentences use different words, but they discuss the same topic.</p><p>A traditional keyword-based search might struggle to identify this relationship. A semantic search system can use numerical representations of meaning to find related information.</p><p>This is where <strong>Text Embeddings</strong> become important.</p><p>In this lesson, we’ll learn how text embeddings work, what embedding models do, how semantic similarity is calculated, and how embeddings help a RAG system retrieve relevant information.</p><p>We’ll also build a Python application that reads the chunks.json file generated in Lesson 03, converts each chunk into an embedding vector, and demonstrates semantic search using a real embedding model.</p><h3>What Are Text Embeddings?</h3><p>Text Embeddings are numerical representations of text generated by machine learning models.</p><p>An embedding model converts text such as words, sentences, or paragraphs into a list of numbers called a <strong>vector</strong>.</p><p>These vectors are designed to capture useful semantic relationships between texts.</p><p><strong>Example</strong></p><p>Input text:</p><pre>Employees receive 20 annual leave days.</pre><p>An embedding model might produce a vector resembling:</p><pre>[0.12, -0.45, 0.78, 0.23, -0.16, ...]</pre><p>This is only a simplified illustration. Real embedding models commonly generate vectors containing hundreds or thousands of numerical values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*qrUcW3qB93ktOg4KgeezbA.png" /></figure><p>The generated vector is not a translation of the sentence into individual numbers. Instead, the embedding model learns a mathematical representation of the text.</p><h4>Why is this useful?</h4><p>Suppose we have two sentences:</p><pre>Sentence A: Employees receive 20 annual leave days.<br><br>Sentence B: Staff members are entitled to 20 days of paid vacation.</pre><p>Although the sentences use different words, they have similar meanings.</p><p>A well-trained embedding model should generally represent these sentences with relatively similar vectors.</p><p>This allows us to search for information based on meaning rather than only exact keywords.</p><h3>Embedding Model</h3><p>An Embedding Model is a machine learning model trained to convert text into numerical vectors that capture useful relationships between texts.</p><p>Different embedding models produce different vector dimensions and have different performance characteristics.</p><p>Popular examples include:</p><ul><li><strong>all-MiniLM-L6-v2</strong> — A lightweight Sentence Transformers model producing 384-dimensional embeddings.</li><li><strong>BAAI/bge-small-en-v1.5</strong> — An English embedding model producing 384-dimensional embeddings.</li><li><strong>text-embedding-3-small</strong> — An OpenAI embedding model with a default output dimension of 1,536.</li><li><strong>Gemini Embedding models</strong> — Google’s embedding models for semantic representations.</li></ul><p>In our practical, we will use sentence-transformers/all-MiniLM-L6-v2.</p><p>Why this model?</p><p>It is relatively small, can run locally, does not require an API key, and is suitable for learning how embeddings and semantic search work.</p><p>The model is primarily designed for English text, which matches our example company policy document.</p><h3>Understanding Embedding Dimensions</h3><p>Embedding Dimensions refer to the number of numerical values contained in an embedding vector.</p><p>For example:</p><pre>embedding = [0.12, 0.34, -0.56]</pre><p>This vector has three dimensions.</p><p>A 384-dimensional embedding contains 384 numerical values.</p><pre>Dimension 1   → 0.12<br>Dimension 2   → -0.45<br>Dimension 3   → 0.78<br>...<br>Dimension 384 → 0.31</pre><p>Each dimension is part of a learned representation. Individual dimensions do not necessarily correspond to human-readable concepts such as “leave”, “salary”, or “security”.</p><h4>Why Do Dimensions Matter</h4><p>Embedding dimensions affect vector storage requirements, computational cost, and the representational capacity of the model.</p><p>Higher dimensions are not automatically better. Retrieval quality depends on the model’s training, data, and evaluation results.</p><p><strong><em>Important:</em></strong><em> Document embeddings and query embeddings should be generated using the same compatible embedding model and processing configuration.</em></p><p><em>Otherwise, comparing their vectors may not produce meaningful similarity scores.</em></p><h3>Semantic Similarity</h3><p>Semantic Similarity measures how closely two pieces of text are related in meaning.</p><p>For example:</p><pre>Text A: How many vacation days do employees receive?<br><br>Text B: Employees are entitled to 20 annual leave days.<br><br>Text C: Employees must report security incidents immediately.</pre><p>Text A and Text B are semantically related because both discuss employee leave.</p><p>Text A and Text C are less related because they discuss different topics.</p><p>Embedding models allow us to estimate these relationships mathematically.</p><h4>Visualizing Semantic Similarity</h4><p>Conceptual illustration: semantically related text may appear closer in embedding space. Real embeddings have many more dimensions, and distances depend on the similarity metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*M20CSdeBNwT12HPJziR61A.png" /></figure><h3>Cosine Similarity</h3><p>Cosine Similarity is a common method used to measure the similarity between two embedding vectors.</p><p>It calculates the cosine of the angle between the vectors.</p><p>The formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*T9GcgklgeuCDFVkcKUXCVw.png" /></figure><p>Where:</p><ul><li>A and B are embedding vectors.</li><li>A . B is the dot product.</li><li>|A| and |B| are the vector magnitudes.</li></ul><p>Cosine similarity normally ranges from -1 to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*giOPUAZWcaSnrC4ViH6kqA.png" /></figure><p>In text retrieval, higher cosine similarity often indicates a stronger semantic relationship.</p><p>However, a similarity score is <strong>not a probability that the answer is correct</strong>.</p><p><strong>Simple Mathematical Example</strong></p><p>Consider two vectors:</p><pre>A = [1, 0]<br>B = [1, 0]<br>C = [0, 1]</pre><p>The cosine similarity between A and B is 1 because they point in the same direction.</p><p>The cosine similarity between A and C is 0 because they are perpendicular.</p><p>In real RAG applications, we calculate similarity using embedding vectors produced by an embedding model rather than manually defined vectors.</p><h3>How Embeddings Work in RAG</h3><p>A RAG system generally uses embeddings in two stages.</p><h4>Stage 1 -Document Indexing</h4><p>The system converts document chunks into embeddings and stores them for later retrieval.</p><pre>PDF Document<br>     ↓<br>Text Extraction<br>     ↓<br>Text Chunking<br>     ↓<br>Embedding Model<br>     ↓<br>Document Embeddings<br>     ↓<br>Vector Database</pre><h4>Stage 2 - Query Retrieval</h4><p>When a user asks a question, the system generates an embedding for the question using the same compatible model.</p><p>It then compares the query embedding with stored document embeddings to identify relevant chunks.</p><pre>User Question<br>     ↓<br>Embedding Model<br>     ↓<br>Query Embedding<br>     ↓<br>Similarity Search<br>     ↓<br>Relevant Document Chunks<br>     ↓<br>LLM Generates Answer</pre><p>For example:</p><p><strong>User Question:</strong></p><blockquote><em>How many vacation days are employees entitled to?</em></blockquote><p>The system creates a query embedding and compares it against the stored embeddings.</p><p>A chunk containing the annual leave policy may receive a higher similarity score than a chunk containing information security rules.</p><p>The retrieved chunk can then be passed to the LLM as supporting context.</p><p><strong><em>Important:</em></strong><em> Embedding similarity helps identify potentially relevant text. It does not guarantee factual correctness or answer completeness.</em></p><h3>Practical — Generate Text Embeddings &amp; Perform Semantic Search</h3><p>In Lesson 03, we generated a file called chunks.json.</p><p>Each entry contains a chunk of text, a unique chunk ID, and metadata.</p><p>In this practical, we will build a Python application that:</p><ol><li>Loads the chunks generated in Lesson 03.</li><li>Loads a local Sentence Transformers embedding model.</li><li>Converts every chunk into a numerical embedding vector.</li><li>Saves the embeddings in JSON format.</li><li>Converts a user question into a query embedding.</li><li>Calculates cosine similarity between the query and document embeddings.</li><li>Displays the most relevant chunks.</li></ol><p>This will be our first working example of semantic retrieval.</p><p>We will not use a vector database yet. Instead, we will calculate similarity directly in Python so we can understand the underlying process.</p><h4>Step 1-Project Structure</h4><p>We will continue using the same repository from the previous lessons.</p><pre>rag-fundamentals-to-advanced/<br>│<br>├── Lesson-02-Document-Loading-Text-Processing/<br>│   └── output/<br>│       └── documents.json<br>│<br>├── Lesson-03-Text-Chunking/<br>│   └── output/<br>│       └── chunks.json<br>│<br>└── Lesson-04-Text-Embeddings/<br>    ├── output/<br>    │   └── embeddings.json<br>    ├── main.py<br>    └── requirements.txt</pre><p><em>The </em><em>output directory will be created automatically when we run the application.</em></p><h4>Step 2-Install Dependencies</h4><p>We will use two external Python packages.</p><p><strong>Sentence Transformers</strong> loads the embedding model and converts text into vectors.</p><p><strong>NumPy</strong> helps us perform numerical calculations on the embedding vectors.</p><p>Create requirements.txt:</p><pre>sentence-transformers<br>numpy</pre><p>Install:</p><pre>pip install -r requirements.txt</pre><p>The embedding model will be downloaded automatically the first time it is loaded. After the model is cached locally, it can be reused without downloading it again.</p><p>No API key is required for this practical.</p><h4>Step 3 -Complete Python Implementation</h4><p>Create main.py inside the Lesson 04 folder.</p><pre><br>import json<br>from pathlib import Path<br><br>import numpy as np<br>from sentence_transformers import SentenceTransformer<br><br><br># Keep paths relative to the lesson folders.<br>BASE_DIR = Path(__file__).parent<br><br># Read the chunks generated in Lesson 03.<br>INPUT_PATH = (<br>    BASE_DIR.parent<br>    / &quot;Lesson-03-Text-Chunking&quot;<br>    / &quot;output&quot;<br>    / &quot;chunks.json&quot;<br>)<br><br># Output configuration.<br>OUTPUT_DIR = BASE_DIR / &quot;output&quot;<br>EMBEDDINGS_OUTPUT_PATH = OUTPUT_DIR / &quot;embeddings.json&quot;<br><br># Embedding model configuration.<br>MODEL_NAME = &quot;sentence-transformers/all-MiniLM-L6-v2&quot;<br><br># Number of relevant chunks to retrieve.<br>TOP_K = 3<br><br><br>def load_chunks(file_path: Path) -&gt; list[dict]:<br>    &quot;&quot;&quot;Load the text chunks generated in Lesson 03.&quot;&quot;&quot;<br><br>    # Check whether the input JSON file exists.<br>    if not file_path.exists():<br>        raise FileNotFoundError(<br>            f&quot;Chunks file not found: {file_path}&quot;<br>        )<br><br>    # Read the JSON file.<br>    with file_path.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as file:<br>        chunks = json.load(file)<br><br>    # Validate the JSON structure.<br>    if not isinstance(chunks, list):<br>        raise ValueError(&quot;Expected a list of chunks.&quot;)<br><br>    if not chunks:<br>        raise ValueError(&quot;No chunks found in the input file.&quot;)<br><br>    return chunks<br><br><br>def load_embedding_model() -&gt; SentenceTransformer:<br>    &quot;&quot;&quot;Load the pre-trained sentence embedding model.&quot;&quot;&quot;<br><br>    print(f&quot;Loading embedding model: {MODEL_NAME}&quot;)<br><br>    # The model is downloaded automatically if not cached.<br>    model = SentenceTransformer(MODEL_NAME)<br><br>    return model<br><br><br>def generate_embeddings(<br>    chunks: list[dict],<br>    model: SentenceTransformer,<br>) -&gt; tuple[list[dict], np.ndarray]:<br>    &quot;&quot;&quot;Generate embeddings for all document chunks.&quot;&quot;&quot;<br><br>    # Extract text from each chunk.<br>    texts = [chunk[&quot;page_content&quot;] for chunk in chunks]<br><br>    # Convert all chunk texts into numerical vectors.<br>    vectors = model.encode(<br>        texts,<br>        convert_to_numpy=True,<br>        normalize_embeddings=True,<br>        show_progress_bar=True,<br>    )<br><br>    # Combine embeddings with their original metadata.<br>    embedded_chunks = []<br><br>    for chunk, vector in zip(chunks, vectors):<br>        embedded_chunks.append(<br>            {<br>                &quot;chunk_id&quot;: chunk[&quot;chunk_id&quot;],<br>                &quot;page_content&quot;: chunk[&quot;page_content&quot;],<br>                &quot;metadata&quot;: chunk[&quot;metadata&quot;],<br>                &quot;embedding&quot;: vector.tolist(),<br>            }<br>        )<br><br>    return embedded_chunks, vectors<br><br><br>def save_embeddings(embedded_chunks: list[dict]) -&gt; None:<br>    &quot;&quot;&quot;Save generated embeddings in JSON format.&quot;&quot;&quot;<br><br>    # Create the output directory.<br>    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)<br><br>    # Save the embeddings and metadata.<br>    with EMBEDDINGS_OUTPUT_PATH.open(<br>        &quot;w&quot;, encoding=&quot;utf-8&quot;<br>    ) as file:<br>        json.dump(<br>            {<br>                &quot;model&quot;: MODEL_NAME,<br>                &quot;dimension&quot;: len(<br>                    embedded_chunks[0][&quot;embedding&quot;]<br>                ),<br>                &quot;chunks&quot;: embedded_chunks,<br>            },<br>            file,<br>            indent=2,<br>            ensure_ascii=False,<br>        )<br><br><br>def cosine_similarity(<br>    query_vector: np.ndarray,<br>    document_vectors: np.ndarray,<br>) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Calculate cosine similarity for normalized vectors.&quot;&quot;&quot;<br><br>    # Both query and document vectors are normalized.<br>    # Their dot product is therefore cosine similarity.<br>    return document_vectors @ query_vector<br><br><br>def semantic_search(<br>    query: str,<br>    model: SentenceTransformer,<br>    chunks: list[dict],<br>    document_vectors: np.ndarray,<br>    top_k: int = TOP_K,<br>) -&gt; list[dict]:<br>    &quot;&quot;&quot;Retrieve the most semantically similar chunks.&quot;&quot;&quot;<br><br>    # Convert the user question into an embedding.<br>    query_vector = model.encode(<br>        query,<br>        convert_to_numpy=True,<br>        normalize_embeddings=True,<br>    )<br><br>    # Calculate similarity with every document chunk.<br>    similarities = cosine_similarity(<br>        query_vector,<br>        document_vectors,<br>    )<br><br>    # Get indices of the highest similarity scores.<br>    top_indices = np.argsort(similarities)[::-1][:top_k]<br><br>    results = []<br><br>    # Collect the most relevant chunks.<br>    for index in top_indices:<br>        chunk = chunks[int(index)]<br><br>        results.append(<br>            {<br>                &quot;chunk_id&quot;: chunk[&quot;chunk_id&quot;],<br>                &quot;page_content&quot;: chunk[&quot;page_content&quot;],<br>                &quot;metadata&quot;: chunk[&quot;metadata&quot;],<br>                &quot;similarity_score&quot;: float(similarities[index]),<br>            }<br>        )<br><br>    return results<br><br><br>def main() -&gt; None:<br><br>    # Load document chunks from Lesson 03.<br>    chunks = load_chunks(INPUT_PATH)<br><br>    print(f&quot;Loaded {len(chunks)} chunks&quot;)<br><br>    # Load the embedding model.<br>    model = load_embedding_model()<br><br>    # Generate document embeddings.<br>    embedded_chunks, document_vectors = generate_embeddings(<br>        chunks,<br>        model,<br>    )<br><br>    # Save embeddings for later lessons.<br>    save_embeddings(embedded_chunks)<br><br>    print(f&quot;Generated {len(embedded_chunks)} embeddings&quot;)<br>    print(f&quot;Embedding dimensions: {document_vectors.shape[1]}&quot;)<br>    print(f&quot;Saved embeddings to: {EMBEDDINGS_OUTPUT_PATH}&quot;)<br><br>    # Ask a question about the company policy.<br>    query = &quot;How many annual leave days do employees receive?&quot;<br><br>    print(f&quot;\nUser Question: {query}&quot;)<br><br>    # Find the most relevant document chunks.<br>    results = semantic_search(<br>        query,<br>        model,<br>        chunks,<br>        document_vectors,<br>    )<br><br>    # Display retrieved results.<br>    print(&quot;\nTop Relevant Chunks:&quot;)<br><br>    for rank, result in enumerate(results, start=1):<br>        print(f&quot;\n--- Result {rank} ---&quot;)<br>        print(f&quot;Chunk ID: {result[&#39;chunk_id&#39;]}&quot;)<br>        print(f&quot;Similarity: {result[&#39;similarity_score&#39;]:.4f}&quot;)<br>        print(f&quot;Metadata: {result[&#39;metadata&#39;]}&quot;)<br>        print(f&quot;Content:\n{result[&#39;page_content&#39;]}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4>Code Explanation</h4><p>Let’s understand how this application works.</p><p><strong>Define File Paths</strong></p><pre>BASE_DIR = Path(__file__).parent<br><br>INPUT_PATH = (<br>    BASE_DIR.parent<br>    / &quot;Lesson-03-Text-Chunking&quot;<br>    / &quot;output&quot;<br>    / &quot;chunks.json&quot;<br>)</pre><p>BASE_DIR identifies the directory containing our current main.py file.</p><p>BASE_DIR.parent accesses the main repository directory.</p><p>From there, we access the chunks.json file generated during Lesson 03.</p><p>This allows us to continue the RAG pipeline without repeating document extraction or chunking.</p><p><strong>Load the Embedding Model</strong></p><pre>model = SentenceTransformer(<br>    &quot;sentence-transformers/all-MiniLM-L6-v2&quot;<br>)</pre><p>This loads a pre-trained embedding model.</p><p>The model converts input text into 384-dimensional numerical vectors.</p><p>We use the same model for both document chunks and user questions so their embeddings can be compared.</p><p><strong>Generate Document Embeddings</strong></p><pre>vectors = model.encode(<br>    texts,<br>    convert_to_numpy=True,<br>    normalize_embeddings=True,<br>    show_progress_bar=True,<br>)</pre><p>The encode() method converts a list of text chunks into embedding vectors.</p><p>Let’s understand the parameters.</p><p>texts-The list of document chunk texts.</p><p>convert_to_numpy=True -Returns the embeddings as a NumPy array.</p><p>normalize_embeddings=True-Normalizes each embedding vector to unit length.</p><p>show_progress_bar=True -Displays progress during embedding generation.</p><p>For example, if we have 8 chunks, the resulting array will have this shape:</p><pre>(8, 384)</pre><p>This means we have 8 embedding vectors, each containing 384 numerical values.</p><p><strong>Preserve Embeddings with Metadata</strong></p><pre>{<br>    &quot;chunk_id&quot;: chunk[&quot;chunk_id&quot;],<br>    &quot;page_content&quot;: chunk[&quot;page_content&quot;],<br>    &quot;metadata&quot;: chunk[&quot;metadata&quot;],<br>    &quot;embedding&quot;: vector.tolist(),<br>}</pre><p>Each chunk is stored together with its embedding and metadata.</p><p>The tolist() method converts a NumPy array into a regular Python list so it can be saved in JSON format.</p><p>This allows us to preserve the relationship between the original text and its numerical representation.</p><p><strong>Generate a Query Embedding</strong></p><pre>query_vector = model.encode(<br>    query,<br>    convert_to_numpy=True,<br>    normalize_embeddings=True,<br>)</pre><p>The user question is converted into an embedding using the same model.</p><p>For example:</p><pre>How many annual leave days do employees receive?</pre><p>The model generates a 384-dimensional vector representing this question.</p><p>We can now compare it with the embeddings generated from the company policy chunks.</p><p><strong>Calculate Cosine Similarity</strong></p><pre>similarities = document_vectors @ query_vector</pre><p>The @ operator performs matrix multiplication.</p><p>Because both the query embedding and document embeddings have been normalized, their dot products are equal to cosine similarity.</p><p>This produces one similarity score for each document chunk.</p><p>For example:</p><pre>Chunk 1 → 0.82<br>Chunk 2 → 0.41<br>Chunk 3 → 0.19</pre><p>These are illustrative scores.</p><p>Higher scores indicate that the embedding model considers the chunk more similar to the question.</p><p><strong>Retrieve the Top-K Chunks</strong></p><pre>top_indices = np.argsort(similarities)[::-1][:top_k]</pre><p>np.argsort() returns the indices of similarity scores in ascending order.</p><p>[::-1] reverses the order so the highest scores appear first.</p><p>[:top_k] selects only the requested number of results.</p><p>For example, if TOP_K = 3, the application returns the three chunks with the highest similarity scores.</p><p>This is a simple form of semantic retrieval.</p><h4>Step 5 -Run the Application</h4><p>Open a terminal inside the Lesson 04 folder.</p><pre>python main.py</pre><p>The application will load the chunks, initialize the embedding model, generate embeddings, save the results, and perform semantic search.</p><p><strong>Expected Console Output</strong></p><pre>Loaded 8 chunks<br>Loading embedding model: sentence-transformers/all-MiniLM-L6-v2<br><br>Generated 8 embeddings<br>Embedding dimensions: 384<br>Saved embeddings to: .../output/embeddings.json<br><br>User Question: How many annual leave days do employees receive?<br><br>Top Relevant Chunks:<br><br>--- Result 1 ---<br>Chunk ID: company_policy_page_1_chunk_1<br>Similarity: 0.8234<br>Metadata: {&#39;source&#39;: &#39;company_policy.pdf&#39;, &#39;page&#39;: 1, &#39;chunk_number&#39;: 1}<br>Content:<br>Employee Leave Policy<br>All permanent employees are entitled to 20 annual leave days.</pre><p>This is illustrative output. Actual chunk counts, ranking, and similarity scores depend on the source document, chunking configuration, and model output.</p><h4>Step 6 -Generated File: embeddings.json</h4><p>After running the application, the output directory will contain:</p><pre>Lesson-04-Text-Embeddings/<br>├── output/<br>│   └── embeddings.json<br>├── main.py<br>└── requirements.txt</pre><p>The JSON file contains the embedding model information and the generated document embeddings.</p><p>Example:</p><pre>{<br>  &quot;model&quot;: &quot;sentence-transformers/all-MiniLM-L6-v2&quot;,<br>  &quot;dimension&quot;: 384,<br>  &quot;chunks&quot;: [<br>    {<br>      &quot;chunk_id&quot;: &quot;company_policy_page_1_chunk_1&quot;,<br>      &quot;page_content&quot;: &quot;Employee Leave Policy...&quot;,<br>      &quot;metadata&quot;: {<br>        &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>        &quot;page&quot;: 1,<br>        &quot;chunk_number&quot;: 1<br>      },<br>      &quot;embedding&quot;: [<br>        0.12,<br>        -0.45,<br>        0.78<br>      ]<br>    }<br>  ]<br>}</pre><p>The embedding values above are illustrative and truncated. The actual output contains 384 values per chunk.</p><p>For this educational project, JSON is useful because we can inspect the generated vectors directly.</p><p>In production systems, embeddings are commonly stored in vector databases rather than large JSON files.</p><h4>Step 7 -Experiment with Different Questions</h4><p>Now let’s test whether semantic search can identify related information even when the question uses different words.</p><p>Replace the query in main() with these questions one at a time:</p><pre>query = &quot;How many vacation days are available to staff?&quot;</pre><pre>query = &quot;Can employees work from home?&quot;</pre><pre>query = &quot;What should employees do if a security incident occurs?&quot;</pre><p>Run the application after each change.</p><p>Observe which chunks appear in the top results.</p><p>For example, the question “Can employees work from home?” may retrieve a chunk containing the Remote Work Policy, even if the document uses the phrase “work remotely” instead of “work from home”.</p><p>This demonstrates the purpose of semantic search.</p><p><strong><em>Note:</em></strong><em> Semantic search may still return irrelevant results. A high similarity score does not guarantee that a chunk contains the correct answer.</em></p><h3>7. Where Do Embeddings Fit in the RAG Pipeline?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*bSE_z0PSI4FFXcoJyXutFg.png" /></figure><p>Our RAG indexing pipeline now has three completed stages:</p><p><strong>Lesson 02:</strong> Document Loading and Text Processing</p><p><strong>Lesson 03:</strong> Text Chunking</p><p><strong>Lesson 04:</strong> Embedding Generation</p><p>We have also implemented a basic semantic search mechanism using cosine similarity.</p><p>In the next lesson, we will replace our manual similarity search with a vector database.</p><h3>Common Problems in Text Embeddings</h3><h4>Problem 1-Using Different Embedding Models</h4><p>Document and query embeddings generated by incompatible models may not share the same meaningful vector space.</p><p>Use the same compatible model and configuration for indexing and retrieval.</p><h4>Problem 2 -Confusing Similarity with Correctness</h4><p>A chunk can have a high similarity score while still containing incomplete or incorrect information.</p><p>Similarity is a retrieval signal, not a factual accuracy guarantee.</p><h4>Problem 3 -Poor Text Chunking</h4><p>Embedding models cannot reliably recover context that was removed during chunking.</p><p>If important information is split incorrectly, retrieval quality may decrease.</p><h4>Problem 4 -Choosing an Unsuitable Model</h4><p>Embedding models differ in supported languages, text lengths, domains, and retrieval performance.</p><p>A model trained primarily on English text may perform less effectively on other languages.</p><h4>Problem 5-High Storage Requirements</h4><p>Large document collections can produce millions of embedding vectors.</p><p>Storing and searching these vectors efficiently requires appropriate indexing and database technologies.</p><h4>Problem 6 -Model Updates</h4><p>If we change the embedding model, previously generated embeddings may no longer be compatible with new query embeddings.</p><p>In many cases, we need to regenerate the document embeddings and rebuild the vector index.</p><h3>Conclusion</h3><p>In this lesson, we learned what text embeddings are, how embedding models convert text into numerical vectors, and how semantic similarity helps a RAG system identify related information.</p><p>We explored embedding dimensions, cosine similarity, document embeddings, and query embeddings.</p><p>We also built a Python application using Sentence Transformers to generate 384-dimensional embeddings from the chunks created in Lesson 03, save those embeddings with metadata, and retrieve relevant chunks using cosine similarity.</p><p><strong>Key takeaway:</strong> Text embeddings enable semantic retrieval by representing text as numerical vectors. This allows RAG systems to search for related meaning instead of relying only on exact keyword matches.</p><p><strong>Next Lesson-RAG Lesson 05: Vector Databases</strong></p><p>In Lesson 05, we’ll learn what vector databases are, why they are useful, how vector indexing and similarity search work, and how to store and retrieve our company policy embeddings using ChromaDB and Python.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad2210fc2bab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Lesson 03 -Text Chunking]]></title>
            <link>https://jinadgamage.medium.com/rag-lesson-03-text-chunking-e1d4d3c465f8?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1d4d3c465f8</guid>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[text]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 06:13:46 GMT</pubDate>
            <atom:updated>2026-11-17T06:13:46.809Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cBo3gOzeP-Hm1oCLqt9QhA.png" /></figure><p>In Lesson 02, we learned how to load PDF documents, extract text, clean unnecessary whitespace, preserve metadata, and save the processed content in JSON format.</p><p>Now we have a new problem.</p><p>Imagine that our company has a 100-page employee policy document containing information about annual leave, sick leave, remote work, security, and employee benefits.</p><p>A user asks: <em>How many annual leave days do employees receive?</em></p><p>Do we need to send the entire 100-page document to the Large Language Model (LLM) to answer this simple question?</p><p>No. We only need the section containing the annual leave policy.</p><p>This is where <strong>Text Chunking</strong> becomes important.</p><p>In this lesson, we will understand how chunking works, explore different chunking strategies, learn about chunk size and overlap, and implement a practical Python application using the output generated in Lesson 02.</p><p>By the end, we will have a collection of smaller document chunks ready for embedding generation in Lesson 04.</p><h3>Text Chunking</h3><p>Text Chunking is the process of dividing large documents into smaller, manageable pieces of text called <strong>chunks</strong>.</p><p>In a Retrieval-Augmented Generation (RAG) system, chunks are typically converted into numerical embeddings and stored in a vector database.</p><p>When a user asks a question, the system searches for the most relevant chunks instead of retrieving an entire document.</p><p><strong>Example :</strong></p><p>Consider this text extracted from company_policy.pdf:</p><pre>Employee Leave Policy<br><br>All permanent employees are entitled to<br>20 annual leave days.<br><br>Employees receive 10 sick leave days.<br><br>Remote Work Policy<br><br>Employees may work remotely with<br>manager approval.<br><br>Information Security Policy<br><br>Employees must report security incidents<br>to the IT department immediately.</pre><p>We can divide this document into three meaningful chunks.</p><p><strong>Chunk 1 - Leave Policy</strong></p><p>All permanent employees are entitled to 20 annual leave days. Employees receive 10 sick leave days.</p><p><strong>Chunk 2 -Remote Work</strong></p><p>Employees may work remotely with manager approval.</p><p><strong>Chunk 3 -Information Security</strong></p><p>Employees must report security incidents to the IT department immediately.</p><p>Conceptual example of splitting a document by topic.</p><p>Now, if a user asks about annual leave, the RAG system can retrieve Chunk 1 because it contains the relevant information.</p><p><strong><em>Important:</em></strong><em> Chunking itself does not search for relevant information. It prepares the document for later embedding and retrieval operations.</em></p><h3>Why Do We Need Text Chunking</h3><p>There are several reasons why chunking is an important stage in a RAG pipeline.</p><h4>Efficient Information Retrieval</h4><p>Large documents often contain information about many unrelated topics.</p><p>For example, a company policy document may contain:</p><ul><li>Annual leave and sick leave policies</li><li>Remote work guidelines</li><li>Employee benefits</li><li>Information security requirements</li><li>Training and development policies</li></ul><p>If we store the entire document as a single embedding, its representation may combine all these topics.</p><p>Smaller chunks allow the retrieval system to identify more focused information.</p><h4>Managing the LLM Context Window</h4><p>Every LLM has a limited context window, measured in tokens.</p><p>Sending unnecessary document content consumes context space that could otherwise be used for relevant information or conversation history.</p><p>Chunking allows the system to select useful sections before generating an answer.</p><h4>Reducing Token Usage</h4><p>Sending a full document to an LLM may require significantly more input tokens than sending a few relevant chunks.</p><p>Retrieving smaller sections can reduce token usage and API costs.</p><p>However, retrieving too many overlapping chunks can also increase costs.</p><h4>Improving Retrieval Quality</h4><p>Good chunking keeps related information together.</p><p>For example, a policy heading and its explanation should ideally remain in the same chunk.</p><p>If the heading is separated from the explanation, the retrieved text may lose important context.</p><p><strong><em>Key idea:</em></strong><em> Chunking is not simply about making text smaller. It is about creating useful units of information for retrieval.</em></p><h3>Understanding Chunk Size</h3><p>Chunk Size defines the maximum amount of text that can be included in one chunk.</p><p>It can be measured in characters, words, or tokens, depending on the implementation.</p><p>For example:</p><pre>CHUNK_SIZE = 500</pre><p>If our splitter measures characters, this means each chunk can contain up to 500 characters.</p><p>It does <strong>not</strong> mean 500 tokens.</p><h4>Small Chunks</h4><p>Example:</p><pre>Employees receive 20 annual leave days.</pre><p>Small chunks are focused and can work well for simple factual questions.</p><p>However, they may not contain enough surrounding context.</p><p>For example, the sentence does not explain whether the policy applies to permanent employees, temporary employees, or everyone.</p><h4>Large Chunks</h4><p>Example:</p><pre>Employee Leave Policy<br><br>All permanent employees are entitled to<br>20 annual leave days.<br><br>Employees receive 10 sick leave days.<br><br>Leave requests must be submitted<br>through the HR portal.<br><br>Managers must approve leave requests<br>before employees take leave.</pre><p>This chunk provides more context.</p><p>However, if chunks become too large, they may contain unrelated information that reduces retrieval precision.</p><h4>Small vs Large Chunks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*2w0bm_7zEGpm8eB0BUxK1g.png" /><figcaption>small chunk vs large chunk</figcaption></figure><p>There is no universal best chunk size. The correct value depends on the document, embedding model, and questions users ask.</p><h3>Understanding Chunk Overlap</h3><p>Chunk Overlap refers to the text repeated between consecutive chunks.</p><p>Why is overlap necessary?</p><p>Consider this paragraph:</p><pre>All permanent employees are entitled to<br>20 annual leave days. Leave requests must<br>be submitted through the HR portal.<br>Managers must approve requests before<br>employees take leave.</pre><p>Imagine splitting the text without overlap.</p><p><strong>Without Overlap</strong></p><p><strong>Chunk 1 : </strong><em>All permanent employees are entitled to 20 annual leave days. Leave requests must be submitted</em></p><p><strong>Chunk 2 : </strong><em>through the HR portal. Managers must approve requests before employees take leave.</em></p><p>The sentence about submitting leave requests has been divided between two chunks.</p><p>A retrieval system might return only one of these chunks, resulting in incomplete information.</p><p>Now consider splitting with overlap.</p><p><strong>With Overlap</strong></p><p><strong>Chunk 1 : </strong><em>All permanent employees are entitled to 20 annual leave days. Leave requests must be submitted through the HR portal.</em></p><p><strong>Chunk 2 : </strong><em>Leave requests must be submitted through the HR portal. Managers must approve requests before employees take leave.</em></p><p>The highlighted sentence is repeated in both chunks to preserve context. This is a conceptual example.</p><p>The repeated content helps preserve information across chunk boundaries.</p><h4>How Much Overlap Should We Use</h4><p>A common starting experiment is an overlap of around 10–20% of the chunk size.</p><p>For example:</p><pre>CHUNK_SIZE = 500<br>CHUNK_OVERLAP = 100</pre><p>Here, the overlap setting is 20% of the chunk size.</p><p>This is a starting point, not a guaranteed optimal configuration. Recursive splitters may produce less actual overlap depending on text boundaries.</p><p>Too little overlap may lose context, while excessive overlap creates duplicated information and increases storage requirements.</p><h3>Text Chunking Strategies</h3><p>There are several approaches to splitting documents. Let’s understand the main strategies before choosing one for our practical.</p><h4>Fixed-Size Chunking</h4><p>Fixed-Size Chunking divides text into chunks using a predefined number of characters or tokens.</p><p>For example:</p><pre>text = &quot;ABCDEFGHIJKLMNOPQRSTUVWXYZ&quot;<br><br>chunk_size = 10<br><br>chunks = [<br>    text[i:i + chunk_size]<br>    for i in range(0, len(text), chunk_size)<br>]<br><br>print(chunks)</pre><p>Output:</p><pre>[&#39;ABCDEFGHIJ&#39;, &#39;KLMNOPQRST&#39;, &#39;UVWXYZ&#39;]</pre><p>This approach is simple and fast.</p><p>However, it does not understand sentences, paragraphs, or document structure.</p><p>A sentence might be divided in the middle, making the resulting chunks harder to interpret.</p><h4>Recursive Character Chunking</h4><p>Recursive Character Chunking attempts to split text using meaningful separators.</p><p>It typically tries:</p><ol><li>Paragraph boundaries (\n\n)</li><li>Line breaks (\n)</li><li>Spaces ( )</li><li>Individual characters when necessary</li></ol><p>For example, consider:</p><pre>Employee Leave Policy<br><br>Employees receive 20 annual leave days.<br><br>Remote Work Policy<br><br>Employees may work remotely with approval.</pre><p>A recursive splitter prefers separating the two paragraphs rather than splitting them at arbitrary character positions.</p><p>This helps preserve natural text structure.</p><p><strong>Why is this useful?</strong></p><p>Most PDF documents contain paragraphs and headings. Recursive chunking provides a practical balance between chunk size control and text readability.</p><p>For this lesson, we will use RecursiveCharacterTextSplitter from LangChain.</p><h4>Semantic Chunking</h4><p>Semantic Chunking attempts to group sentences based on their meaning.</p><p>Instead of splitting only by length or separators, semantic chunking can use embeddings or similarity measurements to identify changes in topics.</p><p>For example:</p><pre>Sentence 1: Employees receive annual leave.<br>Sentence 2: Leave requests require manager approval.<br>Sentence 3: Employees must protect company passwords.<br>Sentence 4: Security incidents must be reported.</pre><p>A semantic chunking approach might group sentences 1-2 into a leave-related chunk and sentences 3-4 into a security-related chunk.</p><p>Semantic chunking can preserve meaningful topic boundaries, but it is more complex and may require additional model calls or computation.</p><p>It is not automatically better than recursive chunking for every dataset.</p><h4>Document Structure-Based Chunking</h4><p>Document Structure-Based Chunking uses headings, sections, paragraphs, tables, or other structural information.</p><p>For example, a policy document might have:</p><pre>1. Employee Leave Policy<br>2. Remote Work Policy<br>3. Information Security Policy<br>4. Employee Benefits</pre><p>Each section could become a separate chunk.</p><p>This approach is particularly useful for structured documents, but it requires reliable extraction of headings and section boundaries.</p><h3>Which Chunking Strategy Should We Use?</h3><p>For our beginner RAG application, we will use <strong>Recursive Character Chunking</strong>.</p><p>Why?</p><p>It is easy to implement, supports configurable chunk size and overlap, and attempts to preserve natural text boundaries.</p><p>We will start with:</p><pre>CHUNK_SIZE = 500<br>CHUNK_OVERLAP = 100</pre><p>Later, when we evaluate retrieval quality, we can compare different configurations and strategies.</p><h3>Practical - Building a Text Chunking Pipeline with Python</h3><p>In Lesson 02, we created a Python application that extracted text from company_policy.pdf and saved the result in documents.json.</p><p>Now we will continue from that exact output.</p><p>Our goal is to build a Python application that:</p><ul><li>Reads documents.json from Lesson 02.</li><li>Splits each document page into smaller text chunks.</li><li>Preserves the original source and page metadata.</li><li>Assigns a unique ID to each chunk.</li><li>Saves the chunks in JSON and readable TXT formats.</li></ul><p>We will also compare the original document with the generated chunks so we can understand what the splitter actually does.</p><h4>Step 1 - Understand the Input</h4><p>Our Lesson 02 application generated a JSON file with this structure:</p><p>documents.json</p><pre>[<br>  {<br>    &quot;page_content&quot;: &quot;ABC Company | Fictional sample for RAG training\nPage 1\nABC Company\nEmployee Handbook - Company Policy\nSample training document | Version 1.0 | Effective: 1 January 2026\nThis is a fictional company policy created for RAG document-loading, chunking, retrieval and citation\nexercises. It is not legal or HR advice.\n1. Annual Leave Policy\n• Permanent full-time employees receive 20 paid annual leave days per calendar year.\n• Annual leave requests must be submitted through the HR portal at least 7 calendar days before the\nintended start date.\n• Managers should approve or reject annual leave requests within 3 working days.\n• Unused annual leave can be carried forward up to 5 days into the next calendar year and must be used by\n31 March.\n2. Sick Leave Policy\n• Employees receive 10 paid sick leave days per calendar year.\n• A medical certificate is required when sick leave lasts more than 2 consecutive working days.\n• Employees should notify their manager before 9:00 AM on the first day of absence whenever possible.\n3. Working Hours and Remote Work\n• Standard working hours are Monday to Friday, 9:00 AM to 5:30 PM, including a 30-minute lunch break.\n• Eligible employees may work remotely up to 2 days per week with manager approval.\n• Employees must follow company security requirements when working remotely.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 1<br>    }<br>  },<br>  {<br>    &quot;page_content&quot;: &quot;ABC Company | Fictional sample for RAG training\nPage 2\nABC Company - Company Policy (continued)\n4. Information Security\n• Employees must use multi-factor authentication for company accounts.\n• Passwords must not be shared with colleagues or third parties.\n• Suspected phishing emails or security incidents must be reported to the IT Security team within 1 hour of\ndiscovery.\n5. Expense Reimbursement\n• Approved business expenses must be submitted through the Finance portal within 30 days of the expense\ndate.\n• Expense claims must include an itemized receipt and manager approval.\n• Finance normally processes complete claims within 10 working days.\n6. Training and Development\n• Each permanent employee has an annual professional development budget of USD 300, subject to\nmanager approval.\n• Training requests must explain how the course supports the employee role or team goals.\n7. Contacts\n• Human Resources: hr@abc-example.com (fictional training address).\n• IT Security: security@abc-example.com (fictional training address).\n• Finance: finance@abc-example.com (fictional training address).&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 2<br>    }<br>  }<br>]</pre><p>Each object represents a PDF page.</p><p>Notice that it contains two important fields:</p><p><strong>page_content</strong> stores the extracted text.</p><p><strong>metadata</strong> stores information about where the text came from.</p><p>Our chunking application will split only the page_content value and preserve the metadata for every generated chunk.</p><p>This means that if page 1 produces three chunks, all three chunks will retain source: company_policy.pdf and page: 1.</p><h4>Step 2 -Project Structure</h4><p>We will keep Lesson 02 and Lesson 03 in separate folders inside the same repository.</p><pre>rag-fundamentals-to-advanced/<br>│<br>├── Lesson-02-Document-Loading-Text-Processing/<br>│   ├── documents/<br>│   │   └── company_policy.pdf<br>│   ├── output/<br>│   │   ├── documents.json<br>│   │   └── processed_policy.txt<br>│   ├── main.py<br>│   └── requirements.txt<br>│<br>└── Lesson-03-Text-Chunking/<br>    ├── output/<br>    │   ├── chunks.json<br>    │   └── chunks.txt<br>    ├── main.py<br>    └── requirements.txt</pre><p><em>The </em><em>output folder for Lesson 03 will be created automatically when we run the program.</em></p><h4>Step 3 -Install Dependencies</h4><p>We will use LangChain’s text splitter package.</p><p>Create a requirements.txt file inside the Lesson 03 folder.</p><pre>langchain-text-splitters</pre><p>Install the dependency:</p><pre>pip install -r requirements.txt</pre><p><em>We do not need PyMuPDF in Lesson 03 because we are not reading the original PDF again. We are using the processed JSON output from Lesson 02.</em></p><h4>Step 4-Complete Python Implementation</h4><p>Create main.py inside the Lesson 03 folder.</p><pre><br>import json<br>from pathlib import Path<br><br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br><br># Keep all paths relative to the lesson folders.<br>BASE_DIR = Path(__file__).parent<br><br># Read the processed documents from Lesson 02.<br>INPUT_PATH = (<br>    BASE_DIR.parent<br>    / &quot;Lesson-02-Document-Loading-Text-Processing&quot;<br>    / &quot;output&quot;<br>    / &quot;documents.json&quot;<br>)<br><br># Output files for Lesson 03.<br>OUTPUT_DIR = BASE_DIR / &quot;output&quot;<br>JSON_OUTPUT_PATH = OUTPUT_DIR / &quot;chunks.json&quot;<br>TEXT_OUTPUT_PATH = OUTPUT_DIR / &quot;chunks.txt&quot;<br><br># Chunking configuration (measured in characters).<br>CHUNK_SIZE = 500<br>CHUNK_OVERLAP = 100<br><br>def load_documents(file_path: Path) -&gt; list[dict]:<br>    &quot;&quot;&quot;Load processed documents generated in Lesson 02.&quot;&quot;&quot;<br><br>    # Check whether the input JSON file exists.<br>    if not file_path.exists():<br>        raise FileNotFoundError(<br>            f&quot;Input JSON file not found: {file_path}&quot;<br>        )<br><br>    # Read the JSON file.<br>    with file_path.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as file:<br>        documents = json.load(file)<br><br>    # Validate the expected JSON structure.<br>    if not isinstance(documents, list):<br>        raise ValueError(&quot;Expected a list of documents.&quot;)<br><br>    return documents<br><br>def create_chunks(documents: list[dict]) -&gt; list[dict]:<br>    &quot;&quot;&quot;Split documents into chunks and preserve metadata.&quot;&quot;&quot;<br><br>    chunks = []<br><br>    # Configure recursive character splitting.<br>    text_splitter = RecursiveCharacterTextSplitter(<br>        chunk_size=CHUNK_SIZE,<br>        chunk_overlap=CHUNK_OVERLAP,<br>        length_function=len,<br>        separators=[&quot;\n\n&quot;, &quot;\n&quot;, &quot;. &quot;, &quot; &quot;, &quot;&quot;],<br>    )<br><br>    # Process each document page separately.<br>    for document in documents:<br><br>        page_content = document[&quot;page_content&quot;]<br>        metadata = document[&quot;metadata&quot;]<br><br>        # Skip empty documents.<br>        if not page_content.strip():<br>            continue<br><br>        # Split page content into smaller chunks.<br>        text_chunks = text_splitter.split_text(page_content)<br><br>        # Store each generated chunk with metadata.<br>        for chunk_number, chunk_text in enumerate(<br>            text_chunks, start=1<br>        ):<br><br>            # Create an identifier for the chunk.<br>            chunk_id = (<br>                f&quot;{Path(metadata[&#39;source&#39;]).stem}&quot;<br>                f&quot;_page_{metadata[&#39;page&#39;]}&quot;<br>                f&quot;_chunk_{chunk_number}&quot;<br>            )<br><br>            chunks.append(<br>                {<br>                    &quot;chunk_id&quot;: chunk_id,<br>                    &quot;page_content&quot;: chunk_text,<br>                    &quot;metadata&quot;: {<br>                        **metadata,<br>                        &quot;chunk_number&quot;: chunk_number,<br>                    },<br>                }<br>            )<br><br>    return chunks<br><br>def save_chunks(chunks: list[dict]) -&gt; None:<br>    &quot;&quot;&quot;Save chunks in JSON and readable text formats.&quot;&quot;&quot;<br><br>    # Create output directory if it does not exist.<br>    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)<br><br>    # Save structured chunks as JSON.<br>    with JSON_OUTPUT_PATH.open(<br>        &quot;w&quot;, encoding=&quot;utf-8&quot;<br>    ) as file:<br>        json.dump(<br>            chunks,<br>            file,<br>            indent=2,<br>            ensure_ascii=False,<br>        )<br><br>    # Create a readable text representation.<br>    processed_text = &quot;\n\n&quot;.join(<br>        f&quot;--- {chunk[&#39;chunk_id&#39;]} ---\n&quot;<br>        f&quot;{chunk[&#39;page_content&#39;]}&quot;<br>        for chunk in chunks<br>    )<br><br>    # Save readable chunks as TXT.<br>    TEXT_OUTPUT_PATH.write_text(<br>        processed_text,<br>        encoding=&quot;utf-8&quot;<br>    )<br><br>def main() -&gt; None:<br><br>    # Load processed documents from Lesson 02.<br>    documents = load_documents(INPUT_PATH)<br><br>    # Split documents into chunks.<br>    chunks = create_chunks(documents)<br><br>    # Save generated chunks.<br>    save_chunks(chunks)<br><br>    # Display summary.<br>    print(f&quot;Loaded {len(documents)} documents&quot;)<br>    print(f&quot;Generated {len(chunks)} chunks&quot;)<br>    print(f&quot;Chunk size: {CHUNK_SIZE} characters&quot;)<br>    print(f&quot;Chunk overlap: {CHUNK_OVERLAP} characters&quot;)<br><br>    print(f&quot;Saved JSON to: {JSON_OUTPUT_PATH}&quot;)<br>    print(f&quot;Saved text to: {TEXT_OUTPUT_PATH}&quot;)<br><br>    # Preview each generated chunk.<br>    for chunk in chunks:<br>        print()<br>        print(f&quot;Chunk ID: {chunk[&#39;chunk_id&#39;]}&quot;)<br>        print(f&quot;Metadata: {chunk[&#39;metadata&#39;]}&quot;)<br>        print(f&quot;Length: {len(chunk[&#39;page_content&#39;])} characters&quot;)<br>        print(chunk[&quot;page_content&quot;][:200])<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4>Code Explanation</h4><p>Let’s understand how this application works.</p><p><strong>Define File Paths</strong></p><pre>BASE_DIR = Path(__file__).parent<br><br>INPUT_PATH = (<br>    BASE_DIR.parent<br>    / &quot;Lesson-02-Document-Loading-Text-Processing&quot;<br>    / &quot;output&quot;<br>    / &quot;documents.json&quot;<br>)</pre><p>BASE_DIR identifies the directory containing the current Python script.</p><p>BASE_DIR.parent accesses the main repository folder.</p><p>From there, we access the documents.json file generated by Lesson 02.</p><p><em>This allows us to reuse the previous lesson’s output instead of processing the PDF again.</em></p><p><strong>Load Documents</strong></p><pre>with file_path.open(&quot;r&quot;, encoding=&quot;utf-8&quot;) as file:<br>    documents = json.load(file)</pre><p>The json.load() function converts the JSON file into Python objects.</p><p><em>Our input is a list of documents, where each document contains text and metadata.</em></p><p><strong>Configure the Text Splitter</strong></p><pre>text_splitter = RecursiveCharacterTextSplitter(<br>    chunk_size=CHUNK_SIZE,<br>    chunk_overlap=CHUNK_OVERLAP,<br>    length_function=len,<br>    separators=[&quot;\n\n&quot;, &quot;\n&quot;, &quot;. &quot;, &quot; &quot;, &quot;&quot;],<br>)</pre><p>This is the most important part of the practical.</p><p>Let’s understand each parameter.</p><p><strong>chunk_size=500 </strong>Sets the maximum length of a chunk to 500 characters.</p><p><strong>chunk_overlap=100 </strong>Requests up to 100 characters of overlap between consecutive chunks. Actual overlap can vary because the splitter tries to respect separators.</p><p><strong>length_function=len </strong>Uses Python’s len() function to measure text length in characters.</p><p><strong>separators </strong>Defines the order of preferred text boundaries:</p><pre>[&quot;\n\n&quot;, &quot;\n&quot;, &quot;. &quot;, &quot; &quot;, &quot;&quot;]</pre><p>The splitter first tries paragraph boundaries. If the resulting sections are still too large, it tries line breaks, then sentence-like boundaries, spaces, and finally individual characters.</p><p><em>This helps avoid splitting text unnecessarily in the middle of a sentence.</em></p><p><strong>Split the Documents</strong></p><pre>text_chunks = text_splitter.split_text(page_content)</pre><p>This method takes the original text and returns a list of smaller strings.</p><p>For example, imagine that one page contains 1,200 characters.</p><p>With a chunk size of 500 and an overlap of 100, the splitter may produce approximately three chunks, depending on where natural text boundaries occur.</p><p>We process each page separately, so a chunk will not contain text from two different PDF pages.</p><p><strong>Preserve Metadata</strong></p><pre>chunks.append(<br>    {<br>        &quot;chunk_id&quot;: chunk_id,<br>        &quot;page_content&quot;: chunk_text,<br>        &quot;metadata&quot;: {<br>            **metadata,<br>            &quot;chunk_number&quot;: chunk_number,<br>        },<br>    }<br>)</pre><p>The expression **metadata copies the original metadata from Lesson 02.</p><p>We then add chunk_number to identify the position of the chunk within its source page.</p><p>For example:</p><pre>{<br>  &quot;chunk_id&quot;: &quot;company_policy_page_1_chunk_1&quot;,<br>  &quot;page_content&quot;: &quot;Employee Leave Policy...&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>    &quot;page&quot;: 1,<br>    &quot;chunk_number&quot;: 1<br>  }<br>}</pre><p>This information will be useful when we retrieve chunks later and want to identify their original source.</p><p><strong>Save Generated Chunks</strong></p><pre>json.dump(<br>    chunks,<br>    file,<br>    indent=2,<br>    ensure_ascii=False,<br>)</pre><p>The application saves the generated chunks in JSON format.</p><p>We also save a readable TXT version so we can inspect the results without opening the JSON structure.</p><h4>Step 6 -Run the Application</h4><p>Open a terminal inside the Lesson 03 folder.</p><pre>python main.py</pre><p>The application will read the input documents, generate chunks, and save the output files.</p><p><strong>Expected Console Output</strong></p><pre>Loaded 2 documents<br>Generated 8 chunks<br><br>Chunk size: 500 characters<br>Chunk overlap: 100 characters</pre><pre>Saved JSON to: .../output/chunks.json<br>Saved text to: .../output/chunks.txt</pre><pre>Chunk ID: company_policy_page_1_chunk_1<br>Metadata: {&#39;source&#39;: &#39;company_policy.pdf&#39;, &#39;page&#39;: 1, &#39;chunk_number&#39;: 1}<br>Length: 487 characters<br>Employee Leave Policy...</pre><p>This is an illustrative output. The actual number of chunks and their lengths depend on the input document.</p><h4>Step 7 -Generated Files</h4><p>After running the program, we will have two output files.</p><p><strong>chunks.json</strong></p><p>This file contains the structured chunks and metadata.</p><p>Example:</p><pre>[<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_1_chunk_1&quot;,<br>    &quot;page_content&quot;: &quot;ABC Company | Fictional sample for RAG training\nPage 1\nABC Company\nEmployee Handbook - Company Policy\nSample training document | Version 1.0 | Effective: 1 January 2026\nThis is a fictional company policy created for RAG document-loading, chunking, retrieval and citation\nexercises. It is not legal or HR advice.\n1. Annual Leave Policy\n• Permanent full-time employees receive 20 paid annual leave days per calendar year.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 1,<br>      &quot;chunk_number&quot;: 1<br>    }<br>  },<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_1_chunk_2&quot;,<br>    &quot;page_content&quot;: &quot;• Permanent full-time employees receive 20 paid annual leave days per calendar year.\n• Annual leave requests must be submitted through the HR portal at least 7 calendar days before the\nintended start date.\n• Managers should approve or reject annual leave requests within 3 working days.\n• Unused annual leave can be carried forward up to 5 days into the next calendar year and must be used by\n31 March.\n2. Sick Leave Policy\n• Employees receive 10 paid sick leave days per calendar year.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 1,<br>      &quot;chunk_number&quot;: 2<br>    }<br>  },<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_1_chunk_3&quot;,<br>    &quot;page_content&quot;: &quot;31 March.\n2. Sick Leave Policy\n• Employees receive 10 paid sick leave days per calendar year.\n• A medical certificate is required when sick leave lasts more than 2 consecutive working days.\n• Employees should notify their manager before 9:00 AM on the first day of absence whenever possible.\n3. Working Hours and Remote Work\n• Standard working hours are Monday to Friday, 9:00 AM to 5:30 PM, including a 30-minute lunch break.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 1,<br>      &quot;chunk_number&quot;: 3<br>    }<br>  },<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_1_chunk_4&quot;,<br>    &quot;page_content&quot;: &quot;• Eligible employees may work remotely up to 2 days per week with manager approval.\n• Employees must follow company security requirements when working remotely.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 1,<br>      &quot;chunk_number&quot;: 4<br>    }<br>  },<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_2_chunk_1&quot;,<br>    &quot;page_content&quot;: &quot;ABC Company | Fictional sample for RAG training\nPage 2\nABC Company - Company Policy (continued)\n4. Information Security\n• Employees must use multi-factor authentication for company accounts.\n• Passwords must not be shared with colleagues or third parties.\n• Suspected phishing emails or security incidents must be reported to the IT Security team within 1 hour of\ndiscovery.\n5. Expense Reimbursement&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 2,<br>      &quot;chunk_number&quot;: 1<br>    }<br>  },<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_2_chunk_2&quot;,<br>    &quot;page_content&quot;: &quot;discovery.\n5. Expense Reimbursement\n• Approved business expenses must be submitted through the Finance portal within 30 days of the expense\ndate.\n• Expense claims must include an itemized receipt and manager approval.\n• Finance normally processes complete claims within 10 working days.\n6. Training and Development\n• Each permanent employee has an annual professional development budget of USD 300, subject to\nmanager approval.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 2,<br>      &quot;chunk_number&quot;: 2<br>    }<br>  },<br>  {<br>    &quot;chunk_id&quot;: &quot;company_policy_page_2_chunk_3&quot;,<br>    &quot;page_content&quot;: &quot;manager approval.\n• Training requests must explain how the course supports the employee role or team goals.\n7. Contacts\n• Human Resources: hr@abc-example.com (fictional training address).\n• IT Security: security@abc-example.com (fictional training address).\n• Finance: finance@abc-example.com (fictional training address).&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 2,<br>      &quot;chunk_number&quot;: 3<br>    }<br>  }<br>]</pre><p><em>This example illustrates the output structure, not the exact chunks produced by the application.</em></p><p><strong>chunks.txt</strong></p><p>This file contains the generated chunks in a readable format.</p><pre>--- company_policy_page_1_chunk_1 ---<br>ABC Company | Fictional sample for RAG training<br>Page 1<br>ABC Company<br>Employee Handbook - Company Policy<br>Sample training document | Version 1.0 | Effective: 1 January 2026<br>This is a fictional company policy created for RAG document-loading, chunking, retrieval and citation<br>exercises. It is not legal or HR advice.<br>1. Annual Leave Policy<br>• Permanent full-time employees receive 20 paid annual leave days per calendar year.<br><br>--- company_policy_page_1_chunk_2 ---<br>• Permanent full-time employees receive 20 paid annual leave days per calendar year.<br>• Annual leave requests must be submitted through the HR portal at least 7 calendar days before the<br>intended start date.<br>• Managers should approve or reject annual leave requests within 3 working days.<br>• Unused annual leave can be carried forward up to 5 days into the next calendar year and must be used by<br>31 March.<br>2. Sick Leave Policy<br>• Employees receive 10 paid sick leave days per calendar year.<br><br>--- company_policy_page_1_chunk_3 ---<br>31 March.<br>2. Sick Leave Policy<br>• Employees receive 10 paid sick leave days per calendar year.<br>• A medical certificate is required when sick leave lasts more than 2 consecutive working days.<br>• Employees should notify their manager before 9:00 AM on the first day of absence whenever possible.<br>3. Working Hours and Remote Work<br>• Standard working hours are Monday to Friday, 9:00 AM to 5:30 PM, including a 30-minute lunch break.<br><br>--- company_policy_page_1_chunk_4 ---<br>• Eligible employees may work remotely up to 2 days per week with manager approval.<br>• Employees must follow company security requirements when working remotely.<br><br>--- company_policy_page_2_chunk_1 ---<br>ABC Company | Fictional sample for RAG training<br>Page 2<br>ABC Company - Company Policy (continued)<br>4. Information Security<br>• Employees must use multi-factor authentication for company accounts.<br>• Passwords must not be shared with colleagues or third parties.<br>• Suspected phishing emails or security incidents must be reported to the IT Security team within 1 hour of<br>discovery.<br>5. Expense Reimbursement<br><br>--- company_policy_page_2_chunk_2 ---<br>discovery.<br>5. Expense Reimbursement<br>• Approved business expenses must be submitted through the Finance portal within 30 days of the expense<br>date.<br>• Expense claims must include an itemized receipt and manager approval.<br>• Finance normally processes complete claims within 10 working days.<br>6. Training and Development<br>• Each permanent employee has an annual professional development budget of USD 300, subject to<br>manager approval.<br><br>--- company_policy_page_2_chunk_3 ---<br>manager approval.<br>• Training requests must explain how the course supports the employee role or team goals.<br>7. Contacts<br>• Human Resources: hr@abc-example.com (fictional training address).<br>• IT Security: security@abc-example.com (fictional training address).<br>• Finance: finance@abc-example.com (fictional training address).</pre><p>This makes it easier to inspect how the original text was divided.</p><h4>Step 8 -Test Different Chunk Sizes</h4><p>Now that the application works, let’s experiment with the configuration.</p><p>Try the following settings one at a time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*f6uRSsSfvohk7XWYLkimoA.png" /></figure><p>After each experiment, run:</p><pre>python main.py</pre><p>Observe how the number and length of chunks change.</p><p>With smaller chunk sizes, the application generally creates more chunks.</p><p>With larger chunk sizes, the application generally creates fewer chunks containing more information.</p><p>Also inspect chunks.txt to see whether sentences and paragraphs remain readable.</p><p>This is important because choosing chunk size should not depend only on the number of chunks generated.</p><p>We also need to consider whether the chunks preserve useful information.</p><h3>Where Does Text Chunking Fit in RAG</h3><p>Text chunking is part of the <strong>indexing pipeline</strong> of a RAG system.</p><p>The original document is loaded and cleaned first. It is then divided into chunks.</p><p>In the next stage, each chunk will be converted into an embedding vector and stored in a vector database.</p><p>Later, when a user asks a question, the retrieval system will search those embeddings to find relevant chunks.</p><h3>Common Problems in Text Chunking</h3><h4>Problem 1 : Chunks Are Too Small</h4><p>Very small chunks may lose important context.</p><p>For example, a chunk containing only “20 annual leave days” does not explain who is eligible for those days.</p><h4>Problem 2 : Chunks Are Too Large</h4><p>Large chunks may contain multiple unrelated topics, reducing retrieval precision.</p><p>For example, a single chunk containing annual leave, remote work, and security policies may be less focused than separate topic-specific chunks.</p><h4>Problem 3 : Insufficient Overlap</h4><p>Without enough overlap, important information may be divided between consecutive chunks.</p><h4>Problem 4 : Excessive Overlap</h4><p>Too much overlap can create unnecessary duplication, increasing the number of stored tokens and the amount of repeated information retrieved.</p><h4>Problem 5 : Losing Document Structure</h4><p>Splitting tables, headings, numbered lists, or code blocks incorrectly may damage their meaning.</p><p>For complex documents, structure-aware chunking may be more appropriate.</p><h4>Problem 6 : Losing Metadata</h4><p>If chunks are saved without their original source and page information, identifying the source of retrieved information becomes difficult.</p><p>Preserving metadata is important for source tracking and citations.</p><h3>Conclusion</h3><p>In this lesson, we learned what text chunking is, why it matters in RAG, and how chunk size and overlap affect retrieval.</p><p>We explored fixed-size, recursive, semantic, and document structure-based chunking strategies.</p><p>We also built a Python application using LangChain’s RecursiveCharacterTextSplitter to process the documents.json file from Lesson 02, preserve metadata, and generate structured chunks in JSON and TXT formats.</p><p><strong>Key takeaway:</strong> Effective text chunking helps a RAG system retrieve focused information while preserving enough context to generate accurate and useful answers.</p><p>In <strong>Lesson 04 -Text Embeddings</strong>, we will learn how text chunks are converted into numerical vectors, how semantic similarity works, and how to generate embeddings for our chunks.json file using Python.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1d4d3c465f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Lesson 02 -Document Loading & Text Processing]]></title>
            <link>https://jinadgamage.medium.com/rag-lesson-02-document-loading-text-processing-52f4cb0f0f09?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/52f4cb0f0f09</guid>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[text-processing]]></category>
            <category><![CDATA[document-loader]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 03:50:39 GMT</pubDate>
            <atom:updated>2026-11-16T03:50:39.874Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s2_s8MWlcL9kfK0blOebLg.png" /></figure><p>In Lesson 01, we explored the fundamentals of Large Language Models (LLMs) and their limitations.</p><p>Now, let’s take our first practical step toward building a RAG system!</p><p>In this lesson, we’ll learn how to load documents, extract and clean text, and preserve metadata. We’ll also build a simple Python application to process PDF documents and save the extracted content in JSON and TXT formats, preparing it for the next stage of RAG.</p><h3>Document Loading</h3><p>Document Loading is the process of reading external files or data sources and extracting their content into a format that a RAG system can process.</p><p>RAG applications can load data from different sources, such as:</p><ul><li>PDF documents</li><li>Word documents (DOCX)</li><li>Text files (TXT)</li><li>Websites and HTML pages</li><li>Databases</li><li>CSV and JSON files</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*e4C1NqqRqFN9QfQbkN_8dQ.png" /><figcaption>document loading architecture</figcaption></figure><h3>Text Extraction</h3><p>Text Extraction is the process of retrieving readable text from documents.</p><p>For example, a PDF may contain paragraphs, tables, images, and headings. A text extractor attempts to retrieve the textual content so it can be processed.</p><blockquote><strong>Example:</strong> company_policy.pdf</blockquote><blockquote><strong>Employee Leave Policy :</strong> All permanent employees are entitled to 20 annual leave days. Leave requests must be submitted through the HR portal.</blockquote><blockquote><strong>After text extraction: </strong>Employee Leave Policy</blockquote><blockquote>All permanent employees are entitled<br>to 20 annual leave days.</blockquote><blockquote>Leave requests must be submitted<br>through the HR portal.</blockquote><p><strong>Important:</strong> Scanned PDFs may contain images instead of selectable text. They can require OCR (Optical Character Recognition) or a document-vision model. Tables and complex layouts may also require specialized extraction.</p><h3>Document Loader</h3><p>A Document Loader is a software component that reads a specific data source and converts its content into a usable document representation.</p><p>Popular Python tools include:</p><ul><li><strong>PyMuPDF</strong> -Extract text from PDF files.</li><li><strong>pypdf</strong> -Read PDF content using Python.</li><li><strong>python-docx-</strong>Read Word documents.</li><li><strong>BeautifulSoup</strong> -Parse HTML content.</li><li><strong>LangChain Document Loaders</strong> -Provide a common interface for loading different document types.</li></ul><h3>Document Preprocessing</h3><p>Document Preprocessing is the process of cleaning and organizing extracted content before using it for retrieval.</p><p>Common preprocessing operations include:</p><ul><li>Removing unnecessary whitespace.</li><li>Removing repeated headers and footers.</li><li>Fixing broken line breaks.</li><li>Removing irrelevant text.</li><li>Preserving meaningful headings and sections.</li></ul><p>Before Cleaning :</p><pre>Employee    Leave     Policy<br><br>Employees receive   20   annual<br>leave days.<br>Page 1</pre><p>After Cleaning :</p><pre>Employee Leave Policy<br>Employees receive 20 annual leave days.</pre><p><strong>Why is this important </strong>: Cleaner, well-structured text can improve retrieval quality and reduce irrelevant content.</p><h3>Metadata</h3><p>Metadata is additional information that describes a document or its content.</p><p>For example:</p><pre>{<br>  &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>  &quot;page&quot;: 1,<br>  &quot;document_type&quot;: &quot;HR Policy&quot;,<br>  &quot;department&quot;: &quot;Human Resources&quot;<br>}</pre><p>Metadata helps a RAG system identify where information came from and filter documents during retrieval.</p><h4>Why is metadata important for RAG</h4><p>Suppose a user asks:</p><blockquote>What is the annual leave policy according to the HR document?</blockquote><p>Metadata allows the system to restrict retrieval to HR documents and identify the source page.</p><h3>Practical-<strong>PDF Document Loading &amp; Text Processing</strong></h3><p><strong>Goal:</strong> Build a Python application that reads a PDF document, extracts text page by page, cleans unnecessary whitespace, preserves document metadata, and saves the processed content in JSON and TXT formats.</p><p><strong>github : </strong><a href="https://github.com/jinad-gamage/rag-fundamentals-to-advanced.git">https://github.com/jinad-gamage/rag-fundamentals-to-advanced.git</a></p><h4>Step 1 - <strong>Project Structure</strong></h4><p>Lesson-02-Document-Loading-Text-Processing/<br> ├── documents/<br> │ └── company_policy.pdf<br> ├── output/<br> │ ├── documents.json<br> │ └── processed_policy.txt<br> ├── main.py<br> └── requirements.txt</p><h4><strong>Complete Python Implementation</strong></h4><p>main.py</p><pre><br>import json<br>import re<br>from pathlib import Path<br><br>import pymupdf<br><br><br># Keep all input and output paths relative to this lesson folder.<br>BASE_DIR = Path(__file__).parent<br>PDF_PATH = BASE_DIR / &quot;documents&quot; / &quot;company_policy.pdf&quot;<br>OUTPUT_DIR = BASE_DIR / &quot;output&quot;<br>JSON_OUTPUT_PATH = OUTPUT_DIR / &quot;documents.json&quot;<br>TEXT_OUTPUT_PATH = OUTPUT_DIR / &quot;processed_policy.txt&quot;<br><br><br>def clean_text(text: str) -&gt; str:<br>    &quot;&quot;&quot;Normalize whitespace while preserving paragraph line breaks.&quot;&quot;&quot;<br><br>    # Replace multiple spaces and tabs with a single space.<br>    text = re.sub(r&quot;[ \t]+&quot;, &quot; &quot;, text)<br><br>    # Remove unnecessary spaces at the beginning and end of each line.<br>    text = re.sub(r&quot; *\n *&quot;, &quot;\n&quot;, text)<br><br>    # Replace three or more consecutive newlines with two.<br>    text = re.sub(r&quot;\n{3,}&quot;, &quot;\n\n&quot;, text)<br><br>    return text.strip()<br><br><br>def load_pdf(file_path: Path) -&gt; list[dict]:<br>    &quot;&quot;&quot;Load a PDF and return one document object for each non-empty page.&quot;&quot;&quot;<br>    documents = []<br><br>    # Open the PDF safely. It will be closed automatically after this block.<br>    with pymupdf.open(file_path) as pdf:<br>        # Process each page separately so page metadata can be preserved.<br>        for page_number, page in enumerate(pdf, start=1):<br>            raw_text = page.get_text(&quot;text&quot;)<br>            cleaned_text = clean_text(raw_text)<br><br>            # Skip pages that do not contain extractable text.<br>            if not cleaned_text:<br>                continue<br><br>            # Store the text and its source information together.<br>            documents.append(<br>                {<br>                    &quot;page_content&quot;: cleaned_text,<br>                    &quot;metadata&quot;: {<br>                        &quot;source&quot;: file_path.name,<br>                        &quot;page&quot;: page_number,<br>                    },<br>                }<br>            )<br><br>    return documents<br><br><br>def main() -&gt; None:<br>    # Stop early with a clear message if the input PDF is missing.<br>    if not PDF_PATH.exists():<br>        raise FileNotFoundError(f&quot;PDF not found: {PDF_PATH}&quot;)<br><br>    # Load and clean the PDF page by page.<br>    documents = load_pdf(PDF_PATH)<br><br>    # Create the output folder if it does not already exist.<br>    OUTPUT_DIR.mkdir(exist_ok=True)<br><br>    # Save page-wise documents in JSON format for future RAG steps.<br>    with JSON_OUTPUT_PATH.open(&quot;w&quot;, encoding=&quot;utf-8&quot;) as file:<br>        json.dump(documents, file, indent=2, ensure_ascii=False)<br><br>    # Also save all processed pages as one readable text file.<br>    processed_text = &quot;\n\n&quot;.join(<br>        f&quot;--- Page {document[&#39;metadata&#39;][&#39;page&#39;]} ---\n{document[&#39;page_content&#39;]}&quot;<br>        for document in documents<br>    )<br>    TEXT_OUTPUT_PATH.write_text(processed_text, encoding=&quot;utf-8&quot;)<br><br>    print(f&quot;Loaded {len(documents)} pages&quot;)<br>    print(f&quot;Saved structured documents to: {JSON_OUTPUT_PATH}&quot;)<br>    print(f&quot;Saved processed text to: {TEXT_OUTPUT_PATH}&quot;)<br><br>    # Preview each document&#39;s metadata and first 200 characters.<br>    for document in documents:<br>        print(document[&quot;metadata&quot;])<br>        print(document[&quot;page_content&quot;][:200])<br>        print()<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4><strong>Code Explanation</strong></h4><h4>Step 1- Install dependencies</h4><p>requirements.txt</p><pre>pymupdf</pre><p>Terminal:</p><pre>pip install -r requirements.txt</pre><h4><strong>Step 2 - Define File Paths</strong></h4><pre><br>BASE_DIR = Path(__file__).parent<br>PDF_PATH = BASE_DIR / &quot;documents&quot; / &quot;company_policy.pdf&quot;p</pre><p>BASE_DIR identifies the directory containing main.py. Other paths are created relative to this directory, so the program does not depend on where the terminal is opened.</p><h4>Step 3- Clean text</h4><pre><br>text = re.sub(r&quot;[ \t]+&quot;, &quot; &quot;, text)<br>text = re.sub(r&quot; *\n *&quot;, &quot;\n&quot;, text)<br>text = re.sub(r&quot;\n{3,}&quot;, &quot;\n\n&quot;, text)</pre><p>These regular expressions normalize repeated spaces, remove unnecessary spaces around line breaks, and reduce excessive blank lines while preserving paragraphs.</p><h4>Step 4- <strong>Load PDF</strong></h4><pre>with pymupdf.open(file_path) as pdf:<br>    for page_number, page in enumerate(pdf, start=1):<br>        raw_text = page.get_text(&quot;text&quot;)</pre><p>The PDF is opened and processed page by page. enumerate() provides page numbers starting from 1.</p><h4>Step 5 - Store Metadata</h4><pre>{<br>    &quot;page_content&quot;: cleaned_text,<br>    &quot;metadata&quot;: {<br>        &quot;source&quot;: file_path.name,<br>        &quot;page&quot;: page_number<br>    }<br>}</pre><p>Each document object stores extracted text and metadata identifying its source file and page.</p><h4>Step 6 - Save Outputs</h4><p>The application creates two files: documents.json for structured data and processed_policy.txt for human-readable text.</p><p><strong>Expected Console Output</strong></p><p>Terminal</p><pre>Loaded 2 pages<br>Saved structured documents to: .../output/documents.json<br>Saved processed text to: .../output/processed_policy.txt<br><br>{&#39;source&#39;: &#39;company_policy.pdf&#39;, &#39;page&#39;: 1}<br>Employee Leave Policy<br>Employees receive 20 annual leave days.<br>...</pre><p>Illustrative output. Actual content and page count depend on the input PDF.</p><p><strong>Generated Files</strong></p><p><strong><em>documents.json</em></strong></p><p>Contains structured page content and metadata. This will be used as input for Lesson 03 — Text Chunking.</p><pre>[<br>  {<br>    &quot;page_content&quot;: &quot;Employee Leave Policy\nEmployees receive 20 annual leave days.&quot;,<br>    &quot;metadata&quot;: {<br>      &quot;source&quot;: &quot;company_policy.pdf&quot;,<br>      &quot;page&quot;: 1<br>    }<br>  }<br>]</pre><p><strong><em>processed_policy.txt</em></strong></p><p>Contains all extracted pages in a readable format, separated by page labels.</p><pre>--- Page 1 ---<br>Employee Leave Policy<br>Employees receive 20 annual leave days.<br>--- Page 2 ---<br>...</pre><h3><strong>Where Does Document Processing Fit in RAG</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*EX559fgjtFf2OKusURmLnA.png" /></figure><h3>Common Problems in Document Processing</h3><p><strong>Problem 1-Scanned PDFs</strong></p><p>Scanned PDFs may not contain an extractable text layer.</p><p><strong>Problem 2-Tables</strong></p><p>Basic PDF text extraction may lose table rows, columns, or reading order.</p><p><strong>Problem 3 -Repeated Headers and Footers</strong></p><p>Repeated page headers, footers, and page numbers may introduce irrelevant content.</p><p><strong>Problem 4-Missing Metadata</strong></p><p>Without metadata, identifying source documents and generating reliable citations becomes difficult.</p><h3>Conclusion</h3><p>In this lesson, we learned how to load external documents, extract text, clean content, preserve metadata, and save structured document data using Python.</p><p><strong>Key takeaway:</strong> Document processing is the first major stage of building a RAG knowledge base. Good document preparation helps improve the quality of later retrieval.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52f4cb0f0f09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RAG Lesson 01 — LLM Fundamentals]]></title>
            <link>https://jinadgamage.medium.com/rag-lesson-01-llm-fundamentals-da119bea0348?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/da119bea0348</guid>
            <category><![CDATA[context-window]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[hallucinations]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 18:19:44 GMT</pubDate>
            <atom:updated>2026-11-15T18:23:49.592Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9QKv_f3_Pp158Ezkj_ydg.png" /></figure><p>Large Language Models (LLMs) are the foundation of modern AI applications. Before learning Retrieval-Augmented Generation (RAG), it’s important to understand how LLMs work and their limitations.</p><p>In this first lesson, we’ll explore LLMs, tokens, context windows, prompts, and hallucinations using simple explanations and practical examples.</p><h3>What is an LLM (<strong>Large Language Model</strong>) ?</h3><p>An LLM is an AI model trained on large amounts of text to understand and generate language.</p><p>LLMs can answer questions, summarize documents, translate languages, and generate code.</p><p>Examples:</p><ul><li>GPT- OpenAI</li><li>Gemini- Google</li><li>Claude- Anthropic</li><li>Llama- Meta</li></ul><p><strong>How does an LLM work?</strong></p><p>An LLM receives input text, converts it into tokens, processes those tokens using a neural network, and predicts the next token repeatedly to generate a response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*KLJux0Ovyipnhbbm-dyxMw.png" /><figcaption>fig: how to work LLM</figcaption></figure><blockquote><strong>Example:</strong></blockquote><blockquote>Input: <em>What is Python?</em></blockquote><blockquote>LLM Output: <em>Python is a high-level programming language used for web development, AI, automation, and data analysis.</em></blockquote><h3>What are Tokens?</h3><p>An LLM receives input text, converts it into tokens, processes those tokens using a neural network, and predicts the next token repeatedly to generate a response.</p><blockquote>Example :</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*Szinmhr49Zjme2PXO9oNkw.png" /></figure><p><strong>Why are tokens important for RAG?</strong></p><ul><li>Token limits affect how much information an LLM can process.</li><li>More tokens can increase API costs.</li><li>RAG retrieves only relevant document content to reduce unnecessary tokens.</li></ul><p><strong>What is a Context Window?</strong></p><p>A context window is the maximum amount of token-based information an LLM can process within a single context.</p><blockquote><strong>Example: </strong>Suppose an LLM supports an 8,000-token context window, but your PDF contains 50,000 tokens. You cannot provide the entire PDF in that context.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*fmY4QbxaFXX4np6adHakbg.png" /></figure><p><strong>RAG Solution:</strong> Retrieve only the most relevant sections from the PDF and send them to the LLM.</p><h3>What is a Prompt?</h3><p>A prompt is the input or instruction provided to an LLM.</p><blockquote><strong>Example 1 - Basic Promp</strong></blockquote><blockquote>Explain Python.</blockquote><blockquote><strong>Example 2 - Better Prompt</strong></blockquote><blockquote>You are a programming tutor.<br><br>Explain Python to a beginner.<br>Use simple English.<br>Include two practical examples.<br>Keep the answer under 150 words.</blockquote><blockquote><strong>Example 3 - RAG Prompt</strong></blockquote><blockquote>You are a helpful assistant.<br><br>Answer the question using only the<br>provided context.<br><br>If the answer is not in the context,<br>say &quot;I don&#39;t know&quot;.<br><br>Context: Employees receive 20 annual leave days.<br>Question: How many annual leave days do employees get?<br>Answer:</blockquote><blockquote><strong>Expected output:</strong></blockquote><blockquote>Employees receive 20 annual leave days.</blockquote><h3>What is Hallucination?</h3><p>Hallucination occurs when an LLM generates incorrect or unsupported information and presents it as factual.</p><blockquote>Example:</blockquote><blockquote>Question: How many leave days does ABC Company provide?</blockquote><blockquote>LLM may answer: 30 days</blockquote><blockquote>Actual company policy: 20 days</blockquote><p>The model generated an incorrect answer.</p><h4><strong>How RAG helps</strong></h4><p>RAG retrieves relevant information from company documents before generating the answer.</p><p><em>Note: RAG reduces hallucinations but does not completely eliminate them.</em></p><h3>Why Do We Need RAG</h3><p>Traditional LLMs have limitations:</p><ul><li>They may not know private company information.</li><li>Their learned knowledge may be outdated.</li><li>They can generate hallucinations.</li><li>They cannot automatically access your local documents.</li></ul><p>RAG addresses these limitations by retrieving relevant information from external knowledge sources.</p><h3>Practical : Calling an LLM with Python</h3><p>We will create a simple Python application that sends a question to Gemini and receives an AI-generated response.</p><p><em>github — </em><a href="https://github.com/jinadInduwithwa/rag-fundamentals-to-advanced.git"><em>https://github.com/jinadInduwithwa/rag-fundamentals-to-advanced.git</em></a></p><p><strong>Step 1- Install libraries</strong></p><pre>pip install google-genai python-dotenv</pre><p><strong>Step 2- Create </strong>.env</p><pre>GEMINI_API_KEY=your_api_key</pre><p><strong>Step 3 — Create </strong>main.py</p><pre><br>import os<br>from dotenv import load_dotenv<br>from google import genai<br><br># Load environment variables<br>load_dotenv()<br><br># Create Gemini client<br>client = genai.Client(<br>    api_key=os.environ[&quot;GEMINI_API_KEY&quot;]<br>)<br><br># Send a question to the LLM<br>response = client.models.generate_content(<br>    model=&quot;gemini-2.5-flash&quot;,<br>    contents=&quot;How many leave days does ABC Company provide?&quot;<br>)<br><br># Display the response<br>print(response.text)</pre><ul><li>load_dotenv() loads environment variables.</li><li>genai.Client() connects to Gemini.</li><li>generate_content() sends the prompt.</li><li>response.text returns the generated answer.</li></ul><h3>Conclusion</h3><p>LLMs are powerful tools for understanding and generating text, but they have limitations, such as outdated knowledge, hallucinations, and limited context windows.</p><p>RAG helps overcome these challenges by retrieving relevant information from external sources, allowing LLMs to generate more accurate and context-aware answers.</p><p><strong>Next Lesson:</strong> Introduction to RAG - What is RAG and How Does It Work?</p><p><strong>Next Lesson:</strong> Introduction to RAG — What is RAG and How Does It Work?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da119bea0348" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Keywords to Semantics: A Beginner’s Guide to Vector Databases]]></title>
            <link>https://jinadgamage.medium.com/from-keywords-to-semantics-a-beginners-guide-to-vector-databases-0b4b61f7e3c3?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b4b61f7e3c3</guid>
            <category><![CDATA[vector-search]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Sun, 31 May 2026 10:00:24 GMT</pubDate>
            <atom:updated>2026-05-31T10:00:24.351Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5voWUXVxmHPWjEQrYPrMAw.png" /></figure><p>A <strong>Vector Database</strong> is a specialized type of database built for a specific purpose for storing and searching through <strong>unstructured data</strong>.</p><p>What do we mean by “unstructured data”? Think of everything that doesn’t fit neatly into rows and columns:</p><ul><li><strong>Images:</strong> Photos of landscapes, objects, faces, etc.</li><li><strong>Text:</strong> Articles, product descriptions, emails, books.</li><li><strong>Audio:</strong> Music, speech, recordings.</li><li><strong>Video:</strong> Clips, movies, etc.</li></ul><h3>Why Can’t We Just Use a Relational Database (SQL)?</h3><p>This is a key question! Traditional relational databases (like SQL Server, MySQL, or PostgreSQL) are excellent for structured data (like a spreadsheet of customer names, ages, and purchase history). However, they struggle with unstructured data for a few big reasons:</p><ol><li><strong>Rigid Schemas:</strong> When you store data in SQL, you have to define a precise schema first meaning you must decide exactly what columns you need.</li><li><strong>Problem</strong> — Let’s say you want to store details about an image. In SQL, you might create columns for ‘color’, ‘object type’, ‘date taken’, ‘location’, and ‘dominant mood’. What happens if a new image arrives that contains details you hadn’t planned for? For example, “is it an indoor or outdoor photo?”. You would have to <strong>manually</strong> add a new column to your entire database and update every single existing entry</li><li><strong>No Semantic Understanding:</strong> A relational database can find an exact match if you search for “apple” in a product column. But it cannot understand that “apple” (the fruit) is related to “fruit,” “healthy snack,” or “red,” unless you manually program those relationships. It has no way to understand the <strong>meaning</strong> or <strong>semantic context</strong> of the unstructured data.</li></ol><h3>The Vector Database Solution</h3><p>This is where the Vector Database pipeline comes in! It provides an automated, scalable way to store and find unstructured data by focusing on its <strong>meaning</strong>.</p><ol><li><strong>Unstructured Data:</strong> You start with your files (images, text, audio).</li><li><strong>High-Dimensional Vectors (Embeddings):</strong> You feed this data into an <strong>Embedding Model</strong> (an AI or Machine Learning model). This model is smart; it “reads” the image or text and converts its core meaning into a long list of numbers. This list is called a <strong>Vector</strong> or <strong>Embedding</strong>. It might have hundreds or thousands of dimensions. The crucial part is that data with similar meanings gets converted into vectors that are “close” to each other in this complex numerical space.</li><li><strong>Vector Database (e.g., Chroma):</strong> You store these lists of numbers (the vectors) in a specialized database like <strong>Chroma</strong>. The database is optimized to perform a “similarity search” very quickly. If you want to find “images of a sunset,” you convert that search query into a vector and ask the database to find the vectors that are closest to it numerically, giving you the most relevant results.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C7a9so-BBc2MTXGqc8-7NA.png" /></figure><h3>When Should You Use a Vector Database?</h3><p>Because vector databases excel at finding items that are <strong>conceptually similar</strong>, they are the backbone of many modern AI applications. Here are three major situations where they shine:</p><h4>1. RAG (Retrieval-Augmented Generation) Applications</h4><ul><li><strong>The Problem:</strong> Large Language Models (LLMs) only know what they were trained on. They don’t know your private company data or recent news.</li><li><strong>The Vector DB Solution:</strong> You use RAG to give the AI a “memory.” You take all your company documents, convert them into vectors, and store them in the database. When a user asks a chatbot a question, the system converts that question into a vector, searches the vector database for the most relevant document chunks, and hands those chunks to the AI to formulate an accurate, customized answer.</li></ul><h4>2. Recommendation Engines</h4><ul><li><strong>The Problem:</strong> Recommending a movie or product based purely on exact tags (like “Action” or “Red”) often leads to poor, robotic suggestions.</li><li><strong>The Vector DB Solution:</strong> You can convert user behaviors, past purchases, and item descriptions into vectors. The database maps these out. If a user’s “preference vector” sits very close to the vector for a specific sci-fi movie, the system recommends that movie even if the user has never explicitly searched for “sci-fi.” It understands the hidden connections between what the user likes and what the item is.</li></ul><h4>3. Classifiers (Semantic Sorting)</h4><ul><li><strong>The Problem:</strong> Sorting data (like support tickets or customer reviews) into categories is hard when people use different words to mean the same thing (e.g., “broken screen” vs. “display cracked”).</li><li><strong>The Vector DB Solution:</strong> Instead of looking for exact keywords, the system looks at the meaning. You create a vector for the category “Hardware Issues.” When a new complaint comes in, it becomes a vector. The database instantly recognizes that the complaint’s vector is closest to the “Hardware Issues” category, automatically classifying it correctly.</li></ul><h3>Popular Examples of Vector Databases</h3><p>The market for vector databases is growing rapidly. Here are four of the most notable options you can mention:</p><ul><li><strong>Chroma:</strong> An open-source vector database that is incredibly popular among developers building AI applications. It is known for being very easy to set up and use on a local machine for testing and prototyping.</li><li><strong>Pinecone:</strong> A fully managed, cloud-native vector database. It is highly favored by enterprises because it is “serverless”.</li><li><strong>Weaviate:</strong> Another powerful open-source option. Weaviate is robust and scales well for production environments. It also has built-in features that allow you to bring your own machine learning models directly into the database workflow.</li><li><strong>FAISS (Facebook AI Similarity Search):</strong> While often grouped with vector databases, FAISS is technically a highly optimized <em>software library</em> developed by Meta. It provides the core algorithms needed to search massive vector datasets incredibly fast. Many actual vector databases use FAISS under the hood.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7RzcbySQf-6Yn2kVYMOS2A.png" /></figure><h3>Getting Started with ChromaDB</h3><p>To see how this works in reality, let’s look at a simple Python example using <strong>Chroma</strong>, a popular open-source vector database.</p><p>Remember our pipeline: <strong>Raw Text → Embedding Model → Vector Embeddings → Vector Database</strong>. Chroma is incredibly user-friendly because, by default, it handles the embedding process for you behind the scenes</p><h4>Step 1: Installation and Setup</h4><p>First, you need to install the Chroma library. In your terminal or notebook, run:</p><pre>!pip install chromadb -q</pre><p>Next, we initialize the Chroma client and create a “collection.” A collection in a vector database is similar to a table in a traditional relational database.</p><pre>import chromadb<br><br># Initialize the client<br>client = chromadb.Client()<br># Create a new collection to store our vectors<br>collection = client.create_collection(name=&quot;my_collection&quot;)</pre><h4>Step 2: Adding Data</h4><p>Now, let’s add some unstructured data. We will use short news summaries covering different topics: Cricket, Football, Elections, and AI.</p><pre>cricket_news = &quot;The T20 World Cup 2024 is in full swing...&quot;<br>football_news = &quot;The world of football is buzzing with excitement...&quot;<br>election_news = &quot;As election season heats up, the latest developments...&quot;<br>ai_revolution_news = &quot;The AI revolution continues to transform industries...&quot;<br><br># Add the documents to our Chroma collection<br>collection.add(<br>    documents=[cricket_news, football_news, election_news, ai_revolution_news],<br>    metadatas=[{&quot;source&quot;: &quot;cricket&quot;}, {&quot;source&quot;: &quot;football&quot;}, {&quot;source&quot;: &quot;election&quot;}, {&quot;source&quot;: &quot;ai revolution&quot;}],<br>    ids=[&quot;id1&quot;, &quot;id2&quot;, &quot;id3&quot;, &quot;id4&quot;]<br>)</pre><p><strong>What just happened?</strong> When we called collection.add(), Chroma automatically took our raw text, passed it through its default embedding model, converted the text into high-dimensional vectors, and stored those vectors alongside our metadata.</p><h4>Step 3: Performing a Similarity Search</h4><p>This is where the vector database shines. Let’s search our database for the concept of “technology.” Notice that the word “technology” doesn’t explicitly appear in our documents, but the <em>meaning</em> relates to one of them.</p><pre>results = collection.query(<br>    query_texts=[&quot;technology&quot;],<br>    n_results=2 # We want the top 2 closest matches<br>)<br>print(results)</pre><p>Because Chroma understands the semantic meaning (the context) of the vectors, it will successfully retrieve the ai_revolution_news document as the closest match to the concept of &quot;technology.&quot;</p><h4>Step 4: Managing Your Data (CRUD Operations)</h4><p>Just like a traditional database, vector databases allow you to manage your records.</p><ul><li><strong>Read:</strong> You can retrieve specific documents by their ID.Python</li></ul><pre>res = collection.get(ids=[&quot;id1&quot;, &quot;id3&quot;])</pre><ul><li><strong>Update:</strong> You can overwrite existing data. The database will automatically recalculate the new vector embedding.Python</li></ul><pre>collection.update(     ids=[&quot;id3&quot;],     documents=[&quot;This is a sample document about generative AI&quot;],     metadatas=[{&quot;source&quot;: &quot;gen ai&quot;}] )</pre><ul><li><strong>Delete:</strong> You can easily remove vectors from your collection.Python</li></ul><pre>collection.delete(ids=[&#39;id2&#39;])</pre><h4>Step 5: Advanced — Using Custom Embedding Models</h4><p>While Chroma’s default model is great, you often want to use specific, highly trained models for your specific use case. You can easily override the default and use external libraries like sentence_transformers.</p><p>Here is how you generate embeddings yourself and pass them directly into a new Chroma collection:</p><pre># Install the library<br>!pip install sentence_transformers -q<br><br>from sentence_transformers import SentenceTransformer<br># Load a specific open-source embedding model<br>embedding_model = SentenceTransformer(&quot;all-mpnet-base-v2&quot;)<br># Manually convert our text into numerical vectors (embeddings)<br>embeddings = embedding_model.encode([cricket_news, football_news])<br># Create a new collection<br>new_collection = client.create_collection(name=&quot;my_new_collection&quot;)<br># Add the pre-calculated embeddings directly to Chroma<br>new_collection.add(<br>    documents=[cricket_news, football_news],<br>    embeddings=embeddings, <br>    metadatas=[{&quot;source&quot;: &quot;cricket&quot;}, {&quot;source&quot;: &quot;football&quot;}],<br>    ids=[&quot;id1&quot;, &quot;id2&quot;]<br>)</pre><p>Now, when you query this new collection, you must also convert your search query into a vector using that same exact model:</p><pre>results = new_collection.query(<br>    query_embeddings=embedding_model.encode([&quot;test worldcup&quot;]),<br>    n_results=1<br>)</pre><h3>Conclusion</h3><p>We are generating more unstructured data: text, images, audio, and video than ever before. As we have seen, traditional relational databases simply weren’t built to handle the complexity and nuance of this information.</p><p>Vector databases bridge this gap. By leveraging embedding models to translate complex, unstructured data into high-dimensional numerical vectors, these databases allow us to search by <strong>concept and meaning</strong> rather than just exact keyword matches.</p><p>Whether you are building a sophisticated Retrieval-Augmented Generation (RAG) application to give a Large Language Model access to private data, designing a hyper-personalized recommendation engine, or creating an automated semantic classifier, vector databases are the foundational infrastructure making it possible.</p><p>The shift from relational rows and columns to vector embeddings represents a massive leap forward in how we process information. And, as the ChromaDB example demonstrates, adding this powerful capability to your modern development stack is more accessible than ever. The AI revolution is here, and vector databases are the engine driving it forward. Now is the perfect time to start experimenting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b4b61f7e3c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval-Augmented Generation (RAG) For Beginers]]></title>
            <link>https://jinadgamage.medium.com/retrieval-augmented-generation-rag-for-beginers-814672fe60c8?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/814672fe60c8</guid>
            <category><![CDATA[data-retrieval]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm-applications]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 14:22:45 GMT</pubDate>
            <atom:updated>2026-04-15T14:34:54.549Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EuL16322vpO0qPjtGtAHNw.png" /></figure><h3>Introduction</h3><p>Have you ever used an AI chatbot and wished it knew about your specific company policies, your personal notes, or a massive textbook you just downloaded? While Large Language Models (LLMs) are incredibly smart, they <strong>only know what they were originally trained on</strong>. They don’t have access to your private files.</p><p>To solve this, developers use a brilliant system architecture often called <strong>Retrieval-Augmented Generation (RAG)</strong>. The diagram you are looking at is a blueprint of exactly how this system works. It acts as a bridge, allowing an AI to quickly “read” your specific documents before it answers your question. Let’s break down how this works in two simple phases: storing the knowledge, and answering the question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*noXBN5UVff9iYtxkZ5Z6jw.png" /></figure><h3>Creating the AI’s Library (Data Preparation)</h3><p>Before you can ask the AI a question, the system needs to organize your documents so the computer can understand them.</p><ul><li><strong>Data Sources:</strong> It all starts with your files. These could be PDFs, Word documents, Excel sheets, or simple text files.</li><li><strong>Split into Chunks:</strong> Imagine trying to swallow a whole cake at once you can’t. AI models have reading limits, too. So, the system takes your long documents and “splits” them into smaller, bite-sized paragraphs or “chunks.”</li><li><strong>Embedding Model:</strong> Computers don’t read English the way we do; they understand numbers. The embedding model acts as a translator. It takes every chunk of text and turns it into a long string of numbers (called an embedding). These numbers capture the actual <em>meaning</em> of the text.</li><li><strong>Vector Database:</strong> Finally, these number strings are saved and indexed in a special filing cabinet called a Vector Database. Think of this as a highly organized library where information is grouped by meaning rather than just by alphabetical order.</li></ul><h3>Asking the Question (Retrieval and Response)</h3><p>Now that your documents are neatly organized in the AI’s library, here is what happens the moment you type a question.</p><ul><li><strong>The Question:</strong> You ask the system something (e.g., “What is our company’s refund policy?”).</li><li><strong>Query Vector Embeddings:</strong> Your question goes through that exact same translator (the Embedding Model) and is turned into a string of numbers.</li><li><strong>Similarity Search:</strong> The system takes your “number question” and runs to the Vector Database. It looks for the document chunks whose numbers closely match your question’s numbers. It’s essentially playing a massive game of matchmaking to find the most relevant information.</li><li><strong>Retrieved Chunks:</strong> The system pulls out the top few paragraphs that contain the answer.</li><li><strong>The Prompt:</strong> This is the magic step. The system builds a new, behind-the-scenes message (the prompt). It combines three things:</li></ul><p><em>A strict set of instructions (e.g., “Answer the user using only the provided text”).</em></p><p><em>The relevant chunks it just found (the “cheat sheet”).</em></p><p><em>Your original question.</em></p><ul><li><strong>The LLM &amp; Final Response:</strong> This combined package is sent to the “brain” of the operation the Large Language Model (LLM). Because the LLM now has your question <em>and</em> the exact paragraphs containing the facts, it can write out a highly accurate, custom response just for you</li></ul><h3>Seeing It in Action: The Python Code</h3><p>Now that we understand the theory, what does this actually look like in practice? Below is a complete, working Python script that builds this exact RAG architecture using <strong>LangChain</strong> and <strong>Hugging Face</strong> (which hosts free, powerful AI models).</p><pre>import argparse<br>import os<br>from typing import List<br>import requests<br><br>from langchain_chroma import Chroma<br>from langchain_community.document_loaders import PyPDFLoader<br>from langchain_huggingface import HuggingFaceEndpointEmbeddings<br>from langchain_text_splitters import RecursiveCharacterTextSplitter<br>from dotenv import load_dotenv<br><br>def get_hf_token() -&gt; str:<br>    # Prefer environment variable; fallback to Colab secret if available.<br>    token = os.getenv(&quot;HUGGINGFACE_API_KEY&quot;)<br>    if token:<br>        return token<br><br>    try:<br>        from google.colab import userdata  # type: ignore<br><br>        token = userdata.get(&quot;HUGGINGFACE_API_KEY&quot;)<br>        if token:<br>            return token<br>    except Exception:<br>        pass<br><br>    raise ValueError(<br>        &quot;Missing Hugging Face API token. Set HUGGINGFACE_API_KEY in env vars &quot;<br>        &quot;or Colab Secrets.&quot;<br>    )<br><br>def build_retriever(pdf_path: str, hf_token: str):<br>    os.environ[&quot;HUGGINGFACEHUB_API_TOKEN&quot;] = hf_token<br><br>    # Initialize Embedding Model<br>    embedding_model = HuggingFaceEndpointEmbeddings(<br>        model=&quot;sentence-transformers/all-mpnet-base-v2&quot;,<br>        huggingfacehub_api_token=hf_token,<br>    )<br><br>    if not os.path.exists(pdf_path):<br>        raise FileNotFoundError(<br>            f&quot;PDF file not found: {pdf_path}\n&quot;<br>            f&quot;Please provide a valid PDF file path using --pdf argument.\n&quot;<br>            f&quot;Example: python rag_hf_api_only.py --pdf /path/to/document.pdf&quot;<br>        )<br>    <br>    # Load PDF Document<br>    print(f&quot;Loading PDF: {pdf_path}...&quot;)<br>    loader = PyPDFLoader(pdf_path)<br>    docs = loader.load()<br>    print(f&quot;✓ Loaded {len(docs)} pages&quot;)<br>    <br>    if not docs:<br>        raise ValueError(f&quot;No pages extracted from PDF: {pdf_path}&quot;)<br><br>    # Split Documents into Chunks<br>    text_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50)<br>    splits = text_splitter.split_documents(docs)<br><br>    # Create Vector Store and Retriever<br>    print(f&quot;Creating embeddings for {len(splits)} chunks (this may take a moment)...&quot;)<br>    vectorstore = Chroma.from_documents(<br>        documents=splits,<br>        embedding=embedding_model,<br>        collection_name=&quot;rag_docs&quot;<br>    )<br>    retriever = vectorstore.as_retriever(search_kwargs={&quot;k&quot;: 4})<br>    print(&quot;✓ Vector store created successfully&quot;)<br><br>    return retriever, len(docs), len(splits)<br><br>def format_docs(retrieved_docs) -&gt; str:<br>    return &quot;\n\n&quot;.join(doc.page_content for doc in retrieved_docs)<br><br>def build_prompt(question: str, context: str) -&gt; str:<br>    # Define Prompt Template<br>    return f&quot;&quot;&quot;<br>Answer this question using the provided context only.<br><br>Question:<br>{question}<br><br>Context:<br>{context}<br><br>Answer:<br>&quot;&quot;&quot;<br><br>def call_hf_inference_api(prompt: str, hf_token: str) -&gt; str:<br>    # Initialize HuggingFace LLM<br>    # Free-tier friendly model fallback list for the current HF router providers.<br>    model_candidates = [<br>        &quot;meta-llama/Llama-3.1-8B-Instruct&quot;,<br>        &quot;Qwen/Qwen2.5-7B-Instruct&quot;,<br>        &quot;deepseek-ai/DeepSeek-R1-Distill-Qwen-7B&quot;,<br>    ]<br><br>    headers = {<br>        &quot;Authorization&quot;: f&quot;Bearer {hf_token}&quot;,<br>        &quot;Content-Type&quot;: &quot;application/json&quot;,<br>    }<br><br>    last_error = &quot;Unknown error&quot;<br>    for model_id in model_candidates:<br>        url = &quot;https://router.huggingface.co/v1/chat/completions&quot;<br>        payload = {<br>            &quot;model&quot;: model_id,<br>            &quot;messages&quot;: [<br>                {<br>                    &quot;role&quot;: &quot;system&quot;,<br>                    &quot;content&quot;: &quot;You are a helpful RAG assistant. Answer only using the provided context.&quot;,<br>                },<br>                {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>            ],<br>            &quot;max_tokens&quot;: 256,<br>            &quot;temperature&quot;: 0.1,<br>        }<br><br>        try:<br>            response = requests.post(url, headers=headers, json=payload, timeout=90)<br>            if response.status_code &gt;= 400:<br>                last_error = f&quot;{model_id}: HTTP {response.status_code} - {response.text[:300]}&quot;<br>                continue<br><br>            data = response.json()<br>            if isinstance(data, dict):<br>                choices = data.get(&quot;choices&quot;) or []<br>                if choices and isinstance(choices[0], dict):<br>                    message = choices[0].get(&quot;message&quot;) or {}<br>                    content = message.get(&quot;content&quot;)<br>                    if content:<br>                        return str(content).strip()<br><br>            if isinstance(data, dict) and data.get(&quot;error&quot;):<br>                last_error = f&quot;{model_id}: {data[&#39;error&#39;]}&quot;<br>                continue<br><br>            last_error = f&quot;{model_id}: Unexpected API response format&quot;<br>        except requests.RequestException as exc:<br>            last_error = f&quot;{model_id}: {exc}&quot;<br><br>    raise RuntimeError(f&quot;All Hugging Face free model calls failed. Last error: {last_error}&quot;)<br><br>def ask_questions(retriever, questions: List[str], hf_token: str) -&gt; None:<br>    # Initialize Output Parser<br>    # Chain Retriever and Prompt Template with LLM<br>    # Invoke RAG Chain with Example Questions<br>    for q in questions:<br>        print(f&quot;Question: {q}&quot;)<br>        print(&quot;Answer:&quot;)<br>        docs = retriever.invoke(q)<br>        context = format_docs(docs)<br>        prompt = build_prompt(q, context)<br>        print(call_hf_inference_api(prompt, hf_token))<br>        print(&quot;-&quot; * 80)<br><br>def main() -&gt; None:<br>    # Load environment variables from a local .env file when present.<br>    load_dotenv()<br><br>    parser = argparse.ArgumentParser(<br>        description=&quot;RAG app using Hugging Face hosted APIs only (no local LLM/embedding).&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--pdf&quot;,<br>        default=&quot;Arch 64.pdf&quot;,<br>        help=&quot;Path to PDF document&quot;,<br>    )<br>    parser.add_argument(<br>        &quot;--question&quot;,<br>        action=&quot;append&quot;,<br>        help=&quot;Question to ask (repeat --question for multiple).&quot;,<br>    )<br>    args = parser.parse_args()<br><br>    hf_token = get_hf_token()<br>    print(&quot;Initializing RAG pipeline with Hugging Face hosted APIs...&quot;)<br>    retriever, doc_count, split_count = build_retriever(args.pdf, hf_token)<br><br>    print(f&quot;\n✓ Setup complete!&quot;)<br>    print(f&quot;  - Loaded pages: {doc_count}&quot;)<br>    print(f&quot;  - Created chunks: {split_count}\n&quot;)<br><br>    default_questions = [<br>        &quot;Who is Arch 64?&quot;,<br>        &quot;What do they offer?&quot;,<br>    ]<br><br>    questions = args.question if args.question else default_questions<br>    ask_questions(retriever, questions, hf_token)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>github repo :<a href="https://github.com/jinadInduwithwa/rag-langchain-huggingface.git"> https://github.com/jinadInduwithwa/rag-langchain-huggingface.git</a></p><h3>Breaking Down the Code</h3><p>If you have never written a RAG application before, this code might look like a lot. But if you look closely, it maps perfectly to the two phases we discussed earlier! Let’s translate this code into plain English.</p><h3>Phase 1: Creating the Library (build_retriever function)</h3><p>This part of the code is responsible for reading your file and organizing it.</p><ul><li><strong>Loading the Data:</strong> The line loader = PyPDFLoader(pdf_path) opens up your specific PDF document so the computer can read the text inside.</li><li><strong>Splitting into Chunks:</strong> The RecursiveCharacterTextSplitter acts like a pair of scissors. It cuts your long document into smaller blocks of 400 characters (with a little bit of overlap so sentences don&#39;t get awkwardly cut in half).</li><li><strong>Embedding and Storing:</strong> We use a tool called Chroma to create our Vector Database. The code takes every 400-character chunk, turns it into a mathematical embedding using a sentence-transformer model, and saves it neatly in Chroma.</li></ul><h3>Phase 2: Answering the Question (ask_questions &amp; call_hf_inference_api)</h3><p>Once the database is built, the system is ready to chat.</p><ul><li><strong>Retrieval (Finding the Facts):</strong> When you ask a question, docs = retriever.invoke(q) searches the Chroma database. Because we set k=4 earlier in the code, it pulls out the top 4 most relevant text chunks that match your question.</li><li><strong>Building the Prompt:</strong> The build_prompt function is where we create our &quot;cheat sheet.&quot; It stitches together a strict rule (&quot;Answer this question using the provided context only&quot;), the 4 relevant chunks it just found, and your actual question.</li><li><strong>The Final Output:</strong> Finally, call_hf_inference_api sends this giant, fact-filled prompt to a powerful AI model (like Llama 3.1 or Qwen) hosted on Hugging Face. The AI reads your document chunks, writes a smart, accurate answer, and prints it to your screen.</li></ul><h3>Conclusion</h3><p>Retrieval-Augmented Generation might sound like an intimidating, highly complex computer science term, but at its core, it is a very logical system. By giving an AI the ability to look up facts in an organized database before it speaks, we turn a general chatbot into a highly specialized, accurate, and secure assistant for our own data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=814672fe60c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essential Dart Concepts for Flutter Developers]]></title>
            <link>https://jinadgamage.medium.com/essential-dart-concepts-for-flutter-developers-5868599d9b12?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/5868599d9b12</guid>
            <category><![CDATA[dart-programming-language]]></category>
            <category><![CDATA[flutter]]></category>
            <category><![CDATA[dart]]></category>
            <category><![CDATA[flutter-app-development]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Sat, 21 Jun 2025 18:33:50 GMT</pubDate>
            <atom:updated>2025-06-21T18:33:50.658Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Id2WHz-UUYjETwlNne4Ccg.jpeg" /></figure><p>When diving into Flutter, the first thing you’ll hear is that it uses a language called <strong>Dart</strong>. While many developers focus on Flutter’s widgets and UI capabilities, understanding Dart is equally important. Why? Because Dart isn’t just the language <em>behind</em> Flutter — it’s the engine that makes it powerful, fast, and flexible.</p><p>Whether you’re new to mobile development or transitioning from JavaScript, Java, or Swift, knowing the essential Dart concepts will make your Flutter journey smoother, help you write cleaner code, and reduce debugging nightmares.</p><p>In this blog, we’ll break down the key Dart concepts every Flutter developer should know — from syntax basics to more advanced features like null safety and asynchronous programming.</p><h3>What is Dart?</h3><p><strong>Dart</strong> is a <strong>client-optimized programming language</strong> developed by <strong>Google</strong>, designed for building fast, modern web and mobile applications.</p><p>Originally released in 2011, Dart has grown in popularity mainly due to its role as the primary language for <strong>Flutter</strong>, Google’s UI toolkit for crafting beautiful apps for mobile, web, and desktop from a single codebase.</p><p><strong>Key Features of Dart:</strong></p><ul><li><strong>Object-Oriented</strong> — Everything in Dart is an object, even numbers and functions.</li><li><strong>Statically Typed</strong> — Types are known at compile time, allowing for better tooling and error checking.</li><li><strong>Just-in-Time (JIT) &amp; Ahead-of-Time (AOT)</strong> compilation — JIT enables fast development and hot reload, while AOT provides optimized performance in production.</li><li><strong>Null Safety</strong> — Eliminates a common source of bugs by helping avoid null reference exceptions.</li><li><strong>Asynchronous Support</strong> — Built-in support for async, await, and Future, making it ideal for responsive UIs and networking.</li></ul><h3>Main Function</h3><p>In Dart, the main() function is the <strong>entry point</strong> of every program. It’s where the execution of your code begins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vm31031bJTWTpNNJqUQuCQ.png" /></figure><h3>String Concatenation &amp; Interpolation in Dart</h3><p>In Dart, instead of using traditional + operators to join strings and variables, you can use <strong>string interpolation</strong>, which is cleaner and more readable. The print() statement uses <strong>$x and </strong><strong>$y</strong> to insert variable values directly into the string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPau1aMxJsJyuXh3Csr22Q.png" /></figure><p><strong>Why Use String Interpolation?</strong></p><ul><li>Easier to read than &quot;Hello world &quot; + x.toString() + y.toString()</li><li>Automatically converts variables to strings</li><li>Supports expressions inside ${} (e.g., ${x + y})</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOUZeUO2EhmIdF0AXbg-Ow.png" /></figure><h3>Variables in Dart</h3><p>In Dart, variables are used to store data, and you can declare them using keywords like var, const, and final. Let&#39;s look at how they work with different types.</p><ul><li><strong>var</strong>: Automatically infers the type at compile time. Here, w is an int, and x is a double.</li><li><strong>const</strong>: A compile-time constant. The value must be known at compile time and cannot change.</li><li><strong>final</strong>: A runtime constant. It can be set once and cannot be changed after it’s assigned.</li></ul><p><strong>runtimeType</strong>: This property shows the type of the variable during execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1_eTFvTDhYr4pZ0JiO2Sw.png" /></figure><h3>Classes in Dart</h3><p>Dart is an <strong>object-oriented language</strong>, which means it supports the use of <strong>classes and objects</strong>. A class is like a <strong>blueprint</strong> for creating objects. it defines what properties and behaviors (methods) those objects will have.</p><ul><li>class Person defines a class named <strong>Person</strong>.</li><li>It has two properties: name and age, both of type String? (nullable, meaning they can be null).</li><li>In main(), we create an object using Person p = Person();.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZOXEyKCqFvM7yVcJBf_lLw.png" /></figure><p>This is the foundation of OOP in Dart. you can assign values, use constructors, and add methods to build more powerful classes.</p><h3>Constructors in Dart</h3><p>In Dart, a <strong>constructor</strong> is a special method used to <strong>initialize objects</strong> when they are created. Constructors help set default values or pass data during object creation.</p><p><strong>default constructor</strong></p><p>Person() constructor is automatically called when you create a new object using Person p = Person();.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1vz8UW4H1anbg3mBNYr1kg.png" /></figure><p><strong>Parameterized constructors</strong> In Dart, <strong>parameterized constructors</strong> allow you to pass values directly when creating an object. This is useful when you want to initialize class properties with specific data instead of hardcoding values.</p><ul><li>Person(String name, int age) is a constructor that takes two parameters.</li><li>The this.name = name; syntax assigns the incoming values to the class properties.</li><li>When you create the object with Person(&quot;Jinad&quot;, 23), the constructor initializes name and age with those values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_91CHoaulrW-hvgyt0KY5A.png" /></figure><p><strong>Constructor with Initializer List</strong></p><p>Dart allows a special syntax called the <strong>initializer list</strong>, which lets you assign values to class properties <strong>before the constructor body runs</strong>. It’s a more concise and preferred way for simple initializations.</p><ul><li>: name = n, age = a is the initializer list.</li><li>It assigns values to name and age <strong>before</strong> the constructor body (if any) runs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2UVV0d3aY6c3ymxfeNNQJQ.png" /></figure><p><strong>Constructor with Named Parameters</strong></p><p>In Dart, constructors can use <strong>named parameters</strong> with curly braces {}. This makes the code more <strong>readable</strong> and <strong>flexible</strong>, especially when you&#39;re passing multiple or optional values — something you&#39;ll see often in Flutter widgets.</p><ul><li>{} defines <strong>named parameters</strong> in the constructor.</li><li>required ensures that the caller must provide a value for that parameter.</li><li>this.name and this.age directly assign values to the class properties.</li><li>text is also a required parameter but not assigned to a class variable — instead, it&#39;s used for printing inside the constructor.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3m4LHYhzZMyLFvuK56ipcw.png" /></figure><p><strong>Why Use Named Parameters?</strong></p><ul><li>Improves readability (especially with many parameters)</li><li>Avoids confusion about parameter order</li><li>Makes constructors look like Flutter widget constructors</li></ul><h3>Functions and Parameters in Dart</h3><p>Functions in Dart allow you to group reusable code into logical blocks. They can accept <strong>parameters</strong> and optionally return values. Let’s look at a simple example that adds two numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Ajoqr7QaDyMV0a-qFyXlg.png" /></figure><ul><li>Functions in Dart are defined using the syntax:returnType functionName(parameters) { ... }</li><li>You can <strong>omit the return type</strong>, but it’s good practice to declare it for clarity.</li><li>Dart is flexible with parameter types unless type safety is enforced.</li></ul><p>Improved (typed) version:</p><pre>int add(int num1, int num2) {<br>  return num1 + num2;<br>}</pre><p>This ensures type safety and helps avoid runtime errors.</p><p><strong>Optional Positional Parameters</strong></p><p>Optional positional parameters are wrapped in square brackets []. They are <strong>optional</strong>, and if you don’t pass a value, it uses null or a default value (if specified).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4RTRWjiuHpmwccQVYgZ1g.png" /></figure><p><strong>Named Parameters</strong></p><p>Named parameters are wrapped in curly braces {} and passed by <strong>name</strong> when calling the function. You can also mark them as <strong>required</strong>. required ensures the parameter must be provided.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEDaDAzhT41jEcNMVZPGAg.png" /></figure><p><strong>Passing Functions as Parameters</strong></p><p>In Dart, you can pass a <strong>function as an argument</strong> to another function — just like you pass numbers, strings, or objects. This is known as a <strong>higher-order function</strong>, and it’s useful for customizing behavior.</p><p>Let’s explore it using a simple math operation example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ncJc5QZfR3frhzN5EIldtw.png" /></figure><ul><li>doMath() is a function that takes three <strong>named parameters</strong>:</li><li>n1 and n2: numbers to operate on</li><li>onClick: a <strong>function</strong> that takes two arguments</li><li>onClick(n1, n2) dynamically calls whatever function was passed</li><li>In main(), we call doMath() and pass the add function as onClick</li></ul><h3>Generic Classes in Dart</h3><p>In Dart, <strong>generic classes</strong> allow you to define classes that work with <strong>any data type</strong>. This makes your code <strong>more reusable, type-safe</strong>, and easy to maintain — especially useful for data structures like lists, stacks, or queues.</p><p><strong>Why Use Generics?</strong></p><ul><li><strong>Type-safe</strong>: Only allows values of the specified type.</li><li><strong>Reusable</strong>: One class works with many data types.</li><li><strong>Clean and scalable</strong>: Great for building custom data structures.</li></ul><p>Let’s look at a custom generic <strong>Stack</strong> class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4jAZNravURL0slgIKd2atQ.png" /></figure><ul><li>Stack&lt;T, V&gt; is a <strong>generic class</strong> with two type parameters (T and V). In this example, only T is being used.</li><li>List&lt;T&gt; _l stores the stack items — this ensures type safety.</li><li>push() adds an item to the list.</li><li>pop() removes and returns the last item.</li><li>printStack() prints the current state of the stack.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5868599d9b12" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[React : A Beginner-Friendly Guide]]></title>
            <link>https://jinadgamage.medium.com/react-a-beginner-friendly-guide-8d9ff77b5afc?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d9ff77b5afc</guid>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[reactjs]]></category>
            <category><![CDATA[react-beginner-guide]]></category>
            <category><![CDATA[react-hook]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 07:56:47 GMT</pubDate>
            <atom:updated>2025-06-20T07:56:47.159Z</atom:updated>
            <content:encoded><![CDATA[<h3>React : A Beginner-Friendly Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bOQbXI8WaeQf0kzoj8cLvQ.jpeg" /></figure><h4>what is React ?</h4><p><strong>React</strong> is an open-source JavaScript library developed by Facebook for building fast, interactive user interfaces (UIs), especially for single-page applications (SPAs). It allows developers to create reusable UI components that automatically update when data changes — making web apps more dynamic and user-friendly.</p><p>Think of React as the <strong>“brain”</strong> behind what users see and interact with in modern web apps like Facebook, Instagram, and Netflix. It’s component-based, declarative, and built for performance and scalability.</p><p><strong>Why React?</strong></p><ul><li>Component-based: Build once, reuse everywhere.</li><li>Fast updates: React updates only the parts that change.</li><li>One-way data flow: Predictable and easier to debug.</li><li>Huge ecosystem: Tools, libraries, and community support.</li></ul><p><strong>Pre-Request</strong></p><p>Before diving into React, it’s helpful to have a basic understanding of the following:</p><ul><li><strong>HTML</strong> — Structure of web pages</li><li><strong>CSS</strong> — Styling the layout and look</li><li><strong>JavaScript (ES6+)</strong> — Variables, functions, arrays, arrow functions, import/export, etc.</li><li><strong>Basic Terminal Usage</strong> — Running npm/yarn commands, creating folders</li><li><strong>Node.js &amp; npm</strong> — Make sure they’re installed for package management</li></ul><h4>Way to build react app</h4><ol><li>create-react-app : Official setup tool for React projects</li><li>Vite (Recomanded): Fast modern build tool with React support</li><li>NextJs: Framework for SSR, routing, and fullstack</li><li>Remix: Fullstack React framework with routing</li><li>Gatsby: React-based static site generator</li></ol><p>Checkout React Documentation — <a href="https://react.dev/learn">https://react.dev/learn</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8njojQSWLvxRX_lES3e_w.png" /><figcaption>fg1: Create react app using Vite</figcaption></figure><h4>Project Structure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*72Fjs_l6iuz91C9nUtfBKw.png" /><figcaption>fg2: Stucture of project</figcaption></figure><h3>Fundamentals</h3><h4>1. React Function</h4><p>In React, we build our app using <strong>components</strong>, and a <strong>component is just a JavaScript function</strong> that returns a piece of UI (like a button, heading).Think of it like : A React function is like a reusable recipe for a UI piece.</p><p><strong>Example:</strong></p><pre>function Product(){<br>    return(<br>        &lt;div&gt;&lt;h1&gt;this is product ccomponent&lt;/h1&gt;&lt;/div&gt;<br>    )<br>}</pre><p>How to use it: Just like HTML tags, use it like this:</p><pre>&lt;Welcome /&gt;</pre><h4>2. JSX</h4><p>JSX is a <strong>special way of writing HTML inside JavaScript</strong>. React uses JSX to describe <strong>what the UI should look like</strong>.</p><p>Without JSX:</p><pre>React.createElement(&#39;h1&#39;, null, &#39;this is product ccomponent&#39;)</pre><p>With JSX (much easier):</p><pre>&lt;h1&gt;this is product ccomponent&lt;/h1&gt;</pre><h4>3. Import/Export</h4><p>When your app grows, you’ll put components in <strong>separate files</strong>. You need export to <strong>share</strong> code and import to <strong>bring it in</strong>.</p><p>Exporting (like making it public):</p><pre>function Product(){<br>    return(<br>        &lt;div&gt;&lt;h1&gt;this is product component&lt;/h1&gt;&lt;/div&gt;<br>    )<br>}<br>export default Product;</pre><p>Importing (like using it elsewhere):</p><p>In App.jsx:</p><pre>import Product from &#39;./component/Product&#39;</pre><pre>function App() {<br>  return (<br>    &lt;Product/&gt;<br>  )<br>}<br>export default App</pre><h4>4. Reuse a Component</h4><p>React lets you <strong>reuse one component multiple times</strong> — like a rubber stamp.</p><p>Example: You want to greet different people</p><pre>function GreetUser() {<br>  return &lt;h3&gt;Hello&lt;/h3&gt;;<br>}</pre><p>Now use it in App.jsx:</p><pre>import Product from &#39;./component/Product&#39;</pre><pre>function App() {<br>  return (<br>    &lt;&gt;<br>    &lt;Product/&gt;<br>    &lt;Product/&gt;<br>    &lt;Product/&gt;<br>    &lt;/&gt;<br>  )<br>}<br>export default App</pre><p>output</p><ul><li>Product is the <strong>same component</strong>, used in <strong>3 different ways</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*4GVGLzhsv1F7EUpW_LXw5g.png" /></figure><h3>Dynamic Value In JSX</h3><p>One of the key benefits of using React is its ability to handle <strong>dynamic content</strong> easily using JSX. In we’ll look at how JSX can render variables, function results, and even simple expressions directly in the UI.</p><p>Let’s understand this better with a practical example:</p><pre>function Product() {<br>    let productName = &quot;Iphone&quot;;</pre><pre>function calculatePrice(productPrice) {<br>        let discountPrice = 1000;<br>        return productPrice - discountPrice;<br>    }</pre><pre>return (<br>        &lt;div&gt;<br>            &lt;h1&gt;This is {productName} product component&lt;/h1&gt;<br>            &lt;p&gt;Total price is {calculatePrice(100000)}&lt;/p&gt;<br>            &lt;p&gt;2 + 2 = {2 + 2}&lt;/p&gt;<br>        &lt;/div&gt;<br>    );<br>}<br>export default Product;</pre><p><strong>What’s Going On?</strong></p><p>Let’s break down how <strong>dynamic values</strong> are being handled in this JSX code.</p><h3>Variable inside JSX</h3><pre>{productName}</pre><p>This inserts the string &quot;Iphone&quot; directly into the HTML.</p><h3>Function call inside JSX</h3><pre>{calculatePrice(100000)}</pre><p>This calls a function that subtracts a discount of 1000 from the original price (100000). The result (99000) is then displayed.</p><h3>Inline expression</h3><pre>{2 + 2}</pre><p>This performs a simple math operation and shows the result (4) directly in the UI.</p><p>Output in the Browser — When rendered, the component will display:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*uqah3mxSxV3xgPeMxZuQTA.png" /></figure><p><strong>JSX Expression Rules</strong></p><p>In JSX:</p><ul><li>You can only use <strong>expressions</strong>, not full statements like if, for, or while.</li><li>Expressions must go inside {} brackets.</li><li>These can include: variables, function calls, arithmetic, ternary operators, etc.</li></ul><p>Valid:</p><pre>&lt;p&gt;{5 * 2}&lt;/p&gt;<br>&lt;p&gt;{name}&lt;/p&gt;<br>&lt;p&gt;{price &gt; 100 ? &quot;Expensive&quot; : &quot;Affordable&quot;}&lt;/p&gt;</pre><p>Invalid:</p><pre>// This will cause an error<br>{let x = 10;}<br>{if (true) { return &quot;Hello&quot;; }}</pre><h3>Styling in JSX</h3><p>React supports multiple ways to style your components. let’s understand <strong>how to apply styles in JSX</strong>, using both <strong>inline styles</strong> and <strong>external CSS files</strong>.</p><p>Here’s a simple example using both methods:</p><pre>import &quot;./Product.css&quot;; // Importing external CSS file</pre><pre>function Product() {<br>    return (<br>        &lt;div&gt;<br>            {/* Method 1: Inline Styling */}<br>            &lt;h1 style={{<br>                fontSize: &quot;20px&quot;,<br>                color: &quot;red&quot;<br>            }}&gt;<br>                This is product component<br>            &lt;/h1&gt;</pre><pre>{/* Method 2: External CSS Class */}<br>            &lt;p className=&quot;productCatelog&quot;&gt;<br>                Product catalog<br>            &lt;/p&gt;<br>        &lt;/div&gt;<br>    );<br>}<br>export default Product;</pre><h4>Method 1: Inline Styling in JS</h4><p>React allows you to use <strong>inline styles</strong> using a special syntax: <strong>JavaScript object</strong> inside the style attribute.</p><pre>&lt;h1 style={{<br>  fontSize: &quot;20px&quot;,<br>  color: &quot;red&quot;<br>}}&gt;<br>  This is product component<br>&lt;/h1&gt;</pre><p><strong>Things to note:</strong></p><ul><li>Use <strong>camelCase</strong> for property names (e.g., fontSize, not font-size)</li><li>Values should be strings (e.g., &quot;20px&quot;, not 20px)</li><li>Useful for quick or dynamic styling (e.g., based on props or state)</li></ul><h4>Method 2: CSS Classes in JSX</h4><p>You can also use an <strong>external CSS file</strong> to write styles and apply them using className.</p><p>In Product.css:</p><pre>.productCatelog {<br>  color: blue;<br>  font-weight: bold;<br>}</pre><p>In your JSX:</p><pre>import &quot;./Product.css&quot;;<br>&lt;p className=&quot;productCatelog&quot;&gt;Product catalog&lt;/p&gt;</pre><h3>Props — Making Components Dynamic and Reusable</h3><p>In React, <strong>props</strong> (short for “properties”) let you <strong>pass data from one component to another</strong>. This makes your components <strong>dynamic, customizable, and reusable</strong>.</p><p>Let’s understand how props work with a practical example.</p><p><strong>Example: </strong><strong>App.js and </strong><strong>Product.js</strong></p><p>App.js</p><pre>import Product from &#39;./component/Product&#39;;</pre><pre>function App() {<br>  return (<br>    &lt;&gt;&lt;Product title=&quot;Iphone 14&quot; price={120000} /&gt;<br>      &lt;Product title=&quot;Samsung S23&quot; price={110000} /&gt;<br>      &lt;Product title=&quot;Google Pixel 9&quot; price={100000} /&gt;<br>    &lt;/&gt;<br>  );<br>}<br>export default App;</pre><p>Product.js</p><pre>function Product(props) {<br>  console.log(props); // for debugging</pre><pre>return (<br>    &lt;div&gt;<br>      &lt;h1&gt;This is {props.title}&lt;/h1&gt;<br>      &lt;p&gt;Price is {props.price}&lt;/p&gt;<br>    &lt;/div&gt;<br>  );<br>}</pre><pre>export default Product;</pre><p><strong>How It Works</strong></p><ul><li>The App component uses the Product component <strong>three times</strong>, passing different title and price values as <strong>props</strong>.</li><li>Inside Product, we access these values using the props object.</li><li>This allows a single component to render <strong>different content</strong> each time.</li></ul><p><strong>What Are Props?</strong></p><ul><li><strong>Props are read-only</strong>: You can’t change them inside the child component.</li><li><strong>Props are passed like HTML attributes</strong>: &lt;Component key=&quot;value&quot; /&gt;</li><li><strong>They allow reuse</strong>: One component, multiple data inputs.</li></ul><p><strong>Console Output</strong></p><p>When you open the browser and check the console (because of console.log(props)), you&#39;ll see:</p><pre>{ title: &quot;Iphone 14&quot;, price: 120000 }<br>{ title: &quot;Samsung S23&quot;, price: 110000 }<br>{ title: &quot;Google Pixel 9&quot;, price: 100000 }</pre><p>Real Output in Browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*8AOgz6EOae23TnERzxJOHg.png" /></figure><h3>Conditional Rendering in React — Show or Hide Content Dynamically</h3><p>Sometimes in UI development, you need to <strong>show different content</strong> based on some condition — like whether a product is in stock or not.</p><p>React makes this easy using <strong>conditional rendering</strong>, just like using if or the ternary ? : operator in JavaScript.</p><p>Let’s look at a practical example with products and stock status.</p><p>App.js – Passing Props</p><pre>import Product from &#39;./component/Product&#39;;</pre><pre>function App() {<br>  return (<br>    &lt;&gt;&lt;Product title=&quot;Iphone 14&quot; price={120000} stock={10} /&gt;<br>      &lt;Product title=&quot;Samsung S23&quot; price={110000} stock={20} /&gt;<br>      &lt;Product title=&quot;Google Pixel 9&quot; price={100000} stock={0} /&gt;<br>    &lt;/&gt;<br>  );<br>}</pre><pre>export default App;</pre><p>Product.js – Conditional Rendering</p><pre>function Product(props) {<br>  console.log(props); // View props in console<br>  return (<br>    &lt;div&gt;<br>      &lt;h1&gt;This is {props.title}&lt;/h1&gt;<br>      &lt;p&gt;Price is {props.price}&lt;/p&gt;<br>      {<br>        props.stock &gt; 0<br>          ? &lt;p&gt;Stock is {props.stock}&lt;/p&gt;<br>          : &lt;p style={{ color: &quot;red&quot; }}&gt;Out of stock&lt;/p&gt;<br>      }<br>    &lt;/div&gt;<br>  );<br>}<br>export default Product;</pre><p><strong>What’s Happening?</strong></p><p>In the Product component:</p><ul><li>We use the ternary operator (condition ? trueCase : falseCase)</li><li>If props.stock &gt; 0, we display the stock number.</li><li>If not, we display “Out of stock” with red styling.</li></ul><p>Output in the Browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*fkdPrl1ifujHRTF_4hb7bw.png" /></figure><h4>Other Ways to Do Conditional Rendering</h4><p><strong>Using </strong><strong>&amp;&amp; operator:</strong></p><pre>{ props.stock &gt; 0 &amp;&amp; &lt;p&gt;Stock is {props.stock}&lt;/p&gt; }</pre><p><strong>Using variables inside JSX:</strong></p><pre>let stockMessage = props.stock &gt; 0<br>  ? `Stock is ${props.stock}`<br>  : &quot;Out of stock&quot;;</pre><p>Then in JSX:</p><pre>&lt;p&gt;{stockMessage}&lt;/p&gt;</pre><h3>Handling Events</h3><p>In React, handling user interactions like button clicks is simple and powerful. Just like HTML and JavaScript, React gives you special attributes like onClick to handle events.</p><p><strong>The Example — Product Component</strong></p><p>Let’s take a look at this simple React component:</p><pre>function Product(props) {<br>    function addToCart() {<br>        alert(&quot;Added to cart&quot;);<br>    }</pre><pre>return (<br>        &lt;div&gt;<br>            &lt;h1&gt;This is {props.title}&lt;/h1&gt;<br>            &lt;p&gt;Price is {props.price}&lt;/p&gt;<br>            {<br>                props.stock &gt; 0<br>                    ? &lt;p&gt;Stock is {props.stock}&lt;/p&gt;<br>                    : &lt;p&gt;Out of stock&lt;/p&gt;<br>            }<br>            &lt;button onClick={addToCart}&gt;Add To Cart&lt;/button&gt;<br>        &lt;/div&gt;<br>    );<br>}<br>export default Product;</pre><p><strong>What’s Happening?</strong></p><pre>function addToCart() {<br>    alert(&quot;Added to cart&quot;);<br>}</pre><p>This is a <strong>simple JavaScript function</strong> that shows an alert when called.</p><pre>&lt;button onClick={addToCart}&gt;Add To Cart&lt;/button&gt;</pre><p>In React, we attach event handlers like this:</p><ul><li>Use onClick (camelCase, not onclick)</li><li>Pass the <strong>function reference</strong> (no ()) → onClick={addToCart}</li><li>When the button is clicked, the function runs</li></ul><p><strong>More Events</strong></p><p>Event Type Description onClick Buttons, toggles, interactivity onChange Input fields, checkboxes onSubmit Forms onMouseOver Hover effects onKeyDown Keyboard shortcuts or input logic</p><h3>Hooks</h3><h3>Use State Hook</h3><p>React is known for its dynamic UI and interactivity. One key feature that enables this is <strong>state</strong>. In this post, we’ll explore how to use the useState hook to create a simple counter app in React.</p><p><strong>What Is State in React?</strong></p><p><strong>State</strong> is a built-in React concept that lets components keep track of changing data over time. When the state changes, the component automatically re-renders to reflect the new values.</p><p><strong>The Example — A Simple Counter</strong></p><p>Here’s a small example that uses state to build a counter:</p><pre>import { useState } from &quot;react&quot;;</pre><pre>function App() {<br>  const [state, setState] = useState(0);</pre><pre>return (<br>    &lt;div&gt;<br>      &lt;h1&gt;Counter&lt;/h1&gt;<br>      &lt;button onClick={() =&gt; setState(state + 1)}&gt;counter&lt;/button&gt;<br>      &lt;p&gt;{state}&lt;/p&gt;<br>    &lt;/div&gt;<br>  );<br>}</pre><pre>export default App;</pre><p><strong>How It Works</strong></p><p>Step 1: Import useState</p><pre>import { useState } from &quot;react&quot;;</pre><p>useState is a <strong>React Hook</strong> that lets you manage state in a functional component.</p><p>Step 2: Declare a State Variable</p><pre>const [state, setState] = useState(0);</pre><ul><li>state: the current value (initially 0)</li><li>setState: a function used to update the state</li><li>useState(0): sets the initial value to 0</li></ul><p>Step 3: Handle the Click Event</p><pre>&lt;button onClick={() =&gt; setState(state + 1)}&gt;counter&lt;/button&gt;</pre><ul><li>Every time the button is clicked, setState(state + 1) increases the counter by 1.</li><li>This triggers a <strong>re-render</strong>, updating the UI with the new count.</li></ul><p><strong>The Example — Live Text</strong></p><pre>import { useState } from &quot;react&quot;;</pre><pre>function App() {<br>  const [text, setText] = useState(&quot;&quot;);</pre><pre>return (<br>    &lt;div&gt;<br>      &lt;inputtype=&quot;text&quot;<br>        onChange={(e) =&gt; setText(e.target.value)}<br>      /&gt;<br>      &lt;p&gt;{text}&lt;/p&gt;<br>    &lt;/div&gt;<br>  );<br>}</pre><pre>export default App;</pre><p><strong>How It Works</strong></p><p>Step 1: useState for Input Value</p><pre>const [text, setText] = useState(&quot;&quot;);</pre><ul><li>text: holds the current value of the input</li><li>setText: updates the value</li><li>&quot;&quot;: initial value is an empty string</li></ul><p>Step 2: Handle Input Change</p><pre>onChange={(e) =&gt; setText(e.target.value)}</pre><ul><li>React listens for changes in the &lt;input&gt; field</li><li>e.target.value: gets the new value typed by the user</li><li>setText(...): updates the state with that value</li></ul><p>Step 3: Display the Updated Value</p><pre>&lt;p&gt;{text}&lt;/p&gt;</pre><p>This makes the component <strong>reactive</strong>: whatever you type shows up immediately.</p><h3>useEffect Hook</h3><p>Fetching external data in React is a very common use case. Whether you’re getting user data, product details, or blog posts, React makes it easy with two powerful tools: useEffect and useState.</p><p>Let’s walk through an example where we fetch a post from a public API and display it on the page.</p><p><strong>Example</strong></p><pre>import { useEffect, useState } from &quot;react&quot;;</pre><pre>function App() {<br>  const [post, setPost] = useState(null);</pre><pre>useEffect(() =&gt; {<br>    fetch(&quot;&lt;https://jsonplaceholder.typicode.com/todos/1&gt;&quot;)<br>      .then((response) =&gt; response.json())<br>      .then((json) =&gt; setPost(json))<br>      .catch((error) =&gt; console.error(&quot;Error fetching data:&quot;, error));<br>  }, []);</pre><pre>return (<br>    &lt;div&gt;<br>      &lt;h1&gt;Post&lt;/h1&gt;<br>      {post ? (<br>        &lt;ul&gt;<br>          &lt;li&gt;ID: {post.id}&lt;/li&gt;<br>          &lt;li&gt;Title: {post.title}&lt;/li&gt;<br>          &lt;li&gt;Completed: {post.completed ? &quot;Yes&quot; : &quot;No&quot;}&lt;/li&gt;<br>        &lt;/ul&gt;<br>      ) : (<br>        &lt;p&gt;Loading...&lt;/p&gt;<br>      )}<br>    &lt;/div&gt;<br>  );<br>}</pre><pre>export default App;</pre><p>Step 1: Setup State</p><pre>const [post, setPost] = useState(null);</pre><ul><li>We start with null because no data has been fetched yet.</li><li>setPost is used to update the state once data is received.</li></ul><p>Step 2: Fetch Data with useEffect</p><pre>useEffect(() =&gt; {<br>  fetch(&quot;&lt;https://jsonplaceholder.typicode.com/todos/1&gt;&quot;)<br>    .then((response) =&gt; response.json())<br>    .then((json) =&gt; setPost(json))<br>    .catch((error) =&gt; console.error(&quot;Error fetching data:&quot;, error));<br>}, []);</pre><p>useEffect runs <strong>once</strong> after the component mounts (because of the empty dependency array []).</p><p>fetch(...) sends a request to the placeholder API.</p><p>.then(...) handles the response and stores it in state.</p><p>.catch(...) handles any errors.</p><h3>useContext Hook</h3><p>In React development, managing how components communicate and share data is essential. For small apps, we often use <strong>props</strong> to pass data from parent to child components. But as your app grows, <strong>prop-drilling</strong> becomes a headache — passing data through several layers of components unnecessarily.</p><p>That’s where <strong>React Context API</strong> shines. It allows us to share data across components without explicitly passing props at every level.</p><p>Let’s dive into a practical example. You’ll learn how to:</p><ul><li>Create a context</li><li>Provide data in a parent component</li><li>Access that data in a child component using useContext</li></ul><p>Step 1: Create &amp; Provide the Context</p><p>In App.js, we create a context and use the Provider to supply data (user name) to child components.</p><pre>// App.js<br>import { createContext, useState } from &quot;react&quot;;<br>import Child from &quot;./component/Child&quot;;</pre><pre>// Create the context<br>export const UserApp = createContext();</pre><pre>function App() {<br>  const [name, setName] = useState(&quot;&quot;);</pre><pre>return (<br>    &lt;UserApp.Provider value={name}&gt;<br>      &lt;div&gt;<br>        &lt;h1&gt;👨‍👧 Parent Component&lt;/h1&gt;<br>        &lt;inputtype=&quot;text&quot;<br>          onChange={(e) =&gt; setName(e.target.value)}<br>          placeholder=&quot;Enter your name&quot;<br>        /&gt;<br>        &lt;p&gt;Your Name: {name}&lt;/p&gt;<br>        &lt;Child /&gt;<br>      &lt;/div&gt;<br>    &lt;/UserApp.Provider&gt;<br>  );<br>}</pre><pre>export default App;</pre><p>What’s Happening Here?</p><ul><li>createContext() sets up a context named UserApp</li><li>UserApp.Provider makes the name value available to all nested components</li><li>The input field lets the user type their name and updates state via setName</li></ul><p>Step 2: Access Context in the Child Component</p><p>Now we use the context value in a separate component without props.</p><pre>// Child.js<br>import React, { useContext } from &#39;react&#39;;<br>import { UserApp } from &#39;../App&#39;;</pre><pre>const Child = () =&gt; {<br>  const userName = useContext(UserApp);</pre><pre>return (<br>    &lt;div&gt;<br>      &lt;h2&gt;👶 Child Component&lt;/h2&gt;<br>      &lt;p&gt;Received Name: {userName}&lt;/p&gt;<br>    &lt;/div&gt;<br>  );<br>};</pre><pre>export default Child;</pre><ul><li>useContext(UserApp) gives us access to the shared name value</li><li>The child automatically updates when the parent input changes — no props required!</li></ul><p><strong>Why Use Context?</strong></p><p>Problem Solution with Context Prop-drilling through many layers Direct access to data Redundant re-passing of props Cleaner, focused components Central state needed by many parts Shared globally via context</p><p><strong>Ideal Use Cases:</strong></p><ul><li>User authentication (logged-in user info)</li><li>App themes (light/dark mode)</li><li>Language settings (i18n)</li><li>Shopping cart or global filters</li></ul><p><strong>When NOT to Use Context</strong></p><ul><li>For passing simple data down just one or two levels</li><li>When the context value changes <strong>very frequently</strong>, causing unnecessary re-renders</li><li>In such cases, state management libraries like Redux or Zustand might be better</li></ul><h4>Conclusion</h4><p>React is more than just a JavaScript library — it’s a powerful way to build modern, interactive UIs with ease. Throughout this guide, we’ve explored its fundamental concepts: building UI using components, writing markup with JSX, reusing code with props, controlling logic through conditional rendering, and handling interactivity with events. We’ve also discovered how to manage dynamic data using useState, fetch external data using useEffect, and avoid prop-drilling with useContext. Whether you&#39;re building a simple counter or a scalable frontend, React offers the flexibility, performance, and developer-friendly tools to bring your ideas to life. With tools like <strong>Vite</strong> and advanced ecosystems like <strong>Next.js</strong>, you&#39;re now well-equipped to begin your React journey and confidently build real-world applications. See you in Redux Blog. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d9ff77b5afc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VERSION CONTROLLING]]></title>
            <link>https://jinadgamage.medium.com/version-controlling-02fa840a0fba?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/02fa840a0fba</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[code-management]]></category>
            <category><![CDATA[version-control]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Sat, 01 Mar 2025 15:38:09 GMT</pubDate>
            <atom:updated>2025-03-01T15:38:09.553Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMt7mhh1VThb9ySOr9NqPQ.jpeg" /></figure><p><strong>Agenda</strong></p><ol><li><strong>Introduction to Version Controlling</strong></li><li><strong>Why Version Controlling is Needed</strong></li><li><strong>Benefits of Version Controlling</strong></li><li><strong>Key Version Control Terminology</strong></li><li><strong>Working with Local Repository</strong></li><li><strong>Working with Remote Repository</strong></li><li><strong>Best Practices in Git Usage</strong></li><li><strong>Commit Message Best Practices</strong></li></ol><h3>What is version controlling ?</h3><p>it is practice of tracking and managing software codes changes. changers identify using a revision number. each revision has its timestamp and person who are done the changes</p><h3>Why is version controlling needed ?</h3><ul><li>Centralized source code repository.</li><li>Easier backups</li><li>Easy collaborative development.</li><li>Overview of changes performed to a file.</li><li>Access control.</li><li>Helps in conflict resolution.</li></ul><h3>Benefits of version controlling</h3><ul><li><strong>Security:</strong> Protects against accidental loss or corruption.</li><li><strong>Clean History:</strong> Keeps track of changes and authors.</li><li><strong>Collaboration:</strong> Enables teams to work on the same project simultaneously.</li><li><strong>Branching &amp; Merging: </strong>Allows feature development without affecting the main code.</li><li><strong>Scalability:</strong> Works efficiently with small and large projects.</li></ul><h3>Version control terminology</h3><ul><li><strong>Repository</strong>– The central location where project files are stored.</li><li><strong>Trunk (Master/Main Branch)</strong>– The most stable version of the project.</li><li><strong>Stage</strong>– The process of marking files for tracking changes.</li><li><strong>Commit</strong>– A snapshot of changes in the repository.</li><li><strong>Merge</strong>– Combining branches together.</li><li><strong>Branch</strong>– A copy of the code used for feature development or bug fixes.</li><li><strong>Merge Conflict</strong>– Occurs when multiple changes conflict and need manual resolution</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*08MaGD1zbkmfpP4I3nF2PQ.png" /></figure><p>his diagram illustrates the Git workflow, showing how changes move between the <strong>Working Directory</strong>, <strong>Staging Area</strong>, <strong>Local Repository</strong>, and <strong>Remote Repository (GitHub)</strong>. It depicts key Git commands: git add moves files to the staging area, git commit saves changes locally, git push uploads commits to GitHub, and git pull retrieves updates from the remote repository. Additional commands like git checkout and git merge help manage branches and integrate changes, ensuring smooth version control and collaboration.</p><h3>Working with Local Repository</h3><h4>Initializing</h4><ul><li><strong>git init</strong> → create a Git repository in a local directory.</li></ul><h4>Cloning</h4><p>git clone is a command used to copy an <strong>existing Git repository</strong> from a remote server (such as GitHub, GitLab, or Bitbucket) to your local machine. This allows you to work on the project locally, make changes, and push updates back to the remote repository.</p><p>If the repository is <strong>public</strong>, you can simply run:</p><pre>git clone https://github.com/your-username/repository-name.git</pre><p>If the repository is <strong>private</strong>,</p><p>you must authenticate using a <strong>Personal Access Token (PAT)</strong> because GitHub no longer supports password authentication over HTTPS.</p><pre>git clone https://your-username:your-personal-access-token@github.com/your-username/repository-name.git</pre><h4>Staging</h4><p>Prepares files for commit but does not save them permanently.</p><p><strong>Why is Staging Important?</strong></p><ul><li>Allows you to review your changes before committing.</li><li>Helps in grouping related changes into a single commit.</li><li>Ensures that only selected files are committed.</li></ul><p><strong>How Staging Works in Git?</strong></p><ol><li>Modify files in your working directory.</li><li>Stage the files using <strong>git add</strong> (this moves the changes to the staging area).</li></ol><ul><li><strong>git add &lt;file-name&gt; </strong>→ add a single file to the staging area.</li><li><strong>git add .</strong> → add multiple files and folders to the staging area.</li><li><strong>git status </strong>→ shows the list of changed files or files that have yet to be staged.</li><li><strong>git reset &lt;file-name&gt;</strong> → Unstage a file</li></ul><h4>Committing</h4><p>Saves the staged changes into the local repository with a commit message.</p><p><strong>How Committing Works in Git?</strong></p><ol><li>Modify files in your working directory.</li><li>Stage the files using <strong>git add</strong> (this moves the changes to the staging area).</li><li>Commit the changes to save them in the repository using <strong>git commit</strong>.</li></ol><ul><li><strong>git commit -m &lt;message&gt; </strong>→ add a single file to the staging area.</li><li><strong>git reset — soft HEAD~1</strong> → Undo the last commit while keeping changes.</li><li><strong>git reset — hard HEAD~1</strong> → Undo the last commit and discard changes.</li><li><strong>git checkout — — &lt;file&gt;</strong> →Discard changes in a specific file.</li></ul><h4>Branching</h4><p>Allows developers to work on different features separately. it Helps maintain a stable main branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*9fImhwm6Ms0Jqh1hWBpJXg.jpeg" /></figure><ul><li><strong>git branch &lt;branch-name&gt; </strong>→Create a new branch.</li><li><strong>git checkout &lt;branch-name&gt; </strong>→ Switch to a branch.</li><li><strong>git branch -M &lt;new-branch-name&gt; </strong>→ change branch name</li><li><strong>git branch –d</strong> <strong>&lt;branch-name&gt; </strong>→ Delete a branch (after merging)</li></ul><h4>Merging</h4><p>Merging is the process of integrating changes from one branch into another. Git automatically merges changes when there are no conflicts, but if the same parts of a file were modified in different branches, conflicts must be resolved manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*DvJJRmxumbjTiHTnsmeIsA.jpeg" /></figure><ul><li><strong>git merge</strong> <strong>&lt;branch-name&gt;</strong> → after resolving merge conflicts merge selected branch into the current branch.</li></ul><h3>Working with Remote Repository</h3><h4>Add GitHub Repository as a Remote</h4><p><strong>first time setting a remote repo,</strong></p><pre>git remote add origin <a href="https://github.com/your-username/MyProject.git">https://github.com/your-username/MyProject.git</a></pre><pre>git remote add origin @github.com/your-username/MyProject.git&quot;&gt;https://:@github.com/your-username/MyProject.git</pre><p><strong>Changing an existing remote URL,</strong></p><pre>git remote set-url origin https://github.com/your-username/MyProject.git</pre><pre>git remote set-url origin @github.com/your-username/MyProject.git&quot;&gt;https://:@github.com/your-username/MyProject.git</pre><p>connects your local repository to a remote repository hosted on platforms like GitHub, GitLab, or Bitbucket.</p><p>origin is a conventional name for the remote repository, but you can use any name.</p><p><strong>git remote add origin https://github.com/your-username/MyProject.git</strong></p><p><strong>Use This When:</strong></p><ul><li>Your repository is <strong>public</strong> and does <strong>not</strong> require authentication for cloning or pulling.</li><li>You are pushing changes from a local repository <strong>but will authenticate separately</strong> (e.g., using Git credential manager or SSH keys).</li><li>You have already authenticated with GitHub via <strong>SSH or a stored HTTPS token</strong>.</li></ul><p><strong>git remote add origin https://&lt;your-github-username&gt;:&lt;your-personal-access-token&gt;@github.com/your-username/MyProject.git</strong></p><p>Use This When:</p><ul><li>Your repository is <strong>private</strong>, and GitHub requires authentication to push or pull changes.</li><li>You want to <strong>bypass interactive login prompts</strong> by embedding your <strong>Personal Access Token (PAT)</strong> in the URL.</li><li>GitHub has <strong>disabled password authentication for HTTPS</strong>, and you must use a PAT instead.</li><li>You are using <strong>automation (CI/CD pipelines, scripts, or remote servers)</strong> to interact with GitHub repositories.</li></ul><h4>Push a Local Git Repository to GitHub</h4><ul><li><strong>git push origin &lt;branch-name&gt;</strong> → Push changes to the remote repository</li></ul><h4>Pull Request (PR)</h4><p>The git pull command is used to <strong>fetch and merge</strong> changes from a remote repository into your local branch. It ensures your local copy is up to date with the latest changes made by other team members or from another device.</p><ul><li><strong>git pull origin</strong> <strong>&lt;branch-name&gt; </strong>→ Pull changes from the remote repository</li></ul><p>How Does git pull Work?</p><p>The git pull command is a combination of two commands:</p><ol><li><strong>git fetch</strong> – Downloads new changes from the remote repository.</li><li><strong>git merge</strong> – Merges those changes into your current branch.</li></ol><h4>BEST PRACTICES Of GIT</h4><ul><li>Use meaningful commit messages.</li><li>Do not commit sensitive data.</li><li>Use <strong>.gitignore</strong> to ignore unnecessary files.</li><li>Always make sure to have the latest version of the file.</li><li>Get the Latest Source Code at Least Once a Day (For Distributed Version Control Systems).</li><li>Merge code with the development branch at least once per day.</li><li>Always make sure code is working as expected and it is not causing any other code to break.</li><li>Follow a formal review process when merging</li></ul><p><strong>The below shows the most common commit types included in the specification</strong></p><p>feat — Introduce a new feature to the codebase</p><p>fix — Fix a bug in the codebase</p><p>docs — Create/update documentation</p><p>style — Feature and updates related to styling</p><p>refactor — Refactor a specific section of the codebase</p><p>test — Add or update code related to testing</p><p>chore — Regular code maintenance</p><p><strong>commit message with the following structure:</strong></p><pre><strong>&lt;type&gt;(optional scope): &lt;subject-description&gt;<br><br>&quot;feat(api): Add support to create coupons&quot;</strong></pre><h3>Conclusion</h3><p>Version controlling is an essential practice in modern software development, enabling teams to efficiently track and manage code changes. It provides a centralized repository, ensures secure backups, and facilitates seamless collaboration. By understanding key concepts such as repositories, commits, branches, and merges, developers can maintain a structured workflow. Tools like Git help streamline development through features like branching, staging, and remote repository management. Following best practices — including meaningful commit messages, proper branching strategies, and regular updates — ensures a smooth and organized development process. Ultimately, version control enhances productivity, security, and project scalability, making it indispensable for software development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02fa840a0fba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Association Mapping in Hibernate Using Spring Boot]]></title>
            <link>https://jinadgamage.medium.com/association-mapping-in-hibernate-using-spring-boot-e0f27ab3278e?source=rss-5ca59f35dc04------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0f27ab3278e</guid>
            <category><![CDATA[spring-boot]]></category>
            <category><![CDATA[spring-framework]]></category>
            <category><![CDATA[association]]></category>
            <category><![CDATA[spring]]></category>
            <dc:creator><![CDATA[Jinad Gamage]]></dc:creator>
            <pubDate>Fri, 10 Jan 2025 12:55:03 GMT</pubDate>
            <atom:updated>2025-01-10T12:55:03.542Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, you’ll explore how to manage relationships between entities in <strong>Spring Boot</strong> using <strong>Hibernate</strong>. Association mapping is a critical aspect of<strong> ORM (Object-Relational Mapping)</strong> and allows you to model real-world relationships such as <strong>one-to-one</strong>, <strong>one-to-many</strong>, <strong>many-to-one</strong>, and <strong>many-to-many</strong>. Using annotations, we’ll demonstrate how to define and implement these associations effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Huce0g6psOI3eS6-TmN5VA.jpeg" /></figure><p>In association mapping with Hibernate and JPA, it’s crucial to determine whether the relationship is <strong>collection-dependent</strong> or <strong>non-collection-dependent</strong> before designing your entities. This distinction affects how the relationship is implemented and managed.</p><p><strong>1. Collection-Dependent Relationships</strong></p><ul><li>A relationship is <strong>collection-dependent</strong> when one entity is associated with a collection (e.g., List, Set) of another entity.</li></ul><p>Common examples:</p><ul><li><strong>One-to-Many</strong>: A Customer has multiple Orders.</li><li><strong>Many-to-Many</strong>: A Student is enrolled in multiple Courses, and a Course has multiple Students.</li></ul><p><strong>2. Non-Collection-Dependent Relationships</strong></p><ul><li>A relationship is <strong>non-collection-dependent</strong> when an entity is associated with a single instance of another entity (not a collection).</li></ul><p>Common examples:</p><ul><li><strong>One-to-One</strong>: A User has one Profile.</li><li><strong>Many-to-One</strong>: Multiple Orders belong to one Customer.</li></ul><h4><strong>Steps to Implement Association Mapping</strong></h4><ol><li><strong>Define Entity Classes</strong><br>Use JPA annotations such as @OneToOne, @ManyToOne, @OneToMany, and @ManyToMany to establish relationships.</li><li><strong>Set Up Repositories</strong><br>Extend JpaRepository to handle CRUD operations for each entity.</li><li><strong>Write Services</strong><br>Implement services to handle business logic, including saving and retrieving associated data.</li><li><strong>Test and Verify</strong><br>Execute the services and check the database for correctly mapped relationships.</li></ol><h3>One to One Mapping</h3><p>A simple one-to-one relationship connects two tables where each entity references only one other. For instance, a User entity with a Profile.<br>Using annotations like @OneToOne and @JoinColumn, we set up the mapping and save data using a repository service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*_cvFpbZ7r5uqlKxSuR2uvA.jpeg" /></figure><h4>Step 1: Create the Entity Classes</h4><p>User Entity</p><pre>@Entity<br>@Data<br>@AllArgsConstructor<br>@NoArgsConstructor<br>@Getter<br>@Setter<br>@Table(name = &quot;user_tbl&quot;)<br>public class User {<br>    @Id<br>    private Integer id;<br><br>    private String name;<br><br>    @OneToOne(cascade = CascadeType.ALL)<br>    @JoinColumn(name = &quot;profile_id&quot;, referencedColumnName = &quot;id&quot;)<br>    private Profile profile;  // Reference to the Profile entity<br><br>}</pre><ul><li><strong>@OneToOne</strong>: Establishes a one-to-one relationship between User and Profile.</li><li><strong>@JoinColumn(name = &quot;profile_id&quot;)</strong>: Specifies that profile_id in the User table acts as a foreign key referencing the id in the Profile table.</li></ul><p>Profile Entity</p><pre>@Entity<br>@Data<br>@AllArgsConstructor<br>@NoArgsConstructor<br>@Getter<br>@Setter<br>@Table(name = &quot;profile_tbl&quot;)<br>public class Profile {<br>    @Id<br>    private Integer id;<br><br>    private String address;<br><br>    private String phoneNumber;<br>}</pre><h4>Step 2: Save Data in the Repository</h4><p>Repositories</p><p>Create repositories for both entities.</p><pre>public interface UserRepository extends JpaRepository&lt;User,Integer&gt; {<br>}</pre><pre>public interface ProfileRepository extends JpaRepository&lt;Profile,Integer&gt; {<br>}</pre><h4>Step 3: Service to Save Data</h4><pre>@Service<br>public class UserService {<br><br>    @Autowired<br>    private UserRepository userRepository;<br><br>    public void saveUserWithProfile() {<br>        // Create a Profile<br>        Profile profile = new Profile();<br>        profile.setId(1);<br>        profile.setAddress(&quot;123 Main Street&quot;);<br>        profile.setPhoneNumber(&quot;123-456-7890&quot;);<br><br>        // Create a User<br>        User user = new User();<br>        user.setId(101);<br>        user.setName(&quot;John Doe&quot;);<br><br>        user.setProfile(profile);<br><br>        // Save the user (the profile will also be saved due to CascadeType.ALL)<br>        userRepository.save(user);<br>    }<br>}</pre><h4>Step 4: Testing</h4><p>Run the <strong>saveUserWithProfile()</strong> method and check the database:</p><p><strong>User Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*wluC_c3M5g2EAfDaZGQUKg.png" /></figure><p><strong>Profile Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*w41g5R0FxLEvehvjlSKwrg.png" /></figure><h3>Many to One Mapping</h3><p>This is used when multiple entities reference a single entity, like Order entities belonging to one Customer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*XCJockee0A-UPOLbvI4-9A.png" /></figure><p>In a <strong>Many-to-One Unidirectional</strong> relationship, the <strong>primary key</strong> of the “one” side (e.g., Customer) is set as a <strong>foreign key</strong> in the &quot;many&quot; side (e.g., Order) table. This allows multiple Order entities to reference the same Customer entity. Since the relationship is unidirectional, the Customer entity does not maintain any reference to the Order entities.</p><h4>Step 1: Create the Entity Classes</h4><p>Customer Entity</p><pre>@Entity<br>public class Customer {<br>    @Id<br>    @GeneratedValue(strategy = GenerationType.IDENTITY)<br>    private Integer id;<br>    private String name;<br>}</pre><p>Order Entity</p><pre>public class Order {<br><br>    @Id<br>    @GeneratedValue(strategy = GenerationType.IDENTITY)<br>    private Integer id;<br><br>    private String product;<br><br>    private double price;<br><br>    @ManyToOne<br>    @JoinColumn(name = &quot;customer_id&quot;, nullable = false) // Defines the foreign key column<br>    private Customer customer;<br><br>}</pre><h4>Step 2: Save Data in the Repository</h4><p>Repositories</p><p>Create repositories for both entities.</p><pre>import org.springframework.data.jpa.repository.JpaRepository;<br><br>public interface CustomerRepository extends JpaRepository&lt;Customer, Integer&gt; {<br>}<br>public interface OrderRepository extends JpaRepository&lt;Order, Integer&gt; {<br>}</pre><h4>Step 3: Service to Save Data</h4><pre>import org.springframework.beans.factory.annotation.Autowired;<br>import org.springframework.stereotype.Service;<br>@Service<br>public class OrderService {<br><br>    @Autowired<br>    private CustomerRepository customerRepository;<br>    @Autowired<br>    private OrderRepository orderRepository;<br>-<br>    public void createOrder() {<br><br>        // Create a Customer<br>        Customer customer1 = new Customer();<br>        customer.setName(&quot;Jane Doe&quot;);<br>        customerRepository.save(customer);<br><br>        // Create Orders<br>        Order order1 = new Order();<br>        order1.setProduct(&quot;Laptop&quot;);<br>        order1.setPrice(1500.0);<br>        order1.setCustomer(customer1); // Set the customer<br><br>        Order order2 = new Order();<br>        order2.setProduct(&quot;Tablet&quot;);<br>        order2.setPrice(800.0);<br>        order2.setCustomer(customer1); // Set the customer<br><br>        // Save Orders<br>        orderRepository.save(order1);<br>        orderRepository.save(order2);<br>    }<br>}</pre><h4>Step 4: Testing</h4><p>Run the createOrder() method and check the database:</p><p><strong>Customer Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*V8HBO2S8VvaYt_ZOrbX-5w.png" /></figure><p><strong>Order Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*kwZsSucN4J9xhYAdYhYUeA.png" /></figure><h3>One to Many Mapping</h3><p>A unidirectional one-to-many relationship is established when the “one” side knows about the “many” side.<br>Example: A Customer entity referencing a collection of Order entities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*DjXZhhsFKPA6jhfPotZkKQ.jpeg" /></figure><p>only the “one” side knows about the relationship. For example, a Customer can have multiple Order entities, but the Order entity does not reference the Customer.</p><p>This is less common than the bidirectional relationship but can simplify certain use cases where the “many” side doesn’t need to reference the “one” side.</p><h4>Step 1: Create the Entity Classes</h4><p>Customer Entity</p><p>The Customer entity will hold the List&lt;Order&gt; to represent the one-to-many relationship. The Order table will include a foreign key reference to the Customer table, but the Customer table will not have any reference to Order.</p><pre>@Entity<br>@Data<br>@AllArgsConstructor<br>@NoArgsConstructor<br>@Getter<br>@Setter<br>@Table(name = &quot;customer_tbl&quot;)<br>public class Customer {<br><br>    @Id<br>    private Integer id;<br><br>    private String name;<br><br>    @OneToMany(cascade = CascadeType.ALL, orphanRemoval = true)<br>    @JoinColumn(name = &quot;customer_id&quot;)<br>    private List&lt;Order&gt; orders;<br><br>}</pre><ul><li><strong>@OneToMany</strong>: Defines a one-to-many relationship between Customer and Order.</li><li><strong>@JoinColumn(name = &quot;customer_id&quot;)</strong>: The foreign key is stored in the Order table, and it references the Customer entity.</li><li><strong>cascade = CascadeType.ALL</strong>: Ensures that any operation on the Customer entity (like persist, remove) is cascaded to the Order entities.</li><li><strong>orphanRemoval = true</strong>: If an Order is removed from the orders list, it will be deleted from the database.</li></ul><p>Order Entity</p><p>The Order entity will <strong>not</strong> reference the Customer entity directly, and it will just include the foreign key in the table.</p><pre>@Entity<br>@Data<br>@AllArgsConstructor<br>@NoArgsConstructor<br>@Getter<br>@Setter<br>@Table(name = &quot;order_tbl&quot;)<br>public class Order {<br>    @Id<br>    private Integer id;<br><br>    private String product;<br><br>    private double price;<br><br>}</pre><h4>Step 2: Save Data in the Repository</h4><p>Repositories</p><p>Create repositories for both entities.</p><pre>import org.springframework.data.jpa.repository.JpaRepository;<br><br>public interface CustomerRepository extends JpaRepository&lt;Customer, Integer&gt; {<br>}<br>public interface OrderRepository extends JpaRepository&lt;Order, Integer&gt; {<br>}</pre><h4>Step 3: Service to Save Data</h4><pre>@Service<br>public class OrderService {<br>    @Autowired<br>    private CustomerRepository customerRepository;<br>    @Autowired<br>    private OrderRepository orderRepository;<br><br>    public void saveOrderForCustomer() {<br>        // Create a Customer<br>        Customer customer = new Customer();<br>        customer.setId(1);<br>        customer.setName(&quot;John Doe&quot;);<br><br>        // Create Orders and associate with the customer<br>        Order order1 = new Order();<br>        order1.setId(101);<br>        order1.setProduct(&quot;Laptop&quot;);<br>        order1.setPrice(1200.0);<br><br>        Order order2 = new Order();<br>        order2.setId(102);<br>        order2.setProduct(&quot;Tablet&quot;);<br>        order2.setPrice(800.0);<br><br>        // Add orders to customer<br>        customer.setOrders(List.of(order1, order2));<br><br>        // Save customer and orders (cascade will save orders)<br>        customerRepository.save(customer);<br>    }</pre><p>In the service, you will save the Customer and Order entities. The Customer entity will have a list of Order entities, and when the Customer is saved, the associated Order entities will be saved automatically due to the cascade.</p><h4>Step 4: Testing</h4><p>Run the e saveOrderForCustomer() method and check the database:</p><p><strong>Customer Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*V8HBO2S8VvaYt_ZOrbX-5w.png" /></figure><p><strong>Order Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*kwZsSucN4J9xhYAdYhYUeA.png" /></figure><h3>Many to Many Mapping</h3><p>When both sides of the relationship reference multiple entities, a join table is used to manage the relationship.<br>Example: Students and Courses, where each can reference the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*F0ZoCMxazkysQjD1nqA-5w.jpeg" /></figure><p>In this relationship:</p><ul><li>A join table is created to manage the relationship between the two entities.</li><li>Each entity uses the @ManyToMany annotation.</li></ul><h4>Step 1: Create the Entity Classes</h4><p>Student Entity</p><pre>@Entity<br>@Data<br>@AllArgsConstructor<br>@NoArgsConstructor<br>@Getter<br>@Setter<br>@Table(name = &quot;student_tbl&quot;)<br>public class Student {<br>    @Id<br>    private Integer id;<br><br>    private String name;<br><br>    @ManyToMany<br>    @JoinTable(<br>            name = &quot;student_course&quot;, // name of join table<br>            joinColumns = @JoinColumn(name = &quot;student_id_FK&quot;),<br>            inverseJoinColumns = @JoinColumn(name = &quot;course_id_FK&quot;)<br>    )<br>    private Set&lt;Course&gt; courses; // Many-to-Many relationship with Course<br>}</pre><ul><li><strong>@ManyToMany</strong>: Defines the Many-to-Many relationship.</li><li><strong>@JoinTable</strong>: Specifies the join table (student_course) and the foreign key columns (student_id and course_id).</li></ul><p>Course Entity</p><pre>@Entity<br>@Data<br>@AllArgsConstructor<br>@NoArgsConstructor<br>@Getter<br>@Setter<br>@Table(name = &quot;course_tbl&quot;)<br>public class Course {<br>    @Id<br>    private Integer id;<br><br>    private String title;<br><br>    @ManyToMany(mappedBy = &quot;courses&quot;)<br>    private Set&lt;Student&gt; students;<br>}</pre><p><strong>@ManyToMany(mappedBy = &quot;courses&quot;)</strong>: Indicates that the Student entity owns the relationship.</p><h4>Step 2: Save Data in the Repository</h4><p>Repositories</p><p>Create repositories for both entities.</p><pre>import org.springframework.data.jpa.repository.JpaRepository;<br><br>public interface StudentRepository extends JpaRepository&lt;Customer, Integer&gt; {<br>}<br>public interface CourseRepository extends JpaRepository&lt;Order, Integer&gt; {<br>}</pre><h4>Step 3: Service to Save Data</h4><pre>@Service<br>public class StudentCourseService {<br><br>    @Autowired<br>    private StudentRepository studentRepository;<br><br>    @Autowired<br>    private CourseRepository courseRepository;<br><br>    public void saveStudentWithCourses() {<br>        // Create Courses<br>        Course course1 = new Course();<br>        course1.setId(1);<br>        course1.setTitle(&quot;Mathematics&quot;);<br><br>        Course course2 = new Course();<br>        course2.setId(2);<br>        course2.setTitle(&quot;Computer Science&quot;);<br><br>        // Save Courses<br>        courseRepository.saveAll(Set.of(course1, course2));<br><br>        // Create Student<br>        Student student = new Student();<br>        student.setId(101);<br>        student.setName(&quot;John Doe&quot;);<br>        student.setCourses(Set.of(course1, course2)); // Associate courses with the student<br><br>        // Save Student<br>        studentRepository.save(student);<br>    }<br>}</pre><h4>Step 4: Testing</h4><p>Run the <strong>saveStudentWithCourses()</strong> method and check the database:</p><p><strong>Student Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/1*V8HBO2S8VvaYt_ZOrbX-5w.png" /></figure><p><strong>Course Table</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*-bVkNZOS0-_1KQT8aX11fQ.png" /></figure><p><strong>Join Table (student_course)</strong> will have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*0TEo3I6EsGslkBP2TSR2SQ.png" /></figure><p>Association mapping in Hibernate is the foundation for creating scalable and maintainable Spring Boot applications. By mastering these relationships and their implementation, you’ll be able to design robust systems that accurately reflect real-world models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0f27ab3278e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>