<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mukesh Baphna on Medium]]></title>
        <description><![CDATA[Stories by Mukesh Baphna on Medium]]></description>
        <link>https://medium.com/@mukesh.baphna_61982?source=rss-d6c15009b4d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*1w5CdHk6AXOWojIF</url>
            <title>Stories by Mukesh Baphna on Medium</title>
            <link>https://medium.com/@mukesh.baphna_61982?source=rss-d6c15009b4d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:00:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mukesh.baphna_61982/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Engineering an Engineering Org]]></title>
            <link>https://medium.com/@mukesh.baphna_61982/engineering-an-engineering-org-070146cd18c2?source=rss-d6c15009b4d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/070146cd18c2</guid>
            <dc:creator><![CDATA[Mukesh Baphna]]></dc:creator>
            <pubDate>Thu, 07 Nov 2024 21:22:08 GMT</pubDate>
            <atom:updated>2024-11-08T18:15:15.509Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*XNnc5z-HIj6cFDys5mrtbg.png" /><figcaption>Effective Organization (generated using Dall-E3)</figcaption></figure><p>Crafting an effective engineering organization is a complex puzzle with no one-size-fits-all solution. The key to success lies in understanding and embracing the trade-offs involved. Drawing from my own experiences with fast-growing, engineering-driven startups, I’ll share insights into what works and what doesn’t.</p><p>A well-structured engineering organization is essential for both sustaining existing products and driving innovation through new developments. Here’s a structured approach to achieve this:</p><h4><strong>Product and Technology Alignment</strong>:</h4><ul><li><strong>Support Existing Products</strong>: Identify core areas of the current product where stability and ongoing maintenance are critical. Allocate experienced engineers to support these areas, ensuring knowledge retention and minimal disruptions. It’s prudent to plan for knowledge transfer to make it effective in terms of engineering cost you pay for such areas. Many times, startups tend to move to next shiny things, ignoring the need to maintain and improve prior work resulting into ‘feature atrophy’ and eventually ‘customer attrition’. Identify all product areas and make sure they continue to have adequate coverage. You cannot have a product feature if it’s not backed by explicit resource investments behind it. It’s like insurance, you don’t need it until you need it but…would you go without it?</li><li><strong>Enable New Product Development</strong>: Create dedicated teams for innovation and new product features. These teams should work closely with product managers (PMs) to ensure new investments are aligned with the roadmap and address customer needs. Each substantial product feature must have dedicated resourcing and explicit plans. You might build a version one or a PoC, with a set of borrowed resources (sometimes called horizontal teams or tiger teams) however at some stage an explicit product feature must have a explicit set of engineering and product people assigned to it.</li><li><strong>Investment in cross-cutting concerns</strong>: Don’t ignore cross-cutting aspects like scaling, performance, observability and reliability. Each team should have a clear mandate and the resources needed to meet the goals in their area. In many of the SaaS companies, this needs to be owned by each feature team (DevOps model) but it depends upon the scale of the features and the stage you are at as a company, and you must explicitly discuss the need to support these sub-area and make sure there is coverage from product and engineering for each sub-areas. Lot of companies will have DevOps culture and still have a set of production engineering/SRE folks to cover for gaps or as an additional line of defense. Do not overlook this aspect as your business starts to scale. The business cost of operational mishaps is highly underestimated.</li></ul><h4><strong>Process and Operational Excellence</strong>:</h4><ul><li><strong>Code and Design Review</strong>: Maintain a minimum number of experienced engineers to ensure rigorous code and design reviews. Establish a peer-review system that’s efficient but thorough, balancing quality with speed. Any sub-areas with fewer than 3–4 engineers is a risk that must be outright discussed and agreed upon.</li><li><strong>Customer Escalations &amp; Incident Management</strong>: Ensure there’s a team with dedicated on-call rotations and incident management protocols. This team should have the skills to handle escalations swiftly, minimizing downtime and ensuring fast recovery. Design a sustainable on-call rotation that spreads the load evenly across teams to prevent burnout, even if there are no actual incidents. This requires each sub-areas to be adequately staffed (typically 12+) or a collection of sub-areas to be clubbed under one on-call rotation. Ideally more than one on-call rotation (assuming it lasts a day or longer) a quarter should be addressed. I understand when you are really small engineering org and building a product, this is hard to live by, but once you reach a size of 50+ engineering org, you must strive to balance on-call burden.</li><li><strong>Maintenance and Contingency Planning</strong>: Set aside resources for maintenance work, particularly to handle technical debt and refactoring. Plan for contingencies like unexpected outages or last-minute new requirements or staffing changes, with strategies for quickly adapting team assignments and resources.</li></ul><h4><strong>People and Organizational Structure</strong>:</h4><ul><li><strong>Balanced Leadership Ratio</strong>: Aim for a roughly 1:1 engineering manager (EM) to PM ratio, allowing for strong product alignment and decision-making. This setup fosters a cross-functional approach where every product decision is technically informed. Think of your PM-EM pair at each level as CPO-CTO thematically and see if you can build that sort of delegation and execution model at each level. If it skews, you’ve to plan conciously to minimize impact of that skew. Also don’t overload your EMs and PMs with multiple product sub-areas, ideally 2–3 adjacent sub-areas are the upper limit depending upon the size of your company. As you grow bigger, at some stage it should come down to EM-PM pair (+ a Staff+ IC) for each sub-area. This is super critical if you want to be agile and responsive to changing market demands and therefore a focused delegated model at each level improves your ability to react to changes, fosters a culture of innovation and experimentation.</li><li><strong>EM to Engineer Ratios</strong>: Adjust the ratio based on company maturity and team needs. For a mature company, a larger ratio (e.g., 1 EM to 10 engineers) may work, while in early growth stages, a smaller ratio (1 EM to 6–8 engineers) helps in scaling the team culture and mentorship. Quite often, this aspect is overlooked in startups resulting into execution and retention challenges usually as surprises.</li><li><strong>Durable Project Assignments</strong>: Assign engineers to projects that last long enough for them to develop relationships with their peers and leaders, learn deeply, and contribute meaningfully. Thrashing is bad in general whether for your compute resources or people resources. (bad pun!) Make sure you resist the urge to restructure or move around people for every roadmap change which is likely to happen more than usual in a fast growing startup.</li><li><strong>Career Growth</strong>: Create a structure that supports long-term career growth, with clear paths for both technical and leadership advancement. Provide mentorship and meaningful projects that align with both business needs and individual aspirations. Ideally have a healthy mix of senior to junior engineers depending upon complexity of sub-areas so there are always people who are growing at each levels in the team. Ideally 3–4 junior engineers to 1 staff can be a good balance but there is absolutely no template that works for all teams. It’s highly subjective.</li><li><strong>Skill Alignment and Self-Sufficiency</strong>: Build teams with a complementary skill set to make them as self-sufficient as possible. For instance, have data engineers, frontend, and backend engineers embedded within teams where possible to handle end-to-end features. Apply this as much as possible for every team however sometimes you can make exceptions for horizontal ownership around say production-engineering/SRE and/or UI teams but again as an exception not as a rule!</li><li><strong>Optimize for local execution</strong>: Having teams spread our across multiple locations is a reality one has to contend with. Build center-of-excellence i.e. each team is self-sufficient (i.e. autonomous) in their entire SDLC (idea, design, execution, delivery, operations) as much as possible. If you can’t make that happen, explicitly discuss the trade offs and document those decisions to ensure everyone is aligned to the trade offs. Trade off usually involve handling challenges in building faster alignment, remote decision making, communication and collaboration.</li></ul><p>Using this structure, you can foster an engineering organization that is both resilient in its support of existing products and agile in driving new initiatives, while prioritizing sustainable growth and career development for your team. This ensures an organization equipped to manage immediate operational demands and scale for future needs.</p><p><strong>Disclaimer </strong>— <em>opinions published here are my own.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=070146cd18c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chunking in RAG: why that’s critical?]]></title>
            <link>https://medium.com/@mukesh.baphna_61982/chunking-in-rag-why-thats-critical-388be71f5997?source=rss-d6c15009b4d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/388be71f5997</guid>
            <dc:creator><![CDATA[Mukesh Baphna]]></dc:creator>
            <pubDate>Wed, 30 Oct 2024 05:19:45 GMT</pubDate>
            <atom:updated>2024-10-31T03:10:50.224Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*mWgKmydKusMDhKKVmlsyFA.png" /><figcaption>Generated using ChatGPT</figcaption></figure><p>Chunking is a crucial part of the Retrieval-Augmented Generation (RAG) architecture. When done correctly, it breaks large documents into manageable pieces while preserving their meaning and context. This makes retrieval more efficient and ensures that the most relevant information is highlighted for the language model, leading to more accurate responses.</p><p>This is a follow up to my earlier blog on intelligent PDF parsing using RAG architecture <a href="https://medium.com/@mukesh.baphna_61982/my-curious-experiments-an-intelligent-pdf-agent-using-local-llm-a974906302fb">https://medium.com/@mukesh.baphna_61982/my-curious-experiments-an-intelligent-pdf-agent-using-local-llm-a974906302fb</a></p><h3>Chunking:</h3><ul><li><strong>Optimal Chunk Size</strong>: If chunks are too large, they may contain unrelated or excessive information, diluting the relevance of retrieved results. If they are too small, important context could be lost, leading to fragmented understanding.</li><li><strong>Dynamic Chunking</strong>: Emerging techniques now use dynamic chunking based on semantic boundaries rather than fixed-size chunks, improving the relevance of retrieved passages.</li><li><strong>Hierarchical Chunking</strong>: Innovations in this space enable hierarchical retrieval, where both coarse and fine-grained chunks are indexed to balance between speed and depth of information.</li></ul><p>In this article, I’d delve deeper into dynamic chunking.</p><p>Dynamic chunking is an evolving technique in the context of RAG (Retrieval-Augmented Generation) architectures, which aims to create more contextually meaningful chunks of text for retrieval rather than using a fixed-size approach. Some of this in practice can overlap with what I call hierarchical chunking.</p><h3>Key Features of Dynamic Chunking:</h3><ol><li><strong>Semantic Boundaries</strong>: Instead of splitting text purely by length or character count, dynamic chunking identifies natural breakpoints in content, such as sentence boundaries or paragraph transitions. This helps to preserve the coherence and meaning of the text within each chunk.</li><li><strong>Context Preservation</strong>: By maintaining the integrity of semantic units (such as full sentences or sections), dynamic chunking ensures that key ideas aren’t split across multiple chunks. This leads to higher quality retrieval, as each chunk retains more relevant context for a query.</li><li><strong>Content-Dependent Chunking</strong>: Depending on the type of text (e.g., technical documentation vs. narrative), dynamic chunking can adapt chunk size and boundaries. For example, in a research paper, the chunking might follow headings and sub-sections, while for a conversation transcript, it could follow dialog turns.</li><li><strong>Improved Retrieval Accuracy</strong>: Because dynamic chunking produces chunks that are contextually rich and complete, the retrieval step in a RAG architecture can return more relevant and accurate results, since the whole idea or unit of meaning is more likely to be captured within a single chunk.</li><li><strong>Adaptive Size</strong>: Chunk size can be adjusted based on the query or task at hand. For example, a long, complex query might trigger larger chunks to ensure enough context is provided, while a simple query might use smaller chunks to focus on precision.</li></ol><h3>Benefits of Dynamic Chunking:</h3><ul><li><strong>Higher Retrieval Precision</strong>: Dynamic chunking reduces the risk of retrieving irrelevant or partially relevant chunks, leading to more accurate responses.</li><li><strong>Reduced Information Loss</strong>: By keeping context intact, dynamic chunking minimizes the risk of losing important details that could occur when chunks are too small or arbitrarily cut.</li><li><strong>Better Query Alignment</strong>: When chunks align with the structure and flow of the text, they can better match the intent behind queries, which enhances the quality of responses in RAG models.</li></ul><h3>Latest Innovations in Dynamic Chunking:</h3><ul><li><strong>Transformer-Based Models</strong>: These models can be used to identify semantic boundaries more intelligently, enabling dynamic chunking to operate at higher levels of complexity, such as detecting topic shifts or subtle context changes within paragraphs.</li><li><strong>Real-Time Chunk Adjustments</strong>: New algorithms are being developed to adjust chunk size and boundaries in real-time based on user feedback or query intent, making the retrieval process even more tailored.</li></ul><p>Dynamic chunking is emerging as a vital optimization in making RAG systems more accurate and effective, especially for long-form or complex documents.</p><p>Here’s a simple example where I applied few heuristics of dynamic chunking. The approach here takes a document and break it down into multiple chunk using few different heuristics. Each iteration based on a heuristics results in a set of chunks, however chunks across iterations could overlap. For example a simple chunking based on ‘punctuation marks’ or ‘fixed-length’ will have many chunks that overlap when ‘dynamic chunking’ is performed based on topic detection using an LLM or when we apply a heuristic of chunking based on presence of common words between adjacent chunks created by other heuristics. Eventually we generate embeddings for all these chunks and store them into vector DB for later extraction or similarity search before we feed them into LLM for inference.</p><h3>Dynamic Chunking using Topics Detection</h3><p>Here’s a simple Python code snippet that chunks a paragraph based on topic detection using an LLM. We’ll use theKMeans clustering algorithm for topic detection.</p><h3>Sample Code</h3><pre>import numpy as np<br>from sklearn.cluster import KMeans<br>import nltk<br>from nltk.tokenize import sent_tokenize<br>import requests<br><br># Download the punkt tokenizer<br>nltk.download(&#39;punkt&#39;)<br><br>def group_sentences(embeddings, num_clusters=3):<br>    # Agglomerative Clustering with cosine affinity<br>    clustering_model = AgglomerativeClustering(n_clusters=num_clusters, affinity=&#39;cosine&#39;, linkage=&#39;complete&#39;)<br>    labels = clustering_model.fit_predict(embeddings)<br>    <br>    # Group sentences by cluster labels<br>    grouped_sentences = {}<br>    for idx, label in enumerate(labels):<br>        if label not in grouped_sentences:<br>            grouped_sentences[label] = []<br>        grouped_sentences[label].append(sentences[idx])<br>    <br>    return grouped_sentences<br><br>def get_embeddings(text):<br>    # Define the endpoint for your local Ollama model<br>    ollama_url = &quot;http://localhost:11434/generate&quot;  # My Ollam setup, change based on your&#39;s<br><br>    # Send a request to the Ollama model for embeddings<br>    response = requests.post(ollama_url, json={&quot;input&quot;: text, &quot;model&quot;: &quot;embedding-ada-002&quot;})<br>    return np.array(response.json()[&#39;embedding&#39;])<br><br>def infer_number_of_topics(text):<br>    # Define the endpoint for your local Ollama model<br>    ollama_url = &quot;http://localhost:11434/generate&quot;  # Adjust based on your Ollama setup<br><br>    # Prompt Ollama to infer the number of topics<br>    prompt = f&quot;Analyze the following text and suggest an appropriate number of distinct topics:\n\n{text}\n\nNumber of topics:&quot;<br>    response = requests.post(ollama_url, json={&quot;input&quot;: prompt, &quot;model&quot;: &quot;llama&quot;})<br>    number_of_topics = int(response.json()[&#39;output&#39;].strip())<br>    return number_of_topics<br><br>def chunk_text(text):<br>    # Tokenize text into sentences<br>    sentences = sent_tokenize(text)<br>    embeddings = np.array([get_embeddings(sentence) for sentence in sentences])<br><br>    # Infer number of topics from Ollama model<br>    num_clusters = infer_number_of_topics(text)<br><br>    # Use KMeans to cluster the sentence embeddings<br>    kmeans = KMeans(n_clusters=num_clusters)<br>    kmeans.fit(embeddings)<br>    labels = kmeans.labels_<br><br>    # Create a dictionary to hold the chunks<br>    topic_chunks = {i: [] for i in range(num_clusters)}<br>    for label, sentence in zip(labels, sentences):<br>        topic_chunks[label].append(sentence)<br><br>    # Combine sentences for each topic<br>    chunks = {f&#39;Topic {i}&#39;: &#39; &#39;.join(chunk) for i, chunk in topic_chunks.items()}<br>    return chunks<br><br># Example children&#39;s story: &quot;Goldilocks and the Three Bears&quot;<br>children_story = &quot;&quot;&quot;<br>Once upon a time, there were three bears who lived together in a house in the woods. <br>There was a Daddy Bear, a Mummy Bear, and a Baby Bear. One day, they made some porridge for breakfast. <br>But it was too hot to eat, so they decided to take a walk in the woods while it cooled down. <br>While they were away, a little girl named Goldilocks came to their house. <br>She knocked on the door, but no one answered. So, she walked right in. <br>Goldilocks tasted the porridge from the Daddy Bear&#39;s bowl, but it was too hot. <br>Next, she tasted the Mummy Bear&#39;s porridge, but it was too cold. <br>Finally, she tasted the porridge from the Baby Bear&#39;s bowl, and it was just right. So, she ate it all up! <br>After eating, Goldilocks felt tired and decided to sit in the bears&#39; chairs. <br>She tried Daddy Bear&#39;s chair, but it was too big. Then she tried Mummy Bear&#39;s chair, but it was too soft. <br>Finally, she tried Baby Bear&#39;s chair, and it was just right. But when she sat in it, it broke! <br>Feeling sleepy, Goldilocks went upstairs to the bears&#39; bedroom. <br>She lay down in Daddy Bear&#39;s bed, but it was too hard. She lay down in Mummy Bear&#39;s bed, but it was too soft. <br>Then she lay down in Baby Bear&#39;s bed, and it was just right. <br>Soon Goldilocks fell asleep. <br>When the three bears came home, they found the door open. <br>Daddy Bear said, &quot;Someone&#39;s been eating my porridge!&quot; <br>Mummy Bear said, &quot;Someone&#39;s been sitting in my chair!&quot; <br>And Baby Bear said, &quot;Someone&#39;s been sleeping in my bed!&quot; <br>Just then, Goldilocks woke up and saw the three bears. <br>She jumped out of bed and ran out of the house as fast as she could, never to return again.<br>&quot;&quot;&quot;<br><br># Dynamic chunking based on the story<br>chunks = chunk_text(children_story)<br><br># Display the input story and output chunks<br>print(&quot;Input Story:\n&quot;)<br>print(children_story)<br>print(&quot;\nOutput Chunks:\n&quot;)<br>for topic, content in chunks.items():<br>    print(f&quot;{topic}:\n{content}\n&quot;)</pre><h3>Chunks based on Topics</h3><pre>Topic 0:<br>Once upon a time, there were three bears who lived together in a house in the woods. There was a Daddy Bear, a Mummy Bear, and a Baby Bear. One day, they made some porridge for breakfast. But it was too hot to eat, so they decided to take a walk in the woods while it cooled down. <br><br>Topic 1:<br>While they were away, a little girl named Goldilocks came to their house. She knocked on the door, but no one answered. So, she walked right in. Goldilocks tasted the porridge from the Daddy Bear&#39;s bowl, but it was too hot. Next, she tasted the Mummy Bear&#39;s porridge, but it was too cold. Finally, she tasted the porridge from the Baby Bear&#39;s bowl, and it was just right. So, she ate it all up! <br><br>Topic 2:<br>After eating, Goldilocks felt tired and decided to sit in the bears&#39; chairs. She tried Daddy Bear&#39;s chair, but it was too big. Then she tried Mummy Bear&#39;s chair, but it was too soft. Finally, she tried Baby Bear&#39;s chair, and it was just right. But when she sat in it, it broke! Feeling sleepy, Goldilocks went upstairs to the bears&#39; bedroom. She lay down in Daddy Bear&#39;s bed, but it was too hard. She lay down in Mummy Bear&#39;s bed, but it was too soft. Then she lay down in Baby Bear&#39;s bed, and it was just right. Soon Goldilocks fell asleep. When the three bears came home, they found the door open. Daddy Bear said, &quot;Someone&#39;s been eating my porridge!&quot; Mummy Bear said, &quot;Someone&#39;s been sitting in my chair!&quot; And Baby Bear said, &quot;Someone&#39;s been sleeping in my bed!&quot; Just then, Goldilocks woke up and saw the three bears. She jumped out of bed and ran out of the house as fast as she could, never to return again.</pre><p>The output provides a structured way to view the story’s content, breaking it down into distinct topics that capture different aspects of the narrative.</p><h3>Explanation</h3><ul><li><strong>Code Functionality</strong>: The code takes a paragraph and splits it into sentences. Each sentence is converted into an embedding using a transformer model, and then KMeans clustering is applied to group the sentences into topics based on their embeddings.</li><li><strong>Output</strong>: The output shows how the sentences are clustered into different topics, allowing for topic detection within the provided paragraph.</li></ul><h3>Dynamic Chunking using grammatical structures</h3><p>Here’s an example where chunks are generated based on punctuation marks and new lines, and then adjacent chunks are combined into additional chunks if they shared more than 20% of their words (excluding common English words).</p><h3>Sample Code</h3><pre>import re<br>from collections import Counter<br><br># List of common English words to exclude<br>COMMON_WORDS = set([<br>    &#39;i&#39;, &#39;me&#39;, &#39;my&#39;, &#39;myself&#39;, &#39;we&#39;, &#39;our&#39;, &#39;ours&#39;, &#39;ourselves&#39;, &#39;you&#39;, &#39;your&#39;, &#39;yours&#39;,<br>    &#39;yourself&#39;, &#39;yourselves&#39;, &#39;he&#39;, &#39;him&#39;, &#39;his&#39;, &#39;himself&#39;, &#39;she&#39;, &#39;her&#39;, &#39;hers&#39;,<br>    &#39;herself&#39;, &#39;it&#39;, &#39;its&#39;, &#39;itself&#39;, &#39;they&#39;, &#39;them&#39;, &#39;their&#39;, &#39;theirs&#39;, &#39;themselves&#39;,<br>    &#39;what&#39;, &#39;which&#39;, &#39;who&#39;, &#39;whom&#39;, &#39;this&#39;, &#39;that&#39;, &#39;these&#39;, &#39;those&#39;, &#39;am&#39;, &#39;is&#39;, &#39;are&#39;,<br>    &#39;was&#39;, &#39;were&#39;, &#39;be&#39;, &#39;been&#39;, &#39;being&#39;, &#39;have&#39;, &#39;has&#39;, &#39;had&#39;, &#39;having&#39;, &#39;do&#39;, &#39;does&#39;,<br>    &#39;did&#39;, &#39;doing&#39;, &#39;a&#39;, &#39;an&#39;, &#39;the&#39;, &#39;and&#39;, &#39;but&#39;, &#39;if&#39;, &#39;or&#39;, &#39;because&#39;, &#39;as&#39;, &#39;until&#39;,<br>    &#39;while&#39;, &#39;of&#39;, &#39;at&#39;, &#39;by&#39;, &#39;for&#39;, &#39;with&#39;, &#39;about&#39;, &#39;against&#39;, &#39;between&#39;, &#39;into&#39;,<br>    &#39;through&#39;, &#39;during&#39;, &#39;before&#39;, &#39;after&#39;, &#39;above&#39;, &#39;below&#39;, &#39;to&#39;, &#39;from&#39;, &#39;up&#39;, &#39;down&#39;,<br>    &#39;in&#39;, &#39;out&#39;, &#39;on&#39;, &#39;off&#39;, &#39;over&#39;, &#39;under&#39;, &#39;again&#39;, &#39;further&#39;, &#39;then&#39;, &#39;once&#39;, &#39;here&#39;,<br>    &#39;there&#39;, &#39;when&#39;, &#39;where&#39;, &#39;why&#39;, &#39;how&#39;, &#39;all&#39;, &#39;any&#39;, &#39;both&#39;, &#39;each&#39;, &#39;few&#39;, &#39;more&#39;,<br>    &#39;most&#39;, &#39;other&#39;, &#39;some&#39;, &#39;such&#39;, &#39;no&#39;, &#39;nor&#39;, &#39;not&#39;, &#39;only&#39;, &#39;own&#39;, &#39;same&#39;, &#39;so&#39;,<br>    &#39;than&#39;, &#39;too&#39;, &#39;very&#39;, &#39;s&#39;, &#39;t&#39;, &#39;can&#39;, &#39;will&#39;, &#39;just&#39;, &#39;don&#39;, &#39;should&#39;, &#39;now&#39;<br>])<br><br>def break_into_chunks(text):<br>    # Step 1: Split based on punctuation marks<br>    punctuation_chunks = re.split(r&#39;(?&lt;=[.!?]) +&#39;, text.strip())<br>    <br>    # Step 2: Further break down chunks based on new lines<br>    new_line_chunks = []<br>    for chunk in punctuation_chunks:<br>        new_line_split = chunk.split(&#39;\n&#39;)<br>        new_line_chunks.extend([line.strip() for line in new_line_split if line.strip()])<br><br>    # Step 3: Create initial chunks list<br>    initial_chunks = new_line_chunks.copy()<br><br>    # Step 4: Combine chunks based on repeated words (excluding common words)<br>    final_chunks = []<br>    for i in range(len(initial_chunks)):<br>        if not final_chunks:  # If final_chunks is empty, add the first chunk<br>            final_chunks.append(initial_chunks[i])<br>        else:<br>            current_chunk = initial_chunks[i]<br>            last_chunk = final_chunks[-1]<br>            <br>            # Count words excluding common words<br>            current_words = [word.lower() for word in re.findall(r&#39;\w+&#39;, current_chunk) if word.lower() not in COMMON_WORDS]<br>            last_words = [word.lower() for word in re.findall(r&#39;\w+&#39;, last_chunk) if word.lower() not in COMMON_WORDS]<br><br>            # Calculate the number of common words<br>            current_count = Counter(current_words)<br>            last_count = Counter(last_words)<br>            common_count = sum((current_count &amp; last_count).values())<br>            <br>            # Calculate total unique words in both chunks<br>            total_unique = len(set(current_words).union(set(last_words)))<br><br>            # Check if they share more than 20% repeated words<br>            if total_unique &gt; 0 and (common_count / total_unique &gt; 0.2):<br>                final_chunks[-1] += &#39; &#39; + current_chunk  # Combine<br>            else:<br>                final_chunks.append(current_chunk)  # Add as new chunk<br><br>    return initial_chunks, final_chunks<br><br># Function to group sentences by grammatical notations and semantic similarity<br>def group_sentences_by_grammar_and_theme(paragraph):<br>    # Step 1: Syntactic Parsing using spaCy<br>    doc = nlp(paragraph)<br>    <br>    # Extract sentences from the parsed document<br>    sentences = [sent.text for sent in doc.sents]<br>    <br>    # Step 2: Analyze grammatical structure<br>    grammatical_groups = {}<br>    for sent in doc.sents:<br>        subject = None<br>        verb = None<br>        for token in sent:<br>            if token.dep_ == &#39;nsubj&#39;:  # Find subject<br>                subject = token.text<br>            elif token.pos_ == &#39;VERB&#39;:  # Find verb<br>                verb = token.text<br>        if subject and verb:<br>            key = f&quot;Subject: {subject}, Verb: {verb}&quot;<br>            if key not in grammatical_groups:<br>                grammatical_groups[key] = []<br>            grammatical_groups[key].append(sent.text)<br><br># Sample paragraph<br>sample_paragraph = &quot;&quot;&quot;<br>Once upon a time, in a land far away, there lived a brave knight. <br>He ventured into dark forests and climbed high mountains. <br>His quest was to find the lost treasure of the ancient king. <br><br>Every day was a new adventure, filled with dangers and surprises. <br>One day, he met a talking dragon who offered to help him. <br>They became friends and set off together on an exciting journey.<br>&quot;&quot;&quot;<br><br># Break the sample paragraph into chunks<br>initial_chunks, combined_chunks = break_into_chunks(sample_paragraph)<br><br># Display the output<br>print(&quot;Chunks based on punctuation marks and new lines:\n&quot;)<br>for idx, chunk in enumerate(initial_chunks, 1):<br>    print(f&quot;Initial Chunk {idx}:\n{chunk}\n&quot;)<br><br>print(&quot;Combined Chunks based on repeated words:\n&quot;)<br>for idx, chunk in enumerate(combined_chunks, 1):<br>    print(f&quot;Combined Chunk {idx}:\n{chunk}\n&quot;)</pre><p>I use a predefined set of common English words is used to filter out from the counting of unique words. The text is first split into chunks based on punctuation marks. It then splits further based on new lines The initial_chunks list contains the chunks formed by the above steps. After that we combine these initial chunks by checking if adjacent chunks share more than 20% of their words (excluding common words). If they do, the chunks are combined. And finally, it also clubs disparate sentences with same subject and verb together as one chunk.</p><h3>Chunks based on punctuation criteria</h3><pre>Chunks based on punctuation marks and new lines:<br><br>Initial Chunk 1:<br>Once upon a time, in a land far away, there lived a brave knight.<br><br>Initial Chunk 2:<br>He ventured into dark forests and climbed high mountains.<br><br>Initial Chunk 3:<br>His quest was to find the lost treasure of the ancient king.<br><br>Initial Chunk 4:<br>Every day was a new adventure, filled with dangers and surprises.<br><br>Initial Chunk 5:<br>One day, he met a talking dragon who offered to help him.<br><br>Initial Chunk 6:<br>They became friends and set off together on an exciting journey.<br><br>Combined Chunks based on repeated words:<br><br>Combined Chunk 1:<br>Once upon a time, in a land far away, there lived a brave knight. He ventured into dark forests and climbed high mountains. His quest was to find the lost treasure of the ancient king.<br><br>Combined Chunk 2:<br>Every day was a new adventure, filled with dangers and surprises. One day, he met a talking dragon who offered to help him. They became friends and set off together on an exciting journey.</pre><h3>Conclusion</h3><p>In my Intelligent PDF parser, I saw the quality of results improved considerably as I applied dynamic chunking. Quite expectedly the scan performance dipped however it was a one time setup and therefore wasn’t visible. The cost of additional embeddings and tracking chunks for each of those literally doubled with each heuristic I added. It’s difficult to measure the quality improvement but visibly the results were far more accurate as the size of documents increased (say 10+ pages) and when content was spread out across multiple documents with common topics and sub-topics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=388be71f5997" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My curious experiments — An Intelligent PDF Agent using local LLM]]></title>
            <link>https://medium.com/@mukesh.baphna_61982/my-curious-experiments-an-intelligent-pdf-agent-using-local-llm-a974906302fb?source=rss-d6c15009b4d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a974906302fb</guid>
            <dc:creator><![CDATA[Mukesh Baphna]]></dc:creator>
            <pubDate>Sun, 20 Oct 2024 22:52:21 GMT</pubDate>
            <atom:updated>2024-10-30T05:30:24.158Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>My curious experiments — An Intelligent PDF Agent using local LLM (Part 1)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/0*yiJ1RX1aO1VQ5atr" /></figure><h3>Introduction</h3><p>I decided to build an application with Ollama to query PDFs locally because, well, why not? I was curious about all the AI buzz, wanted to keep my data close, and figured I might as well learn something cool along the way. Plus, with all these fancy tools out there, it felt like cheating not to use them to speed things up!</p><h3>Why</h3><p>Imagine if one could leverage the power of modern LLM locally to get better insights and then make decisions accordingly. I tried it and while I was mildly amused (I have a high bar) with what I discovered, I was mighty pleased with possibilities.</p><p>Take a look at what came to light after I ran this over my cell phone bills for past few months -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iyT54uwxgapkAeVy" /></figure><p>It wasn’t a huge surprise! I had turned on international roaming on March 20th owing to my international travel. What it helped me infer was that I’d forgotten to turn it off and my cell-phone carrier was too happy to keep charging me for that. Of course, I could’ve taken a glance at my bill and detected it myself but in the world of auto-payments and other distractions, this gets lost.</p><h3>Workflow</h3><p>To minimize hallucinations (incorrect responses), LLMs can be paired with private datasets. Retrieval Augmented Generation (RAG) framework is a popular method for achieving this, as it provides relevant documents to the LLM model for context. I also wanted to build a full-stack RAG application locally, that lets me keep my documents secure without relying on cloud based infrastructure or sharing my documents with any third parties.</p><p>I chose to implement RAG using LangChain, a framework that streamlines LLM application development, and integrate it with Chroma to build a vector database. A high level overview of the workflow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCe9VkUufSnd_OOm-m5mKw.png" /></figure><h3><strong>Technologies under the hood</strong></h3><p>Intelligent PDF Agent leverages cutting-edge technologies to provide users with a seamless and intelligent experience. Here’s a brief overview of the key technologies and frameworks</p><p>1. <strong>PyQt5 for User Interface</strong></p><p>The graphical user interface (GUI) is crafted using PyQt5, a set of Python bindings for the Qt application framework. This allows for the creation of modern, responsive, and visually appealing applications. PyQt5 facilitates smooth interactions and provides a native look and feel on macOS.</p><p><strong>2</strong>. <strong>ChromaDB for Vector Storage</strong></p><p>To manage and query document data, I chose ChromaDB, a vector database designed for data storage and retrieval. ChromaDB enables efficient embedding of PDF content, making it easy to search and retrieve relevant information.</p><p><strong>3. Natural Language Processing (NLP)</strong></p><p>The intelligent question-answering feature relies on natural language processing techniques to understand user queries. By analyzing the semantics of the questions, it can accurately fetch and present answers from the stored document embeddings.</p><p><strong>4. Ollama for LLM Integration</strong></p><p>Ollama, a language model that powers the application’s ability to generate contextual responses. Harnessing the capabilities of large language models (LLMs) enables insightful and accurate answers to user inquiries.</p><h3><strong>Environment</strong></h3><p>First, follow <a href="https://github.com/jmorganca/ollama?trk=article-ssr-frontend-pulse_little-text-block">these instructions</a> to set up and run a local Ollama instance:</p><p>Download Ollama locally <a href="https://github.com/ollama/ollama?tab=readme-ov-file">https://github.com/ollama/ollama?tab=readme-ov-file<br></a>Download model you want to run — <strong><em>ollama run mistral</em></strong></p><p>Pull appropriate model <strong><em>— ollama pull ollama2</em></strong></p><p>1. Setup your workspace</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/0*GdWvxDa-QfMWlfS5" /></figure><p>2. Clone the repository</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*raaVcxjUdl9wbXiL" /></figure><p>3. Setup your virtual environment</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/0*UJayWP3B3V_ZazkR" /></figure><p>4. Install all dependencies</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/0*FKEbj_dmuYvg9bXT" /></figure><p>5. Run Application</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/226/0*9cnNmS-WIrWxNL3F" /></figure><h3><strong>Future Work</strong></h3><p>One can enhance this code base to take multiple files (or folders) to read a collection of files and those files need not be just PDF but any format. If I take it to extreme lengths, you can read your entire document repository but then a local LLM (on your laptop won’t cut it) and you have to use cloud resources. You can build a chat interface rather than just a simple Q&amp;A interface. And integrating it with some dash-boarding tools to build analytics is not far fetched either.</p><blockquote><em>Part 2 — Improve chunking in document parsing </em><a href="https://medium.com/@mukesh.baphna_61982/chunking-in-rag-why-thats-critical-388be71f5997">https://medium.com/@mukesh.baphna_61982/chunking-in-rag-why-thats-critical-388be71f5997</a><em><br>Part 3(coming soon) — Build a local knowledge graph by ingesting multi modal content</em></blockquote><h3>References</h3><p>Besides of course using ChatGPT, I’ve to mention amazing blogs written on this topic by <a href="https://callumjmac.github.io/posts/2024/04/22/RAG_intro/">https://callumjmac.github.io/posts/2024/04/</a></p><p>My GitHub repo is here — <a href="https://github.com/mukeshbaphna/IntelliPDF">https://github.com/mukeshbaphna/IntelliPDF</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a974906302fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>