<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sangitapokhrel on Medium]]></title>
        <description><![CDATA[Stories by Sangitapokhrel on Medium]]></description>
        <link>https://medium.com/@sangitapokhrel911?source=rss-9540fb596d80------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ncKRQhayE6HXhVfVouybGQ.jpeg</url>
            <title>Stories by Sangitapokhrel on Medium</title>
            <link>https://medium.com/@sangitapokhrel911?source=rss-9540fb596d80------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 14:12:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sangitapokhrel911/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multi-Agent AI Systems: The Next Evolution of Intelligent Business Automation]]></title>
            <link>https://medium.com/@sangitapokhrel911/multi-agent-ai-systems-the-next-evolution-of-intelligent-business-automation-b9744fefe6f5?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9744fefe6f5</guid>
            <category><![CDATA[ai-automation]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 15:40:49 GMT</pubDate>
            <atom:updated>2026-07-30T15:40:49.944Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aBGJpjP5UNcv5J_7wkjaUw.png" /><figcaption>(Image generated by Gen AI Tool)</figcaption></figure><p>Artificial Intelligence has come a long way in a remarkably short time. Just a few years ago, organizations were experimenting with machine learning models to automate repetitive tasks. Today, Large Language Models (LLMs) can write reports, generate software code, analyse documents, and answer complex questions in seconds.</p><p>Yet, despite these impressive capabilities, most AI applications still rely on single model performing multiple tasks. While this works well for straightforward problems, it begins to struggle when organizations need AI to execute complex workflows involving planning, reasoning, validation, decision-making, and collaboration.</p><blockquote>This is where Multi-Agent AI Systems are changing the game.</blockquote><p>Rather than depending on one AI model to perform every task, a multi-agent system consists of several specialized AI agents working together towards a common objective. Each agent has a clearly defined responsibility, communicates with other agents, and contributes to a larger workflow. The result is an intelligent ecosystem capable of solving problems that would be difficult or even impossible for a single AI assistant to handle effectively.</p><h4><strong>Moving Beyond a Single AI Assistant</strong></h4><p>Imagine asking an AI to produce a market analysis report.</p><p>A traditional AI assistant receives the prompt, generates a response, and returns the result. While this may be useful, the entire process relies on one model making every decision from gathering information to analysing data and writing conclusions.</p><p>A multi-agent system approaches the same task very differently.</p><p>One agent searches for reliable information from trusted sources.</p><p>Another evaluates the credibility and relevance of that information.</p><p>A third analyses trends and extracts meaningful insights.</p><p>A fourth agent prepares a professional report.</p><p>Finally, another agent reviews the output for factual consistency, formatting, and quality before presenting it to the user.</p><p>Each agent performs one specialised role, much like members of a project team collaborating within an organization. Instead of replacing teamwork, AI begins to emulate it.</p><h4>The Importance of Workflow</h4><p>At the heart of every multi-agent system lies a <strong>workflow</strong>.</p><p>A workflow defines how tasks move between agents, when decisions are made, which agent should act next, and how information flows throughout the system.</p><p>Without a structured workflow, multiple AI agents simply become isolated models operating independently.</p><p>With a well-designed workflow, however, these agents can coordinate efficiently, share context, adapt to changing situations, and solve increasingly sophisticated business problems.</p><p>Modern frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, and Microsoft’s Agent Framework are making it easier for developers to design these collaborative workflows. Instead of building isolated AI applications, organizations can now create intelligent pipelines where specialised agents work together from start to finish.</p><h4>Why Businesses Are Paying Attention</h4><p>Businesses are beginning to realise that the true value of AI lies not only in generating content but in automating entire business processes.</p><p>Consider a customer service operation.</p><p>Instead of one chatbot answering every question, multiple agents can collaborate:</p><ul><li>A customer intent agent identifies the user’s request.</li><li>A retrieval agent searches company knowledge bases.</li><li>A policy agent checks organizational rules.</li><li>A reasoning agent determines the most appropriate solution.</li><li>A communication agent records the interaction of future learning and compliance.</li><li>A monitoring agent records the interaction for future learning and compliance.</li></ul><p>The customer experiences one seamless conversation, while behind the scenes several AI agents are collaborating in real time.</p><p>The same principle can be applied to healthcare, finance, cybersecurity, education, logistics, manufacturing, legal services, and almost every knowledge-intensive industry.</p><p>Rather than automating individual tasks, organizations can automate complete workflows.</p><h4>Automation Becomes Intelligent</h4><p>Traditional automation follows predefined rules.</p><p>If X happens, perform Y.</p><p>While this works well for repetitive processes, it struggles when situations become uncertain or require reasoning.</p><p>Multi-agent systems introduce a different form of automation.</p><p>Each agent can observe, reason, communicate, and adapt before decising on the next action. Instead of rigid automation, organizations gain intelligent automation capable of responding to changing circumstances.</p><p>This shift represents one of the biggest transformation in enterprise AI.</p><p>Rather than replacing humans, AI increasingly acts as a collaborative workforce that supports decision-making, reduces repetitive effort, and accelerates business operations.</p><h4>Challenges We Should Not Ignore</h4><p>Despite their enormous potential, multi-agent systems introduce new challenges.</p><p>As more AI agents collaborate, organizations must answer important questions:</p><ul><li>Which agent should make the final decision?</li><li>How should conflicting recommendations be resolved?</li><li>How do agents establish trust with one another?</li><li>Who is accountable when something goes wrong?</li><li>How can organizations ensure transparency and governance across the entire workflow?</li></ul><p>These questions become particularly important in regulated industries where decisions can directly affect people’s lives, finances, or safety.</p><p>Building more intelligent AI is only part of the solution.</p><p>Building <strong>responsible AI ecosystems </strong>is equally important.</p><h3>Looking Ahead</h3><p>We are entering a new era of AI where intelligence is no longer measured solely by the capabilities of a single model. Instead, it is defined by how effectively multiple specialised agents collaborate to achieve shared goals.</p><p>The organisations that succeed will not simply deploy more AI , they will design better <strong>frameworks</strong>, more efficient <strong>workflows</strong>, and trustworthy <strong>governance mechanisms</strong> that enable AI agents to work together safely and effectively.</p><p>Multi-agent AI systems are more than a technological trend. They represent a fundamental shift in how businesses will automate processes, make decisions, and deliver value over the coming decade.</p><p>As researchers, developers, and business leaders continue to explore this exciting field, one thing is becoming increasingly clear:</p><blockquote><strong>The future of enterprise AI is not one intelligent assistant , it is an intelligent team of AI agents working together.</strong></blockquote><p><em>What excites you most about Multi-Agent AI Systems? Is it workflow automation, intelligent collaboration, or the opportunity to build more autonomous enterprises? I’d love to hear your thoughts in the comments.</em></p><p>If you are interested in conversations around AI, Generative AI, and the future of digital marketing, let’s connect on LinkedIn and continue exchanging ideas and insights.</p><p>Connect in LinkedIn: <a href="https://www.linkedin.com/in/sangitapokhrel/">Sangita Pokhrel</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9744fefe6f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Rise of AI in Marketing: From Automation to Intelligent Creativity]]></title>
            <link>https://medium.com/@sangitapokhrel911/the-rise-of-ai-in-marketing-from-automation-to-intelligent-creativity-c118eac97ce4?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/c118eac97ce4</guid>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[ai-generated-content]]></category>
            <category><![CDATA[gen-ai-in-marketing]]></category>
            <category><![CDATA[ai-in-marketing]]></category>
            <category><![CDATA[marketing-automation]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 21:07:25 GMT</pubDate>
            <atom:updated>2026-03-08T21:07:25.813Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jj5VdFugnc3Bjdj6yArQXw.png" /><figcaption>(Image Generated by AI)</figcaption></figure><p>Artificial Intelligence is no longer a futuristic concept in marketing; it is already reshaping how brands communicate, create content, analyze customers, and make decisions. What once required large teams of marketers, analysts, and creative professionals can now be augmented, accelerated, and sometimes entirely generated by AI systems.</p><p>From recommendation engines and predictive analytics to generative AI models that produce videos, captions, and entire marketing campaigns, AI is rapidly transforming marketing from a human-driven discipline into a human-AI collaborative ecosystem.</p><blockquote>The real question today is no longer ‘<em>Will AI change marketing?</em>’ <em>but rather</em> ‘How should markets adapt to an AI-driven world?”</blockquote><h4><strong>The Evolution of Marketing Technology</strong></h4><p>Marketing has always evolved alongside technology.</p><p>The early digital marketing era focused primarily on websites, email campaigns, and basic analytics. Marketers relied on tools that provided descriptive insights, such as which customers clicked, which emails were opened, and how many visitors landed on a webpage.</p><p>The next phase introduced <strong>data-driven marketing</strong>, where large datasets enable marketers to understand customer behavior better. Machine learning algorithms began helping organisations identify patterns, segment customers, and predict purchasing behaviour.</p><p>Today, we are entering the <strong>Generative AI era</strong>, where artificial intelligence is not only analysing data but also creating content, designing campaigns, and interacting with customers in real time.</p><p>This shift fundamentally changes the role of marketing professionals. Instead of manually producing every piece of content, marketers are increasingly becoming strategists, curators, and orchestrators of AI-powered systems.</p><h4>How AI is Transforming Marketing</h4><p>AI technologies are being integrated into nearly every stage of the marketing pipeline. These transformations can be broadly categorised into several key areas.</p><ol><li><strong>Hyper-Personalization</strong></li></ol><p>One of the most powerful applications of AI in marketing is personalization at scale.</p><p>Traditional marketing campaigns often targeted broad audience segments, but AI enables companies to tailor messages for individual customers. By analysing behavioural data, browsing history, purchase patterns, and engagement signals, AI systems can predict what customers want even before they explicitly express it.</p><p>Streaming platforms such as Netflix and e-commerce companies like Amazon demonstrate how recommendation systems powered by machine learning can drive user engagement and sales.</p><p>In marketing, hyper-personalisation means that two customers visiting the same website may see entirely different product recommendations, advertisements, or promotional messages based on their preferences and behaviours.</p><p><strong>2. Predictive Analytics and Customer Insights</strong></p><p>AI allows marketers to move beyond reactive decision-making toward predictive strategies.</p><p>Predictive models analyse historical data to forecast future outcomes, such as customer churn, purchase likelihood, and campaign performance. This allows companies to optimise marketing resources and target customers at the right moment.</p><p>For example, predictive analytics can help businesses identify:</p><ul><li>Customers are most likely to convert</li><li>Users at risk of abandoning a service</li><li>Optimal timing for promotional offers</li><li>Emerging market trends</li></ul><p>Instead of guessing which campaign might succeed, marketers can now rely on data-driven predictions supported by machine learning models,</p><p><strong>3. AI-Generated Content</strong></p><p>Generative AI has introduced a major shift in content creation.</p><p>Large Language Models (LLMs) such as GPT-based systems can produce blog posts, product descriptions, email campaigns, and social media captions within seconds. Meanwhile, image and video generation models can create visuals, advertisements, and promotional videos.</p><p>This dramatically reduces the time required to produce marketing assets.</p><p>However, AI -generated content does not eliminate the role of human creativity. Instead, it acts as a creative accelerator, allowing marketers to experiment with multiple variations of content quickly and efficiently.</p><p>For example, a marketing team could generate:</p><ul><li>Multiple social media captions</li><li>Alternative product description</li><li>AI-generated promotional images and videos</li><li>Personalised email campaigns</li></ul><p>The final selection and strategic alignment still depend on human judgment.</p><p><strong>4. Conversational Marketing and AI Assistants</strong></p><p>AI-powered chatbots and conversational agents are becoming central to modern customer engagement.</p><p>Instead of waiting for human customer support, customers can now interact with intelligent virtual assistants capable of answering questions, recommending products, and guiding them through purchasing decisions.</p><p>These systems operate 24/7 and can handle thousands of interactions simultaneously.</p><p>Recent advancements in natural language processing allow chatbots to understand context, sentiment, and intent, making conversations increasingly natural and useful.</p><p>In marketing contexts, conversational AI can support:</p><ul><li>Lead generation, customer support, product recommendations, and personalized shopping experiences</li></ul><p><strong>5. Marketing Automation and Campaign Optimization</strong></p><p>AI is also transforming how marketing campaigns are designed and optimized.</p><p>Marketing automation platforms now incorporate machine learning algorithms that continuously analyze campaign performance and adjust parameters such as targeting, budget allocation, and messaging.</p><p>This allows campaigns to evolve dynamically based on real-time data.</p><p>For example, AI systems can automatically test multiple variations of advertisements (A/B testing), identify the most effective version, and allocate more budget to high-performing campaigns.</p><p>The result is more efficient marketing strategies with improved return on investment.</p><h3>Challenges and Ethical Considerations</h3><p>Despite its transformative potential, AI in marketing raises several challenges that organizations must address.</p><h4>1. Data Privacy</h4><p>AI systems rely heavily on user data to deliver personalized experiences. However, collecting and processing large amounts of personal data raises privacy concerns and regulatory challenges.</p><p>Regulations such as GDPR emphasize the need for transparency, consent, and responsible data usage.</p><h4>2. Bias and Fairness</h4><p>AI models are trained on historical data, which may contain biases. If not carefully monitored, these biases can influence marketing decisions, potentially leading to unfair targeting or exclusion of certain demographic groups.</p><p>Responsible AI development requires continuous monitoring, auditing, and ethical oversight.</p><h4>3. Authenticity and Trust</h4><p>As AI-generated content becomes more widespread, maintaining authenticity becomes increasingly important.</p><p>Consumers value genuine brand voices and human connection. Over-reliance on automated content may risk reducing trust if audiences perceive communications as artificial or impersonal.</p><p>The future of marketing therefore, lies not in replacing human creativity but in balancing AI efficiency with authentic storytelling.</p><h4>The Future of AI-Driven Marketing</h4><p>The next generation of marketing will likely be shaped by <strong>multimodal AI systems</strong> capable of generating and integrating text, images, video, and audio simultaneously.</p><p>Advanced AI architectures such as Retrieval-Augmented Generation (RAG) are also emerging as powerful tools for ensuring that AI-generated marketing content remains factually accurate and aligned with brand knowledge.</p><p>In this emerging landscape, marketers will increasingly collaborate with AI systems to:</p><ul><li>Generate personalized multimedia campaigns</li><li>Retrieve brand knowledge and product information</li><li>Create interactive and immersive marketing experiences</li><li>Deliver content tailored to individual users in real time</li></ul><p>Rather than replacing marketers, AI will expand their capabilities, enabling them to focus on strategy, innovation, and customer relationships.</p><h3>Final Thoughts</h3><blockquote>Artificial Intelligence is transforming marketing from a reactive, intuition-driven discipline into a data-driven, intelligent, and highly personalized ecosystem.</blockquote><blockquote>Organizations that embrace AI strategically will gain significant competitive advantages through improved customer insights, efficient content production, and optimized marketing campaigns.</blockquote><blockquote>However, the most successful marketing strategies will not be fully automated. Instead, they will combine human creativity, ethical responsibility, and AI-powered intelligence.</blockquote><blockquote>The future of marketing is not human versus machine , it is human creativity amplified by artificial intelligence.</blockquote><ol><li>What do you think about the future of marketing in the age of AI?</li><li>Do you see AI as a powerful opportunity, a challenge, or both?</li></ol><blockquote>Feel free to share your thoughts in the comments and join the discussion.</blockquote><p>If you are interested in conversations around AI, Generative AI, and the future of digital marketing, let’s connect on LinkedIn and continue exchanging ideas and insights.</p><p>Connect in LinkedIn: <a href="https://www.linkedin.com/in/sangitapokhrel/">Sangita Pokhrel</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c118eac97ce4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parsing Word, CSV, Excel, JSON, and SQL Data for Retrieval-Augmented Generation (RAG)]]></title>
            <link>https://medium.com/@sangitapokhrel911/parsing-word-csv-excel-json-and-sql-data-for-retrieval-augmented-generation-rag-a0798b8d5405?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0798b8d5405</guid>
            <category><![CDATA[wordparser]]></category>
            <category><![CDATA[json-parser]]></category>
            <category><![CDATA[split-text]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[excelparser]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 21:32:19 GMT</pubDate>
            <atom:updated>2025-11-23T21:32:19.278Z</atom:updated>
            <content:encoded><![CDATA[<p>Retrieval-Augmented Generation (RAG) systems enhance LLMs by feeding them external data, but that data often lives in varied formats like Word documents, spreadsheets, JSON APIs, or databases. Before an LLM can use this knowledge, we need to <strong>parse and prepare those files</strong> into a chunked, metadata-rich text format that’s easy to embed and retrieve. In this post, we’ll walk through how to ingest real-world data sources (DOCX, CSV, XLSX, JSON, SQL) into a RAG pipeline. We have already written about <a href="https://medium.com/@sangitapokhrel911/different-methods-to-read-pdf-files-in-langchain-5d547206bcef">Pdf Files</a>. We’ll cover why parsing is critical, tackle format-specific challenges, demo 2–3 Python tools for each (including LangChain &amp; LlamaIndex loaders), and show code snippets (with outputs) along the way. We’ll also discuss best practices for chunking and metadata, common pitfalls (like flattening nested JSON or parsing tables), and how to integrate the processed data into a RAG workflow. By the end, you’ll have both an in-depth understanding and a concise summary cheat-sheet for parsing these common file types for RAG.</p><h4>Why File Parsing is Essential in RAG Pipelines</h4><p>In a RAG pipeline, <em>document loaders</em> (parsers) are the foundation, they take files in various formats and convert them into plain text that an LLM can work with. If data isn’t loaded and cleaned properly, the AI may miss key information or return wrong answers. Real-world files can be messy (e.g. a DOCX with complex formatting or a CSV with inconsistent entries), so robust parsing ensures we extract useful content and structure while discarding noise. Key reasons to prioritize parsing include:</p><p>In short, parsing is the first step that organizes a “messy desk” of data so your AI assistant can find what it needs. Below is a typical ingestion flow in a RAG pipeline:</p><p><em>Ingestion stage of a RAG pipeline: raw data from various sources (documents, databases, etc.) is</em> <em>parsed and cleaned</em> <em>into text, then</em> <em>split into chunks</em> <em>with metadata and</em> <em>embedded</em> <em>into a vector store for semantic search.</em></p><ul><li><strong>File ➔ Parsing:</strong> Extract text and structure from the source file (e.g. DOCX, CSV).</li><li><strong>Cleaning ➔ Chunking:</strong> Remove noise (formatting, irrelevant text), then break the text into manageable <em>chunks</em> (e.g. paragraphs or rows) for embedding.</li><li><strong>Embedding ➔ Vector Store:</strong> Convert each chunk into an embedding vector and store it along with metadata. This vector database enables semantic retrieval of relevant chunks at query time.</li></ul><p>With the overall pipeline in mind, let’s dive into format-specific techniques and tools.</p><h3>1. Parsing Word Documents (DOCX)</h3><p>Word documents often contain knowledge bases, policies, reports, or other unstructured text that we want to query in a RAG application. For example, a company might have manuals or FAQs in .docx files that an internal QA bot should leverage.</p><p>DOCX files can have complex formatting, headings, lists, tables, footnotes, images, which purely textual extraction might flatten or omit. Important context (like a heading title) could be lost if not handled. Ensuring that <strong>all textual content</strong> (including headers/footers or text in tables) is extracted is key. We also need to watch out for artifacts like soft hyphens or special characters. Another challenge is dividing the content logically (e.g. by paragraphs or sections) for chunking.</p><p><strong>Tools &amp; Libraries:</strong> Several Python libraries can extract text from Word documents, each with pros/cons:</p><ol><li><strong>python-docx:</strong> A popular library to read and write DOCX files. It gives structured access to document elements (paragraphs, runs, tables).</li></ol><p><strong><em>Pros:</em> </strong>Can selectively extract or iterate through content (paragraph by paragraph), preserve some structure, and doesn’t require Word to be installed.</p><p><strong><em>Cons:</em></strong> Requires a couple dependencies (e.g. lxml) and by default only pulls main body text, headers, footers, and some list formatting might be missed unless you handle them separately. It also doesn’t extract images (only text).</p><p><strong>2. docx2txt:</strong> A lightweight, pure-python utility specifically for extracting text (and images) from .docx.</p><p><strong><em>Pros</em></strong><em>:</em> Very easy to use (essentially one function call), and it <strong>captures links, headers, footers, and even images</strong> (images are saved separately). It tends to concatenate all text into a single string.</p><p><strong><em>Cons</em></strong><em>:</em> Loses fine-grained structure, e.g. all paragraphs get joined (with newlines), and you can’t easily tell a heading from body text except by formatting cues. Numbered or bulleted lists may lose their numbering or indentation.</p><p><strong>3. LangChain Document Loaders (Word):</strong> LangChain provides loaders like Docx2txtLoader and UnstructuredWordDocumentLoader for Word files</p><p><strong><em>Pros</em></strong><em>:</em> These integrate with RAG pipelines out-of-the-box, returning Document objects with text and metadata. The Unstructured loader uses the <strong>Unstructured</strong> library to chunk the document into elements (paragraphs, titles, etc.) if desired. This can preserve more structure, e.g. you can get a list of elements like “Title: …” and “Paragraph: …” instead of one blob.</p><p><strong><em>Cons</em></strong><em>:</em> The Unstructured approach has external dependencies and can be heavy. If used in “single” mode, it will output a single text like docx2txt (no column/row context). Additional setup may be needed for Unstructured. If using Docx2txtLoader, it shares the same limitations as docx2txt discussed above.</p><p><strong>Example: Using python-docx:</strong> Below we use python-docx to read a .docx and print the combined text of all paragraphs:</p><pre>from docx import Document<br><br>doc = Document(&quot;example.docx&quot;)  # Load the Word file<br>full_text = &quot;\n&quot;.join(paragraph.text for paragraph in doc.paragraphs)<br>print(full_text[:100])  # Print first 100 characters<br># Output (example): &quot;ACME Corp Policy Handbook\nIntroduction\nACME Corp is committed to ... &quot;</pre><p><em>Pros:</em> We can access each paragraph (paragraph.text) and handle them as needed (e.g., keep headings separate or inspect doc.tables).</p><p><em>Cons:</em> By default this misses header/footer text and text inside tables. We’d have to manually extract those (e.g., doc.sections[0].header or iterate doc.tables), which complicates things.</p><p><strong>Example: Using docx2txt:</strong> With docx2txt, extracting all text is straightforward:</p><pre>import docx2txt<br><br>text = docx2txt.process(&quot;example.docx&quot;)<br>print(text[:100])<br># Output (example): &quot;ACME Corp Policy Handbook\nIntroduction\nACME Corp is committed to ... [image1.png] ... &quot;</pre><p>This returns the full document text in one string. You might see placeholders for images (e.g. “[image1.png]”) and all content (headers, footers) in sequence. It’s great for quick ingestion, indeed, <em>“python-docx2txt… can also extract text from links, headers and footers… and images.</em></p><p><strong>Example: Using LangChain Loader:</strong> If you’re using LangChain, you can load a Word document directly into Documents:</p><pre>from langchain.document_loaders import Docx2txtLoader<br>loader = Docx2txtLoader(&quot;example.docx&quot;)<br>docs = loader.load()<br>print(docs[0].page_content[:100])<br># Output: &quot;ACME Corp Policy Handbook\nIntroduction\nACME Corp is committed to ... &quot;</pre><p>LangChain’s loader above just wraps docx2txt (as shown by the identical output). Alternatively, using the Unstructured loader:</p><pre>from langchain.document_loaders import UnstructuredWordDocumentLoader<br>docs = UnstructuredWordDocumentLoader(&quot;example.docx&quot;, mode=&quot;elements&quot;).load()<br>print(len(docs), &quot;elements&quot;)<br># Output (example): 10 elements<br>print(docs[0].page_content)  <br># e.g., &quot;ACME Corp Policy Handbook&quot;  (first element, perhaps a title)</pre><p>With mode=&quot;elements&quot;, you would get a list of document chunks – each might be a title, a paragraph, etc., instead of one big text. Each chunk comes with metadata like category (e.g. “Title” or “Paragraph”) and possibly language or other info. This is useful if you want to preserve that a line was a heading (you could store that in metadata, or use it in chunk text). The trade-off is an extra step to recombine or handle multiple chunks, whereas the “single” mode (or Docx2txtLoader) gives one document.</p><h3>2. Parsing CSV Files</h3><p>CSV (Comma-Separated Values) files are essentially tables of data. They appear everywhere, exports from databases or spreadsheets, lists of records (e.g. product catalogs, user lists), survey results, etc. In RAG, CSVs can provide structured facts that an LLM can retrieve. For example, you might have a CSV of employee information, and you want the RAG system to answer “Who is the manager of the Sales department?” by looking up that table.</p><p><strong>Challenges:</strong> CSVs are simpler than Word docs, but they pose their own issues. A CSV might have <strong>dozens of columns</strong>, some of which aren’t relevant or are hard to interpret out of context. Data types can vary (dates, numbers, text) and might need formatting. <strong>Parsing pitfalls</strong> include fields containing commas or newlines (which can break naive split logic, though the CSV format handles it if properly read with a library). Also, very <strong>large CSVs</strong> (thousands of rows) are problematic to embed in entirety, you may need to filter or chunk them. Another consideration: a single row on its own might be meaningless unless we include the column names as context. For instance, a row “Alice, 30, Engineering” needs headers (“Name: Alice, Age: 30, Department: Engineering”) to make sense to the LLM.</p><p><strong>Tools and Libraries</strong></p><ul><li><strong>csv module (built-in):</strong><br> Lightweight, no installation needed, can stream large files line-by-line. But it’s low-level, so you must manually loop and format rows</li><li><strong>pandas:</strong><br> Reads CSVs into DataFrames easily (pd.read_csv). Great for data cleaning, filtering, and handling missing values. But large files can use a lot of memory, and you still need to format output yourself.</li><li><strong>LangChain CSVLoader:</strong><br> Automatically converts each row into a Document object with “column: value” text, ideal for RAG pipelines. But it treats each row separately and loads the whole file (via pandas), so huge files may need custom chunking.</li><li><strong>Other options:</strong><br> You can load CSVs with pandas and feed them into frameworks like <strong>LlamaIndex</strong>. For very large datasets, tools like <strong>Apache Arrow</strong> or <strong>Dask</strong> exist but aren’t needed for most standard use cases.</li><li>Additionally, if you are using frameworks like LlamaIndex, you might just use pandas to read CSV then feed it to LlamaIndex’s ingestion utilities (like creating documents or using a SimpleDirectoryReader on the file — which might just treat it as text). There’s also Apache Arrow or Dask for big data, but for beginner/mid-level scope, the above tools suffice.</li></ul><p><strong>Example: Using </strong><strong>csv module:</strong> Here’s how to read a CSV and turn each row into a text string:</p><pre>import csv<br><br>with open(&quot;employees.csv&quot;, newline=&#39;&#39;) as f:<br>    reader = csv.DictReader(f)  # reads header row as field names<br>    for row in reader:<br>        text = &quot; | &quot;.join(f&quot;{key}: {value}&quot; for key, value in row.items())<br>        print(text)<br>        break  # print only the first row for demo<br># Output (example): &quot;Name: Alice | Age: 30 | Department: Engineering&quot;</pre><p>In this snippet, csv.DictReader gives each row as a dict mapping column names to values, then we join them into a single line. We use &quot; | &quot; as a separator for clarity (commas could be confused with the CSV delimiter itself). The output shows the first row with each column labeled. This is exactly the context an LLM needs – <em>documents like these give the LLM the context to understand the meaning behind data</em> (column labels with values). Without the keys, “Alice 30 Engineering” might confuse the model.</p><p><strong>Example — Using pandas:</strong> We can achieve something similar (all rows as text) with pandas:</p><pre>import pandas as pd<br><br>df = pd.read_csv(&quot;employees.csv&quot;)<br>for i, row in df.iterrows():<br>    text = &quot; | &quot;.join(f&quot;{col}: {row[col]}&quot; for col in df.columns)<br>    print(text)<br>    break  # first row for demo<br># Output: &quot;Name: Alice | Age: 30 | Department: Engineering&quot;</pre><p>This produces the same result. Pandas also has a convenient df.to_string() method to get a pretty-printed table in text form, but that tends to align columns in fixed widths. For RAG, it’s usually better to have the “Column: Value” format as above, so that if a chunk is retrieved, the model clearly sees the field labels (even if spacing is lost). We could also use df.to_dict(orient=&quot;records&quot;) to get a list of dictionaries and then format similarly.</p><p>If the CSV is large and you don’t want to load all at once, you can iterate in chunks with pandas (pd.read_csv(..., chunksize=1000)) or stream with the csv module as shown. Consider filtering or selecting only relevant columns before converting to text (e.g. if some columns are irrelevant or highly numeric, maybe exclude them to save token space).</p><p>, but that tends to align columns in fixed widths. For RAG, it’s usually better to have the “Column: Value” format as above, so that if a chunk is retrieved, the model clearly sees the field labels (even if spacing is lost). We could also use df.to_dict(orient=&quot;records&quot;) to get a list of dictionaries and then format similarly.</p><p>If the CSV is large and you don’t want to load all at once, you can iterate in chunks with pandas (pd.read_csv(..., chunksize=1000)) or stream with the csv module as shown. Consider filtering or selecting only relevant columns before converting to text (e.g. if some columns are irrelevant or highly numeric, maybe exclude them to save token space).</p><p><strong>Example: Using LangChain CSVLoader:</strong> With LangChain, it’s very straightforward:</p><pre>from langchain.document_loaders import CSVLoader<br><br>loader = CSVLoader(file_path=&quot;employees.csv&quot;)<br>docs = loader.load()<br>print(docs[0].page_content)<br># Output: &quot;Name: Alice, Age: 30, Department: Engineering&quot;</pre><p>By default, each docs[i] corresponds to one CSV row. The content is a comma-separated string of Column: Value pairs (notice it used commas and newline or space as separator – the exact format may be &quot;, &quot; between pairs as above). The loader also automatically includes the CSV file path in metadata, and you can specify parameters like source_column to use a particular column as the Document.metadata[&#39;source&#39;] (for instance, using an “ID” or name column as the source label).</p><h3>3. Parsing Excel Files (XLSX)</h3><p>Excel files often contain multi-sheet business data, tables, reports, summaries, and charts, that you may want to query in a RAG system.</p><p><strong>Challenges:</strong></p><ul><li><strong>Multiple sheets:</strong> Treat each sheet separately and keep sheet names in metadata.</li><li><strong>Complex layout:</strong> Merged cells, formulas, and mixed sections don’t convert cleanly to text.</li><li><strong>Large size:</strong> Slow to read; must chunk by rows or sections.</li><li><strong>Losing structure:</strong> If not labeled properly, cell values lose column/row context.</li></ul><blockquote><strong>Tools and Libraries</strong></blockquote><ul><li><strong>pandas + openpyxl:</strong> Best for sheets that behave like tables; easy to load per sheet. Doesn’t preserve formatting and may show NaNs for merged areas.</li><li><strong>openpyxl (direct):</strong> More control when you need to inspect merged cells or unusual layouts.</li><li><strong>LangChain Excel loaders:</strong></li></ul><p><em>-UnstructuredExcelLoader:</em> Easy, but may output flattened text with little structure.</p><ul><li><em>StructuredExcelLoader (community):</em> Preserves exact sheet structure with cell labels (A1, B1, etc.)great for accuracy but verbose.</li><li><strong>Example: Using pandas:</strong> Suppose we have an Excel file with two sheets: “Employees” and “Departments” as in the earlier example. We’ll load each sheet and output rows similarly to CSV:</li></ul><pre>import pandas as pd<br><br>xls = pd.ExcelFile(&quot;company_data.xlsx&quot;)<br>for sheet_name in xls.sheet_names:<br>    df = pd.read_excel(xls, sheet_name=sheet_name)<br>    for _, row in df.iterrows():<br>        text = &quot; | &quot;.join(f&quot;{col}: {row[col]}&quot; for col in df.columns)<br>        print(f&quot;[{sheet_name}] &quot; + text)<br>    break  # demo: break after first sheet for brevity<br># Output example:<br># [Employees] Name: John Doe | Department: Sales | Salary: 50000<br># [Employees] Name: Jane Smith | Department: Marketing | Salary: 55000</pre><p>Here we prefixed the text with the sheet name in brackets for clarity. The output shows two rows from the “Employees” sheet, each labeled with headers. We would do the same for “Departments”. In practice, you might want to keep each sheet’s data separate (e.g., treat “Employees” records and “Departments” records as different sets, since queries might be specific). Attaching the sheet name either in the text or at least in metadata ({&quot;sheet&quot;: &quot;Employees&quot;}) is important for context. For instance, if a user asks “What is the salary of Jane Smith?”, the chunk with “Name: Jane Smith | ... Salary: 55000” is relevant – and we know it’s from the Employees sheet.</p><p><strong>Example: LangChain UnstructuredExcelLoader:</strong> Using the unstructured loader in element mode:</p><pre>from langchain.document_loaders import UnstructuredExcelLoader<br><br>loader = UnstructuredExcelLoader(&quot;company_data.xlsx&quot;, mode=&quot;elements&quot;)<br>docs = loader.load()<br>print(len(docs), &quot;documents loaded&quot;)<br># Output: e.g. &quot;4 documents loaded&quot;<br>for d in docs:<br>    print(d.metadata.get(&quot;page_name&quot;), d.metadata.get(&quot;category&quot;))<br># Possible output:<br># &quot;Employees&quot; Title<br># &quot;Employees&quot; Table<br># &quot;Departments&quot; Title<br># &quot;Departments&quot; Table</pre><p>As this hypothetical output suggests, the loader might produce a document for the sheet title and one for the sheet’s table content for each sheet. Each doc’s page_name metadata holds the sheet name, and category might be &#39;Title&#39; or &#39;Table&#39;. The page_content of the &#39;Table&#39; docs would be the textual table (likely with columns and rows separated by newlines). For example, docs[1].page_content might contain something like:</p><pre>Team  Location  Stanley Cups  <br>Blues STL       1  <br>Flyers PHI      2  <br>Maple Leafs TOR 13</pre><p><strong>Example: StructuredExcelLoader output:</strong> To illustrate the structured approach, here’s an example output for a tiny spreadsheet from the StructuredExcelLoader docs:</p><pre>SHEET: &quot;Employees&quot;<br><br>ROW 1:<br>CELL A1: Name<br>CELL B1: Department<br>CELL C1: Salary<br><br>ROW 2:<br>CELL A2: John Doe<br>CELL B2: Sales<br>CELL C2: 50000<br><br>ROW 3:<br>CELL A3: Jane Smith<br>CELL B3: Marketing<br>CELL C3: 55000</pre><p><em>Example of structured text output for an Excel sheet, preserving cell positions.</em> Each sheet becomes a document with every cell labeled. An LLM reading this would clearly know, for instance, that “John Doe” is in the Name column (A2 corresponds to Name header at A1). The downside is that it’s quite verbose, imagine 100 rows of this. It can blow up the token count. In many cases, a slightly less granular approach (like just “Name: John Doe, Department: Sales, Salary: 50000”) is sufficient. Nonetheless, this approach is useful if relative positions or multiple tables need to be distinguished.</p><p><strong>Best Practices:</strong></p><ul><li><strong>Treat each sheet separately:</strong><br> Store sheet name in metadata so context isn’t lost.</li><li><strong>Always include headers:</strong><br> Keep column names with each row. If a sheet has no clear headers, create your own labels.</li><li><strong>Skip non-useful content:</strong><br> Ignore empty sheets, notes, charts, or irrelevant cells.</li><li><strong>Use metadata wisely:</strong><br> Include file name, sheet name, and optional row/cell info for traceability.</li><li><strong>Chunk sensibly:</strong><br> For table-like sheets, use one row per chunk. For narrative or mixed layouts, chunk by logical sections (e.g., a text block, a table, etc.).</li><li><strong>Watch out for pitfalls:</strong></li><li>Merged cells → fill down or add context manually.</li><li>Hidden sheets → exclude if not needed.</li><li>Formulas → you get stored values, which is usually fine.</li><li><strong>Optional:</strong> Convert simple sheets to CSV for easier ingestion.</li></ul><h3>4. Parsing JSON Data</h3><p>JSON is common for structured/semi-structured data (APIs, product info, logs, FAQs, nested categories). In RAG, you may need to turn JSON objects into readable text chunks.</p><p><strong>Key Challenges:</strong></p><ul><li><strong>Nested structure:</strong> Needs flattening, raw JSON is hard for LLMs.</li><li><strong>Deciding what to index:</strong> Skip irrelevant fields (IDs, flags).</li><li><strong>Large files:</strong> Break into smaller documents; use streaming for huge JSON.</li><li><strong>Maintaining context:</strong> Always include key names so values aren’t ambiguous.</li></ul><p><strong>Tools &amp; Approaches:</strong></p><ul><li><strong>Python json module:</strong> Simple and flexible; you manually extract needed fields.</li><li><strong>pandas (json_normalize):</strong> Quick flattening for record-like JSON.</li><li><strong>LangChain JSONLoader:</strong> Pulls specific elements using JQ; gives one Document per JSON element.</li><li><strong>Custom flattening:</strong> Useful for irregular or deeply nested structures.</li></ul><p><strong>Example: Using </strong><strong>JSONLoader (LangChain):</strong> Suppose we have a JSON file products.json with content: {&quot;products&quot;: [ {&quot;id&quot;: 1, &quot;name&quot;: &quot;Widget&quot;, &quot;price&quot;: 25, &quot;details&quot;: {&quot;color&quot;: &quot;red&quot;, &quot;in_stock&quot;: true}}, {...} ] }. We want each product as a document:</p><pre>from langchain.document_loaders import JSONLoader<br><br>loader = JSONLoader(file_path=&quot;products.json&quot;, jq_schema=&quot;.products[]&quot;)<br>docs = loader.load()<br>print(docs[0].page_content)<br># Output (example): &#39;{&quot;id&quot;: 1, &quot;name&quot;: &quot;Widget&quot;, &quot;price&quot;: 25, &quot;details&quot;: {&quot;color&quot;: &quot;red&quot;, &quot;in_stock&quot;: true}}&#39;</pre><p>Now docs[0] contains the JSON string for the first product. This is fine but for better LLM utilization, we might transform that content. For instance, we could post-process each docs[i].page_content by loading it back to Python dict and formatting as a nicer text. Alternatively, we could have provided text_content=False when loading (so that it keeps it as Python objects in metadata), but it’s easier to just manually format:</p><pre>import json<br>record = json.loads(docs[0].page_content)<br>text = f&#39;Name: {record[&quot;name&quot;]}\nPrice: ${record[&quot;price&quot;]}\nColor: {record[&quot;details&quot;][&quot;color&quot;]}\nIn stock: {record[&quot;details&quot;][&quot;in_stock&quot;]}&#39;<br>docs[0].page_content = text<br>print(docs[0].page_content)<br># Output:<br># Name: Widget<br># Price: $25<br># Color: red<br># In stock: True</pre><p>This yields a human-friendly text representation of the JSON object. We incorporated nested fields (details.color etc.) into key: value lines. You would do this for each doc. This manual step is needed because, as noted, <em>“complex or nested JSON needs careful [extraction] to get the right data”</em> – an automated loader might not present it in the ideal way for an LLM.</p><p><strong>Example: Custom parsing with Python </strong><strong>json:</strong> Without LangChain, you could do:</p><pre>import json<br><br>with open(&quot;products.json&quot;) as f:<br>    data = json.load(f)<br>documents = []<br>for item in data[&quot;products&quot;]:<br>    text = f&#39;Name: {item[&quot;name&quot;]}, Price: ${item[&quot;price&quot;]}, Color: {item[&quot;details&quot;][&quot;color&quot;]}, In stock: {item[&quot;details&quot;][&quot;in_stock&quot;]}&#39;<br>    documents.append(text)<br>    print(text)<br>    break  # first item<br># Output: &quot;Name: Widget, Price: $25, Color: red, In stock: True&quot;</pre><p>This is straightforward since we knew the structure (a list under “products”). If the JSON were more irregular, we might need to recursively iterate. Another scenario: JSON Lines (.jsonl), where each line is a JSON object. In that case, you can read line by line and parse each to an object, treating each as a record.</p><p><strong>Best Practices:</strong></p><ul><li><strong>One document per logical item:</strong> e.g., each object in a list = one chunk.</li><li><strong>Flatten nested fields:</strong> Make all important values explicit (e.g., education.college: X University).</li><li><strong>Readable lists:</strong> Convert arrays into clear comma-separated or bulleted lists.</li><li><strong>Use metadata:</strong> File name, IDs, categories, or paths.</li><li><strong>Chunk long text fields:</strong> If text inside JSON is very long, split it but keep object metadata.</li></ul><h3>5. Parsing SQL Databases</h3><p>Use SQL data in RAG by exporting table rows as readable text chunks. Useful for customer records, products, transactions, etc. SQL dump files (.sql) need to be restored or parsed before extracting data.</p><p><strong>Key Challenges:</strong></p><ul><li><strong>Relational links:</strong> Data is in multiple tables; foreign keys need joining to produce meaningful text (e.g., show “Engineering” instead of dept_id=5).</li><li><strong>Large volume:</strong> Databases can be huge, avoid embedding massive tables; focus on important/semantic fields.</li><li><strong>Stale data:</strong> Static embeddings get outdated when DB updates.</li><li><strong>Dump files:</strong> Must restore or parse INSERT statements; they’re not data-ready.</li></ul><p><strong>Tools:</strong></p><ul><li><strong>sqlite3 / DB connectors / SQLAlchemy:</strong> Run SELECT queries and fetch rows.</li><li><strong>pandas.read_sql:</strong> Good for DataFrame workflows</li><li><strong>LlamaIndex DatabaseReader / LangChain SQL tools:</strong> Mostly for dynamic queries, but can also pull tables for indexing.</li></ul><p><strong>Example: Using SQLite (Python):</strong> Let’s say we have a SQLite database company.db with an employees table.</p><pre>import sqlite3<br><br>conn = sqlite3.connect(&quot;company.db&quot;)<br>cursor = conn.cursor()<br>cursor.execute(&quot;SELECT name, age, department FROM employees;&quot;)<br>rows = cursor.fetchall()<br>for row in rows[:2]:<br>    text = f&quot;Name: {row[0]}, Age: {row[1]}, Department: {row[2]}&quot;<br>    print(text)<br># Output (example):<br># Name: Alice, Age: 30, Department: Engineering<br># Name: Bob, Age: 25, Department: Sales</pre><p>This prints two example rows as text. In practice, you’d loop all rows and create Document chunks similarly. If the table had foreign keys (e.g., department was an ID referencing a departments table), you have a choice: do a join in the SQL query to get the actual name (preferred), or index the ID (less useful for natural questions). It’s usually better to join and have human-readable info in the text chunk. For example, use SELECT e.name, e.age, d.dept_name FROM employees e JOIN departments d ON e.dept_id = d.id; so that the chunk contains &quot;Department: Engineering&quot; instead of &quot;Department: 5&quot;.</p><p>If connecting to a different database (say PostgreSQL), you’d use that DB’s connector or an SQLAlchemy engine in place of the sqlite3 connection, but the concept remains: execute a SELECT, get rows.</p><p><strong>Example: Creating Documents from SQL with LangChain/LlamaIndex:</strong> While not demonstrating full code, here’s a conceptual flow:</p><ul><li>Use LangChain’s CSVLoader or DataFrameLoader after pulling data from SQL. For instance, if rows is a list of tuples as above, you could turn it into a DataFrame df = pd.DataFrame(rows, columns=[&quot;Name&quot;,&quot;Age&quot;,&quot;Department&quot;]) and then do docs = DataFrameLoader(df).load(). This would yield docs like &quot;Name: Alice\nAge: 30\nDepartment: Engineering&quot; for each row (assuming DataFrameLoader splits by row with one doc per row).</li><li>LlamaIndex’s DatabaseReader could be invoked as:</li></ul><pre>from llama_hub import SimpleDatabaseReader<br>reader = SimpleDatabaseReader(sql_url=&quot;sqlite:///company.db&quot;)<br>documents = reader.load_data(table_names=[&quot;employees&quot;])</pre><ul><li>This might return the entire table content as one or multiple Document nodes (depending on implementation). You’d likely still need to chunk them.</li></ul><p>For direct RAG indexing, manual control is usually fine.</p><h3>Final Project Using LangChain and LlamaIndex</h3><p>After parsing and chunking the data from Word, CSV, Excel, JSON, and SQL sources, the next steps are <strong>embedding these chunks</strong> and using them in an LLM chain for question answering. The high-level process is similar regardless of source: you have a list of text chunks (with metadata) that represent your knowledge base. You embed each chunk into a vector store. Then, at query time, you embed the query, find similar chunks, and feed them (plus possibly their metadata) into the LLM prompt.</p><p>Let’s outline integration using two popular frameworks: <strong>LangChain</strong> and <strong>LlamaIndex</strong>.</p><blockquote><strong>Using LangChain:</strong> LangChain makes it easy to go from documents to answers:</blockquote><ol><li><strong>Load documents:</strong> Use the Document Loaders we discussed (or your own code) to get a list of Document objects. For example, docs = CSVLoader(&quot;employees.csv&quot;).load() or for a custom parse, you’d create Document(page_content=text, metadata=meta) for each chunk.</li></ol><p><strong>2. Split documents (if not already chunked):</strong> For sources like Word docs or long text fields, you may need to chunk further. LangChain provides text splitters such as RecursiveCharacterTextSplitter which splits by tokens/characters (honoring sentence boundaries, etc.), or MarkdownHeaderTextSplitter if you have structured text with headings. For our cases, we mostly already chunked by row/paragraph, so a splitter might not be needed except for very large paragraphs.</p><pre>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)<br>split_docs = splitter.split_documents(docs)</pre><p>If docs are already small (like one row each), this will either leave them as is or break any that exceed 500 characters.</p><p><strong>3. Embed and store:</strong> Choose an embedding model (e.g., OpenAI’s text-embedding-ada-002 via OpenAIEmbeddings, or Cohere, etc.) and a vector store (FAISS for local in-memory, or Pinecone/Chroma for more persistent storage).</p><pre>from langchain.embeddings import OpenAIEmbeddings<br>from langchain.vectorstores import FAISS<br><br>embeddings = OpenAIEmbeddings()  # assumes API key is set<br>vector_store = FAISS.from_documents(split_docs, embedding=embeddings)</pre><p>This generates embeddings for each document chunk and stores them in a FAISS index. Metadata from the Document objects will be kept alongside vectors.</p><p><strong>4. Build Retrieval QA chain:</strong> LangChain offers RetrievalQA or similar chain that wraps the retrieval + LLM call.</p><pre>from langchain.llms import OpenAI<br>from langchain.chains import RetrievalQA<br>qa_chain = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type=&quot;stuff&quot;, retriever=vector_store.as_retriever())</pre><p>Here, .as_retriever() gives a retriever interface for the vector store (with default top-k settings, etc.). chain_type=&quot;stuff&quot; means it will stuff all retrieved chunks into the prompt (there are other options like map-reduce, refine, but “stuff” is straightforward for smaller numbers of docs).</p><p><strong>5. Ask questions:</strong></p><pre>query = &quot;Who is the manager of the Engineering department?&quot;<br>result = qa_chain.run(query)<br>print(result)<br># Output (example): &quot;The manager of the Engineering department is Alice.&quot;</pre><p>The chain will embed the query, find relevant chunks (e.g., maybe the “Department: Engineering, Manager: Alice” row from a CSV or JSON we indexed), and then the LLM composes the answer using that context.</p><p>The above pipeline would work similarly regardless of the original file type, because by this point, everything is just chunks of text in the vector store. The main difference format-to-format is how you parse and chunk.</p><blockquote><strong>Using LlamaIndex (GPT Index):</strong> LlamaIndex provides a slightly different abstraction. You would:</blockquote><ol><li><strong>Create Documents/Nodes:</strong> Use LlamaIndex’s readers or just manually create Document objects from text. For example:</li></ol><pre>from llama_index import Document, VectorStoreIndex, StorageContext<br>documents = [Document(text_chunk, metadata=meta) for text_chunk, meta in list_of_chunks]</pre><p><strong>2. Construct an index:</strong></p><pre>index = VectorStoreIndex.from_documents(documents)</pre><p>By default this will use a default embeddings model (often OpenAI) and a default vector store (in-memory). You can configure it to use external vector DBs like Qdrant, Chroma, etc., by providing a StorageContext with the desired VectorStore.</p><p><strong>3. Query the index:</strong></p><pre>query_engine = index.as_query_engine()<br>response = query_engine.query(&quot;Who is the manager of the Engineering department?&quot;)<br>print(response)<br># Output: &quot;The manager of the Engineering department is Alice.&quot;</pre><p>LlamaIndex will retrieve relevant nodes and by default synthesize an answer (it does a similar stuffing of context and uses an LLM to generate a response). LlamaIndex also supports more complex query types, e.g., you could do a structured query against the index or get source citations easily.</p><p><strong>Sources:</strong></p><ol><li>Marimuthu, S. <em>Document Loaders: Feeding Data into RAG</em> (Medium, 2025) — Explanation of document loaders and examples for CSV, JSON, etc.<a href="https://medium.com/@msriarunm/document-loaders-feeding-data-into-rag-91e3ff36ff60#:~:text=Document%20loaders%20take%20your%20files,no%20data%20to%20work%20with">medium.com</a><a href="https://medium.com/@msriarunm/document-loaders-feeding-data-into-rag-91e3ff36ff60#:~:text=Loaders%20are%20the%20foundation%20of,Here%E2%80%99s%20why%20they%E2%80%99re%20key">medium.com</a><a href="https://medium.com/@msriarunm/document-loaders-feeding-data-into-rag-91e3ff36ff60#:~:text=CSV%20files%20are%20like%20spreadsheets%2C,the%20total%20sales%20for%202024%3F%E2%80%9D">medium.com</a><a href="https://medium.com/@msriarunm/document-loaders-feeding-data-into-rag-91e3ff36ff60#:~:text=">medium.com</a></li><li>LangChain Documentation — Microsoft Word and Excel loaders, demonstrating use of docx2txt and Unstructured loaders<a href="https://docs.langchain.com/oss/python/integrations/document_loaders/microsoft_word#:~:text=Using%20Docx2txt">docs.langchain.com</a><a href="https://docs.langchain.com/oss/python/integrations/document_loaders/microsoft_word#:~:text=from%20langchain_community">docs.langchain.com</a><a href="https://docs.langchain.com/oss/python/integrations/document_loaders/microsoft_excel#:~:text=The%20,setting%20up%20required%20system%20dependencies">docs.langchain.com</a><a href="https://docs.langchain.com/oss/python/integrations/document_loaders/microsoft_excel#:~:text=%5BDocument%28page_content%3D%27Stanley%20Cups%27%2C%20metadata%3D%7B%27source%27%3A%20%27.%2Fexample_data%2Fstanley,cups.xlsx%27%2C%20%27file_directory">docs.langchain.com</a></li><li>Stack Overflow — Discussion on extracting text from .docx, noting python-docx limitations and docx2txt ability to get headers/footers<a href="https://stackoverflow.com/questions/25228106/how-to-extract-text-from-an-existing-docx-file-using-python-docx#:~:text=guerda%20Over%20a%20year%20ago">stackoverflow.com</a><a href="https://stackoverflow.com/questions/25228106/how-to-extract-text-from-an-existing-docx-file-using-python-docx#:~:text=28">stackoverflow.com</a></li><li><em>langchain-excel-loader</em> Project Description (PyPI, 2025) — Highlights the need to preserve structure in Excel parsing and shows an example of labeled cell output<a href="https://pypi.org/project/langchain-excel-loader/#:~:text=is%20by%20using%20the%20Unstructured,This%20has%20two%20disadvantages">pypi.org</a><a href="https://pypi.org/project/langchain-excel-loader/#:~:text=The%20StructuredExcelLoader%20will%20create%20separate,documents%20for%20each%20sheet">pypi.org</a></li><li>amazee.io Blog — <em>Data Pipelines for RAG</em> (2024) — Overview of RAG pipeline stages (ingestion, chunking, embedding) and importance of cleaning data<a href="https://www.amazee.io/blog/post/data-pipelines-for-rag/#:~:text=1">amazee.io</a><a href="https://www.amazee.io/blog/post/data-pipelines-for-rag/#:~:text=Whether%20taken%20from%20a%20webpage,sections%20irrelevant%20to%20your%20needs">amazee.io</a></li><li>LangChain CSV Loader Example — Shows row-wise document creation and importance of including column titles<a href="https://pypi.org/project/langchain-excel-loader/#:~:text=Output%3A">pypi.org</a></li><li>LangChain JSON Loader Example — Demonstrates using jq to extract records and notes on handling nested JSON<a href="https://medium.com/@msriarunm/document-loaders-feeding-data-into-rag-91e3ff36ff60#:~:text=,for%20basic%20text">medium.com</a><a href="https://medium.com/@msriarunm/document-loaders-feeding-data-into-rag-91e3ff36ff60#:~:text=Challenges%3A%20Complex%20or%20nested%20JSON,to%20get%20the%20right%20data">medium.com</a></li></ol><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0798b8d5405" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Different Methods to Read PDF Files in LangChain]]></title>
            <link>https://medium.com/@sangitapokhrel911/different-methods-to-read-pdf-files-in-langchain-5d547206bcef?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d547206bcef</guid>
            <category><![CDATA[documentloaderandreader]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langchaincommunity]]></category>
            <category><![CDATA[rag-application]]></category>
            <category><![CDATA[pdfloader]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Sun, 16 Nov 2025 20:05:28 GMT</pubDate>
            <atom:updated>2025-11-16T20:09:39.250Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9E1OmjfgO7H2x_L9V6OZag.png" /></figure><p>Connect with me and follow my journey: <a href="https://www.linkedin.com/in/sangitapokhrel/">Linkedin</a></p><p>Handling PDF documents is one of the most common tasks when building AI-powered applications, whether you’re creating chatbots, document search engines, summarization tools, or knowledge-retrieval systems. Langchain, as a framework for building LLM-enabled apps, provided several simple and powerful ways to load PDF files and prepare them for downstream tasks.</p><p>In this article, we explore all the major methods available in LangChain for reading PDFs, explain how each loader works, when to use which method, and provide working code examples.</p><p>First thing to understand is why we need PDF loaders?</p><p>Well, the answer is simple: most PDFs are not straightforward text files. They may include complex layouts, tables, images, multi-column formats, and even scanned pages (OCR required).</p><p>Langchain PDFLoaders abstracts away these complexities using <strong><em>document loaders</em></strong><em>, </em>transforming PDF content into a uniform format (usually a list of Document objects) that can be embedded, indexed, or queried.</p><p>The library will be based on this:</p><pre>from langchain_community.document_loaders import (Please insert the Type of PDf Loader)</pre><h3>Different Types of PDF Loaders:</h3><h3><strong>1. PyPDFLoader</strong></h3><p>PyPDFLoader is the default and most widely used loader in LangChain. It extracts text from PDF pages using the pypdf Python package. It is a tool in the LangChain framework that loads content from PDF files by using the pypdf library under the hood. It works by <strong>converting each page of the PDF into a separate Document object</strong>, which contains the <strong>text content</strong> and <strong>metadata</strong> like the page number and source file. You should use PyPDFLoader for text-based PDFs when you need to load them into a program to extract the text for tasks like building a chatbot, performing text analysis, or using them in a Retrieval-Augmented Generation (RAG) pipeline.</p><p>How it works:</p><p><strong>1.1 Initialization</strong>: You create an instance of PyPDFLoader, passing the path to the PDF file as an argument.</p><p><strong>1.2 Page extraction</strong>: The loader reads the PDF and uses the pypdf library to extract the text from each page.</p><p><strong>1.3 Document creation</strong>: Each page is converted into a Document object.</p><p><strong>1.4 Content and metadata</strong>: Each Document object has two main parts:</p><p><strong><em>page_content</em></strong>: A string containing the text from that specific page.</p><p><strong><em>metadata:</em></strong> A dictionary that includes the source file name and the page number, e.g., {‘page’: 0, ‘source’: ‘book.pdf’}.</p><p><strong>1.5 Loading</strong>: The <strong>load()</strong> method returns a list of these Document objects, with one document per page.</p><pre>from langchain_community.document_loaders import PyPDFLoader<br><br>loader = PyPDFLoader(&quot;sample.pdf&quot;)<br>documents = loader.load()      # Returns list of Document objects</pre><p>What we get here is: documents, which are a list of document objects. Each document consists of page_content(text) and metadata such as page number and file source, etc. Let’s see that here:</p><pre>print(&quot;Number of pages loaded:&quot;, len(documents))<br>print(&quot;Metadata on first page:&quot;, documents[0].metadata)<br>print(&quot;Page 1 text snippet:\n&quot;, documents[0].page_content[:500])</pre><p>I have uploaded a PDF document, which is a research paper named <a href="https://arxiv.org/html/2410.15944v1">sample.pdf</a>. The library extracted its text content, page by page. The output looks like:</p><pre>Number of pages loaded: 36<br>Metadata on first page: {&#39;producer&#39;: &#39;pdfTeX-1.40.25&#39;, &#39;creator&#39;: &#39;LaTeX with hyperref&#39;, &#39;creationdate&#39;: &#39;2024-10-22T01:56:19+00:00&#39;, &#39;author&#39;: &#39;&#39;, &#39;keywords&#39;: &#39;&#39;, &#39;moddate&#39;: &#39;2024-10-22T01:56:19+00:00&#39;, &#39;ptex.fullbanner&#39;: &#39;This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5&#39;, &#39;subject&#39;: &#39;&#39;, &#39;title&#39;: &#39;&#39;, &#39;trapped&#39;: &#39;/False&#39;, &#39;source&#39;: &#39;sample.pdf&#39;, &#39;total_pages&#39;: 36, &#39;page&#39;: 0, &#39;page_label&#39;: &#39;1&#39;}<br>Page 1 text snippet:<br> Developing Retrieval Augmented Generation<br>(RAG) based LLM Systems from PDFs: An Experience Report Ayman Asad Khan Tampere University ayman.khan@tuni.fi<br>Md Toufique Hasan Tampere University mdtoufique.hasan@tuni.fi Kai Kristian Kemell Tampere University kai-kristian.kemell@tuni.fi<br>Jussi Rasku Tampere University jussi.rasku@tuni.fi Pekka Abrahamsson Tampere University pekka.abrahamsson@tuni.fi<br>Abstract. This paper presents an experience report on the development of Retrieval Augmented Generation</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*O6WCVsahlsnBa4nD8-mc_g.png" /></figure><h3><strong>2. PyMuPDFLoader</strong></h3><p>The PyMuPDFLoader is a document loader within the LangChain framework that uses the high-performance <a href="https://pymupdf.readthedocs.io/">PyMuPDF</a> library to load and parse PDF documents. It is known for its speed and efficient handling of complex PDF layouts compared to other loaders, such as PyPDF. PyMuPDF is optimized for:</p><ul><li>High speed</li><li>Large files</li><li>Better handling of textual structure</li><li>Extracting embedded images if needed</li></ul><p>When you have long engineering manuals, research theses, legal documents, corporate policy books, or multi-hundred-page ebooks, it’s better to use this PDF loader.</p><pre><br>from langchain_community.document_loaders import PyMuPDFLoader<br><br>loader = PyMuPDFLoader(&quot;sample.pdf&quot;)<br>documents = loader.load()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*PBoue9nSB3SJOu-bCw-Ydg.png" /></figure><h3>3. PDFPlumber</h3><p><strong>PDFPlumber</strong> is a Python library designed to extract <strong>text</strong>, <strong>tables</strong>, and <strong>layout information</strong> from PDF files. It works best on <strong>digital PDFs</strong> where the text is already selectable (not scanned images).</p><p>In LangChain, the loader that uses this library is called <strong>PDFPlumberLoader</strong>. PDFPlumber is known for its <strong>precision</strong> with structured text and <strong>excellent table extraction</strong>, making it ideal for business and data-heavy documents. It can access layout details (word positions, lines, shapes).</p><pre><br>from langchain_community.document_loaders import PDFPlumberLoader<br><br>loader = PDFPlumberLoader(&quot;sample.pdf&quot;)<br>docs = loader.load()<br><br>print(docs[0].page_content[:200])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*fEDGLLHZOxeKEYaFb0R91g.png" /></figure><h3>4. PyPDFium2Loader</h3><p>Best for Slide decks, marketing PDFs, visually designed documents, a fast PDF rendering engine (pypdfium2)</p><pre>“If the PDF looks like a designed graphic or slide, try PyPDFium2Loader.”</pre><h4>Why this loader exists</h4><p>Some PDFs have a <em>bad or missing text layer</em>, for example:</p><ul><li>PowerPoint slides exported to PDF</li><li>Marketing brochures, flyers</li><li>Highly designed reports and posters</li><li>PDFs with lots of shapes, text boxes, and overlays</li></ul><p>Traditional text parsers (like PyPDFLoader) may return broken, jumbled text or even miss content.<br> Because PyPDFium2Loader works from <em>rendered pages</em>, it often recovers a more natural reading order for this kind of content. It can also extract <strong>images</strong> and be combined with OCR (e.g., Tesseract, RapidOCR) for image-based text.</p><p>I have uploaded this pdf file as <a href="https://www.nestle.com/sites/default/files/2025-02/2024-financial-statements-en.pdf">slides.pdf</a> to read the content using PyPDFium2Loader</p><pre>from langchain_community.document_loaders import PyPDFium2Loader<br><br>file_path = &quot;slides.pdf&quot;<br><br>loader = PyPDFium2Loader(file_path)<br>documents = loader.load()  # one Document per page by default<br><br>print(f&quot;Pages loaded: {len(documents)}&quot;)<br>print(&quot;\nFirst page preview:&quot;)<br>print(documents[0].page_content[:400].replace(&quot;\n&quot;, &quot; &quot;) + &quot;...&quot;)<br>print(&quot;\nMetadata:&quot;)<br>print(documents[0].metadata)</pre><p>This will:</p><ul><li>Render each page using pypdfium2</li><li>Extract text from the rendered output</li><li>Return a list of Document objects (usually one per page) with metadata like source and page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*-AmKlR3hbXCNo4HVQpEsNA.png" /></figure><h3><strong>5. UnstructuredPDFLoader</strong></h3><p><strong>UnstructuredPDFLoader</strong> is a LangChain loader powered by the <strong>unstructured</strong> library. It is designed to extract <strong>clean, structured text</strong> from PDFs, especially <strong>messy</strong>, <strong>mixed-content</strong>, or <strong>scanned</strong> PDFs.</p><p>Unlike other loaders, it doesn’t just extract text; it also <strong>classifies</strong>, <strong>segments</strong>, and <strong>cleans</strong> the content.</p><p>Most PDF loaders simply extract text. <strong>UnstructuredPDFLoader both extracts <em>and organizes</em> text. </strong>It tries to identify elements such as Titles</p><ul><li>Headings</li><li>Paragraphs</li><li>Tables</li><li>Lists</li><li>Captions</li><li>Page breaks</li></ul><p>It creates structured elements rather than raw text, which is powerful for <strong>RAG pipelines</strong> and <strong>NLP tasks</strong>.</p><pre>from langchain_community.document_loaders import UnstructuredPDFLoader<br><br>loader = UnstructuredPDFLoader(&quot;sample.pdf&quot;)<br>docs = loader.load()<br><br>print(&quot;First element:&quot;)<br>print(docs[0].page_content[:300])<br><br>print(&quot;\nMetadata:&quot;)<br>print(docs[0].metadata)</pre><h4>How It Works (Simple)</h4><ol><li>PDF page is converted into an image (optional, depending on strategy).</li><li>OCR is applied if needed (e.g., Tesseract, layoutparser).</li><li>Text is broken into meaningful segments.</li><li>Each segment becomes a structured “element.”</li><li>LangChain wraps each element into a Document.</li></ol><p>There are other PDF loaders as well, which in short is below:</p><blockquote><strong>PDFMinderLoader</strong>: PDFMinerLoader uses the long-standing PDFMiner engine, which excels with older PDFs or files that have unusual character encodings. While it&#39;s not as fast or modern as PyPDF or PyMuPDF, it often succeeds where other loaders fail, especially when dealing with legacy documents, quirky spacing, or PDFs generated by outdated systems.</blockquote><blockquote><strong>UnstructuredPDFLoader (OCR Mode): </strong>For scanned or image-only PDFs, the OCR version of UnstructuredPDFLoader extracts text using optical character recognition. This mode is essential for documents where no selectable text exists, such as scanned contracts, printed forms, government documents, or any PDF created from images, making it the go-to option for true “picture-only” files.</blockquote><blockquote><strong>Docling PDF Loader: </strong>The Docling-based PDF loader provides AI-powered, highly structured extraction, preserving elements like section hierarchy, headings, tables, lists, and more. This makes it ideal for academic papers, technical reports, and complex documents where maintaining the original structure improves downstream tasks such as summarisation or RAG-based retrieval.</blockquote><blockquote><strong>MathPixPDFLoader</strong>: MathPixPDFLoader is designed specifically for math-heavy or STEM documents, converting formulas, equations, and symbols into clean LaTeX representations. Regular PDF parsers struggle with mathematical content, but MathPix excels at accurately capturing complex expressions, perfect for scientific papers, textbooks, and engineering documents.</blockquote><blockquote><strong>AmazonTextractPDFLoader: </strong>This loader integrates with AWS Textract, a powerful cloud OCR and document analysis service that extracts text along with structured elements like tables, forms, and key–value pairs. It’s best suited for enterprise scenarios involving invoices, receipts, identity documents, and compliance paperwork that require high accuracy and well-defined data extraction.</blockquote><blockquote><strong>PyPDFDirectoryLoader: </strong>PyPDFDirectoryLoader is a convenience loader that automatically imports all PDFs within a folder, using PyPDF under the hood. It’s particularly useful when building large document datasets, organizational knowledge bases, or multi-file RAG systems, eliminating the need to manually loop through each file.</blockquote><p>LangChain also offers several niche loaders, such as those for Upstage, Zerox, and other cloud document-processing platforms, that cater to specialized workflows or enterprise-level use cases. These are useful when integrating with existing document intelligence services or when very high accuracy or custom processing is required.</p><p><em>As you experiment, try different loaders on the same PDF. The differences can be surprising, and sometimes the right loader makes all the difference.</em></p><p>Thank you for reading. I hope you liked it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d547206bcef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is Chunking in RAG (Retrieval-Augmented Generation) and Why It Matters for AI]]></title>
            <link>https://medium.com/@sangitapokhrel911/chunking-in-rag-the-secret-sauce-behind-smarter-ai-responses-71c99ef70f9a?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/71c99ef70f9a</guid>
            <category><![CDATA[why-chunking-matters]]></category>
            <category><![CDATA[chunking-in-rag]]></category>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[overlapping]]></category>
            <category><![CDATA[rag-application]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 00:11:36 GMT</pubDate>
            <atom:updated>2025-11-10T16:12:56.566Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYghZ2mdUPfihdmkLQB5jw.png" /></figure><p>If you’ve ever tried to build a <strong>Retrieval-Augmented Generation (RAG)</strong> system, where an AI model retrieves relevant context from your own documents before answering a question, you’ve likely come across the term <strong>“chunking.”</strong></p><p>At first glance, chunking sounds like a simple preprocessing task: just split a document into pieces, right? But in reality, chunking is one of the most critical design choices in a RAG pipeline. It directly influences retrieval accuracy, response coherence, and system efficiency.</p><p>In this post, we’ll explore:</p><ul><li>What chunking means and why it matters</li><li>Different types of chunking (with code examples and outputs)</li><li>How chunk size and overlap affect retrieval</li><li>How to choose the right chunking strategy for your RAG application</li></ul><p>Whether you’re a beginner experimenting with LangChain or a practitioner optimizing your enterprise RAG pipeline, this guide will make chunking <em>clear</em>.</p><p><strong>What is Chunking?</strong></p><p>In plain terms, <strong>chunking</strong> means dividing a long document into smaller, manageable pieces called <em>chunks</em>.</p><p>When a user asks a question, your RAG system searches for and retrieves the most relevant chunks, instead of the entire document and feeds them to the LLM (like GPT-4 or Claude) to generate a final answer.</p><p>So, what do we do? <br>We split documents into smaller pieces called chunks.</p><p>Each chunk is then:</p><ol><li><strong>Embedded</strong> into a numerical vector (using tools like OpenAI Embeddings or Sentence Transformers)</li><li><strong>Stored</strong> in a vector database (like Pinecone, FAISS, or Chroma)</li><li><strong>Retrieved</strong> based on similarity to the user query</li><li><strong>Passed</strong> to the LLM to generate the final answer</li></ol><p>In short, chunking helps your model <strong>remember smartly</strong> rather than remembering everything.</p><p><strong>Why Chunking Matters in RAG Systems</strong></p><p>A typical RAG workflow looks like this:</p><ol><li><strong>Text Ingestion</strong> — Collect raw documents.</li><li><strong>Chunking</strong> — Split them into smaller sections.</li><li><strong>Embedding</strong> — Convert each chunk into a vector representation.</li><li><strong>Retrieval</strong> — Search for relevant chunks using semantic similarity.</li><li><strong>Generation</strong> — Pass retrieved chunks to the LLM to form an answer.</li></ol><p>If your chunks are:</p><ul><li><strong>Too large</strong> → retrieval becomes less precise, and irrelevant content sneaks in.</li><li><strong>Too small</strong> → the model misses key context and produces incomplete answers.</li></ul><p>Chunking is therefore the <em>balance point; </em>it determines whether your RAG model feels <em>intelligent</em> or <em>confused</em>.</p><p><strong>Types of Chunking</strong></p><pre>sample_text = &quot;&quot;&quot; <br>Artificial Intelligence (AI) is revolutionizing industries across the world. <br>From healthcare to education, AI systems are improving efficiency and personalization. <br>However, challenges such as data privacy, bias, and interpretability remain key concerns. <br>Organizations are increasingly adopting AI-driven decision systems to stay competitive. <br>&quot;&quot;&quot;</pre><p><em>This sample text has been used while defining the different types of chunking.</em></p><ol><li><strong>Fixed-size Chunking</strong></li></ol><p>This is the simplest method. You split text based on <strong>a fixed number of characters or tokens</strong>.</p><pre>from langchain_text_splitters import CharacterTextSplitter<br><br>splitter = CharacterTextSplitter(<br>    separator=&quot;&quot;,           # force pure character slicing<br>    chunk_size=100,<br>    chunk_overlap=20<br>)<br>chunks = splitter.split_text(sample_text)<br><br>print(f&quot;Total chars: {len(sample_text)} | Chunks: {len(chunks)}&quot;)<br>for i, c in enumerate(chunks, 1):<br>    print(f&quot;\nChunk {i} (len={len(c)}):\n{c}\n&quot; + &quot;-&quot;*60)</pre><p>From the above code, below is the output where you can see 5 chunks has been taken from the sample text.</p><pre>Total chars: 348 | Chunks: 5<br><br>Chunk 1 (len=98):<br>Artificial Intelligence (AI) is revolutionizing industries across the world. <br>From healthcare to e<br>------------------------------------------------------------<br><br>Chunk 2 (len=100):<br>From healthcare to education, AI systems are improving efficiency and personalization. <br>However, cha<br>------------------------------------------------------------<br><br>Chunk 3 (len=100):<br>ation. <br>However, challenges such as data privacy, bias, and interpretability remain key concerns. <br>O<br>------------------------------------------------------------<br><br>Chunk 4 (len=100):<br>ain key concerns. <br>Organizations are increasingly adopting AI-driven decision systems to stay compet<br>------------------------------------------------------------<br><br>Chunk 5 (len=26):<br>stems to stay competitive.</pre><p>This method ignores sentence structure. It’s fast and easy but can <strong>split sentences awkwardly</strong>, which sometimes harms semantic coherence.</p><p><strong>When to use:</strong></p><ul><li>When your documents are well-structured (like logs, transcripts, or formatted reports)</li><li>When you need simple, fast splitting</li></ul><p>⚠ <strong>Limitation:</strong></p><ul><li>Breaks sentences mid-way</li><li>Can reduce retrieval accuracy if the cut happens in important sentences</li></ul><p><em>Tip:</em> For fixed-size splitting, always add a <strong>small overlap (10–20%)</strong> so the model retains continuity.</p><blockquote>We will talk about the <a href="http://overlapping/">overlapping </a>later in this blog.</blockquote><p><strong>2. Recursive Character Chunking</strong></p><p><strong>Recursive chunking</strong> is a smarter version of the fixed-size approach.<br> It splits text <strong>semantically</strong>, trying paragraph breaks, then sentences, then spaces, before resorting to arbitrary cuts.</p><pre>from langchain_text_splitters import RecursiveCharacterTextSplitter<br><br>splitter = RecursiveCharacterTextSplitter(<br>    chunk_size=200,<br>    chunk_overlap=50,<br>    separators=[&quot;\n\n&quot;, &quot;\n&quot;, &quot;.&quot;, &quot; &quot;, &quot;&quot;]<br>)<br>chunks = splitter.split_text(sample_text)<br><br>for i, c in enumerate(chunks):<br>    print(f&quot;Chunk {i+1}:\n{c}\n{&#39;-&#39;*60}&quot;)</pre><p>Output:</p><pre>Chunk 1:<br>Artificial Intelligence (AI) is revolutionizing industries across the world. <br>From healthcare to education, AI systems are improving efficiency and personalization.<br>------------------------------------------------------------<br>Chunk 2:<br>However, challenges such as data privacy, bias, and interpretability remain key concerns. <br>Organizations are increasingly adopting AI-driven decision systems to stay competitive.<br>------------------------------------------------------------</pre><p>Recursive chunking maintains <strong>natural sentence flow</strong> while keeping chunks within a size limit. This balance improves retrieval accuracy since each chunk holds complete thoughts.</p><p><strong>When to use:</strong></p><ul><li>Documents with variable structure, like articles, reports, research papers</li><li>When you want to preserve meaning and sentence integrity</li></ul><p><strong>Advantage:</strong></p><ul><li>Keeps related sentences together while controlling chunk size.</li><li>Provides better semantic quality for retrieval.</li></ul><p><strong>3. Token-based Chunking</strong></p><p>Some LLMs process text by <strong>tokens</strong> (units of words/subwords), not characters, such as GPT-4. A token roughly equals 4 characters or 0.75 words (depending on language). <strong>Token-based chunking</strong> ensures your chunks fit exactly within model token limits.</p><pre>from langchain_text_splitters import TokenTextSplitter<br>splitter = TokenTextSplitter(<br>    encoding_name=&quot;cl100k_base&quot;,<br>    chunk_size=40,<br>    chunk_overlap=10<br>)<br><br>chunks = splitter.split_text(sample_text)<br><br>print(f&quot;Total Chunks: {len(chunks)}\n{&#39;=&#39;*60}&quot;)<br>for i, c in enumerate(chunks, 1):<br>    print(f&quot;Chunk {i}:\n{c}\n{&#39;-&#39;*60}&quot;)</pre><p>Output:</p><pre>Total Chunks: 4<br>============================================================<br>Chunk 1:<br>Artificial Intelligence is transforming industries rapidly. From healthcare to finance, AI systems enhance<br>------------------------------------------------------------<br>Chunk 2:<br>efficiency and decision-making. Yet, privacy, bias, and transparency remain critical concerns that must<br>------------------------------------------------------------<br>Chunk 3:<br>be addressed. Organizations are deploying AI responsibly to gain competitive advantages while<br>------------------------------------------------------------<br>Chunk 4:<br>safeguarding users.<br>------------------------------------------------------------</pre><p>We used LangChain’s TokenTextSplitter to break a long text into smaller, overlapping chunks based on tokens (the units that language models process). This ensures that each chunk stays within a manageable size for an LLM while preserving context between segments, an essential step for tasks like summarization, retrieval, or embedding generation.</p><p><strong>When to use:</strong></p><ul><li>When using APIs with strict token limits (e.g., OpenAI, Anthropic)</li><li>When you want full control over the model’s input size</li></ul><p><em>Tip:</em> Use libraries like tiktoken for precise token counting before chunking.</p><p><strong>4. Semantic Chunking (Advanced)</strong></p><p>The most advanced approach, <strong>Semantic Chunking,</strong> uses embeddings to detect <strong>topic or meaning boundaries</strong>, splitting text where ideas change rather than based on size. It groups sentences that are conceptually related, regardless of length.</p><pre>from langchain_experimental.text_splitter import SemanticChunker<br>from langchain_openai import OpenAIEmbeddings<br><br>splitter = SemanticChunker(OpenAIEmbeddings())<br>chunks = splitter.split_text(sample_text)<br><br>for i, c in enumerate(chunks):<br>    print(f&quot;Semantic Chunk {i+1}:\n{c}\n{&#39;-&#39;*60}&quot;)</pre><p><em>Install: !pip install langchain-experimental langchain-openai</em></p><p>Output:</p><pre>Semantic Chunk 1:<br>Artificial Intelligence (AI) is revolutionizing industries across the world.<br>From healthcare to education, AI systems are improving efficiency and personalization.<br><br>Semantic Chunk 2:<br>However, challenges such as data privacy, bias, and interpretability remain key concerns.<br><br>Semantic Chunk 3:<br>Organizations are increasingly adopting AI-driven decision systems to stay competitive.</pre><p><strong>When to use:</strong></p><ul><li>For knowledge bases, FAQs, research papers, or long-form content</li><li>When accuracy and context quality matter more than speed</li></ul><p><strong>Benefit:</strong></p><ul><li>Groups conceptually similar sentences</li><li>Boosts retrieval accuracy by keeping related ideas together</li></ul><p>⚠<strong>Trade-off:</strong></p><ul><li>Computationally heavier (requires embeddings to be generated first)</li></ul><h4>Understanding Chunk Overlap</h4><p>Chunk overlap means <strong>repeating part of the previous chunk</strong> at the beginning of the next one. It ensures smooth context flow, preventing the loss of crucial information that spans chunk boundaries. For example:</p><pre>Chunk 1: &quot;AI is transforming industries across the world. From healthcare to education...&quot;<br>Chunk 2: &quot;From healthcare to education, AI systems are improving efficiency and personalization...&quot;</pre><p>Here, the overlapping sentence <em>“From healthcare to education”</em> preserves continuity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*9xTehHoDthfIXboLHHEhzQ.png" /><figcaption>How much overlap to use?</figcaption></figure><h3>Choosing the Right Chunking Strategy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*o64EaOoD0BDZAzAxCC_0_A.png" /></figure><p><em>No one-size-fits-all:</em> Always test your RAG retrieval accuracy after chunking; the “best” chunking method depends on your dataset’s structure and query style.</p><blockquote>Final Thoughts</blockquote><p>Chunking isn’t just a preprocessing step; it’s the <strong>backbone</strong> of effective RAG systems.</p><p>Good chunking means:</p><ul><li>Better retrieval accuracy</li><li>More coherent model responses</li><li>Faster and cheaper inference</li></ul><p>Think of it this way:</p><blockquote><em>“Chunking teaches your AI to remember wisely, not just to remember everything.”</em></blockquote><p>Experiment, measure, and iterate; your chunks will determine how smart your RAG truly feels.</p><p>References:</p><ul><li><a href="https://www.ibm.com/think/tutorials/chunking-strategies-for-rag-with-langchain-watsonx-ai">IBM: Chunking Strategies for RAG with LangChain and WatsonX</a></li><li><a href="https://stackoverflow.blog/2024/12/27/breaking-up-is-hard-to-do-chunking-in-rag-applications/">StackOverflow Blog: Breaking Up is Hard to Do — Chunking in RAG Applications</a></li><li><a href="https://priyalwalpita.medium.com/how-important-is-chunking-in-rag-the-foundation-of-effective-ai-systems-69c4983289fb">Priyal Walpita: How Important is Chunking in RAG?</a></li><li><a href="https://latenode.com/blog/ai-frameworks-technical-infrastructure/rag-retrieval-augmented-generation/rag-chunking-strategies-complete-guide-to-document-splitting-for-better-retrieval">Latenode: Complete Guide to RAG Chunking Strategies</a></li></ul><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p><strong><em>Thank you for reading till the end. If you like this blog, follow for more.</em></strong></p><p><strong><em>Get connected on LinkedIn: </em></strong><a href="https://www.linkedin.com/in/sangitapokhrel/"><strong><em>https://www.linkedin.com/in/sangitapokhrel/</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71c99ef70f9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Interconnected Evolution: Exploring AI, ML, GenAI, LLMs, and Agentic AI]]></title>
            <link>https://medium.com/@sangitapokhrel911/the-interconnected-evolution-the-intelligent-continuum-exploring-ai-ml-genai-llms-and-agentic-55ef60239793?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/55ef60239793</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Sun, 19 Oct 2025 11:59:18 GMT</pubDate>
            <atom:updated>2025-11-16T10:56:20.432Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FGyr4Cuskn399fNfEXOFaA.png" /><figcaption>Image created by Google AI Studio: Nano Banana</figcaption></figure><p>These technologies form a stack, not separate branches:</p><ul><li><strong>AI</strong> is the broadest category, any system exhibiting intelligence.</li><li><strong>Machine Learning</strong> is AI that learns from data.</li><li><strong>Generative AI</strong> is ML that creates new content.</li><li><strong>LLMs</strong> are generative AI systems using transformer architecture to understand and generate language.</li><li><strong>Agentic AI</strong> builds on LLMs, adding planning, tool use, and autonomous action.</li></ul><p>Each layer builds on the previous one, and progress at one level enables breakthroughs at higher levels. The emergence of powerful LLMs was necessary to make truly capable agentic systems possible.</p><p><strong>The Evolution of AI: From Machine Learning to Agentic Intelligence</strong></p><p>Artificial Intelligence has evolved from science fiction fantasy to an integral part of our daily lives. Yet many people use AI without truly understanding the distinctions between AI, Machine Learning, Generative AI, Large Language Models, and Agentic AI. These terms are often used interchangeably, but each represents a distinct layer of technological advancement. This blog explores each concept, traces their evolution, and demonstrates how they’re reshaping our world.</p><h3><strong>1. What Is Artificial Intelligence (AI)?</strong></h3><p>Artificial Intelligence (AI) is the branch of computer science that focuses on creating machines or software capable of performing tasks that normally require human intelligence, such as learning, reasoning, problem-solving, perception, understanding language, and decision-making.</p><p>In simple terms, <strong>AI is when computers start to “think” and “act” like humans, or at least mimic human intelligence.</strong></p><p>These processes include:</p><ol><li><strong>Learning</strong> — acquiring knowledge or rules from data</li><li><strong>Reasoning</strong> — using rules to reach conclusions or make decisions</li><li><strong>Perception</strong> — interpreting information from the world (e.g., images, sound)</li><li><strong>Language understanding</strong> — processing and generating human language</li><li><strong>Action</strong> — performing a task or response based on what’s learned</li></ol><p><strong>Simple Example</strong></p><p>When you use <strong>Google Maps</strong>, the app predicts the <strong>fastest route</strong> based on traffic data, that’s AI. <br> When <strong>Netflix</strong> recommends movies you might like, that’s AI. <br> When <strong>ChatGPT</strong> writes or answers questions, that’s AI.</p><p>All of these systems <em>analyze data, learn from it, and make intelligent decisions. </em>It’s the foundation of modern technologies like <strong>Machine Learning</strong>, <strong>Generative AI</strong>, and <strong>Agentic AI</strong>, which are just more advanced or specialized forms of artificial intelligence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*RNhDD9SbCBmrbKucGVdqNw.png" /><figcaption>Table: AI types based on capability</figcaption></figure><h3>2. <strong>What Is Machine Learning (ML)?</strong></h3><p><strong>Machine Learning (ML)</strong> is a branch of <strong>Artificial Intelligence (AI)</strong> that enables computers to <strong>learn from data</strong> and <strong>improve automatically</strong> through experience, <em>without being explicitly programmed.</em></p><p>In other words, instead of giving the computer step-by-step instructions, we give it <strong>data and examples</strong>, and it learns <strong>patterns</strong> or <strong>relationships</strong> on its own.</p><p>In simple terms, <strong>Machine Learning</strong> is the science of teaching computers to learn from data and make decisions or predictions without being explicitly told what to do.</p><p><strong>Simple Example:</strong></p><p>Let’s imagine you want to teach your email system to <strong>detect spam</strong>.</p><ul><li>In traditional programming (old AI), you’d write hundreds of rules like:</li></ul><p>If an email contains “win money,” mark it as spam.</p><ul><li>In Machine Learning, you simply feed the system <strong>thousands of examples</strong> of emails labeled “spam” and “not spam.” <br> The algorithm studies these and learns the <em>patterns</em> that distinguish one from the other, like suspicious words, sender reputation, and formatting.</li></ul><p>Over time, it becomes better and better at predicting whether a new email is spam, just like a student improving with practice.</p><p><strong>The ML process generally involves four key steps on how it works:</strong></p><ol><li><strong>Data Collection</strong> — Gather relevant data (e.g., emails, images, transactions).</li><li><strong>Training</strong> — Feed the data into an ML algorithm so it can learn patterns.</li><li><strong>Testing</strong> — Evaluate the model on new (unseen) data to check accuracy.</li><li><strong>Prediction</strong> — Use the model to make decisions or predictions in the real world.</li></ol><p><strong>Example Story: The Cat Detector</strong></p><p>A fun way to understand ML is through the famous <strong>“cat vs. dog” classifier</strong> story.</p><p>A programmer doesn’t teach the computer what a cat <em>is, </em>no rules like “cats have pointy ears.” <br> Instead, the system sees <strong>thousands of cat and dog images</strong>, each labeled correctly. <br> It then learns patterns in pixels, colors, and shapes.</p><p>After training, if you show it a new picture, it can say:</p><p>“That’s a cat — with 95% confidence.”</p><p>This process, learning from examples, is what makes ML so powerful.</p><h4>🧩 Types of Machine Learning</h4><p>Machine Learning can be broadly categorized into four main types: <strong>Supervised</strong>, <strong>Semi-Supervised</strong>, <strong>Unsupervised</strong>, and <strong>Reinforcement Learning.</strong> Each type differs in how it learns from data, the kind of feedback it receives, and the problems it solves.</p><h4><strong>1. Supervised Learning</strong></h4><p>In <strong>supervised learning</strong>, the model learns from <strong>labeled data, </strong>meaning that both the input (X) and the correct output (y) are known. The algorithm’s goal is to learn a mapping between the two so it can make accurate predictions for new, unseen data.</p><p>For example, a supervised model might learn to <strong>predict house prices</strong> based on features such as location, number of rooms, and area size. Common algorithms include <strong>Linear and Logistic Regression</strong>, <strong>Support Vector Machines (SVM)</strong>, <strong>Random Forests</strong>, <strong>XGBoost</strong>, and <strong>Convolutional Neural Networks (CNNs)</strong> for image-related tasks.</p><p>Supervised learning is widely used in <strong>spam detection</strong>, <strong>pricing models</strong>, <strong>medical diagnosis</strong>, and other prediction-based applications where labeled data is available.</p><h4>2. Semi-Supervised Learning</h4><p><strong>Semi-supervised learning</strong> lies between supervised and unsupervised learning. It uses a <strong>small amount of labeled data</strong> along with a <strong>large amount of unlabeled data</strong>. The labeled portion helps guide the model, while the unlabeled data helps it generalize and improve accuracy.</p><p>This approach is especially useful when labeling data is <strong>costly or time-consuming</strong>, as in medical imaging. For example, a system might train on <strong>500 labeled MRI scans</strong> and use <strong>20,000 unlabeled ones</strong> to enhance its disease detection capability.</p><p>Common algorithms include <strong>Self-training</strong>, <strong>Co-training</strong>, <strong>Label Propagation (Graph Methods)</strong>, and <strong>Consistency Regularization</strong> techniques like <strong>FixMatch</strong>. Semi-supervised learning is widely applied in <strong>speech recognition</strong>, <strong>medical diagnostics</strong>, and <strong>web content classification</strong>, where labeled data is limited but unlabeled data is abundant.</p><h4>3. Unsupervised Learning</h4><p>In <strong>unsupervised learning</strong>, the data has <strong>no labels</strong> — the system must find hidden patterns, structures, or relationships entirely on its own. The goal is not prediction, but <strong>discovery</strong>.</p><p>For example, an unsupervised algorithm can analyze customer purchase data and <strong>group shoppers into behavior clusters</strong>, helping marketers identify audience segments. Common algorithms include <strong>k-Means Clustering</strong>, <strong>DBSCAN</strong>, <strong>Principal Component Analysis (PCA)</strong>, and <strong>Autoencoders</strong>.</p><p>Unsupervised learning is often used for <strong>customer segmentation</strong>, <strong>anomaly detection</strong>, and <strong>dimensionality reduction, </strong>uncovering insights where predefined categories are unavailable.</p><h4>4. Reinforcement Learning</h4><p><strong>Reinforcement learning (RL)</strong> is quite different from the previous types. Instead of learning from labeled data, an RL model learns by <strong>interacting with an environment</strong> and receiving <strong>rewards or penalties</strong> for its actions. Its goal is to discover the best strategy or “policy” to maximize long-term rewards.</p><p>For instance, RL is used to <strong>train an AI agent to play video games like Atari, </strong>the model tries different actions, learns from feedback, and improves over time. Common RL algorithms include <strong>Q-Learning</strong>, <strong>Deep Q-Networks (DQN)</strong>, <strong>Proximal Policy Optimization (PPO)</strong>, and <strong>Soft Actor-Critic (SAC)</strong>.</p><p>Reinforcement learning powers applications in <strong>robotics</strong>, <strong>autonomous vehicles</strong>, <strong>game playing</strong>, and <strong>recommendation ranking</strong>, where decision-making through trial and feedback is key.</p><h3>3. <strong>Generative AI: When Machines Became Creators</strong></h3><p><strong>Generative AI (GenAI)</strong> refers to a branch of artificial intelligence that can <strong>generate new content, such as </strong>text, images, videos, music, or code, that mimics human creativity.</p><p>Unlike traditional AI, which only analyzes or classifies information, <strong>GenAI learns patterns from massive datasets</strong> and uses that knowledge to produce <em>original outputs.</em></p><p>In short, it doesn’t just <em>recognize</em> data, it <em>creates</em> it.</p><p><strong>Example: The Rise of Creative Machines</strong></p><ul><li><strong>DALL·E</strong> can generate an image from a simple sentence like <em>“A fox reading a book under the moonlight.”</em></li><li><strong>ChatGPT</strong> can write an article, a movie script, or a business proposal in seconds.</li><li><strong>Jukebox (by OpenAI)</strong> composes music in the style of any artist.</li></ul><p>Generative AI models learn by <em>training on enormous datasets</em> of human-created examples, then use probabilities to predict what “fits next.” <br> The result feels <em>astonishingly human.</em></p><p><strong>📖 Story: The Designer and Her Digital Muse</strong></p><p>A graphic designer named Amira once spent hours brainstorming logo ideas. <br> Now, she types a few prompts into <strong>Midjourney</strong>, explores dozens of AI-generated concepts, and refines the one that resonates most.</p><p>The AI doesn’t replace her, it amplifies her imagination. <br> She calls it her <em>digital muse.</em></p><p>That’s the magic of Generative AI: <strong>human creativity, supercharged by algorithms.</strong></p><h3><strong>4. Large Language Models (LLMs): The Brains Behind the Words</strong></h3><p><strong>Large Language Models (LLMs)</strong> are a specific type of Generative AI trained on <strong>massive text datasets, </strong>books, articles, websites, and conversations, to understand and generate human language.</p><p>They use a neural network architecture called the <strong>Transformer</strong> (introduced by Google in 2017) that captures <em>context</em> and <em>relationships</em> between words.</p><p>This enables them to:</p><ul><li>Understand natural language prompts</li><li>Generate coherent, contextually relevant responses</li><li>Translate, summarize, and even reason logically</li></ul><p><strong>How They Work (Simplified)</strong></p><p>LLMs work by predicting <strong>the next word</strong> in a sequence. <br> For example, given “The sky is…”, an LLM predicts “blue” based on learned probabilities from billions of examples.</p><p>The models have <em>parameters, </em>essentially adjustable weights, that store knowledge. <br> GPT-4, for instance, has <strong>over a trillion parameters</strong> fine-tuned to understand language, reasoning, and intent.</p><p><strong>Example: ChatGPT and the Power of Language</strong></p><p>When you chat with <strong>ChatGPT</strong>, you’re interacting with an LLM that:</p><ol><li>Understands your question,</li><li>Predicts the most likely and coherent answer,</li><li>Generates it in fluent, human-like language.</li></ol><p>It doesn’t “think” like a human, it doesn’t <em>feel</em> or <em>know, </em>but it’s remarkably good at <em>sounding</em> intelligent because it understands patterns in how humans express knowledge.</p><p><strong>Story: The Research Assistant That Never Sleeps</strong></p><p>A PhD student named Ravi uses ChatGPT to summarize hundreds of papers for his literature review. <br> Instead of spending months reading every detail, he gets concise summaries and thematic insights in hours.</p><p>He still does the critical analysis himself, but AI accelerates his research exponentially.</p><p>That’s the power of LLMs, <strong>knowledge retrieval at the speed of thought.</strong></p><h3>5. <strong>Agentic AI: When Machines Start to Act</strong></h3><p>If Generative AI <em>creates</em> and LLMs <em>understand</em>, then <strong>Agentic AI</strong> <em>acts.</em></p><p><strong>Agentic AI</strong> represents the next stage, AI systems that can <strong>autonomously plan, reason, and take actions</strong> to achieve goals without continuous human prompting.</p><p>These systems combine:</p><ul><li><strong>Reasoning (via LLMs)</strong></li><li><strong>Memory (to remember context and results)</strong></li><li><strong>Tool use (access to the web, APIs, or files)</strong></li><li><strong>Autonomy (self-directed task completion)</strong></li></ul><p>In essence, Agentic AI transforms a conversational model into an <strong>independent digital agent</strong> that can perform multi-step tasks.</p><p><strong>Example: AutoGPT and AI Assistants That Work for You</strong></p><p>Imagine you ask your AI:</p><p>“Research competitors, write a summary, and draft an email to my team.”</p><p>A basic LLM like ChatGPT can respond once. <br> But an <strong>Agentic AI system</strong> (like <strong>AutoGPT</strong>, <strong>LangChain agents</strong>, or <strong>OpenAI’s GPTs</strong>) can:</p><ol><li>Search online for competitors,</li><li>Gather key data,</li><li>Write the summary,</li><li>Draft the email,</li><li>Ask for your approval<br> and <strong>do it autonomously.</strong></li></ol><p>It’s not just replying — it’s <em>executing.</em></p><p><strong>Story: The Entrepreneur’s Digital Team</strong></p><p>Maya runs a small startup. She builds an <strong>Agentic AI system</strong> named “BizMate.” <br> It automatically:</p><ul><li>Tracks online trends,</li><li>Analyzes customer reviews,</li><li>Generates weekly marketing reports, and</li><li>Suggests ad campaign ideas.</li></ul><p>What once required three employees, now happens overnight.</p><p>Maya says,</p><p>“BizMate doesn’t replace people, it lets my team focus on strategy while AI handles the grind.”</p><p>That’s <strong>Agentic AI, </strong>AI that takes initiative, not just instructions.</p><p>What do you think about the following quote? Comment down.</p><blockquote><strong>“AI won’t replace people, but people who use AI wisely will shape the future.”</strong></blockquote><p>Thank you for reading till the end. Let’s get connected on LinkedIn: <a href="https://www.linkedin.com/in/sangitapokhrel/">https://www.linkedin.com/in/sangitapokhrel/</a></p><p>Follow me to see more blogs on AI, Gen AI and LLMs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55ef60239793" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retrieval-Augmented Generation (RAG): Architecture, Real-World Applications, and Future Directions]]></title>
            <link>https://medium.com/@sangitapokhrel911/retrieval-augmented-generation-rag-architecture-real-world-applications-and-future-directions-7ed88ef5e53e?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ed88ef5e53e</guid>
            <category><![CDATA[generation]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[retrieval]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[augmented]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Sun, 12 Oct 2025 12:19:14 GMT</pubDate>
            <atom:updated>2025-10-13T13:10:27.875Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Knowledge Problem in LLM: </strong>Imagine asking about a recent Black Friday purchase you now want to return, but you’re unsure about the policy. If you simply ask a general-purpose LLM (ChatGPT, Claude, Llama), you’ll likely get something generic:</p><p><strong>Without RAG</strong></p><p><strong><em>Customer: </em></strong>“What’s your return policy for items bought during the Black Friday sale?”<br> <strong><em>AI:</em></strong> “Generally, most companies offer a 30-day return, but policies may vary…”<br> <em>[Generic, unhelpful response]</em></p><p><strong>With RAG (LLM + Vector Database)</strong></p><p><strong><em>Customer:</em></strong> “What’s your return policy for items bought during the Black Friday sale?”</p><p><em>[RAG searches company policy database]</em><br> <strong><em>AI:</em></strong> “According to our current policy (Policy Doc v2.5, uploaded Jan 2023), Black Friday purchases have an extended 60-day return window until January 31. Items must be unused with original tags. Electronics have a 20-day return period due to rapid depreciation. Would you like me to start a return for a specific item?”</p><p>That leap in usefulness is what RAG delivers: the fluency of an LLM grounded in your latest, domain-specific knowledge rather than generic patterns or outdated training data.</p><p><strong>What RAG Is (in one Minute)</strong></p><p>Retrieval-Augmented Generation (RAG) is an advanced technique that enhances large language models (LLMs) by enabling them to pull in up-to-date, authoritative information from external data sources before generating their responses. Instead of solely relying on internal training data, RAG-equipped systems first retrieve relevant content such as policy manuals, customer records, or scientific articles then augment the model’s prompt with these findings, resulting in more accurate, relevant, and trustworthy responses.</p><h3><strong><em>RAG Architecture</em></strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*295OQIJYPKshAFKfbS5Buw.png" /><figcaption>Image Source: <a href="https://aws.amazon.com/what-is/retrieval-augmented-generation/">Retrieval Augmented Generation</a></figcaption></figure><p>A typical RAG system can be summarized in the following workfloe:</p><p>- <strong>User submits a prompt.</strong></p><p>- The system’s <strong>retriever</strong> model queries external or enterprise-specific sources (eg, vector databases, internal knowledge bases).</p><p>- <strong>Relevant documents are selected</strong> using embedding similarity or other ranking techniques.</p><p>- These documents are then fused with the original prompt (prompt augmentation).</p><p>- The <strong>LLM generates a response</strong> that combines newly retrieved knowledge and its internal data.</p><p>A modular RAG architecture often includes a chunking/data ingest step (breaking documents into retrievable chunks), a vector/indexing layer. A retrieval engine, prompt engineering, and (sometimes) reranking and filtering before final text output.</p><h3>The RAG Pipeline (5 practical stages)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*eDRqK3KWn16rjL8qxuZzTQ.png" /><figcaption>Image Source: <a href="https://www.capellasolutions.com/blog/rag-and-vector-search-better-together-for-ai?ref=chitika.com">Architecture of RAG</a></figcaption></figure><ol><li><strong>Document ingestion &amp; preprocessing</strong></li></ol><p>This is where your data journey begins. The goal is to take raw, messy documents like PDFs, FAQs, Manuals, or policy notes, or any form of data and make them clean, structured, and ready for search.</p><p><strong><em>What happens in this stage:</em></strong></p><p>- <strong>Collect and clean:</strong> Extract text from PDFs, emails, web pages, and databases while removing irrelevant formatting like headers, footers, and HTML tags.</p><p>- <strong>Chunking:</strong> Break large documents into smaller, meaningful text units (called chunks), typically 200–1000 tokens, so each piece can be searched efficiently.</p><p>- <strong>Overlap for context:</strong> Add 10–20% overlap between chunks to ensure continuity, preventing the model from missing information that spans across paragraph breaks.</p><p>- <strong>Add metadata: </strong>Attach information such as document title, source, version, author, publication date, access level etc. This metadata later helps in filtering and compliance.</p><p>LLM perform best when given concise, context-preserving snippets. Proper chunking and metadata ensure that RAG retrieves exactly the right passage, not an entire 30-page PDF.</p><blockquote><strong>Example:</strong><br> Imagine you’re indexing your company’s “Holiday Returns Policy (v3.5, updated Nov 2024).”</blockquote><p>You extract the following section:</p><p><em>“Black Friday purchases have a 60-day return window until January 31. Electronics must be returned within 15 days with original packaging and receipt.”</em></p><p>This becomes a <strong>chunk</strong> with metadata like:</p><pre>{<br>  &quot;source&quot;: &quot;Holiday_Returns_Policy_v3.5.pdf&quot;,<br>  &quot;section&quot;: &quot;Black Friday Exceptions&quot;,<br>  &quot;effective_date&quot;: &quot;2024-11-15&quot;,<br>  &quot;tags&quot;: [&quot;returns&quot;, &quot;black_friday&quot;, &quot;electronics&quot;]<br>}</pre><p>Now your database knows <em>where</em> this fact originated and <em>when</em> it was valid, which is crucial for accurate retrieval later.</p><p><strong>2. Embedding generation</strong></p><p>Once documents are chunked, they need to be transformed into numerical representations (vectors) that capture their meaning. This is done using embedding models.</p><p>What happened in this stage:</p><p>- Each chunk is converted into a high-dimensional vector (often 768–1536 dimensions).</p><p>- The vector encodes <em>semantic similarity chunks with related meanings are close together in this ‘embedding space’.</em></p><p>- <em>These vectors are stored alongside their metadata in preparation for fast retrieval.</em></p><p>Popular embedding models are OpenAI text-embedding-3-large, Cohere embed-v3, and Sentence-Transformers (open source). Embeddings let your AI understand ‘meaning’ instead of relying on exact word matches. For example, a query containing ‘refund’ can still find chunks about ‘returns’.</p><blockquote>Example:</blockquote><blockquote>The ‘Black Friday Policy’ paragraph is transformed into a vector, a list of floating-point numbers such as [0.22, -0.15, 0.73,…..]. When someone asks ‘Can I return electronics bought on Black Friday&gt;’, their query is also embedded. The two vectors end up close in space, signalling a strong match.</blockquote><p>3. <strong>Vector storage &amp; indexing</strong></p><p>These vectors now need a home, a database designed for high-speed similarity search.</p><p><em>What happens in this stage:</em></p><p>- Chunks and their embeddings are stored in a vector database like Pinecone, Weaviate, Qdrant, Chroma, or FAISS.</p><p>- These databases use algortihms such as HNSW (Hierarchical Navigable Small World) or IVF ( Inverted File Index) to perform lightning-fast lookups across millions of vectors.</p><p>- Metadata filters can limit searches by source, data, or access level (eg, internal vs public).</p><p>Efficient indexing ensures that when a user asks a question, the right information surfaces in milliseconds, even if your company stores millions of knowledge snippets.</p><blockquote>Example: Your indexed vector store contains 10,000 policy paragraphs. A search for ‘return policy after Black Friday’ instantly retrieves the chunk we prepared earlier, because its vector sits near the query’s vector in embedding space.</blockquote><p>4. <strong>Query Processing and Retrieval</strong></p><p>Now comes the retrieval moment, when a user’s question is matched against the knowledge base.</p><p><strong>What happens in this stage:</strong></p><p>1. The user’s query is embedded using the same embedding model as before.</p><p>2. The system performs a <strong>Top-K similarity search</strong> (typically K = 3–10) to find the closest-meaning chunks.</p><p>3. It can use <strong>hybrid search</strong>, combining vector similarity with traditional keyword search (BM25) to improve recall for numeric or exact-term queries.</p><p>4. Retrieved chunks are optionally <strong>re-ranked</strong> for precision using small transformer models (cross-encoders).</p><p>5. Filters ensure relevance, e.g., “only policies effective after 2024.”</p><p>This step determines what knowledge the LLM will see. Poor retrieval means poor answers, no matter how powerful the LLM.</p><blockquote>Example:User Query: ‘<strong>What’s the return window for Black Friday Electronics?</strong>’</blockquote><blockquote>- Vector Search finds the ‘<strong>Black Friday Exceptions’ paragraph</strong>.</blockquote><blockquote>- Keyword Search ensures ‘<strong>electronics</strong>’ appears in the text.</blockquote><blockquote>- Metadata filter limits results to the latest version (<strong>v3.5, 2024</strong>).</blockquote><blockquote>- Re-ranking pushes the chunk with both ‘<strong>Black Friday</strong>’ and ‘<strong>electronics</strong>’ to the top.</blockquote><p>The system now has the right evidence to pass the language model.</p><p><strong>5. Context-Augmented Generation</strong></p><p>Finally, the magic happens, the system fuses the retrieved evidence with the user’s question and lets the LLM craft a grounded response.</p><p><strong>What happens in this stage:</strong></p><ul><li>The retrieved text snippets are inserted into a structured <strong>prompt template</strong> along with the original query.</li><li>Clear <strong>instructions</strong> guide the LLM to use only that context, avoid speculation, and cite sources and say ‘I don’t know’ if the context is insufficient.</li><li>The LLM then generates the final, context-aware answer.</li></ul><p>· <strong>Prompt template (simplified):</strong></p><pre>You are an AI assistant. Use only the following context to answer.<br>If the answer is not in the context, say “I don’t know.”<br>Cite the source after each fact.</pre><pre>Context:<br>[Holiday Returns Policy v3.5 – Updated 2024-11-15]<br>&quot;Black Friday purchases have a 60-day return window until Jan 31...&quot;<br>Question:<br>What&#39;s your return policy for Black Friday purchases?</pre><pre>Generated Answer:<br>According to Holiday Returns Policy v3.5 (Updated Nov 2024), Black Friday <br>purchases can be returned within 60 days, until January 31.<br>Electronics must be returned within 15 days with original packaging and <br>receipt.</pre><p>The LLM is no longer “guessing.” It grounds every sentence in retrieved evidence. This eliminates hallucinations and enables traceable, transparent AI communication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*HPARqXyIpLT0VhDFWJEgIQ.png" /></figure><h3><strong>Real-World Case Studies (Cross-Sector)</strong></h3><p><strong>1.</strong> <strong>Higher Education — Harvard Business School (ChatLTV)</strong></p><p>A Harvard Business School lecturer created an AI teaching assistant called <strong>ChatLTV, which stands for “Chat Learning Through Venture.”</strong> Built with Slack integration, it uses RAG to index course materials, lecture transcripts, case studies, and prior Q&amp;A. When students ask, “What framework did we use for early-stage valuation?”, the system retrieves the exact paragraph from class notes and responds instantly, grounded in the course’s curated content. It acts as a 24/7 virtual tutor for students, frees faculty time from repetitive clarifications, and ensures responses align precisely with the official syllabus, not generic web sources.</p><p><strong>2.</strong> <strong>Analytics Self-Service — Pinterest (Text-to-SQL Assistant)</strong></p><p>Pinterest’s data teams wanted non-technical employees to query analytics databases in natural language. The challenge: databases contain thousands of tables with cryptic names, and generic LLMs often guess or hallucinate schema details. Pinterest indexed its internal <strong>table schemas, documentation, and sample queries</strong> into a vector database. When a user asks, “How many users pinned travel content last week?”, the retriever fetches relevant schema descriptions (tables like <em>user_activity</em> and <em>pin_category</em>). The LLM then composes a correct SQL query referencing those tables.</p><p>It greatly reduced dependency on data engineers for simple questions and significantly fewer SQL errors and retries. It also improved data literacy across teams, so analysts and product managers can self-serve. This is an elegant use of RAG for <strong>structured-data reasoning</strong>, bridging natural language and databases, a key productivity use case for GenAI in analytics.</p><p><strong>3.</strong> <strong>Analytics &amp; Reporting — Grab</strong></p><p>Grab, Southeast Asia’s “super-app” for transport, delivery, and fintech, produces massive internal reports daily: financial metrics, fraud analysis, operational dashboards. Analysts were spending hours manually compiling and summarizing data. Grab’s engineering team built a <strong>RAG-powered “Report Summarizer”</strong> tool. It pulls fresh figures from APIs and data warehouses, retrieves contextual metrics (targets, anomalies, KPIs), and then prompts an LLM to generate an executive-style summary in plain language.</p><p>It saves analysts 3–4 hours per report and produces standardized, high-quality summaries for management. It frees human analysts for deeper interpretation instead of manual drafting. This showcases RAG beyond text retrieval, integrating <strong>live structured data retrieval</strong> (via APIs) with generative summarization, a growing pattern in enterprise analytics.</p><p><strong>4.</strong> <strong>Customer Service — LinkedIn</strong></p><p>LinkedIn’s support agents handle millions of member queries covering account access, billing, and policy questions. The company’s knowledge base was huge but scattered, making it hard for both human agents and chatbots to find precise answers quickly. LinkedIn developed an internal <strong>RAG + knowledge-graph system</strong> (described in an arXiv paper). The knowledge graph represents entities like “policy,” “feature,” “membership tier,” and “error type,” while the retriever pulls supporting paragraphs from help-center articles and ticket archives. The LLM fuses structured graph data with unstructured text to answer customer questions or suggest next steps for human agents.</p><p>28.6% reduction in median issue-resolution time and there is dramatic drop in ‘ticket escalations’ to senior support teams. The consistency has been improved and customers get the same correct answer regardless of channel or agent. LinkedIn’s example proves that <strong>combining retrieval with structured graphs</strong> yields more precise, explainable support AI. It also highlights how RAG can coexist with human workflows, not replace them.</p><p><strong>5. Finance — Morgan Stanley Wealth Management</strong></p><p>Morgan Stanley, one of the world’s leading investment banks, manages a vast library of proprietary research reports, product documentation, and compliance materials. Advisors constantly need quick, accurate answers for clients, yet searching through tens of thousands of internal PDFs or intranet pages was time-consuming and error-prone. In 2023, Morgan Stanley partnered with <strong>OpenAI</strong> to deploy a <strong>GPT-4-powered, retrieval-augmented assistant</strong> for its 16,000+ financial advisors. The system indexes more than <strong>100,000 internal research and policy documents</strong>. When an advisor types a question (“What are the ESG investment criteria for European clients?”), the system retrieves the most relevant passages from the bank’s internal knowledge base, then GPT-4 generates a concise, compliant answer with document citations.</p><p>This is one of the largest <strong>enterprise-scale RAG deployments</strong> in finance, showing that LLMs can be used safely in high-stakes environments when every answer is grounded in verified, version-controlled documents.</p><blockquote><strong>Key Advantages of RAG Applications:</strong></blockquote><p>· Reduces “AI hallucination” by grounding responses in verifiable, retrievable facts.​</p><p>· Eliminates need for frequent retraining, as external data sources can simply be updated, lowering both cost and latency for real-time applications.​</p><p>· Boosts explainability by enabling citation of retrieved documents, promoting trust and regulatory compliance.​</p><blockquote><strong>Future Directions</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*qvNwd-O_5X-H3TxwP7dHrw.png" /><figcaption>Image source: <a href="https://www.capellasolutions.com/blog/rag-and-vector-search-better-together-for-ai?ref=chitika.com">https://www.capellasolutions.com/blog/rag-and-vector-search-better-together-for-ai?ref=chitika.com</a></figcaption></figure><p>RAG technology is poised for progression in several main areas:</p><p><strong>Multimodal Integration:</strong> Future RAG systems will combine text, images, audio, and structured data, enabling applications in healthcare diagnostics, AR-guided engineering, and education with tailored learning paths.​</p><p><strong>Real-time and Autonomous Reasoning:</strong> Integrating RAG into real-time systems (e.g., AR field assistance or dynamic supply chains) for instant, context-aware support.​</p><p><strong>Explainability and Governance:</strong> As regulatory demands grow, RAG’s ability to trace and cite information aligns with explainable AI mandates in finance, healthcare, and governance.​</p><p><strong>Adaptive, Modular Frameworks: </strong>To address scalability and domain-specific needs, new architectures will emphasize plug-and-play modules for indexing, retrieval, and synthesis, enabling tailored solutions across industries.​</p><blockquote><strong>References</strong>:</blockquote><p>1. <a href="https://research.facebook.com/publications/retrieval-augmented-generation-for-knowledge-intensive-nlp-tasks/?utm_source=chatgpt.com">https://research.facebook.com/publications/retrieval-augmented-generation-for-knowledge-intensive-nlp-tasks/?utm_source=chatgpt.com</a></p><p>2. <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG?utm_source=chatgpt.com">https://research.ibm.com/blog/retrieval-augmented-generation-RAG?utm_source=chatgpt.com</a></p><p>3. <a href="https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/?utm_source=chatgpt.com">https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/?utm_source=chatgpt.com</a></p><p>4. <a href="https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/?utm_source=chatgpt.com">https://www.microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost/?utm_source=chatgpt.com</a></p><p>5. <a href="https://openai.com/index/morgan-stanley/?utm_source=chatgpt.com">https://openai.com/index/morgan-stanley/?utm_source=chatgpt.com</a></p><p>6. <a href="https://arxiv.org/abs/2404.17723?utm_source=chatgpt.com">https://arxiv.org/abs/2404.17723?utm_source=chatgpt.com</a></p><p>7. <a href="https://engineering.grab.com/transforming-the-analytics-landscape-with-RAG-powered-LLM?utm_source=chatgpt.com">https://engineering.grab.com/transforming-the-analytics-landscape-with-RAG-powered-LLM?utm_source=chatgpt.com</a></p><p>8. <a href="https://medium.com/pinterest-engineering/how-we-built-text-to-sql-at-pinterest-30bad30dabff">https://medium.com/pinterest-engineering/how-we-built-text-to-sql-at-pinterest-30bad30dabff</a></p><p>9.<a href="https://www.businesswire.com/news/home/20251010008494/en/Retrieval-Augmented-Generation-RAG-Industry-Report-2025-2035-Global-RAG-Market-to-Surpass-%2440-Billion-by-2035-as-Enterprises-Accelerate-AI-Integration---ResearchAndMarkets.com?utm_source=chatgpt.com">https://www.businesswire.com/news/home/20251010008494/en/Retrieval-Augmented-Generation-RAG-Industry-Report-2025-2035-Global-RAG-Market-to-Surpass-%2440-Billion-by-2035-as-Enterprises-Accelerate-AI-Integration---ResearchAndMarkets.com?utm_source=chatgpt.com</a></p><p>10. <a href="https://www.pymnts.com/news/retail/2025/the-weekender-pumpkin-spice-has-gone-off-rails-again/">https://www.pymnts.com/news/retail/2025/the-weekender-pumpkin-spice-has-gone-off-rails-again/</a></p><p>11. <a href="https://www.redlinegroup.com/insight-details/retrieval-augmented-generation-engineering-a-smarter-ai-future">https://www.redlinegroup.com/insight-details/retrieval-augmented-generation-engineering-a-smarter-ai-future</a></p><p>12. <a href="https://www.chitika.com/future-trends-in-retrieval-augmented-generation-what-to-expect-in-2025-and-beyond/">https://www.chitika.com/future-trends-in-retrieval-augmented-generation-what-to-expect-in-2025-and-beyond/</a></p><p>13. <a href="https://aws.amazon.com/what-is/retrieval-augmented-generation/">https://aws.amazon.com/what-is/retrieval-augmented-generation/</a></p><blockquote><strong>Thank you for reading till the end. If you like this blog, follow for more. Get connected on LinkedIn: </strong><a href="https://www.linkedin.com/in/sangitapokhrel/"><strong>https://www.linkedin.com/in/sangitapokhrel/</strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ed88ef5e53e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open-Source and Closed-Source LLMs: A Detailed Comparison]]></title>
            <link>https://medium.com/@sangitapokhrel911/open-source-and-closed-source-llms-a-detailed-comparison-58d3107722e8?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/58d3107722e8</guid>
            <category><![CDATA[closed-vs-open-source]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[open-source-llm]]></category>
            <category><![CDATA[closed-source-llms]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Tue, 13 Aug 2024 21:48:47 GMT</pubDate>
            <atom:updated>2024-08-13T21:51:17.900Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yj4N4tuPvDr6A7mJ1p7w3w.png" /></figure><p>In recent years, large language models (LLMs), have become a cornerstone of AI research and application, transforming fields like natural language processing, content creation, and customer support. As these models become increasingly influential, the debate between open-source and closed-sourced LLMs is more relevant than ever. This blog post will explore the key differences, advantages, and challenges associated with each approach, helping you understand which might be best suited for your models.</p><h3><strong>Section 1. Overview of LLMs</strong></h3><p>Large Language Models (LLMs), the name mentioned itself that it’s a LARGE-LANGUAGE-MODEL which means they are trained on large amounts of text data, it is a sophisticated AI system designed to understand and generate human language. It helps to perform various tasks such as engaging in human-like conversations (chatbots), creating articles, stories, and marketing copy (content generation), converting text from one language to another (translation), and outlining long documents into key points (summarization), etc.</p><h3><strong>Section 2. Open Source LLMs</strong></h3><p>Open-Source LLMs are models with publicly accessible source code and model weights. This openness allows anyone to inspect, modify, and contribute to the development of the model. The collaborative nature of open-source projects creates innovation and transparency.</p><blockquote><strong>Examples</strong>:</blockquote><p><a href="https://www.eleuther.ai/"><strong>GPT-Neo and GPT-J</strong></a> — Developed by <a href="https://www.eleuther.ai/">EleutherAI</a>, these models offer alternatives to proprietary models like GPT-3. GPT-Neo is available in various sizes, while GPT-J is known for its 6 billion parameter architecture, providing robust performance for a range of applications.</p><p><a href="https://huggingface.co/bigscience/bloom"><strong>BLOOM</strong></a><strong> </strong>— Created by the BigScience Project, BLOOM is a multilingual model with 176 billion parameters. It is able to understand and generate text in 46 languages and 13 programming languages.</p><p><a href="https://github.com/google-research/text-to-text-transfer-transformer"><strong>T5 (Text-To-Text Transfer Transformer) </strong></a><strong>— </strong>Developed by Google Research, T5 transforms all NLP tasks into a text-to-text format. It is an encoder-decoder model which has been pre-trained on a multi-task mixture of unsupervised and supervised tasks.</p><p><a href="https://llama.meta.com/"><strong>LLaMA (Large Language Model Meta AI)</strong></a><strong> — </strong>Meta AI’s LLaMA models include LLaMA-7B, LLaMA-13B, LLaMA-30B, and LLaMA-65B, offering various scalable options for various use cases. As of today (August 2024), the latest instruction-tuned model in LlaMA 3.1 is available in 8B, 70B, and 405B versions.</p><blockquote><strong>Advantages:</strong></blockquote><p><strong>1.</strong> <strong>Transparency: </strong>Full visibility into the models’ architecture and training data encourages trust and enables thorough evaluation.</p><p><strong>2.</strong> <strong>Community Contributions: </strong>Contributions from a global community of developers can enhance the model’s capabilities and address issues like biases or bugs.</p><p><strong>3.</strong> <strong>Customization: </strong>Users can tailor the model to specific needs, whether for niche applications or improved performance in particular contexts.</p><p><strong>4.</strong> <strong>Cost: </strong>Open-source models are typically free to use, although there may be costs associated with computing resources and infrastructure.</p><blockquote><strong>Challenges:</strong></blockquote><p><strong>1.</strong> <strong>Maintenance: </strong>Open-source models may lack the consistent support and updates provided by commercial entities, depending on community engagement.</p><p><strong>2.</strong> <strong>Performance Variability: </strong>The quality of open-source models can vary, and advanced optimization may not always be available.</p><h3>Section 3. Closed-Source LLMs</h3><p>Closed-source LLMs are proprietary models developed and controlled by private companies. The source code and model rights are not publicly available, and access is usually provided through paid APIs or licensing agreements.</p><blockquote><strong>Examples:</strong></blockquote><ol><li><a href="https://platform.openai.com/docs/models"><strong>OpenAI’s GPT-3, GPT-3.5, GPT-4, and GPT-4o</strong></a><strong> — </strong>All are advanced closedsource models that provide powerful capabilities via API. GPT-3, with 175 billion parameters, laid the groundwork for subsequent iterations, enabling a wide range of applications from conversational agents to creative writing. GPT-3.5 improved upon its predecessor with enhanced performance, while GPT-4 introduced even more advanced features and accuracy. GPT-4o, as a variant of GPT-4, offers optimized performance for specific use cases, continuing to push the boundaries of what these models can achieve.</li><li><a href="https://claude.ai/new"><strong>Claude </strong></a><strong>— </strong>Developed by Anthropic, Claude represents a series of advanced conversational models. Access is provided through an API, with a focus on ethical AI deployment and user safety.</li><li><a href="https://mistral.ai/"><strong>Mistral</strong></a><strong> — </strong>Mistral offers powerful closed-source models optimized for performance and efficiency. These models are available under commercial licenses, providing tailored solutions for businesses.</li><li><a href="https://deepmind.google/technologies/gemini/"><strong>Gemini</strong></a><strong> — </strong>The Gemini series by Google includes several advanced models such as Gemini Ultra, Pro, Flash, and Nano, each designed to offer different levels of performance and capabilities. Notably, Gemini 1.5 Pro and 1.5 Flash stand out with their exceptional context window of up to one million tokens, the longest among large-scale foundation models. This expansive context window enables near-perfect recall on tasks involving long documents, extensive codebases, and large multimedia files like audio and video. For developers and enterprise clients, Gemini 1.5 Pro offers an even larger context window of up to two million tokens, further enhancing its ability to handle complex and lengthy data.</li></ol><blockquote><strong>Advantages</strong>:</blockquote><p><strong>1.</strong> <strong>Professional Support: </strong>Closed-source models often come with dedicated support teams, regular updates, and maintenance services.</p><p><strong>2.</strong> <strong>Performance and Optimization: </strong>Companies behind closed-source models can leverage extensive resources to fine-tune performance and efficiency.</p><p><strong>3.</strong> <strong>Ease of Use: </strong>Closed-source models typically offer user-friendly APIs and integration tools, streamlining implementation.</p><blockquote><em>Challenges</em>:</blockquote><p><strong>1.</strong> <strong>Cost: </strong>Licensing fees or subscription costs can be substantial, depending on usage and model capabilities.</p><p><strong>2.</strong> <strong>Limited Transparency: </strong>Users have less insight into the model’s inner workings, which can impact trust and understanding.</p><p><strong>3.</strong> <strong>Less Customization: </strong>Customizing closed-source models is generally limited compared to open-source alternatives.</p><h3>Section 4. Comparative Analysis</h3><p><strong>1.</strong> <strong>Innovation and Development</strong></p><p>Open-source models benefit from collaborative innovation, allowing rapid advancements driven by a diverse group of contributors. Closed-source models, however, may leverage extensive internal research and development resources, leading to highly optimized solutions.</p><p><strong>2.</strong> <strong>Control and Customization</strong></p><p>Open-source models offer greater control and the ability to customize for specific use cases. Closed-source models often provide less flexibility but come with polished, out-of-the-box features designed for broad applicability.</p><p><strong>3.</strong> <strong>Cost and Accessibility</strong></p><p>Open-source models are generally more accessible due to their free nature, though infrastructure costs may apply. Closed-source models might involve higher costs but often include professional support and advanced features.</p><p><strong>4.</strong> <strong>Ethics and Transparency</strong></p><p>Open-source models promote transparency and ethical use by allowing scrutiny and community involvement. Closed-source models might have strong ethical guidelines but offer less visibility into their operational mechanisms.</p><p>In the evolving landscape of LLMs, both open-source and closed-source models offer unique benefits and challenges. Open-source models excel in transparency and customization, while closed-source models provide robust performance and professional support. Choosing between them depends on your specific needs, budget, and preferences. As AI technology advances, we can expect continued evolution in both open-source and closed-source domains. Staying informed about new developments will help you make the best decisions for leveraging LLMs in your projects.</p><p><strong><em>What are your thoughts on open-source versus closed-source LLMs? Have you used either type in your projects? Share your experiences and insights in the comments below!</em></strong></p><p><em>Connect with me on Linkedln : </em><a href="https://www.linkedin.com/in/sangitapokhrel/"><em>Sangita Pokhrel</em></a></p><p><em>Like this post if you enjoy reading this blog, follow me for more: </em><a href="https://medium.com/@sangitapokhrel911"><em>Sangita Pokhrel</em></a></p><h4>Thank You! 🙂</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58d3107722e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating Fake Data in Python with Faker and DrawData : Synthetic Data Generation]]></title>
            <link>https://medium.com/@sangitapokhrel911/generating-fake-data-in-python-with-faker-and-drawdata-synthetic-data-generation-dc77d73e8521?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc77d73e8521</guid>
            <category><![CDATA[faker-library]]></category>
            <category><![CDATA[drawdata]]></category>
            <category><![CDATA[synthetic-data-generation]]></category>
            <category><![CDATA[fake-data]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Sun, 26 May 2024 22:26:50 GMT</pubDate>
            <atom:updated>2024-05-26T22:26:50.286Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Generating Fake Data in Python with Faker and DrawData : Synthetic Data Generation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Udg5oVj5z0g_ksn6OL63zA.jpeg" /></figure><p>Synthetic data refers to artificially generated data that mimics the characteristics of real-world data while containing no identifiable information about individuals or entities.</p><p>In the world of data science and machine learning, having access to large datasets is crucial for developing and testing models. However, acquiring real data can sometimes be challenging due to privacy concerns, limited availability, or the need for specific data patterns. This is where synthetic data generation comes into play. Synthetic data steps in to help by creating data that looks and acts like the real thing but doesn’t have any personal info. By creating these datasets, one can freely experiment with diverse datasets without compromising privacy or incurring substantial costs. To make our work easier, we have Python libraries like Faker and DrawData which help generate data that fits for specific situations we’re interested in.</p><blockquote>In this blog post, we’ll explore how to generate synthetic data using two powerful Python libraries: <strong>Faker</strong> and <strong>DrawData</strong></blockquote><ol><li><strong>Introduction to Python Faker Library</strong></li></ol><p>The Python Faker library is an excellent tool for generating realistic fake data. It offers a wide range of methods to create synthetic data for names, addresses, phone numbers, email addresses, and more.</p><p>First, you need to install the Faker library. You can do this using pip:</p><pre>!pip install faker</pre><p>Once installed, you can start generating fake data. Here’s a quick example to demonstrate how to use Faker:</p><pre>#Import the module<br>from faker import Faker<br><br># Initialize the Faker object<br>fake = Faker()<br><br># Generate a fake name<br>fake_name = fake.name()<br>print(f&quot;Fake Name: {fake_name}&quot;)</pre><p>Here, I have shown you just the name generated using fake.name() with the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*cBuXUOyoAKzDudpRz-7wAg.png" /><figcaption>Figure 1: Name generated using faker</figcaption></figure><p>Faker supports generating a wide range of entities. Some of the top entities you can generate using Faker are:</p><ul><li>Person (Names, Address, Email, Phone Number, Birthdate, Society Security Number (SSN), Job title)</li><li>Address (Street address, City, State, Zip Code, Country, Latitude and longitude)</li><li>Company (Company name, Company suffix (eg, Inc, Ltd.), Job title etc)</li><li>Internet ( Email address, Domain name, URL, IP address, MAC address, Username , Password)</li><li>Date and Time ( Date, Time, DateTime, Timezone, Date range)</li><li>Finance (Credit card number, Credit card provider, Bank account number, Currency, Bitcoin address)</li><li>Commerce ( Product name, Price, Department etc.)</li></ul><blockquote>Let’s generate a list of 10 fake names using the Faker library, then prints them to verify the generated names.</blockquote><pre>#Let&#39;s create 10 fake names<br>fake_names = [fake.name() for _ in range(10)]<br><br>#Print the first 10 names to verify<br>for name in fake_names:<br>    print(name)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*ots8asV3ODAFpsqoolNi8A.png" /><figcaption>Figure 2: 10 Fake names has been generated</figcaption></figure><p>Now, we can save the names to a file or a DataFrame for further use, using the panda’s library to convert the generated names into a data frame and to save it in a CSV file so that it can be used in the future.</p><blockquote>Let’s create 100 entires of name, email, phone number, location and profession the same way we have generated in previous figure for name with 10 entities. This is a kind of proper data or the information of a 100 persons information which can be used for the further testing, analysis and so on.</blockquote><pre># Function to generate a dictionary with 100 entries of name, email, phone number, location and profession<br><br>def generate_fake_data(num_entries=100):<br>    data={<br>        &#39;Name&#39;:[],<br>        &#39;Email&#39;:[],<br>        &#39;Phone Number&#39;:[],<br>        &#39;Location&#39;:[],<br>        &#39;Profession&#39;:[]<br>    }<br><br>    for _ in range(num_entries):<br>        data[&#39;Name&#39;].append(fake.name())<br>        data[&#39;Email&#39;].append(fake.email())<br>        data[&#39;Phone Number&#39;].append(fake.phone_number())<br>        data[&#39;Location&#39;].append(fake.country())<br>        data[&#39;Profession&#39;].append(fake.job())A<br><br>    return data<br>    <br>#Generate the data<br>fake_data = generate_fake_data(100)<br><br>#Create a DataFrame<br>df_fake_data = pd.DataFrame(fake_data)<br><br>#Save to a CSV file<br>df_fake_data.to_csv(&#39;fake_data.csv&#39;, index=False)<br><br>print(df_fake_data.head())</pre><p>The fake data for 100 rows and 5 columns have been generated and coverted that into a dataframe using the pandas library. The dataframe is saved into csv format with the name ‘<strong>fake_data.csv</strong>’ which can be used for further analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*3Y1YoH4LzyfqPr5ZALalfA.png" /><figcaption>Figure 3: Fake data generated for 100 entries</figcaption></figure><p>Here are some key aspects and features of Faker:</p><ul><li><strong><em>Data Generation</em></strong>: Faker can generate various types of fake data, including names, addresses, emails, phone numbers, dates, times, text, and more. This data can be customized and tailored to specific needs, such as generating data in different languages, formats, or locales.</li><li><strong>Integration</strong>: Faker seamlessly integrates with various Python frameworks and libraries, such as Django, Flask, SQLAlchemy, and Pandas. This enables developers to easily incorporate fake data generation into their projects and workflows.</li><li><strong>Testing:</strong> Faker is commonly used in software testing to create realistic test scenarios and datasets. It helps developers simulate different user profiles, edge cases, and data scenarios to thoroughly test their applications’ functionality and robustness.</li><li><strong>Data Masking</strong>: Faker can also be used for data masking or anonymization purposes, where sensitive or personally identifiable information (PII) needs to be replaced with fake but realistic-looking data for privacy and security reasons.</li></ul><p>2. <strong>Introduction to Python DrawData Library</strong></p><p>While Faker is great for generating realistic text data, you might need to create synthetic datasets with specific patterns for testing machine learning algorithms. The DrawData library allows you to generate visually understandable synthetic data. This library in python is a tool designed to assist the visualization and creation of synthetic datasets by allowing users to draw data points directly onto a canvas.</p><blockquote>Key Features of ‘<strong>drawdata</strong>’ library</blockquote><ul><li><strong>Interactive Data Drawing</strong>: Users can draw data points manually on a canvas, which can then be converted into a dataset.</li><li><strong>Synthetic Data Creation</strong>: Helps in generating synthetic datasets for machine learning experiments and testing.</li><li><strong>Visualization</strong>: Provides immediate visual feedback of the data being created, helping users to understand the distribution and structure of their data.</li></ul><p>Basic Example of Using ‘drawdata’</p><p>Here’s a simple example to illustrate how you might use the ‘drawdata’ library in python.</p><p><strong><em>i. First you need to install the library by following command.</em></strong></p><pre>!pip install drawdata</pre><p>ii. <strong><em>Import the necessary module based on what type of data that you would like to generate</em></strong></p><p>After running <strong>draw_scatter()</strong>, an interactive window will appear where you can draw points. Once done, you can retrieve the points as a dataset for further processing.</p><pre>from drawdata import draw_scatter<br>draw_scatter()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*KDu34KZJF-VU6UwNGWHRyQ.png" /><figcaption>Figure 5: Scatter plot created in canvas using drawdata library</figcaption></figure><p>As you can see in the figure 5, you can create the customized scatter plot in the blank canva and generate the data. Once you execute the cell and sketch the points, click ‘<strong>Copy CSV</strong>’ and your data points, presented as x,y,z comma-seperated values will be copied to the clipboard. Now you can read the data using pandas DataFrame, or directly download the csv file and start working on that.</p><pre>import pandas as pd<br>pd.read_clipboard(sep=&#39;,&#39;)</pre><p>Similarly, drawdata provides other visuals such as line chart, histogram, bar charts and so on which you can sketch and download the datasets. See the below line chart.</p><pre>from drawdata import draw_line<br>line_data = draw_line()<br>draw_line()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*ACBJ0Ga4wFzPddJwx9gL-g.png" /><figcaption>Figure 6: Figure 5: Line plot created in canvas using drawdata library</figcaption></figure><blockquote><strong>Conclusion:</strong></blockquote><p>Generating synthetic data in Python is straightforward with the right tools. The Faker library allows you to create realistic textual data, while DrawData helps generate structured numerical data with specific patterns. Combining these tools can provide you with versatile datasets for a variety of applications.</p><p>Whether you are testing applications, protecting user privacy, or developing machine learning models, synthetic data can be a valuable resource. Start experimenting with Faker and DrawData to see how they can enhance your projects.</p><p>If you like this blog, hit like and follow me for more.</p><blockquote>Let’s get connected on Linkeldn: <a href="https://www.linkedin.com/in/sangitapokhrel/"><strong>Sangita Pokhrel</strong></a></blockquote><blockquote>Happy Learning, see you in next blog. 😊</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc77d73e8521" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power of GPT-4o — Transforming Everyday Life with AI Integration]]></title>
            <link>https://medium.com/@sangitapokhrel911/power-of-gpt-4o-transforming-everyday-life-with-ai-integration-fd1d11a61743?source=rss-9540fb596d80------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd1d11a61743</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpt-model]]></category>
            <category><![CDATA[gpt-4o]]></category>
            <category><![CDATA[latestnewsonai]]></category>
            <category><![CDATA[openai-gpt4]]></category>
            <dc:creator><![CDATA[Sangitapokhrel]]></dc:creator>
            <pubDate>Tue, 14 May 2024 10:05:53 GMT</pubDate>
            <atom:updated>2024-05-14T10:12:11.372Z</atom:updated>
            <content:encoded><![CDATA[<h3>Power of GPT-4o — Transforming Everyday Life with AI Integration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*DeSx3Jr75znRPN2YBUEj8g.png" /><figcaption>Image Source ( <a href="https://openai.com/index/hello-gpt-4o/">https://openai.com/index/hello-gpt-4o/</a>)</figcaption></figure><p>OpenAI’s latest revelation, the GPT-4o, marks yet another groundbreaking leap in AI advancement, shocking the world with its capacity to reason seamlessly across audio, vision, and text in real time. The ‘o’ in GPT-4o signifies ‘omni,’ symbolizing a significant step toward more natural human-computer interactions. With the ability to produce a multitude of text, audio, and image outputs, this innovation represents a great milestone in AI evolution.</p><p>One of the most remarkable achievements of GPT-4o is its rapid response time, capable of processing audio inputs in as little as 232 milliseconds, similar to human conversational speed. Surpassing its predecessors, GPT-4 Turbo, GPT-4o not only matches performance on English text and code but also exhibits substantial enhancements in handling non-English languages, all while being significantly faster and 50% more cost-effective in API usage. Moreover, GPT-4o showcases superior prowess in comprehending vision and audio inputs compared to existing models.</p><blockquote><strong><em>Imagine this model fitting into businesses and daily routines effortlessly: in smart glasses, chatting with AI that helps with everything from finding directions and solving math problems to translating languages, practicing interviews, and even entertaining with jokes and songs. It makes life easier by turning everyday tasks into smooth experiences enhanced by AI support.</em></strong></blockquote><p>Here are some of the information about GPT-4o:</p><ol><li><strong>GPT-4o’s Versatility:</strong> GPT-4o seamlessly reasons across audio, vision, and text, offering a more natural human-computer interaction experience. It generates diverse outputs including text, audio, and images, enhancing its usability across various domains.</li><li><strong>Rapid Response Time: </strong>With an impressive response time of as little as 232 milliseconds for audio inputs, GPT-4o matches human conversational speed. It surpasses its predecessors, GPT-4 Turbo, in performance on English text and code, while also excelling in handling non-English languages.</li><li><strong>Enhanced Interactions:</strong>GPT-4o’s unified model eliminates latency issues encountered in previous Voice Mode interactions. By consolidating all modalities into a single model, it enables richer interactions without information loss.</li><li><strong>Performance Evaluation:</strong>GPT-4o achieves GPT-4 Turbo-level proficiency in text, reasoning, and coding intelligence. It sets new benchmarks in multilingual capabilities, audio processing, and vision understanding, demonstrating unparalleled versatility and performance across diverse linguistic and perceptual domains.</li><li><strong>Accessibility and Availability:</strong>The release of GPT-4o signifies a significant step in democratizing AI access. OpenAI optimizes efficiency across all layers of the AI stack, making GPT-4 level capabilities more widely accessible. Text and image functionalities are already available in ChatGPT’s free tier and for Plus users, with plans for further integration into Voice Mode.</li></ol><p>The below figure is the vision understanding evals, you can explore more about <strong><em>Text Evaluation, Audio ASR Performance, Audio translation performance and M3Exam Zero-Shot Results.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IW2rQOoCjOesbirudL1QtQ.png" /><figcaption>Image Source ( <a href="https://openai.com/index/hello-gpt-4o/">https://openai.com/index/hello-gpt-4o/</a>)</figcaption></figure><blockquote><strong><em>Which aspect of GPT-4o’s capabilities are you most excited to incorporate into your personal or professional life?</em></strong></blockquote><p>To see detail about GPT-4o and demos, see here: <a href="https://openai.com/index/hello-gpt-4o/">Hello GPT-4o</a>.</p><p>If you enjoy reading this post, do like it and follow me for more here: <a href="https://medium.com/@sangitapokhrel911"><strong>Sangita Pokhrel</strong></a></p><p>Feel free to get connected on <strong>Linkeldn </strong>: <a href="https://www.linkedin.com/in/sangitapokhrel/">Sangita Pokhrel</a></p><p>Thank You. Keep exploring 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd1d11a61743" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>