<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kaushik Shakkari on Medium]]></title>
        <description><![CDATA[Stories by Kaushik Shakkari on Medium]]></description>
        <link>https://medium.com/@kaushikshakkari?source=rss-8b97eb1e4ee2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*InYhkK4PUrTC-mIm0CrGRQ@2x.jpeg</url>
            <title>Stories by Kaushik Shakkari on Medium</title>
            <link>https://medium.com/@kaushikshakkari?source=rss-8b97eb1e4ee2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:32:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaushikshakkari/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fine-Tuning Embeddings: The Practical Middle Ground Between Giant LLMs and Generic Embeddings]]></title>
            <link>https://generativeai.pub/fine-tuning-embeddings-the-practical-middle-ground-between-giant-llms-and-generic-embeddings-58308fa497b6?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/58308fa497b6</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Fri, 20 Jun 2025 02:54:17 GMT</pubDate>
            <atom:updated>2025-06-20T16:42:01.951Z</atom:updated>
            <content:encoded><![CDATA[<h4>An End-to-End Guide to Fine-Tuning Embedding Models with Real Code, Loss Functions, and Evaluation</h4><h4>Why LLMs Are Overkill, Generic Embeddings Fall Short, and Fine-Tuning Still Wins in Scalable, Real-World AI</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQYm9utVOEBe-McHgVX78g.png" /><figcaption>Source: Generated using OpenAI</figcaption></figure><h3>Table of Contents:</h3><ol><li><a href="#381c"><strong>Introduction</strong></a></li><li><a href="#0cee"><strong>Why LLMs Aren’t Good Enough (For Everything)</strong></a></li><li><a href="#5eaa"><strong>Why Generic Embeddings Also Fall Short</strong></a></li><li><a href="#2a1d"><strong>How Do I Fine-Tune Embeddings? (in 5 simple steps)</strong></a></li><li><a href="#e211"><strong>Conclusion</strong></a></li></ol><h3>Introduction:</h3><p>In the current AI landscape, large language models (LLMs) like GPT-4, Claude, and Gemini have undoubtedly taken center stage. These models are powerful, offering flexible natural language understanding, contextual reasoning, and few-shot learning capabilities. From semantic search and summarization to task automation, the LLM stack seems like the default path for many teams.</p><p>But as real-world demands push systems from demos to production, teams are starting to ask: <strong><em>Is an LLM always the right tool for the job?</em></strong></p><p>For many high-throughput use cases including matching, retrieval, clustering, or deduplication, <strong><em>the answer is often no</em></strong>. LLMs, while impressive, come with trade-offs in latency, cost, and consistency. Meanwhile, generic embedding models offer speed and stability, but struggle to capture domain nuance or task-specific meaning.</p><blockquote>✅ This is where <strong>fine-tuned embeddings</strong> step in, not as a fallback, but as a deliberate, strategic middle ground.</blockquote><p>In this guide, I will explore why fine-tuning embeddings still matters in 2025, where it fits between LLMs and generic models, and how you can make it work in your stack.</p><h3>Why LLMs Aren’t Good Enough (For Everything)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gwhfgp5B4f2E2CB2WVJx0A.png" /><figcaption>Source: Generated using OpenAI</figcaption></figure><p>At first glance, <strong>LLMs look like the Swiss Army knife of AI</strong>, they can generate text, summarize documents, reason over context, and even perform basic retrieval. It’s no surprise that many teams reach for them as the default engine behind search, recommendation, and information extraction tasks.</p><p>For many applications including semantic search, deduplication, entity matching, or vector clustering:</p><ul><li><strong>❌ LLMs are slow</strong></li><li><strong>❌ LLMs are expensive</strong></li><li><strong>❌ LLMs are inconsistent</strong></li><li><strong>❌ LLMs are often overkill</strong></li></ul><p>When people move from prototypes to production, they realize:</p><ul><li>⚠️ <strong>Latency</strong>: Even smaller models like GPT-3.5 often introduce few seconds for predictions, far too slow for real-time interactions.</li><li>💸 <strong>Cost</strong>: LLM usage is priced by token volume. At scale, especially in high-volume, low-margin environments, this adds up quickly.</li><li>🔄 <strong>Inconsistency</strong>: Outputs can vary between calls, leading to noisy behavior in systems that require deterministic logic.</li><li>📉 <strong>Scaling constraints</strong>: Running large models locally or on edge is impractical, and API dependency introduces infrastructure complexity and vendor lock-in.</li></ul><blockquote>So what’s the alternative? <strong>Let’s turn to embeddings.</strong></blockquote><h3>Why Generic Embeddings Also Fall Short</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aBbYEMqMoeftI4x8W5cNhQ.png" /><figcaption>Source: Generated using OpenAI</figcaption></figure><p>If LLMs are too heavy-handed, off-the-shelf embedding models seem like the obvious lightweight fix.</p><ul><li>✅ They’re fast</li><li>✅ They’re deterministic or predictable</li><li>✅ They can be run on CPUs -&gt; cheap to run at scale</li><li>✅ They are easy to cache &amp; index</li></ul><blockquote>⚠️ But here’s the catch: <strong>these models weren’t trained on your data or optimized for your tasks.</strong></blockquote><p>Generic embeddings often fail to grasp the subtle, domain-specific relationships that make or break real-world performance. For example:</p><ul><li>In the <strong>recruitment domain</strong>, terms like <strong>“Developer I” </strong>and <strong>“Entry-Level Software Engineer”</strong> often refer to the same role. However, generic embeddings frequently assign them low similarity scores.</li><li>In the <strong>legal domain</strong>, <strong>“force majeure” </strong>and <strong>“act of God”</strong> are widely recognized as interchangeable terms in contracts, yet generic embeddings may not link them effectively.</li><li>In the <strong>medical domain</strong>, <strong>“heart attack”</strong> and <strong>“myocardial infarction”</strong> are medically equivalent, but many pre-trained models struggle to connect these phrases.</li></ul><p>These mismatches lead to missed matches, irrelevant results, and poor recall especially in specialized or high-stakes environments.</p><blockquote>So what’s the missing piece? <strong>Fine-tuning it is ✅</strong></blockquote><h3>How Do I Fine-Tune Embeddings? (in 5 simple steps)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7n8bjPQQ0p30pWeOcwkUtw.png" /><figcaption>Source: Generated using OpenAI</figcaption></figure><p>Fine-tuning embedding models may sound complex, but the process is surprisingly approachable especially with modern libraries like <a href="https://www.sbert.net/">SentenceTransformers</a>, <a href="https://huggingface.co/">Hugging Face</a>, and <a href="https://pytorch.org/">PyTorch</a>. Here’s a high-level breakdown of the workflow:</p><h4>Step 1: Gather Positive (and sometimes Negative) Pairs</h4><p>Start by creating a dataset of similar and dissimilar text pairs. These can be curated from:</p><ul><li>🟢 <strong>Open Source Datasets:</strong> Use publicly available datasets with labeled text pairs relevant to your task or domain.</li><li>🟡 <strong>Pseudo Labelling via LLMs:</strong> Generate candidate pairs using LLMs, optionally followed by manual review to correct or confirm similarity.</li><li>🟠 <strong>Weak Supervision:</strong> Leverage simple rules or fuzzy matches (e.g., similar product titles, synonyms, abbreviations) to auto-label data at scale.</li><li>🔵 <strong>Behavioral Signals: </strong>Extract pairs from user activity such as clicks, selections, search refinements, or edits could be valuable implicit feedback and labels.</li><li>🔴 <strong>Human Annotations: </strong>Create high-quality, labeled pairs from scratch with human annotators which is more costly, but gives possibly best gold-standard evaluation sets.</li></ul><p><strong>Examples:</strong></p><ul><li><strong>Positive / Label = 1 </strong>✅ <strong>:</strong> (“What is diabetes?”, “Explain type 2 diabetes”)</li><li><strong>Negative / Label =0 </strong>❌<strong>:</strong> (“What is diabetes?”, “Symptoms of asthma”)</li></ul><p><strong>Code:</strong></p><pre>from sentence_transformers import SentenceTransformer, InputExample, losses<br>from torch.utils.data import DataLoader<br><br># Step 1: Prepare training examples<br>train_examples = [<br>    InputExample(texts=[&quot;myocardial infarction&quot;, &quot;heart attack&quot;], label=1.0),<br>    InputExample(texts=[&quot;diabetes mellitus&quot;, &quot;type 2 diabetes&quot;], label=1.0),<br>    InputExample(texts=[&quot;hypertension&quot;, &quot;high blood pressure&quot;], label=1.0),<br>    InputExample(texts=[&quot;fever&quot;, &quot;broken arm&quot;], label=0.0),<br>    InputExample(texts=[&quot;asthma&quot;, &quot;chronic back pain&quot;], label=0.0),<br>]<br><br># Create DataLoader<br>train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)</pre><blockquote>⭐ Instead of labeling from scratch, combination of weak supervision (using approximate matches using rules) + LLM generations + verifying and refining them with human review is often the fastest way to bootstrap quality training data.</blockquote><h4>Step 2: Pick a Pre-trained Embedding Model</h4><p>Select a base model to fine-tune. Start with a well-performing general-purpose model such as:</p><ul><li><a href="http://sentence-transformers/all-MiniLM-L6-v2">all-MiniLM-L6-v2</a></li><li><a href="http://sentence-transformers/all-mpnet-base-v2">mpnet-base-v2</a></li><li><a href="https://huggingface.co/Qwen/Qwen3-Embedding-0.6B">qwen3-embedding-0.6B</a></li><li>Or a domain-specific variant if available</li></ul><p>Look for models that support sentence-level embedding tasks and are compatible with your infrastructure (e.g., CPU-friendly or multilingual).</p><p><strong>Code:</strong></p><pre># Step 2: Load a pre-trained model<br>model = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)</pre><h4>Step 3: Choose a Loss Function</h4><p>Each loss function has strengths and weaknesses depending on your labels, batch size, and downstream use case. Below is a breakdown of the most common fine-tuning objectives</p><h4><strong>3.a Cosine Similarity Loss (</strong>for graded similarity scores<strong>):</strong></h4><p>This loss directly aligns the cosine similarity between two embeddings with a ground truth similarity score. It’s most suitable when your labels are not binary (“yes” or “no”) but are continuous (e.g., between 0 and 1), reflecting varying degrees of similarity.</p><blockquote>Loss = (cos(u, v) — y)²</blockquote><blockquote>where <strong>u</strong> and <strong>v</strong> are the embeddings for the source and target texts, and <strong>y</strong> is the ground truth similarity label.</blockquote><p><strong>Code:</strong></p><pre># Step 3: Choose a loss function<br>train_loss = losses.CosineSimilarityLoss(model=model)</pre><h4><strong>3.b Contrastive Loss (</strong>for binary labeled pairs — 0/1<strong>):</strong></h4><p>This loss is designed for binary classification tasks where each text pair is either similar (label = 1) or dissimilar (label = 0). It pushes dissimilar pairs apart beyond a margin and pulls similar pairs closer together.</p><blockquote>Loss = y * d² + (1 — y) * max(0, m — d)²</blockquote><blockquote>Where <strong>d</strong> is the distance between embeddings, <strong>m</strong> is the margin, and <strong>y</strong> is the ground truth similarity label.</blockquote><p><strong>Code:</strong></p><pre># Step 3: Choose a loss function<br>train_loss = losses.ContrastiveLoss(model=model)</pre><h4><strong>3.c Triplet Loss (</strong>when you have anchor/positive/negative triplets):</h4><p>Triplet loss helps the model learn relative similarities. It ensures that an anchor is closer to a positive example (same class) than to a negative example (different class) by a specified margin.</p><blockquote>Loss = max(0, d(a,p) − d(a,n) + m)</blockquote><p><strong>Code:</strong></p><pre># Step 3: Choose a loss function<br>train_loss = losses.TripletLoss(model=model)</pre><blockquote>Where <strong>m</strong> is a margin that separates (anchor, positive) and (anchor, negative) pairs, <strong>d(a,p)</strong> is the distance between anchor and positive pair, <strong>d(a,n)</strong> is the distance between anchor and negative pair. (<strong>a</strong> is basically anchor or query)</blockquote><h4><strong>3.d Multiple Negatives Ranking Loss (</strong>for retrieval-style setups):</h4><p>This loss function is used in retrieval tasks. For each positive pair in a batch, it treats all other items in the batch as negative examples. The goal is to train the model to assign a higher similarity score to the correct (positive) match than to any of the negatives.</p><blockquote>Loss = -log (exp(sim(a, p⁺)/τ) / Σ_j exp(sim(a,nⱼ)/τ) )</blockquote><blockquote>Where<strong> p⁺</strong> is the positive, n<strong>ⱼ</strong> are all negatives in the batch, <strong>a</strong> is basically anchor, and <strong>τ (tau)</strong> is a temperature parameter.</blockquote><blockquote>Lower τ (e.g., 0.05) makes the model focus more strongly on distinguishing the most similar (or confusing) negatives.</blockquote><blockquote>High τ (like 0.2 or 0.5) makes the model more relaxed. It spreads attention more evenly and doesn’t punish close mistakes as much.</blockquote><p><strong>Code:</strong></p><pre># Step 3: Choose a loss function<br>train_loss = losses.MultipleNegativesRankingLoss(model)</pre><h4><strong>3.f </strong>Online Hard Negative Mining:</h4><p>Online Hard Negative Mining is a <strong>training strategy, not a loss function</strong> by itself . It’s often used alongside contrastive or triplet loss to improve learning. Instead of sampling negatives randomly, the model actively selects the most difficult (confusing or borderline) negative examples from within each batch during training.</p><p><strong>Typical Workflow:</strong></p><ul><li>⭐ Use cosine similarity or Euclidean distance to score all negative candidates</li><li>⭐ Select the top-N most similar negatives per anchor for training</li><li>⭐ Combine this with triplet loss or contrastive loss</li></ul><p><strong>Why it works better:</strong></p><p>Without hard negatives, models often converge quickly on <strong>easy examples</strong>, but fail to generalize to near-miss or ambiguous cases. Hard negatives stretch the model’s capacity and sharpen its decision boundaries.</p><blockquote>⚠️ <strong><em>Note:</em></strong><em> Be cautious as </em><strong><em>too many hard negatives too early</em></strong><em> in training can destabilize learning. It’s common to start with random negatives and </em><strong><em>gradually introduce hardness</em></strong><em> as the model improves.</em></blockquote><p><strong>Code:</strong></p><pre># Step 4: Finetune modeel<br>model.fit(<br>    train_objectives=[(train_dataloader, train_loss)],<br>    epochs=1,<br>    warmup_steps=10<br>)</pre><h4>Step 4: Fine-Tune the Model</h4><p>With your labeled data and loss function in place, the next step is to train the model. Using libraries like sentence-transformers, you can kick off training by passing your dataset through a DataLoader and optimizing over several epochs. The goal is to learn an embedding space where semantically similar texts are pulled closer, and dissimilar ones are pushed apart.</p><p><strong>During training, check for:</strong></p><ul><li>📉 <strong>Validation loss</strong>: Is it consistently decreasing? A steady drop usually signals the model is learning a meaningful similarity function.</li><li>📈 <strong>Embedding behavior</strong>: Are similar pairs getting closer in cosine space? You can log example similarities over epochs.</li><li>🛑 <strong>Signs of overfitting</strong>: If training loss continues to drop but validation loss plateaus or rises, your model is starting to memorize. Use <strong>early stopping</strong> based on validation performance to prevent this.</li><li>📊 <strong>Batch dynamics</strong>: Monitor hard negatives, margin violations (for triplet loss), or prediction confidence to understand where the model struggles.</li></ul><p><strong>Code:</strong></p><pre># Step 3: Choose a loss function<br>train_loss = losses.MultipleNegativesRankingLoss(model=model)</pre><h4>Step 5: Evaluate the Results</h4><p>Once training is complete, it’s time to verify that your fine-tuned embeddings actually reflect the kind of similarity your application needs.</p><p><strong>🧪 Use quantitative metrics like:</strong></p><ul><li><a href="https://medium.com/@kaushikshakkari/open-domain-question-answering-series-part-4-answer-quality-metrics-evaluating-the-reader-ff7fa20736bf"><strong>Cosine similarity</strong></a> on known similar/dissimilar pairs</li><li><a href="https://medium.com/@kaushikshakkari/open-domain-question-answering-series-part-4-answer-quality-metrics-evaluating-the-reader-ff7fa20736bf"><strong>ROC-AUC</strong>, <strong>F1 score</strong>, or <strong>Precision/Recall</strong></a> for classification-style tasks</li><li><a href="https://medium.com/@kaushikshakkari/understanding-semantic-search-part-5-ranking-metrics-for-evaluating-question-answering-systems-f3150872d986"><strong>Recall@K</strong>, <strong>MRR</strong>, or <strong>nDCG</strong></a> for retrieval systems</li></ul><p><strong>🔬 Use logging tools </strong>like <strong>TensorBoard</strong>, <strong>Weights &amp; Biases</strong>, or <strong>MLflow for deeper insight:</strong></p><ul><li>⭐ Visualize similarity distributions</li><li>⭐ Compare training vs. validation performance over time</li><li>⭐ Track experiments, checkpoints, and hyperparameter changes</li></ul><p><strong>Code:</strong></p><pre># Step 5: Evaluation<br>from sentence_transformers import InputExample, evaluation<br><br># Define evaluation examples (label = 1.0 for similar, 0.0 for dissimilar)<br>test_examples = [<br>    InputExample(texts=[&quot;heart attack&quot;, &quot;cardiac arrest&quot;], label=0.0), <br>    InputExample(texts=[&quot;fever&quot;, &quot;elevated body temperature&quot;], label=1.0),<br>    InputExample(texts=[&quot;flu&quot;, &quot;influenza&quot;], label=1.0),<br>    InputExample(texts=[&quot;fractured bone&quot;, &quot;kidney stone&quot;], label=0.0),<br>    InputExample(texts=[&quot;blood sugar&quot;, &quot;glucose level&quot;], label=1.0),<br>]<br><br>evaluator = evaluation.BinaryClassificationEvaluator.from_input_examples(<br>    test_examples,<br>    name=&quot;medical-similarity-eval&quot;<br>)<br><br>evaluator(model)</pre><blockquote><strong>🎯 Remember the ultimate goal:</strong></blockquote><blockquote>It’s not just about achieving lower loss but it’s about creating embeddings that <strong>align with human understanding</strong>, <strong>generalize to unseen examples</strong>, and <strong>hold up in your real-world domain</strong>.</blockquote><h3>Conclusion:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GBiE0NWq_Z6owkJKweXgSg.png" /><figcaption>Source: Generated using OpenAI</figcaption></figure><p>Yes, LLMs are magical and powerful. But in production, magic comes with trade-offs: cost, latency, inconsistency, and operational complexity. If your goal is semantic search, clustering, deduplication, or matching, you don’t need full-blown generative intelligence. You need precision. You need speed. You need stability.</p><ul><li>✅ Start with embeddings.</li><li>✅ Fine-tune them your data.</li><li>✅ Use vector search for fast, scalable retrieval</li><li>✅ Call on LLMs only when you need generation or deep reasoning</li></ul><p>Fine-tuned embeddings aren’t a compromise. They’re the <strong>strategic core</strong> of efficient, intelligent, and production-ready AI in 2025 and beyond. They’re not just a bridge between old and new but they’re the <strong>missing middle</strong>. And often, the smartest choice.</p><h3>Add me on <a href="https://www.linkedin.com/in/kaushik-shakkari/">LinkedIn</a>. Thank you!</h3><p>If you are interested in the document parsing topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/2030d58d5fe9">Understanding Document Parsing</a></p><p>If you are interested in semantic search topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/9df1566193b0">Understanding Semantic Search</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*NEix3emWMzpp480x.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*yqWGfeP8gHWLmJK1.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58308fa497b6" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/fine-tuning-embeddings-the-practical-middle-ground-between-giant-llms-and-generic-embeddings-58308fa497b6">Fine-Tuning Embeddings: The Practical Middle Ground Between Giant LLMs and Generic Embeddings</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding AI Agents — (Part 2: Building a Multi-Agent System for Stock Market Analysis)]]></title>
            <link>https://generativeai.pub/understanding-ai-agents-part-2-building-a-multi-agent-system-for-stock-market-analysis-024ded4e7709?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/024ded4e7709</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Tue, 01 Apr 2025 09:03:47 GMT</pubDate>
            <atom:updated>2025-04-01T09:03:47.371Z</atom:updated>
            <content:encoded><![CDATA[<h4>Practical AI Agent Engineering Guide</h4><h3>Understanding AI Agents — (Part 2: Building a Multi-Agent System for Stock Market Analysis)</h3><h4>Leverage CrewAI to orchestrate specialized agents for data retrieval, market research, sentiment analysis, and automated financial reporting.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZWWeE6_QGXiJL9n8EmwVSw.jpeg" /><figcaption>Image generated using <a href="https://openai.com/index/dall-e-3/">DALL.E</a></figcaption></figure><p>In our previous article of <a href="https://medium.com/generative-ai/understanding-ai-agents-part-1-what-are-they-how-they-work-their-challenges-and-future-f8d3e4223e11">“<strong><em>Understanding AI Agents: Part 1</em></strong>”</a>, we explored the fundamentals of AI agents, their core components, challenges, and potential. In this article, we will delve into constructing a robust multi-agent system for stock market analysis using CrewAI.</p><p>We will also explore and simplify key CrewAI concepts such as <strong>Role Playing, Focus, Tools, Cooperation, Guardrails, and Memory</strong>, using practical examples and code snippets specifically designed for stock market analysis.</p><h3>Why Not a Single Agent? Why We Need Multi-Agent Systems?</h3><p>Single-agent systems struggle with complex workflows involving multiple specialized tasks. Multi-agent systems address these limitations through:</p><ul><li><strong>Role Specialization:</strong> Each agent specializes in a specific task.</li><li><strong>Parallel &amp; Sequential Processing:</strong> Tasks can run concurrently or sequentially.</li><li><strong>Enhanced Accuracy &amp; Efficiency:</strong> Specialized agents produce higher-quality outputs faster.</li></ul><h3>Key Concepts of CrewAI Explained with Stock Market Analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gkWJqTdVUKzighIR9tzZFA.png" /><figcaption>Source: Image generated using OpenAI’s DALL.E</figcaption></figure><blockquote>Before building agents, lets load a LLM. I am using OpenAI model but this can be replaced with <a href="https://docs.crewai.com/concepts/llms#provider-configuration-examples">any LLM of your choice from wide providers</a>.</blockquote><pre>from crewai import LLM<br><br>llm = LLM(<br>    model=&quot;gpt-4-turbo&quot;,<br>    api_key=config[&quot;OpenAI&quot;][&quot;OPEN_API_KEY&quot;], # replace with your API key<br>    api_base=&quot;https://api.openai.com/v1&quot;<br>)<br><br>llm.call(&quot;hello&quot;) # returns &quot;Hello! How can I assist you today?&quot;</pre><p>As we loaded our LLM, now let’s explore each concept step-by-step with practical implementations:</p><h3>1. Role Playing</h3><p>In CrewAI, <strong>Role Playing</strong> refers to the ability of AI agents to assume specific identities within a collaborative framework. An agent’s <strong>role</strong>, <strong>goal</strong>, and <strong>backstory</strong> setup Role Playing by collectively defining its identity, purpose, and behavior. The <strong>role</strong> functions like a job title, signaling to the system and other agents what function it performs (e.g., Data Analyst, Legal Reviewer). The <strong>goal</strong> outlines the agent’s mission like what it is trying to achieve, often in a clear, outcome-oriented format (such as “generate concise financial summaries”). The <strong>backstory</strong> provides contextual depth, describing the agent’s expertise, personality, background, and reasoning style, which shape how it communicates and makes decisions.</p><p>Together, these elements ensure the agent behaves consistently and effectively in both autonomous and collaborative scenarios. This role-based architecture enables agents to coordinate seamlessly, leveraging their specialized skills to tackle complex, multi-step tasks with greater accuracy and efficiency.</p><p>In a stock market analysis scenario, you might have agents playing the following specialized roles:</p><ul><li>🧩 <strong>Financial Data Retriever: </strong>Gathers accurate and structured historical stock data from financial APIs like Yahoo Finance.</li><li>🌐 <strong>Industry Insights Analyst: </strong>Researches external market trends, news, and competitor activities using real-time web data.</li><li>🧠 <strong>Sentiment Analyst: </strong>Evaluates the tone of financial news and reports using NLP models like FinBERT to detect positive, negative, or neutral sentiment.</li><li>📈 <strong>Quantitative Stock Analyst: </strong>Analyzes historical price data using statistical and technical methods to uncover trends, signals, and forecasts.</li><li>🗞️ <strong>Financial Reporter: </strong>Synthesizes all findings into clear, professional investment reports tailored for stakeholders and decision-makers.</li></ul><blockquote>Here’s how we define our specialized stock market agents using CrewAI:</blockquote><pre>from crewai import Agent<br><br># 1. Financial Data Retriever<br>data_retriever = Agent(<br>    role=&quot;Financial Data Retriever&quot;,<br>    goal=&quot;Fetch reliable and accurate historical stock data for given symbols and time ranges.&quot;,<br>    backstory=(<br>        &quot;A detail-oriented data specialist focused on retrieving high-quality financial data &quot;<br>        &quot;from trusted sources like Yahoo Finance. Ensures data completeness and correctness &quot;<br>        &quot;for use in downstream analysis and reporting tasks.&quot;<br>    ),<br>    llm=llm,<br>    allow_delegation=False<br>)<br><br># 2. Industry Insights Analyst<br>insights_analyst = Agent(<br>    role=&quot;Industry Insights Analyst&quot;,<br>    goal=&quot;Provide up-to-date industry trends, news, and competitor analysis using real-time web data.&quot;,<br>    backstory=(<br>        &quot;A strategic market researcher skilled in scanning global news, blogs, and financial portals &quot;<br>        &quot;to identify emerging trends, technologies, and risks in the AI and tech sectors. Brings context &quot;<br>        &quot;and sentiment to complement quantitative analysis.&quot;<br>    ),<br>    llm=llm<br>)<br><br># 3. Sentiment Analyst<br>sentiment_analyst = Agent(<br>    role=&quot;Sentiment Analyst&quot;,<br>    goal=&quot;Analyze financial news content to determine sentiment and quantify market tone.&quot;,<br>    backstory=(<br>        &quot;A financial NLP specialist trained on industry-specific language to assess the sentiment of &quot;<br>        &quot;news headlines, articles, and reports. Helps contextualize market conditions by identifying &quot;<br>        &quot;positive, negative, or neutral sentiment trends in the financial media landscape.&quot;<br>    ),<br>    llm=llm<br>)<br><br># 4. Quantitative Stock Analyst<br>quant_analyst = Agent(<br>    role=&quot;Quantitative Stock Analyst&quot;,<br>    goal=&quot;Analyze historical stock data to uncover trends, indicators, and price forecasts using statistical techniques.&quot;,<br>    backstory=(<br>        &quot;An experienced financial modeler and time-series analyst who specializes in extracting insights &quot;<br>        &quot;from structured stock data. Skilled in technical indicators, regressions, and volatility analysis.&quot;<br>    ),<br>    llm=llm<br>)<br><br># 5. Financial Reporter<br>reporter = Agent(<br>    role=&quot;Financial Reporter&quot;,<br>    goal=&quot;Generate professional and concise financial reports based on research and analysis findings.&quot;,<br>    backstory=(<br>        &quot;A seasoned financial writer with a journalism and economics background, skilled at synthesizing &quot;<br>        &quot;complex analysis into clear, actionable reports for investors and executives.&quot;<br>    ),<br>    llm=llm<br>)</pre><blockquote>data_retriever agents has allow_delegation = False, which means:</blockquote><ul><li>The agent must perform its task directly.</li><li>It cannot call on or pass responsibility to another agent.</li></ul><p>Basically, it ensures the task is executed solely by the assigned agent, enhancing traceability, clarity, and overall control of the workflow.</p><h3>2. Focus &amp; Tasks</h3><p>In CrewAI, <strong>Focus &amp; Tasks</strong> allow complex workflows to be broken down into modular, goal-driven steps. Each task has a clearly defined objective and expected output, and it’s assigned to the most appropriate agent based on their specialization. This architecture not only streamlines execution but also improves traceability, accuracy and scalability of AI-driven workflows.</p><p>In our stock market analysis scenario, we assign tasks across our five specialized agents:</p><ul><li>📥 <strong>Fetch historical stock data</strong> from reliable sources, ensuring completeness and accuracy over a defined time period (e.g., the past year).</li><li>🔬 <strong>Research current market trends, news, and competitor activity</strong> to provide relevant industry context for the target stock or sector.</li><li>💭 <strong>Analyze the sentiment</strong> of collected news and reports to assess the overall market tone (positive, negative, or neutral) using financial NLP models.</li><li>📈 <strong>Perform technical and statistical analysis</strong> on the stock’s historical data to extract meaningful signals such as moving averages, volatility, or trend reversals.</li><li>📃 <strong>Generate a comprehensive financial report</strong> that synthesizes data insights, sentiment trends, and market context into a clear and actionable investment summary.</li></ul><blockquote>Here is how you can define tasks in CrewAI:</blockquote><pre>from crewai import Task<br><br># 1. Fetch Historical Stock Data<br>fetch_data_task = Task(<br>    description=&quot;Get historical stock data for {symbol} (past 1 year).&quot;,<br>    expected_output=&quot;JSON with open, close, high, low, and volume.&quot;,<br>    agent=data_retriever<br>)<br><br># 2. Research Market Trends<br>market_research_task = Task(<br>    description=&quot;Search for recent news articles or financial reports about {symbol} from reliable sources like Bloomberg, Reuters, Yahoo Finance, or CNBC.”&quot;,<br>    expected_output=&quot;Brief summary of key news and industry trends.&quot;,<br>    agent=insights_analyst<br>)<br><br># 3. Analyze News Sentiment<br>sentiment_analysis_task = Task(<br>    description=&quot;Assess sentiment of news related to {symbol}.&quot;,<br>    expected_output=&quot;Sentiment label with confidence scores.&quot;,<br>    agent=sentiment_analyst<br>)<br><br># 4. Run Technical Analysis<br>quant_analysis_task = Task(<br>    description=&quot;Analyze {symbol}&#39;s price data for trends and indicators.&quot;,<br>    expected_output=&quot;Summary of trends, volatility, and key metrics.&quot;,<br>    agent=quant_analyst<br>)<br><br># 5. Generate Report<br>report_generation_task = Task(<br>    description=&quot;Create a report summarizing data, sentiment, and insights for {symbol}.&quot;,<br>    expected_output=&quot;Concise investor report with key findings.&quot;,<br>    agent=reporter<br>)</pre><h3>3. Tools</h3><p>In CrewAI, <strong>Tools</strong> extend an agent’s capabilities by allowing it to interact with external systems such as APIs, databases, or web sources. This empowers agents to retrieve data, perform calculations, and make informed decisions based on real-time or historical information. Tools can be reused across agents and are essential for grounding LLM-driven agents in external knowledge.</p><p>CrewAI also enhances tool usage through features like cross-agent caching, which avoids redundant requests, speeds up execution, and helps mitigate rate-limiting from external services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*42xzU5OoTCbKyZMj2UxYnA.png" /><figcaption>Source: <a href="https://docs.crewai.com/concepts/tools">https://docs.crewai.com/concepts/tools</a></figcaption></figure><p>In our stock market analysis system, we use a mix of internal and external tools, such as:</p><ul><li>🔨 A <strong>yfinance</strong>-based tool retrieves structured historical stock data, including open, close, high, low, and volume metrics over a defined period (e.g., 1 year).</li><li>🔨 Web-enabled tools like <strong>SerperDevTool</strong> and <strong>WebsiteSearchTool</strong> are used to gather real-time industry news, competitor activity, and macroeconomic trends from across the web. Also, we can filter websites to search data.</li><li>🔨 A <strong>RoBERTa-based sentiment tool</strong> analyzes financial text to classify sentiment as positive, neutral, or negative with confidence scores.</li></ul><blockquote>Here’s how we define and assign tools for our application:</blockquote><pre>from crewai_tools import SerperDevTool, WebsiteSearchTool<br>from crewai.tools import tool<br>from crewai import Agent<br>import yfinance as yf<br><br># twitter-roberta-base-sentiment dependencies<br>from transformers import AutoTokenizer, AutoModelForSequenceClassification<br>from torch.nn.functional import softmax<br>import torch<br><br># Tool 1: yfinance<br>@tool<br>def get_stock_data(symbol: str, period: str = &quot;1y&quot;) -&gt; str:<br>    &quot;&quot;&quot;<br>    Fetch historical stock data for a given symbol and time period using yfinance.<br><br>    Returns:<br>        str: JSON-formatted OHLCV data or error message.<br>    &quot;&quot;&quot;<br>    try:<br>        data = yf.download(symbol, period=period)<br>        if data.empty:<br>            return f&quot;No data found for symbol: {symbol}&quot;<br>        return data.to_json()<br>    except Exception as e:<br>        return f&quot;Error fetching data for {symbol}: {str(e)}&quot;<br><br><br># Tool 2, 3: Web Search &amp; RAG<br>os.environ[&quot;SERPER_API_KEY&quot;] = config[&quot;SerperSearch&quot;][&quot;api&quot;] #update with your key<br>search_tool = SerperDevTool()<br>web_rag_tool = WebsiteSearchTool()<br><br>from crewai.tools import tool<br>from urllib.parse import urlparse<br><br>ALLOWED_DOMAINS = [<br>    &quot;bloomberg.com&quot;,<br>    &quot;reuters.com&quot;,<br>    &quot;cnbc.com&quot;,<br>    &quot;finance.yahoo.com&quot;,<br>    &quot;marketwatch.com&quot;,<br>    &quot;seekingalpha.com&quot;<br>]<br><br>@tool<br>def safe_website_summary(url: str) -&gt; str:<br>    &quot;&quot;&quot;<br>    filter unsupported URL<br>    &quot;&quot;&quot;<br>    domain = urlparse(url).netloc<br>    if not any(d in domain for d in ALLOWED_DOMAINS):<br>        return f&quot;Skipped unsupported source: {domain}. Please use financial news sources.&quot;<br>    <br>    # fallback to default scraper (already defined in web_rag_tool)<br>    return web_rag_tool.run(url)<br><br># Tool 4: FinBERT Sentiment Analyzer<br>tokenizer = AutoTokenizer.from_pretrained(&quot;cardiffnlp/twitter-roberta-base-sentiment&quot;)<br>model = AutoModelForSequenceClassification.from_pretrained(&quot;cardiffnlp/twitter-roberta-base-sentiment&quot;)<br><br>@tool<br>def analyze_financial_sentiment(text: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Analyze sentiment of financial text using FinBERT.<br><br>    Returns:<br>        dict: sentiment label, confidence score, and full probability breakdown.<br>    &quot;&quot;&quot;<br>    inputs = tokenizer(text, return_tensors=&quot;pt&quot;, truncation=True)<br>    with torch.no_grad():<br>        logits = model(**inputs).logits<br>        probs = softmax(logits, dim=1)[0]<br><br>    labels = [&quot;negative&quot;, &quot;neutral&quot;, &quot;positive&quot;]<br>    sentiment = labels[torch.argmax(probs)]<br>    return {<br>        &quot;sentiment&quot;: sentiment,<br>        &quot;confidence&quot;: round(probs.max().item(), 3),<br>        &quot;probabilities&quot;: dict(zip(labels, [round(p.item(), 3) for p in probs]))<br>    }<br><br># Agent Tool Assignments<br>data_retriever.tools.append(get_stock_data)<br>insights_analyst.tools.extend([search_tool, safe_website_summary])<br>sentiment_analyst.tools.append(analyze_financial_sentiment)</pre><h3>4. Cooperation</h3><p>In CrewAI, <strong>cooperation</strong> refers to how agents coordinate to complete a larger task. CrewAI supports different cooperation models:</p><ul><li><strong>Sequential: </strong>Agents execute tasks in a defined order, passing information downstream</li><li><strong>Parallel</strong> : Agents work independently on separate tasks simultaneously.</li><li><strong>Hierarchical: </strong>One or more agents (e.g., a manager or coordinator) delegate and oversee subtasks across other agents.</li></ul><blockquote>For our stock market analysis system, a <strong>sequential</strong> approach is ideal:</blockquote><pre>from crewai import Crew, Process<br><br>stock_analysis_crew = Crew(<br>    agents=[<br>        data_retriever,<br>        insights_analyst,<br>        sentiment_analyst,<br>        quant_analyst,<br>        reporter<br>    ],<br>    tasks=[<br>        fetch_data_task,<br>        market_research_task,<br>        sentiment_analysis_task,<br>        quant_analysis_task,<br>        report_generation_task<br>    ],<br>    process=Process.sequential,<br>    verbose=True # Useful for tracking steps and debugging<br>)</pre><h3>5. Guardrails</h3><p><strong>Guardrails</strong> in CrewAI help ensure your agents produce safe, valid, and expected outputs. They act as validation checkpoints that prevent agents from going off-track — whether that means producing incorrect data, hallucinating content, or falling into unproductive loops.</p><p>Guardrails are especially useful in:</p><ul><li>👮 Preventing downstream errors caused by malformed data</li><li>👮 Ensuring outputs stay within defined boundaries (e.g., date ranges, formats)</li><li>👮 Enforcing domain-specific constraints (e.g., confidence thresholds)</li></ul><blockquote>Here is an example of guardrail for our application:</blockquote><pre># Guardrail: Minimum Sentiment Confidence Threshold<br>def validate_sentiment_confidence(result):<br>    try:<br>        if isinstance(result, dict) and result.get(&quot;confidence&quot;, 0) &lt; 0.6:<br>            return False, {&quot;error&quot;: &quot;Sentiment confidence too low.&quot;}<br>        return True, result<br>    except Exception as e:<br>        return False, {&quot;error&quot;: str(e)}<br><br>sentiment_analysis_task.guardrail = validate_sentiment_confidence</pre><h3>6. Memory</h3><p>The goal of memory in CrewAI is to enable AI agents to self-improve and maintain continuity across interactions. CrewAI provides a sophisticated memory management system that allows agents to retain and recall context, making their behavior more coherent and adaptive over time.</p><ul><li>🧠 <strong>Short-Term Memory</strong>: Stores recent interactions using Retrieval-Augmented Generation (RAG), allowing agents to reference previous steps within the same session.</li><li>🧠 <strong>Long-Term Memory</strong>: Provides persistent storage using SQLite, enabling agents to remember key facts or prior analysis across sessions.</li><li>🧠 <strong>Entity Memory</strong>: Tracks entities (e.g., stock symbols, companies) encountered during tasks to enhance relevance and reduce redundancy.</li><li>🧠 <strong>Contextual Memory</strong>: Maintains conversation flow and state across complex multi-step workflows for more natural collaboration between agents.</li></ul><blockquote>CrewAI supports both short-term (RAG-style) and long-term (persistent) memory through a simple configuration flag. However, for this application, I’ve chosen not to enable memory to keep the execution lightweight and reduce context overhead.</blockquote><pre>stock_analysis_crew.memory = False</pre><h3>Dynamic Crew Kickoff with Stock Symbol Input</h3><p>To make our system dynamic and user-driven, we can adjust the task descriptions based on the stock symbol input:</p><pre>def run_stock_analysis(symbol: str):<br>    # Dynamically update task descriptions based on the stock symbol<br>    fetch_data_task.description = f&quot;Fetch historical stock data for {symbol} over the past year.&quot;<br>    market_research_task.description = f&quot;Conduct web research to gather recent news, market developments, and competitor trends related to {symbol} and the AI industry.&quot;<br>    sentiment_analysis_task.description = f&quot;Analyze the sentiment of recent news related to {symbol} to assess overall market tone.&quot;<br>    quant_analysis_task.description = f&quot;Analyze {symbol}&#39;s historical stock data to identify technical trends and indicators.&quot;<br>    report_generation_task.description = f&quot;Generate a comprehensive financial report summarizing data trends, sentiment analysis, and market insights for {symbol}.&quot;<br><br>    # Run the crew with the dynamic input<br>    inputs = {&quot;symbol&quot;: symbol}<br>    result = stock_analysis_crew.kickoff(inputs=inputs)<br>    print(result)<br><br>if __name__ == &quot;__main__&quot;:<br>    symbol_input = input(&quot;Enter Stock Symbol (e.g., AAPL): &quot;)<br>    run_stock_analysis(symbol=symbol_input)</pre><pre># Run your script from terminal<br>python main.py</pre><p>The image below shows a successful end-to-end execution of a <strong>CrewAI-based multi-agent system</strong> analyzing NVIDIA (NVDA):</p><p>Each agent ranging from <strong>Financial Data Retriever</strong> to <strong>Financial Reporter, </strong>executes its specialized task in sequence. The green check marks indicate that all agents:</p><ul><li>Retrieved and analyzed relevant data</li><li>Completed their tasks without error</li><li>Used the correct tools (like get_stock_data, Serper, and analyze_financial_sentiment)</li><li>Passed information down the pipeline seamlessly</li><li>(Number) → Number is how many times tool was called</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*OzsAuXRA5LmoOWgKMAnYow.png" /></figure><p>This clear structure and transparency are part of what makes CrewAI so powerful for orchestrating complex AI workflows like stock market analysis, where data, insights, sentiment, and reporting must be tightly coordinated.</p><p>Also, following is the final report generated by agents for <strong>NVIDIA (NVDA)</strong>.</p><blockquote>Analyzing the historical stock data and market variables for NVIDIA (NVDA) between the periods indicated, we observe a promising trend and market sentiment towards the stock. Notably, NVIDIA’s strategic investments in AI and technological advancements suggest a robust future outlook:</blockquote><blockquote>1. **Price Analysis**:<br> — **Closing Price Trend**: Starting from approximately $90.33 to a recent $109.67, NVIDIA’s stock has demonstrated strength with an ascending trajectory over time.<br> — **Volatility Insight**: The range between the highest and lowest prices ($112.87 and $89.17 respectively) indicates notable intra-day swings, potentially driven by market reactions and updates from NVIDIA.</blockquote><blockquote>2. **Volume Trends**:<br> — Trading volume showed a decline from 452,441,000 to 229,177,400 by the close of the period. This decrease may point towards a stabilization in investors’ perception concerning NVIDIA’s market position.</blockquote><blockquote>3. **Market News and Sentiment**:<br> — **Positive Developments**: Investments in a quantum computing initiative and enhancements in AI microservices position NVIDIA at the forefront of technological innovation.<br> — **Sentiment Analysis**: A high positive sentiment score of 0.866 signals robust investor confidence fueled by NVIDIA’s strategic direction and market activities.</blockquote><blockquote>**Recommendation**:<br>Investors are advised to consider NVIDIA as a strong prospect for long-term growth based on its consistent upward price movement, strategic market position in AI and technology, and strong investor sentiment. Continued focus on NVIDIA’s engagement with innovative technologies and market expansion strategies is recommended to leverage potential gains amidst AI advancements and tech industry growth.</blockquote><blockquote>This comprehensive analysis aims to equip investors with critical insights into NVIDIA’s stock performance, underpinned by data-driven analysis and recent market activities.</blockquote><p><strong>Disclaimer: </strong>I am not affiliated with NVIDIA in any professional capacity. NVIDIA was selected purely for demonstration purposes in this project.</p><h3>Benefits of Using CrewAI for Multi-Agent Approach:</h3><ul><li>🛠️ <strong>Fault Tolerance: </strong>It is designed to continue execution even when individual agents fail, avoiding full pipeline crashes.</li><li><strong>⚡ Efficiency</strong>: Automates repetitive tasks like data fetching, news scraping, and report generation.</li><li><strong>🎯 Accuracy</strong>:<strong> </strong>Specialized agents with clearly defined roles produce higher-quality outputs than generic monolithic systems.</li><li><strong>📈 Scalability</strong>: New agents, tools, or tasks can be added easily to adapt to expanding analytical needs (e.g., risk modeling, options pricing).</li><li><strong>🔒 Reliability</strong>: Guardrails ensure each task’s output meets minimum standards before passing results downstream, reducing error propagation.</li></ul><h3>Conclusion</h3><p>In this guide, I showcased how CrewAI can be used to build a modular, explainable multi-agent system for financial market analysis. While I’m not affiliated with <a href="https://www.crewai.com/">CrewAI</a>, I found its framework intuitive and effective for orchestrating specialized agents each handling tasks like data retrieval, sentiment analysis, quantitative evaluation, and report generation.</p><p>By combining key building blocks like:</p><ul><li><strong>Role-based agent architecture</strong></li><li><strong>Focused, goal-driven task assignments</strong></li><li><strong>External tool integration</strong> (e.g., <strong>yfinance</strong>, <strong>Serper</strong>, <strong>RoBERTa</strong>)</li><li><strong>Cooperation strategies</strong> (sequential task execution)</li><li><strong>Memory for contextual awareness</strong></li><li><strong>Guardrails for output validation</strong></li></ul><p>I was able to create a flexible pipeline that delivers reliable, structured investment insights.</p><p>Multi-agent systems like this represent a compelling step forward in AI-driven analytics offering a more modular, transparent alternative to black-box monoliths.</p><p>Stay tuned for more articles in this series!</p><h3>Add me on <a href="https://www.linkedin.com/in/kaushik-shakkari/">LinkedIn</a>. Thank you!</h3><p>If you are interested in the document parsing topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/2030d58d5fe9">Understanding Document Parsing</a></p><p>If you are interested in semantic search topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/9df1566193b0">Understanding Semantic Search</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*8raU3upnnqYiHdQr.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pscISK_yQFJRcEsv.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=024ded4e7709" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/understanding-ai-agents-part-2-building-a-multi-agent-system-for-stock-market-analysis-024ded4e7709">Understanding AI Agents — (Part 2: Building a Multi-Agent System for Stock Market Analysis)</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding AI Agents — (Part 1: What Are They, How They Work, Their Challenges, and Future…]]></title>
            <link>https://generativeai.pub/understanding-ai-agents-part-1-what-are-they-how-they-work-their-challenges-and-future-f8d3e4223e11?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8d3e4223e11</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 13:14:44 GMT</pubDate>
            <atom:updated>2025-06-22T07:00:55.483Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding AI Agents — (Part 1: What Are They, How They Work, Their Challenges, and Future Trends)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1yA-ynVidlX6ei-kCxqdFw.png" /><figcaption>Figure form the Author</figcaption></figure><h3>Introduction:</h3><p>AI agents are rapidly transforming how we interact with technology, automating complex workflows, solving multi-step problems, and enabling seamless user experiences across industries.</p><blockquote><a href="https://www.capgemini.com/wp-content/uploads/2024/11/Generative-AI-in-Organizations-Refresh_25112024.pdf">According to a Capgemini survey </a>of over 1000 executives at large enterprises reveal that 82%<strong> of organizations planning AI agents adoption within the next 3 years.</strong></blockquote><p>In <a href="https://medium.com/generative-ai/understanding-ai-agents-part-0-are-ai-agents-just-another-buzzword-for-reusable-components-e7888d43faca">the last article</a>, I introduced AI Agents. At its core, an AI agent is a program designed to automatically <strong>observe, decide, act, and learn from its environment</strong>. In most of today’s AI agents, typically the brain is a Large Language Model (LLM). However, building robust AI agents requires more than just plugging a LLM into an interface. It involves a careful orchestration of tools, planning frameworks, memory systems, and reflection mechanisms to ensure the agent performs reliably and efficiently in real-world scenarios.</p><p>In this article, I will introduce the concepts like tools, memory, decision-making mechanisms which enable AI agents to function autonomously and adaptively in dynamic environments. By the end of this read, you’ll have a clear understanding of how AI agents work, the challenges involved in their design, and the emerging trends shaping their future.</p><h3>How AI Agents Work?</h3><p>An AI agent typically operates within a defined environment (e.g., a web browser or a chat interface) and interacts with it using tools (e.g., web scraping tool, search api, or even a calculators) to complete desired task. Tools extend the agent’s capabilities beyond what its internal model can do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6tsBBW4rgHJ1ffPIbItoXA.png" /><figcaption>Figure form the Author | Trading Agent</figcaption></figure><p>For example, the agent could use web scraping tools like Beautiful Soup to gather the latest articles from financial news websites, and simultaneously query Yahoo Finance API for real-time stock prices. It would then process this data using natural language processing (NLP) models to extract key insights, such as sentiment analysis of news articles, and combine these insights with quantitative stock data to generate comprehensive market reports or trading recommendations.</p><h3>Core Components of AI Agents:</h3><p>There are 4 main components in AI agents — Tools, Planning, Memory Systems &amp; Reflection Mechanisms. Lets understand each of them.</p><h4>1. Tools:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*73VUsY4cZ1Mla7ZzMneXOw.png" /><figcaption>Source: <a href="https://activantcapital.com/research/ai-agents">https://activantcapital.com/research/ai-agents</a></figcaption></figure><p>Tools are basically external APIs or functions the agent can use to enhance its capabilities. AI agents can rely on tools to overcome their inherent limitations. For instance, while LLMs are excellent at generating text-based responses, they may lack up-to-date knowledge or computational precision.</p><blockquote>Here are a few examples of tools:</blockquote><ul><li><strong>Web Browsers:</strong> For retrieving real-time information (e.g., ChatGPT’s integration with Bing Search).</li><li><strong>Code Interpreters:</strong> For executing Python scripts or SQL queries in real-time.</li><li><strong>Write Operations:</strong> For modifying databases or sending emails autonomously.</li><li><strong>Calculators:</strong> For performing precise numerical operations.</li></ul><blockquote>The effectiveness of an agent often depends on selecting the right tools for its task domain. However, adding too many tools can lead to inefficiencies and errors due to increased complexity in decision-making. <a href="https://www.gabormelli.com/RKB/AI_System_Ablation_Study">Research shows that optimal tool selection requires rigorous testing through ablation studies to identify which tools contribute most to task performance.</a></blockquote><h4>2. Planning:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fD03w5C1zpfpxZhMXgj0wg.png" /><figcaption>Source: <a href="https://huyenchip.com/2025/01/07/agents.html">https://huyenchip.com/2025/01/07/agents.html</a></figcaption></figure><p>Planning is a process of breaking down complex tasks into smaller steps and executing them in sequence or parallel. It is one of the most critical aspects of building an effective AI agent. Unlike single-turn interactions where an LLM generates a response in one go, agents often need to handle multi-step workflows that require reasoning over time.</p><blockquote>Some examples of agents use planning include:</blockquote><ul><li>A <strong>coding assistant</strong> might break down <em>“</em><strong><em>fix bug in cod</em></strong><em>e”</em> into steps like <em>“</em><strong><em>identify error</em></strong><em>,” “</em><strong><em>generate fix</em></strong><em>,” and “</em><strong><em>run tests</em></strong><em>.”</em></li><li>A <strong>customer service agent</strong> might decompose <em>“</em><strong><em>resolve customer complaint</em></strong><em>”</em> into subtasks like <em>“</em><strong><em>retrieve customer history</em></strong><em>,” “</em><strong><em>analyze complaint</em></strong><em>,” and “</em><strong><em>propose resolution</em></strong><em>.”</em></li><li>An <strong>AI tutor agent</strong> could break down <em>“</em><strong><em>prepare student for math exam</em></strong><em>”</em> into <em>“</em><strong><em>assess current knowledge</em></strong><em>,” “</em><strong><em>identify weak areas</em></strong><em>,” “</em><strong><em>provide targeted practice problems</em></strong><em>,” “</em><strong><em>explain complex concepts</em></strong><em>,” and “</em><strong><em>conduct mock tests</em></strong><em>.”</em></li><li>A <strong>project management agent </strong>might decompose <em>“</em><strong><em>launch new product</em></strong><em>”</em> into <em>“</em><strong><em>conduct market research</em></strong><em>,” “</em><strong><em>develop prototype</em></strong><em>,” “</em><strong><em>create marketing strategy</em></strong><em>,” “</em><strong><em>set up production line</em></strong><em>,” and “</em><strong><em>coordinate distribution channels</em></strong><em>.”</em></li></ul><blockquote>Frameworks like <a href="https://research.google/blog/react-synergizing-reasoning-and-acting-in-language-models/">ReAct (Reasoning + Acting)</a> combine reasoning with action-taking in iterative loops. The agent observes its environment after each action and adjusts its plan accordingly. This reduces errors caused by hallucinations or invalid assumptions.</blockquote><h4>3. Memory Systems:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P2HI94RPEavnD4NukCnc5Q.png" /><figcaption>Source: <a href="https://www.linkedin.com/pulse/long-term-memory-agenticai-debmalya-biswas-56ioe/">https://www.linkedin.com/pulse/long-term-memory-agenticai-debmalya-biswas-56ioe/</a></figcaption></figure><p>Memory systems store contextual information that allows the agent to recall past interactions or information across sessions. Without memory, an agent would treat every interaction as isolated, leading to repetitive or irrelevant responses.</p><blockquote>There are primarily 3 types of memory:</blockquote><ul><li><strong>Sensory Memory:</strong> Sensory memory in AI agents is similar to human sensory inputs from eyes, noise, and ears etc. In agents, it captures and processes responses from tools the agents employ, filtering vast amounts of data to retain only significant information.</li><li><strong>Short-Term Memory:</strong> The essential information from sensory memory is transferred to short-term or working memory in AI agents. This working memory has a limited capacity like context windows in Large Language Models (LLMs).</li><li><strong>Long-Term Memory: </strong>The information from short term memory is encoded and passed to long-term memory for retaining information over extended periods. This type of memory enables agents to handle long-running processes, support human-in-the-loop workflows, and maintain state across sessions.</li></ul><blockquote>Below are some examples of how AI agents use memory for efficiency:</blockquote><ul><li>A<strong> coding assistant agent</strong> could leverage memory by maintaining a database of past bugs, their solutions, and associated test results to quickly identify patterns and apply proven fixes to similar issues in the future.</li><li>By storing and analyzing past customer interactions, complaint resolutions, and individual customer preferences, the <strong>customer service agent</strong> can provide more personalized and efficient service, anticipating needs and offering tailored solutions.</li><li>An <strong>AI tutor agent</strong> could use memory to track each student’s learning progress, store effective teaching strategies for different concepts, and maintain a bank of practice problems, allowing for personalized learning paths and more targeted instruction.</li><li>Memory could be utilized to store data from past projects, including timelines, outcomes, and lessons learned, enabling a <strong>project management agent</strong> agent to make more informed decisions, anticipate potential issues, and optimize processes for future product launches.</li></ul><blockquote>Memory plays a crucial role. However, it introduces challenges such as scalability (how much data can be stored efficiently) and privacy (ensuring sensitive information is protected).</blockquote><h4>4. Reflection Mechanisms:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*273ogzod6NtlGGjduHCDVg.png" /><figcaption>Source: <a href="https://blog.langchain.dev/reflection-agents/">https://blog.langchain.dev/reflection-agents/</a></figcaption></figure><p>Even with robust planning and memory systems, agents are inherently prone to errors especially when tasked with ambiguous, dynamic, or highly complex problems. These challenges arise because agents often operate in environments where the correct course of action is not immediately clear, or where multiple conflicting objectives must be balanced. In such cases, traditional forward-planning approaches and static memory retrieval alone may not be sufficient to ensure consistent task success.</p><p>Reflection allows agents to critically evaluate their own performance, recognize patterns in their decision-making processes, and refine their strategies over time. By incorporating reflection, agents can iteratively improve, learning not only from explicit feedback but also from introspection about their successes and failures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X_Y_LmbYHMImhl1spryQxQ.png" /><figcaption>Figure from the Author | Reflection Mechanism Example</figcaption></figure><p>To understand it better, let’s consider an example of an agent designed to automate data analysis tasks. If agent fails to generate accurate reports due to incorrect tool usage or misinterpretation of data formats, a reflection mechanism would prompt the agent to analyze the root cause of the failure. Was the error due to selecting an inappropriate analysis method? Did the agent misunderstand the input data schema? Was there data quality issue? Upon identifying the issue, the agent can adjust its approach such as selecting a more appropriate tool or understanding schema better or revising its data processing pipeline before its next attempt.</p><blockquote>Studies show that incorporating reflection mechanisms can significantly boost task success rates — for example, the <a href="https://arxiv.org/html/2410.02052v3">ExAct paper introduces the Reflective Monte Carlo Tree Search (R-MCTS) framework and reports a 6% to 30% relative improvement in task performance across various benchmarks</a> compared to previous state-of-the-art methods.</blockquote><h3>Challenges in Building AI Agents:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a_RH15oPG73N8NkIPkDU0A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Markus Spiske</a> on <a href="https://unsplash.com/photos/a-triangle-shaped-sign-on-a-yellow-wall-VO5w2Ida70s?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Despite their potential, AI agents face several challenges that limit their effectiveness in real-world applications:</p><ul><li><strong>Error Propagation &amp; Compounding:</strong> Small mistakes in early steps can cascade into larger failures as tasks progress in multi-step workflows.</li><li><strong>Decision Making &amp; Incorrect Tool Selection:</strong> Agents sometimes invoke incorrect tools due to poor reasoning or ambiguous instructions.</li><li><strong>Accountability:</strong> Write-enabled agents like the ones that modify databases pose risks if they act on incorrect assumptions or hallucinated data.</li><li><strong>Safety Concerns: </strong>Being highly autonomous also makes them vulnerable to adversarial attacks, data breaches, and manipulation.</li><li><strong>Transparency &amp; Explainability: </strong>The “<strong>black box</strong>” nature of many AI models makes it difficult to understand how decisions are made, raising concerns in critical applications like healthcare and finance.</li><li><strong>Generalization &amp; Bias:</strong> Agents trained on specific tasks often struggle when faced with novel scenarios outside their training data distribution. Moreover, they often inherit biases from their training data, which can lead to unfair or discriminatory outcomes.</li></ul><blockquote>The above challenges are particularly problematic when agents lack mechanisms for reflection (error correction), too many tools, and data quality issues leading to bad performance and even task failure. Also, the lack of transparency further complicates debugging, auditing, and regulatory compliance.</blockquote><h3>Emerging Trends in AI Agents:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlVaOKhCibvim3YPT08M5A.jpeg" /><figcaption>Picture generate using Dall-E</figcaption></figure><p>The field of AI agents is evolving rapidly, with several transformative trends shaping their future. In my view, below key trends are the most impactful which are also backed by insights from recent research and industry developments.</p><h4>1. Multi-Agent Systems:</h4><p><a href="https://www.researchgate.net/publication/367228899_Past_Present_and_Future_Trends_in_Multi-Agent_System_Technology">Multi-agent systems (MAS)</a> involve multiple specialized agents collaborating to solve complex tasks. Instead of relying on a single generalist agent, MAS enables agents with distinct expertise to work together, improving scalability and efficiency.</p><ul><li><strong>Application Example: </strong>In supply chain management, AI agents expected to handle increasingly dynamic environments where coordination among agents is critical.</li><li><strong>Potential challenges:</strong> Coordination increases latency especially when the number of agents grows eventually requiring advanced orchestration mechanisms to handle scale.</li></ul><h4>2. Explainable AI (XAI) in Agents:</h4><p>Explainability is crucial for addressing the need for transparency and trust in increasingly complex AI systems.</p><ul><li><strong>Application Example: </strong><a href="https://arxiv.org/abs/2404.14394">MIT researchers have introduced an automated interpretability agent (AIA)</a> that mimics a scientist’s experimental processes to explain other AI systems. This agent can plan and perform tests on computational systems ranging from individual neurons to entire models, producing explanations in various forms including language descriptions and code that reproduces the system’s behavior.</li><li><strong>Current challenges:</strong> AI agents involve complex, short-lived decision chains based on real-time data, making their decision paths harder to document, analyze and understand.</li></ul><h4>3. Agentic Evaluation Frameworks:</h4><p>Next-gen AI agent evaluation systems combine automated benchmarking with enterprise-grade compliance monitoring to assess performance across technical, operational, and ethical dimensions. These frameworks address the growing complexity of AI agents handling sensitive workflows.</p><ul><li><strong>Application Example: </strong><a href="https://aisera.com/blog/ai-agent-evaluation/">Aisera’s CLASSic Framework</a> evaluates agents across Cost, Latency, Accuracy, Security, and Stability (CLASS) metrics.</li><li><strong>Current challenges:</strong> Standardizing evaluation process and protocols across industries is challenging. Evaluation is still custom and application centric.</li></ul><h3>Conclusion:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*li6cUaw8_SmgheEbi1I91A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@aaronburden?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Aaron Burden</a> on <a href="https://unsplash.com/photos/road-tunnel-wallpaper-gmy25xvSkq8?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>AI agents are quickly changing how we use technology by automating difficult tasks and making user experiences better. Companies are already planning to use them widely. Building these agents well means carefully thinking about tools, planning, memory, and ways for the agents to reflect on their work.</p><p>AI agents have a lot of potential, but there are problems to solve. These include mistakes getting worse over time, making good decisions, responsibility, being clear about how decisions are made, and being able to handle new situations. New ideas like multi-agent systems (where many agents work together), explainable AI (which makes AI decisions easier to understand), and ways to evaluate agents offer good solutions. As this field grows, more research and development will help AI agents reach their full potential in the real world.</p><p>The journey toward truly intelligent autonomous systems is far from over but it’s one of the most exciting developments in artificial intelligence today. Stay tuned for more articles in this series!</p><h4>Add me on <a href="https://www.linkedin.com/in/kaushik-shakkari/">LinkedIn</a>. Thank you!</h4><p>If you are interested in document parsing topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/2030d58d5fe9">Understanding Document Parsing</a></p><p>If you are interested in semantic search topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/9df1566193b0">Understanding Semantic Search</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*YiC0OmnBQQ_YvjBl.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*I2oHu8UGrO2Cs8Km.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8d3e4223e11" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/understanding-ai-agents-part-1-what-are-they-how-they-work-their-challenges-and-future-f8d3e4223e11">Understanding AI Agents — (Part 1: What Are They, How They Work, Their Challenges, and Future…</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding AI Agents — (Part 0:  Are AI Agents Just Another Buzzword for Reusable Components…]]></title>
            <link>https://generativeai.pub/understanding-ai-agents-part-0-are-ai-agents-just-another-buzzword-for-reusable-components-e7888d43faca?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7888d43faca</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 15:54:19 GMT</pubDate>
            <atom:updated>2025-01-28T15:54:19.762Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding AI Agents — (Part 0: Are AI Agents Just Another Buzzword for Reusable Components and Pipelines?)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MFUtA1dUQ2UQhY49E_7Upg.png" /><figcaption>Picture from the Author</figcaption></figure><h3>Table of Contents:</h3><ol><li><a href="#6245">Introduction</a></li><li><a href="#67ab">What is a reusable component?</a></li><li><a href="#c69b">What is a pipeline?</a></li><li><a href="#20c5">What is an AI agent?</a></li><li><a href="#83f9">Do we always need AI Agents?</a></li><li><a href="#fb79">Conclusion</a></li></ol><h3>Introduction:</h3><p>Artificial Intelligence is evolving everyday and new buzzwords are everywhere. One of the latest hot terms in 2025 is <strong>“AI Agent.”</strong> You’ve probably seen it in <a href="https://www.theverge.com/2024/11/13/24295879/openai-agent-operator-autonomous-ai"><strong><em>product announcements</em></strong></a>, <a href="https://arxiv.org/abs/2402.01968"><strong><em>research papers</em></strong></a>, and <a href="http://social media platform"><strong><em>online communities</em></strong></a><strong><em> </em></strong><em>etc</em>. But what exactly are AI agents, and are they really something new?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7EeWS6qtbjsOzDP3k5kDvg.png" /><figcaption>Source: <a href="https://blog.samaltman.com/reflections">Sam Altman, CEO Open AI</a></figcaption></figure><p>Some argue that AI agents are genuinely innovative, capable of reshaping how we build intelligent systems. Others see them as a fancy way of repackaging familiar concepts like <strong>reusable components and pipelines</strong>. So, who’s right?</p><p>In this article, I’ll try to:</p><ul><li><strong>Unpack what AI agents are</strong></li><li><strong>Explain how they differ from traditional AI pipelines</strong></li><li><strong>Help you figure out if the excitement around them is truly warranted</strong></li></ul><p>Let’s dive in!</p><h3>What is a Reusable Component?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QaTEh3Wu9xuY6_785MVKXg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@calebeangel?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Caleb Angel</a> on <a href="https://unsplash.com/photos/child-pointing-to-red-interlocking-brick-toy-Hq32VUY0xMg?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Before diving into AI agents, let’s clarify a foundational idea of reusable components. These are <strong>modular building blocks designed to perform a specific task</strong> and be easily integrated into various applications or systems. Think of them as tools in a well-stocked toolbox where each tool is specialized for a particular function but can be reused in multiple projects.</p><p>To understand this better, let’s consider a smart home example.</p><p>Imagine setting up a <strong>smart home</strong>. You might have:</p><ul><li>A <strong>thermostat</strong> module for temperature control</li><li>A <strong>lighting</strong> module for adjusting brightness</li><li>A <strong>security</strong> module for locking doors</li><li>A <strong>speaker </strong>module to play songs</li></ul><p>Each component or module can be reused in different smart home setups. However, they <strong>don’t automatically collaborate or adapt to one another unless someone explicitly program</strong> that coordination. For instance, if you want the thermostat to adjust the temperature based on occupancy data, you must code that interaction.</p><blockquote>💡 <strong>Key Takeaway</strong>: <strong>Reusable components provide flexibility and efficiency, but require manual coordination when you want them to work together.</strong></blockquote><h3>What is a Pipeline?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3yYK-51Wlb9mzGakojMDtw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Kelly Sikkema</a> on <a href="https://unsplash.com/photos/toddlers-playing-building-block-toys-JRVxgAkzIsM?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>While reusable components act as building blocks, a pipeline orchestrates these blocks into a structured workflow. Pipelines typically follow a <strong>linear or rule-based sequence of tasks</strong>. This is especially useful for predictable processes.</p><p>Let’s see some scenarios of how a pipeline could manage a smart home:</p><h4>Example 1: Evening Routine Pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFFJQqINjbgK8y9Lqqzv2Q.png" /><figcaption>Source: Picture from the Author</figcaption></figure><p><strong>Trigger</strong>: 7:00 PM Monday to Friday</p><p><strong>Actions</strong>:</p><ol><li><strong>Dim</strong> the lights to <strong>30%</strong></li><li><strong>Increase</strong> the thermostat by <strong>2 degrees</strong></li><li><strong>Lock</strong> all <strong>exterior doors</strong></li></ol><p>The above pipeline runs <strong>everyday tasks</strong> — each component (lighting, thermostat, and security) performs a specific action in a fixed order.</p><h4>Example 2: Weekend Party Pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6goODc-a8HA9nWP6XOTTLQ.png" /><figcaption>Source: Picture from the Author</figcaption></figure><p><strong>Trigger</strong>: Let’s say you press “Party Mode” on a smart home app or configure by default at 6:00 PM, Saturday</p><p><strong>Actions</strong>:</p><ul><li><strong>Activate</strong> multi-room speakers with a <strong>party playlist</strong></li><li><strong>Adjust</strong> living room lights to a <strong>party color</strong> setting</li></ul><p>The above pipeline run <strong>occasional tasks </strong>— only speaker and light components performs a specific action in a fixed order.</p><h4>Example 3: Vacation Pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0NIrALuIng33Duj2yhEiw.png" /><figcaption>Source: Picture from the Author</figcaption></figure><p><strong>Trigger</strong>: You press “Vacation Mode” before leaving to holiday</p><p><strong>Actions</strong>:</p><ol><li><strong>Turn off</strong> all interior <strong>lights</strong></li><li><strong>Set</strong> the thermostat to an <strong>energy-saving temperature</strong></li><li><strong>Lock</strong> all <strong>doors</strong> and <strong>arm</strong> the <strong>security system</strong></li></ol><p>The above pipeline is for <strong>vacations</strong> — it saves energy and secures your home by executing a list of preconfigured components.</p><blockquote><strong>💡 Key Takeaway: Pipelines effectively handle structured, predefined processes. Each module does its job in a linear workflow, giving the illusion of an automated system. They can be manually configured but they don’t adapt on their own.</strong></blockquote><blockquote>For a deeper dive into real world reusable components and pipeline-based AI solutions check out my article on <a href="https://medium.com/generative-ai/understanding-document-parsing-part-2-modern-document-parsing-explained-modular-pipelines-bb605c786293"><strong>Modern Document Parsing with Modular Pipelines</strong></a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I-njGSKPU6EeWnUJXASQOg.png" /><figcaption>Source: Picture from Author’s Previous Article: <a href="https://medium.com/generative-ai/understanding-document-parsing-part-2-modern-document-parsing-explained-modular-pipelines-bb605c786293">https://medium.com/generative-ai/understanding-document-parsing-part-2-modern-document-parsing-explained-modular-pipelines-bb605c786293</a></figcaption></figure><h3>What Is an AI agent?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HHmlcWhprT79fp7sICRfZg.jpeg" /><figcaption>Source: Picture generated using <a href="https://openai.com/index/dall-e-3/">OpenAI’s DALL·E model</a></figcaption></figure><p>At its core, an AI agent is a program designed to <strong>observe, decide, act, and learn from its environment</strong>. Unlike traditional software systems (and pipeline-based systems) that follow static, predefined instructions, AI agents are <strong>autonomous and can adapt to new data or changing conditions in real time</strong>.</p><p>To better understand this concept, let’s again take smart home example. Instead of following a fixed schedule (like running everyday pipeline at 7 PM), a smart home AI agent might:</p><ol><li><strong>Observe real-time data</strong> such as weather conditions, occupancy, calendar events, and activity patterns like entertainment usage (e.g., TV or music streaming).</li><li><strong>Make decisions and act dynamically </strong>like if it’s cold outside and you typically come home at 6 PM, it might start heating the house in advance so it’s comfortable upon arrival.</li><li><strong>Learn</strong> from your behavior over time — such as your preference for warmer temperatures on weekend mornings, or your habit of staying up late to watch movies on Fridays. Might learn and can even pre-heat your smart car like Tesla before you go to office in the morning.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETiIWg6abvh0sF-v0bI5KA.png" /><figcaption>Source: Picture from the Author</figcaption></figure><blockquote><strong>💡 Key Takeaway: </strong>The AI agent orchestrates various modules and dynamically integrates additional tools or data sources without any manual reconfiguration. It goes beyond linear workflows to make context-aware decisions, adapting and improving with experience.</blockquote><h3>Do We Always Need AI Agents?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J-AE9CnF4O_K1qGm8OvzNw.png" /><figcaption>Source: <a href="https://trends.google.com/trends/explore?date=today%205-y&amp;q=AI%20Agent,Pipelines&amp;hl=en">Google Trends</a></figcaption></figure><p>The short answer is not always. Whether you need an AI agent depends largely on the complexity of your use case and the level of adaptability required.</p><ol><li><strong>Stable, Predictable Tasks: </strong>If you have a well-defined, repetitive workflow like standard data preprocessing or nightly batch tasks, pipelines may be enough. They’re simpler to implement, easier to maintain, and typically require less computational overhead.</li><li><strong>Highly Dynamic Environments: </strong>If your application environment changes frequently and unpredictably like adaptive control in self-driving vehicles — an AI agent can be a better choice. Agents can automatically refine their strategies based on new data, reducing the need for manual updates.</li><li><strong>Hybrid Approaches: </strong>In many cases, a hybrid approach works best. You could use pipelines for predictable tasks but layer on an AI agent that only steps in for difficult real-time decision-making or if there is a need for orchestrating multiple pipelines when conditions change.</li><li><strong>Cost vs. Benefit: </strong>While AI agents offer flexibility and potential for continuous improvement, they can be more costly and complex to develop. The training and ongoing adaptation require more robust data pipelines, computational resources, data sources and monitoring tools. Before jumping on the AI agent buzz, its important to weigh the benefits against the added complexity.</li></ol><blockquote><strong>💡 Key Takeaway: Always ask — “Does the environment demand continuous learning and adaptation?” If the answer is yes, consider an AI agent. Otherwise, a hybrid or simpler pipeline might be the right solution.</strong></blockquote><h3>Conclusion</h3><p>Even though AI agents look like just another tech buzzword, they represent a significant shift from <strong>fixed, rule-driven systems to adaptive, learning-oriented</strong> solutions. Although reusable components and pipelines remain essential in many applications, the agility and autonomy offered by AI agents can be a game-changer in environments where conditions are constantly evolving.</p><blockquote>Whether you need an AI agent or not or a hybrid solution highly depends on the nature of your problem and the resources you can commit.</blockquote><p>In upcoming articles in the series, we will explore how AI agents are architected, different types of them, their common use cases, and best practices for deploying them in real-world environments.</p><h4><strong>Add me on </strong><a href="https://www.linkedin.com/in/kaushik-shakkari/"><strong>LinkedIn</strong></a><strong>. Thank you!</strong></h4><p>If you are interested in document parsing topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/2030d58d5fe9">Understanding Document Parsing</a></p><p>If you are interested in semantic search topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/9df1566193b0">Understanding Semantic Search</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*WBdcKvCg0v8bAMTZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*6c_oC6kC7vu_TjdU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7888d43faca" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/understanding-ai-agents-part-0-are-ai-agents-just-another-buzzword-for-reusable-components-e7888d43faca">Understanding AI Agents — (Part 0:  Are AI Agents Just Another Buzzword for Reusable Components…</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hope or Hype?]]></title>
            <link>https://generativeai.pub/hope-or-hype-f14e06770fa9?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f14e06770fa9</guid>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 06:28:57 GMT</pubDate>
            <atom:updated>2025-01-16T06:28:57.698Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring Customer Mindsets, Industry Perspectives, and the Future of AI and LLM Innovation for AI applications</h4><h3>Hope or Hype? — Will Big General-Purpose LLMs Beat Todays Small Custom Domain Specialized AI Solutions in 2025?</h3><h4>Why Businesses are Betting Big on GenAI, and What’s Holding Them Back in 2025?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RKLjTna_Rav1w8sk3gCHcw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@solenfeyissa?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Solen Feyissa</a> on <a href="https://unsplash.com/photos/a-person-holding-a-cell-phone-in-their-hand-hWSNT_Pp4x4?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><h3>Table of Contents</h3><ol><li><a href="#097b">Introduction</a></li><li><a href="#ad1d">How Large Language Models (LLMs) Have Evolved?</a></li><li><a href="#5883">What Customers See in LLMs?</a></li><li><a href="#8fb2">Strengths of General-Purpose LLMs</a></li><li><a href="#bcae">Challenges of Relying on LLMs</a></li><li><a href="#7a07">Emerging Trends in LLM Development</a></li><li><a href="#05fc">Hope or Hype? The Verdict</a></li></ol><h3>Introduction:</h3><p>AI is evolving rapidly. My recent discussions with applied AI customers, developers and thought leaders about off-the-shelf general purpose large language models reveal a fascinating pattern. Initially, people are amazed by how effortlessly these big models function. They’re drawn to the fact that these powerful tools, supported by major tech players, seem to understand dynamic requests with minimal effort. However, as the conversation goes on, some doubts start to surface.</p><p><strong>Businesses question whether they’re:</strong></p><ol><li>Using LLMs that are <a href="https://arxiv.org/pdf/2412.21187"><strong>overkill for simpler tasks</strong></a>?</li><li>Assuming that a bigger model is always better, <a href="https://www.reddit.com/r/ChatGPTPro/comments/1hy7pdc/is_chatgpt_pro_worth_200/"><strong>but at what cost</strong></a>?</li><li>Over-investing in LLMs for <a href="https://chatgpt.com/c/67836b88-435c-800f-a3ad-36cabf05161e"><strong>niche problems</strong></a>?</li></ol><blockquote>Are these models <strong>ideal for production</strong>, or are they just another instance of <strong>market hype</strong>?</blockquote><p>This article delves into the evolution of LLMs, their strengths, and challenges, while examining how these advancements shape customer adoption and perceptions of their potential.</p><h3>How Large Language Models (LLMs) Have Evolved?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G4PhgqVzwkPRcxef3m6rQw.png" /><figcaption>Source: Original BERT paper (<a href="https://arxiv.org/pdf/1810.04805">https://arxiv.org/pdf/1810.04805</a>)</figcaption></figure><p>Large language models (LLMs) have undergone significant advancements over the past few years. The journey began with <a href="https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"><strong><em>attention mechanisms</em></strong></a>, which laid the foundation for <a href="https://paperswithcode.com/method/transformer"><strong><em>transformer architectures</em></strong></a>. This pivotal concept was followed by foundational models like <a href="https://arxiv.org/abs/1810.04805"><strong><em>BERT (Bidirectional Encoder Representations from Transformers)</em></strong></a>, which introduced bidirectional context understanding.</p><p>Subsequently, the emergence of <a href="https://arxiv.org/abs/2305.10435"><strong><em>GPT</em></strong></a> models demonstrated remarkable capabilities in text generation. Each iteration of these models — from <a href="https://llmmodels.org/blog/chatgpt-and-openais-evolution-from-gpt-2-to-gpt-4/"><strong><em>GPT-2 to GPT-4</em></strong></a> — saw exponential increases in parameter counts, improved contextual understanding, and enhanced coherence in outputs. These innovations allowed models to process vast amounts of data more efficiently and capture nuanced relationships between words, resulting in outputs that closely resemble human language.</p><h4>Key advancements include:</h4><ul><li><strong>Pre-training methodologies:</strong> Pre-training methodologies evolved significantly. Early models relied primarily on unsupervised learning from massive datasets, while newer models employ techniques such as <a href="https://neptune.ai/blog/reinforcement-learning-from-human-feedback-for-llms"><strong><em>Reinforcement Learning from Human Feedback (RLHF)</em></strong></a> to refine their outputs.</li><li><strong>Open-source innovations:</strong> The rise of <a href="https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/"><strong><em>open-source models</em></strong></a> including <a href="https://www.deepseek.com/"><strong><em>DeepSeek</em></strong></a>, <a href="https://www.google.com/search?q=LLaMA&amp;oq=LLaMA&amp;gs_lcrp=EgZjaHJvbWUyDAgAEEUYORixAxiABDINCAEQLhjUAhixAxiABDIKCAIQLhixAxiABDIMCAMQIxgnGIAEGIoFMhIIBBAAGBQYgwEYhwIYsQMYgAQyDQgFEAAYgwEYsQMYgAQyDQgGEAAYgwEYsQMYgAQyBggHEEUYPNIBBzEzMGowajSoAgCwAgE&amp;sourceid=chrome&amp;ie=UTF-8"><strong><em>LLaMA</em></strong></a>, <a href="https://mistral.ai/news/mixtral-of-experts/"><strong><em>Mixtral</em></strong></a> and <a href="https://bigscience.huggingface.co/blog/bloom"><strong><em>Bloom</em></strong></a> has democratized access to cutting-edge LLM technology. This open ecosystem expands innovation and enables smaller organizations to leverage advanced AI capabilities without incurring huge costs.</li><li><strong>Fine-tuning for niche applications:</strong> <a href="https://www.datacamp.com/tutorial/fine-tuning-large-language-models"><strong><em>Fine-tuning on domain-specific data</em></strong></a> has bridged the gap between general-purpose models and specialized tasks.</li></ul><h3>What Customers See in LLMs?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*njanlD_8P8JOgQMct_aEaw.jpeg" /></figure><p>Today’s customers see LLMs as a way to simplify and future-proof their AI strategies. Organizations are drawn to LLMs for several reasons:</p><ul><li><strong>Interest for Simplification of Processes / Unified Solutions:</strong> Instead of breaking a problem into multiple subproblems solved by separate models, many businesses prefer <strong>one powerful LLM</strong> to handle everything — reducing the maintenance overhead of multiple processes.</li><li><strong>Ease of Use:</strong> LLMs are often presented as ready-to-use solutions, requiring minimal setup and enabling faster deployment.</li><li><strong>Strategic Differentiation</strong>: Early adopters can gain a competitive edge by offering unique AI-powered services, standing out in crowded markets.</li><li><strong>Credibility and Innovation:</strong> Backed by the largest names in tech, LLMs come with the assurance of ongoing updates and cutting-edge developments, instilling trust in their capabilities — and drawing in investors.</li></ul><blockquote>These motivations drive initial adoption. Customers often see these models as a solution to every problem, but a deeper look reveals where LLMs excel and where they may fall short. This perspective is key to aligning their potential with actual business needs.</blockquote><h3>Strengths of General-Purpose LLMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDwJoLoDDLiTYAKH_QrNSw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@solenfeyissa?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Solen Feyissa</a> on <a href="https://unsplash.com/photos/a-person-holding-a-cell-phone-with-icons-on-the-screen-sTq_Iuri06Q?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a> | Added ✅ to original photo</figcaption></figure><p>From a broader perspective, the strengths of LLMs go beyond what some customers initially perceive:</p><ul><li><strong>Personalization: </strong>LLMs can interpret context to deliver granular and tailored responses, aligning closely with individual user needs.</li><li><strong>Cross-Domain Learning</strong>: Exposure to varied datasets enables LLMs to transfer knowledge across domains, potentially reducing the need for multiple specialized systems.</li><li><strong>Multilingual Capabilities and Scalability:</strong> Many LLMs can understand and generate text in multiple languages, helping organizations expand globally without needing separate models per region.</li><li><strong>Ideation:</strong> By brainstorming ideas and creating plans, LLMs solve the initial <strong>cold start</strong> problem in product design or content creation.</li><li><strong>Low-Code and No-Code Solutions</strong>: Many LLM-powered platforms provide interfaces that allow non-technical users to build AI-driven applications, enabling broader organizational participation.</li><li><strong>Continuous Innovation:</strong> Backed by tech giants and startups alike, LLMs benefit from frequent updates, ensuring they stay relevant for emerging challenges.</li></ul><h3>The Challenges of Relying on LLMs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjjA2wp274YXoj8r-tTZeQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@solenfeyissa?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Solen Feyissa</a> on <a href="https://unsplash.com/photos/a-person-holding-a-cell-phone-with-icons-on-the-screen-sTq_Iuri06Q?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a> | Added ❌ to original photo</figcaption></figure><p>Despite their advantages, LLMs come with challenges that businesses must carefully navigate:</p><ol><li><strong>Opaque Decision-Making and Explainability:</strong> LLMs operate as <strong>black boxes</strong>, making it hard to understand how they arrive at specific outputs — an issue in sensitive or high-stakes domains.</li><li><strong>Regulatory and Ethical Concerns:</strong> In regulated industries, the lack of transparency and accountability can hinder deployment without proper customization or compliance measures.</li><li><strong>Trust in Specialized Domain:</strong> While LLMs are versatile, they may lack the precision of domain-specific models (e.g., in medical diagnostics, legal reccomendations or defense analysis).</li><li><strong>High Computation Requirements:</strong> Running LLMs for every use case can become prohibitively expensive, especially if fine-tuning is required.</li><li><strong>Data Privacy Concerns</strong>: Fine-tuning LLMs often involves sensitive company data. Without robust safeguards, data breaches or misuse can become a serious risk.</li><li><strong>Operational Cost &amp; Overkill for Simple Tasks:</strong> For straightforward tasks such as basic classification or tagging, LLMs may be overpowered and more expensive than specialized lightweight solutions.</li><li><strong>Environmental Impact</strong>: Training and deploying large models require enormous computational resources, raising concerns about energy consumption and carbon footprint.</li></ol><blockquote>Organizations should weigh these limitations carefully and adopt LLMs where their strengths truly outweigh their potential downsides.</blockquote><h3>Emerging Trends in LLM Development</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IFUaXO8Et85gRouCvp3gkg.png" /><figcaption>Source: <a href="https://trends.google.com/trends/explore?date=all&amp;q=GenAI,%2Fg%2F11kc9956b3&amp;hl=en">Google Trends</a></figcaption></figure><p>Several key trends are shaping the future of LLMs, aimed at addressing their current limitations:</p><ol><li><strong>Smaller and Specialized Models (SLMs)</strong>: There is a growing interest in developing <a href="https://www.salesforce.com/blog/small-language-models/?bc=HA"><strong><em>smaller models</em></strong></a> that retain the core strengths of LLMs while being more efficient at cost, control and latency. These models, often trained and focused for specific tasks or industries, offer a balance between performance and resource consumption. They are easier to ensure and manage compliance than LLMs because of size.</li><li><strong>Efforts Toward Explainability</strong>: Researchers are actively working on making <a href="https://arxiv.org/abs/2402.01761"><strong><em>LLMs more interpretable</em></strong></a>. Techniques such as attention visualization and layer-wise relevance propagation aim to provide insights into how models arrive at their outputs, enhancing user trust.</li><li><strong>Federated Learning</strong>: This approach allows models to be trained locally on decentralized data, enhancing privacy by ensuring that sensitive information never leaves the device. <a href="https://www.digica.com/blog/privacy-first-ai-exploring-federated-learning.html"><strong><em>Federated learning</em></strong></a> is especially promising for industries dealing with sensitive data, such as healthcare and finance.</li><li><strong>Edge Deployment</strong>: Advances in hardware and model optimization are making it possible to deploy <a href="https://arxiv.org/html/2408.00462v1"><strong><em>LLMs on edge devices</em></strong></a>, enabling real-time AI solutions in remote or resource-constrained environments. This opens up new possibilities for applications in areas like IoT and autonomous systems.</li><li><strong>Hybrid Solution</strong>: Combining LLMs with rule-based systems or domain-specific models or SLMs is <a href="https://medium.com/@kaushikshakkari/understanding-document-parsing-part-2-modern-document-parsing-explained-modular-pipelines-bb605c786293"><strong><em>emerging</em></strong></a> as a viable approach to achieve both breadth and precision.</li><li><strong>AI Agents:</strong> <a href="https://www.anthropic.com/research/building-effective-agents"><strong>AI Agents</strong></a> autonomously interact with users or systems, breaking down complex problems into smaller tasks and executing the appropriate models or workflows. They dynamically select optimal approaches, enhancing efficiency and enabling real-time decision-making across domains like customer service, logistics, and healthcare</li><li><strong>Efforts Toward Energy-Efficient Models</strong>: Given the high computational demands of LLMs, <a href="https://arxiv.org/html/2410.09241v1"><strong><em>significant research</em></strong></a> is focused on reducing their energy consumption. Techniques such as model distillation, pruning, and quantization aim to create more efficient models without sacrificing performance.</li></ol><h3>Hope or Hype? The Verdict</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m_cjcJX0cW3Ow1X24I7l2Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Markus Spiske</a> on <a href="https://unsplash.com/photos/a-close-up-of-a-sticker-on-a-pole-jZQmWPV5THw?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>The rapid evolution of large language models (LLMs) is reshaping how businesses approach AI — from streamlined workflows and faster time-to-value to advanced multilingual capabilities. However, the ongoing dialogue among industry leaders and practitioners underlines a critical point: <strong>Generic Purpose LLMs are not a one-size-fits-all solution. </strong>Challenges such as high costs, operational complexity, managing compliance, and limited transparency have fueled growing interest in smaller, specialized models (SLMs), which are easier to manage, more cost-effective, and better suited for specific tasks.</p><p>Also, as technology continues to advance — with innovations in explainable AI, energy-efficient models, and privacy-centric solutions — Language Models will become increasingly specialized and practical. By adopting a thoughtful and goal-oriented approach, companies can unlock real value from these Language Models and their future advancements.</p><p><strong>Add me on </strong><a href="https://www.linkedin.com/in/kaushik-shakkari/"><strong>LinkedIn</strong></a><strong> for any questions and discussions. Thank you!</strong></p><p>If you are interested in document parsing topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/2030d58d5fe9">Understanding Document Parsing</a></p><p>If you are interested in semantic search topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/9df1566193b0">Understanding Semantic Search</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mD_Bi840gE5rxqKZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BLBgQeqnOWm2v3TU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f14e06770fa9" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/hope-or-hype-f14e06770fa9">Hope or Hype?</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Document Parsing — (Part 2: Modern Document Parsing Explained— Modular Pipelines &…]]></title>
            <link>https://generativeai.pub/understanding-document-parsing-part-2-modern-document-parsing-explained-modular-pipelines-bb605c786293?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb605c786293</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[document-parsing]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Tue, 07 Jan 2025 19:01:06 GMT</pubDate>
            <atom:updated>2025-01-16T06:30:18.989Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Understanding Document Parsing — (Part 2: </strong>Modern Document Parsing Explained— Modular Pipelines &amp; Vision-Language Models (Multimodal AI))</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YdUmuovlbMzSCP_yNUacUA.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2410.21169">arXiv:2410.21169</a></figcaption></figure><h3>Introduction:</h3><p>If you’ve ever had to deal with stacks of documents — whether it’s invoices, contracts, or research papers — you know how overwhelming it can be to pull out and organize the information you actually need. That’s where document parsing comes in. In <a href="https://medium.com/@kaushikshakkari/understanding-document-parsing-part-1-the-evolution-of-document-parsing-from-pre-ocr-to-llm-51f216f14045">the previous article of the series</a>, we explored the evolution of document parsing technologies — from manual techniques to advanced AI-driven systems. In this article, we’re diving into two of the most exciting modern approaches to document parsing: <strong>Modular Pipeline Systems</strong> and <strong>End-to-End Vision-Language Models (VLMs)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RrEKVG1oh62G6dCLtQDuCg.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2410.21169v1">https://arxiv.org/pdf/2410.21169v1</a></figcaption></figure><p>Modular pipelines divide the process of information extraction into clear, manageable steps, giving you “<strong><em>more control and flexibility”</em></strong>. On the other hand, VLMs take a <strong>“<em>do it all at once</em>”</strong> approach, combining text and layout understanding into a single powerful system.</p><p>We’ll explore how these methods work, where they shine, and the challenges they still face. Whether you’re a data scientist, software developer, sales executive, product manager or business leader, this deep dive into modern document parsing will help you navigate the ever-evolving landscape of information extraction.</p><h3>Modular Pipeline Systems: Breaking Down the Process</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qA5ar7x90O4h5o8MlGSqOA.png" /><figcaption>Picture created by Author | Slides Design Credits: <a href="https://slidesgo.com/theme/timeline-infographics#search-Infographics&amp;position-8&amp;results-2856">slidesgo.com/theme/timeline-infographics</a></figcaption></figure><p>Modular pipeline systems follow a step-by-step approach, where each module performs a distinct task. The output from one module becomes the input for the next, creating a streamlined workflow from raw documents to structured data. This modularity allows individual components to be analyzed, tested, fine-tuned, or replaced, enabling tailored solutions for specific document types, use-cases or even industries.</p><blockquote><strong>Below are some key modules for building modular pipeline systems.</strong></blockquote><h4><strong>1. Input Preprocessing &amp; Conversions:</strong></h4><p>Raw documents, including scanned PDFs and Word documents are transformed into standardized formats such as images for downstream processing. Some techniques in this module include:</p><p><strong>1.a PDF-to-Image Conversion:</strong></p><p>Libraries like <a href="https://pypi.org/project/pdf2image/"><strong><em>PDF2Image</em></strong></a> and <a href="https://pymupdf.readthedocs.io/en/latest/"><strong><em>PyMuPDF</em></strong></a> convert PDF pages into high-resolution images.</p><p><strong>1.b Metadata Extraction:</strong></p><p>Libraries like <a href="https://pypi.org/project/PyPDF2/"><strong><em>PyPDF2</em></strong></a> or <a href="https://github.com/jsvine/pdfplumber"><strong><em>PDFPlumber</em></strong></a> can extract document metadata (like language, authors, or creation dates), which might be useful for downstream modules when extracting key-value pairs.</p><h4><strong>2. Optical Character Recognition (OCR):</strong></h4><p>OCR module converts textual content in images into machine-readable text with bounding box information. Popular OCR engines include <a href="https://github.com/tesseract-ocr/tesseract"><strong><em>Tesseract</em></strong></a>, <a href="https://cloud.google.com/vision"><strong><em>Google Vision</em></strong></a>, <a href="https://aws.amazon.com/textract/"><strong><em>AWS Textract</em></strong></a> and <a href="https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence"><strong><em>Azure Document Intelligence</em></strong></a>, which can handle diverse fonts, scripts, and even handwriting. Some techniques in OCR include:</p><p><strong>2.a. Image Cleanup:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9thsLunGPrTQ5oO2Tht3oA.png" /><figcaption><strong>Raw Scanned Image on Left and Cleaned Image on Right | </strong>Source: <a href="https://huggingface.co/spaces/to-be/Scanned_document_denoise_reconstruct">https://huggingface.co/spaces/to-be/Scanned_document_denoise_reconstruct</a> (Restormer: Efficient Transformer for High-Resolution Image Restoration<strong>)</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJsSG5EhqQcXlYJRQs86Ow.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2203.02378v3">https://arxiv.org/pdf/2203.02378v3</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTP-g1Vy1ESHWe8AGCIAgw.png" /><figcaption><strong>Raw Scanned Image on Left and Cleaned Image on Right</strong> | Source: <a href="https://huggingface.co/spaces/qubvel-hf/documents-restoration">https://huggingface.co/spaces/qubvel-hf/documents-restoration</a> (DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks<strong>)</strong></figcaption></figure><p>Image cleanup improves OCR accuracy. It involves several steps to enhance the quality of the scanned document. Some steps are following:</p><ul><li><strong>De-skewing: </strong>Corrects any tilt or skew in the image for proper text alignment</li><li><strong>Binarization: </strong>Converts color images into black-and-white images, which is often more suitable for OCR processing</li><li><strong>Despeckling:</strong> Removes small specks or noise from the image, which can interfere with text recognition</li><li><strong>Border removal:</strong> Eliminates border artifacts that may have been introduced during scanning documents.</li><li><strong>Contrast &amp; Brightness Adjustment:</strong> Ensures text stands out against the background.</li><li><strong>Character / Edge Strengthening:</strong> Sharpens faint characters to improve OCR accuracy.</li></ul><p>Cleanup tools like <a href="https://pypi.org/project/opencv-python/"><strong><em>OpenCV-Python</em></strong></a> or <a href="https://pypi.org/project/pillow/"><strong><em>Pillow</em></strong></a> ensures proper text alignment and contrast.</p><p><strong>2.b. Image Segmentation:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vYtna1RpotavTwjMrnwGaw.png" /><figcaption><strong>Image on Left and Line Segments on Right</strong> | Source: <a href="https://aws.amazon.com/textract/">https://aws.amazon.com/textract/</a></figcaption></figure><p>Segmentation divides the input image into meaningful components like line segments, words and characters to facilitate accurate recognition. Multiple python libraries like <a href="https://github.com/arthurflor23/text-segmentation"><strong><em>OpenCV</em></strong></a><strong><em> </em></strong>perform Image Segmentation.</p><p><strong>2.c Text Recognization:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5mYdyZjDfgE8I-wzwb5W7A.jpeg" /><figcaption>Photo by Author | Iphone’s feature to detect text from image</figcaption></figure><p>Once segmentation is complete, the OCR system applies text recognition algorithms to identify and convert each segmented component into machine-readable characters. Modern techniques such as deep learning-based models (e.g., <a href="https://www.kaggle.com/code/ritvik1909/text-recognition-crnn"><strong>CRNN</strong></a><strong> </strong>and<strong> </strong><a href="https://www.sciencedirect.com/science/article/abs/pii/S0957417422007278"><strong><em>Attention-based Sequence Recognition</em></strong></a>) are often employed to improve accuracy. Many libraries including <a href="https://github.com/JaidedAI/EasyOCR"><strong><em>EasyOCR</em></strong></a> perform Text Recognization in OCR.</p><h4><strong>3. Layout Analysis:</strong></h4><p>This module identifies document elements — text blocks, tables, images, headers, footers — and determines their reading order and spatial relationships. Some techniques include:</p><p><strong>3.a. Structural Detection:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h5BwxkFIg7e9KdZc8AkbMg.png" /><figcaption>Source: <a href="https://github.com/microsoft/unilm/tree/master/dit">https://github.com/microsoft/unilm/tree/master/dit</a></figcaption></figure><p>Libraries like <a href="https://layout-parser.github.io/"><strong><em>LayoutParser</em></strong></a>, <a href="https://camelot-py.readthedocs.io/en/master/"><strong><em>Camelot</em></strong></a>, and <a href="https://github.com/tabulapdf/tabula"><strong><em>Tabula</em></strong></a> detect text blocks, tables, and images etc. Commercial OCR engines also provide services to detect layout structure in documents.</p><p><strong>3.b. Reading Order &amp; Spatial Mapping:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5texSSBYdPB13B2cj9c_cg.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2108.11591v2">https://arxiv.org/pdf/2108.11591v2</a></figcaption></figure><p>Logical sequences and content alignment are established for coherent interpretation. <a href="https://github.com/microsoft/unilm/tree/master/layoutreader"><strong><em>LayoutReader</em></strong></a><strong><em> </em></strong>and most popular commercial OCR also do reading order detection.</p><blockquote>💡<strong>Note: </strong>Some modules could be omitted based on document type. For example, for parsing <a href="https://nlsblog.org/2020/06/12/three-types-of-pdfs/"><strong>text-based PDF document</strong></a> , Input Preprocessing, OCR and Layout Analysis modules could often be skipped and replaced with tools like <a href="https://pypi.org/project/pdfminer/"><strong>PDFMiner</strong></a> for direct layout analysis and structural extraction as it’s not an image-based file.</blockquote><h4>4. Content Type Classification:</h4><p>After layout analysis, this module classifies each detected element into a document-specific content type, such as PII (personally identifiable information) in a bank statement, terms &amp; conditions in a contract, a conclusion section in a research paper, itemized list in an invoice, or skills section in a resume. This module is typically specific to the domain and use case. Some techniques include:</p><p><strong>4.a. Domain-Specific Classifiers:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_kGwFghqyYItthbU5RPVtA.png" /><figcaption>Source: Try demo at <a href="https://huggingface.co/tasks/zero-shot-classification">https://huggingface.co/tasks/zero-shot-classification</a></figcaption></figure><p>Pre-trained or fine tuned machine learning models from libraries like <a href="https://huggingface.co/facebook/bart-large-mnli"><strong><em>Hugging Face</em></strong></a><strong>, </strong><a href="https://github.com/centre-for-humanities-computing/stormtrooper"><strong><em>Stromtrooper</em></strong></a> and <a href="https://scikit-learn.org/stable/"><strong><em>Scikit-Learn</em></strong></a><strong><em> </em></strong>etc.<strong><em> </em></strong>can be used to classify content from a layout element.</p><p><strong>4.b. Layout Features:</strong></p><p>One could use layout-based features (like position, element type, font style, and size etc) and combine them with ML model to detect content type. For instance, in a dataset of academic research papers, references are typically found at the end of the document, often formatted as a numbered list or a block of text with italicized or bold titles and smaller font sizes.</p><h4><strong>5. Key-Value Extraction:</strong></h4><p>This module extracts structured key-value pairs from the classified content. The previous module simplifies this step. For example, only specific keys (like email, name, and phone-number etc) need to be extracted from document element with PII content type, instead of<strong> </strong>extracting all keys in the document. Some techniques include:</p><p><strong>5.a. Pattern Matching:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xDaOjoTgVr8KUwTrPfVjg.png" /><figcaption>Picture by Author</figcaption></figure><p>Regular expressions libraries like <a href="https://pypi.org/project/regex/"><strong><em>regex</em></strong></a>, <a href="https://github.com/google/re2/wiki/Syntax"><strong><em>re2</em></strong></a> and <a href="https://github.com/hanickadot/compile-time-regular-expressions"><strong><em>CTRE</em></strong></a><strong><em> </em></strong>detect predefined formats like email addresses, age, phone numbers, and dates etc.</p><p><strong>5.b. Named Entity Recognition (NER):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KlnAzX3bpqE5ZJHwKmSkew.png" /><figcaption>Source: <a href="https://demos.explosion.ai/displacy-ent">https://demos.explosion.ai/displacy-ent</a></figcaption></figure><p>Use tools like <a href="https://towardsdatascience.com/named-entity-recognition-with-nltk-and-spacy-8c4a7d88e7da"><strong><em>NLTK</em></strong></a><strong>, </strong><a href="https://demos.explosion.ai/displacy-ent"><strong><em>Spacy/Explosion</em></strong></a>, <a href="https://github.com/flairNLP/flair"><strong><em>Flair</em></strong></a><strong> and </strong><a href="https://help.alteryx.com/current/en/designer/tools/alteryx-intelligence-suite/text-mining/named-entity-recognition.html"><strong><em>Alteryx</em></strong></a><strong> </strong>to detect entities like names, dates, and monetary values. You can also train a model to detect custom entities like email which is missing in the pre-trained AI/ML model shown in the above picture.</p><p><strong>5.c. Rule-Based Mapping in Tables:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4NnIoq8KRi7345nDtRGBEg.png" /><figcaption>Source: <a href="https://aws.amazon.com/textract/">https://aws.amazon.com/textract/</a></figcaption></figure><p>Libraries like <a href="https://camelot-py.readthedocs.io/en/master/"><strong><em>Camelot</em></strong></a>, and <a href="https://github.com/tabulapdf/tabula"><strong><em>Tabula</em></strong></a><strong><em> </em></strong>extract tables, and you can use specific rules to get key value pairs. For example, in the above picture, the second column of above detected table gives all product name values.</p><blockquote>💡<strong>Note: </strong>After<strong> Content Type Classification</strong>, elements with same content types can be merged before <strong>Key-Value Extraction. </strong>For instance, PII in an academic transcript could be present across header and footer section, so merging them prior to extracting PII-related key-values is ideal.</blockquote><h4><strong>6. Post-Processing &amp; Validation:</strong></h4><p>This final stage ensures extracted data meets business rules, maintains accuracy, and adheres to standards. Techniques include:</p><p><strong>6.a. Error Correction &amp; Quality Checks:</strong></p><p>Utilize rule-based validation or QA models to ensure data accuracy and integrity. Some examples include,</p><ul><li><strong>Range Validation:</strong> An <strong>age ≥ 130</strong> is unlikely.</li><li><strong>Null/Missing Value Detection:</strong> Flag empty entries for the <strong>Name</strong> key.</li><li><strong>Data Type Verification</strong>: A date value of <strong>30/02/2025 is invalid</strong> because February only has 28 or 29 days.</li><li><strong>Cross-Field Consistency:</strong> Check if the order shipment date is after the order placement date.</li><li><strong>Outlier Identification: </strong>Highlight unusually large bank transactions, which might indicate fraudulent activity.</li></ul><p><strong>6.b. Integration &amp; Formatting:</strong></p><p>This ensures compatibility by formatting refined outputs for ingestion into databases, ERP systems, or analytics platforms. For example, the key value pairs may be filtered, nested, or flattened based on system requirements.</p><h4>Advantages of Modular Pipeline Systems</h4><ul><li><strong>Modularity:</strong> Each stage operates as an independent module, making it easy to upgrade or replace specific components without affecting the entire pipeline.</li><li><strong>Reusability:</strong> Individual modules can be reused across different workflows or projects. For example, OCR could be used across invoice parsing and resume parsing.</li><li><strong>Troubleshooting:</strong> Errors are easier to locate and fix since each module performs a well-defined task.</li><li><strong>Transparency: </strong>The modular design provides clear visibility into each stage of the process, making it easier to understand how data flows and transforms.</li><li><strong>Flexibility:</strong> Quickly adapt to new document types or languages by introducing or adjusting modules without overhauling the entire system.</li></ul><h4>Limitations of Modular Pipeline Systems</h4><ul><li><strong>Error Propagation: </strong>Errors in one module (e.g., OCR) can propagate, affecting the accuracy of downstream modules (e.g., Key-Value Extractions) and final result.</li><li><strong>Complexity for Irregular Layouts:</strong> Highly variable or unstructured documents demand frequent changes including configuration of modules.</li><li><strong>Maintenance Overhead:</strong> Continuously evaluating and tuning multiple modules can be challenging to manage.</li><li><strong>Latency Issues:</strong> Sequential processing across modules can lead to slower performance for high-volume tasks.</li><li><strong>Scalability Challenges:</strong> Adding modules for new document formats or languages can increase system complexity to scale.</li><li><strong>Limited Contextual Integration:</strong> Modules work in isolation, potentially missing cross-step cues.</li></ul><h3><strong>Vision-Language Models (VLMs): An End-to-End Multimodal Approach</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5nbbF49X28ovDEc18E1bnw.png" /><figcaption>Picture created by Author | Code for above process: <a href="https://medium.com/gopenai/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa">https://medium.com/gopenai/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa</a></figcaption></figure><p>Unlike modular pipelines, which divide the document parsing process into discrete steps, Vision-Language Models (VLMs) offer a holistic, unified, end-to-end solution. They internally integrate multiple document processing tasks (like text extraction, layout understanding, and semantic interpretation etc) into a single model. This approach is particularly advantageous for handling complex, multi-modal documents, as it captures textual, structural, and visual relationships in a single pass.</p><h4>Prompts Role in VLMs:</h4><p>Prompts play a crucial role toward guiding VLMs for specific document parsing tasks. They help the model focus on particular outputs, ensuring task-specific accuracy and relevance. Below are some common prompts used across different parsing scenarios:</p><ul><li><em>Extract the key-value pairs for all personal information fields, including name, address, and phone number.</em></li><li><em>Extract key data fields from the provided invoice and return the output in JSON format. Ensure strict validation rules as followed…</em></li><li><em>Identify and parse the main sections of this legal contract, including clauses and terms.</em></li><li><em>Retrieve all tables and convert them into a structured JSON format.</em></li></ul><blockquote><strong>Below are some key concepts on how VLMs internally</strong> <strong>work in general</strong></blockquote><h4><strong>1. Unified Feature Extraction and Fusion:</strong></h4><p>This concept involves capturing and combining both visual and textual features to create a unified representation.</p><p><strong>1.a. Textual Feature Extraction:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-0arPechG0UD1UzkHuiYA.png" /><figcaption>Source: <a href="https://openai.com/index/introducing-text-and-code-embeddings/">https://openai.com/index/introducing-text-and-code-embeddings/</a></figcaption></figure><p>Language Models like <a href="https://arxiv.org/pdf/1706.03762"><strong><em>Transformers</em></strong></a><strong><em> </em></strong><em>and</em><strong><em> </em></strong><a href="https://arxiv.org/pdf/1810.04805"><strong><em>BERT</em></strong></a><strong><em> </em></strong>extract the underlying semantic meaning of text by generating embeddings that effectively represent its content.</p><p><strong>1.b. Visual Feature Extraction:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZo8w6pTxF55cqRwKTUrDA.png" /><figcaption>In the above picture, we can see the Image is split into embeddings which represent spatial features. These embeddings could be used for multiple tasks including classification, parsing and similarity etc. | Source: <a href="https://arxiv.org/pdf/2010.11929v2">https://arxiv.org/pdf/2010.11929v2</a></figcaption></figure><p>Simultaneously, VLMs leverage pre-trained vision models (e.g., <a href="https://www.ibm.com/think/topics/convolutional-neural-networks"><strong><em>CNNs</em></strong></a>, <a href="https://arxiv.org/pdf/2010.11929v2"><strong><em>Vision Transformers</em></strong></a>, or <a href="https://arxiv.org/pdf/2403.04473v1"><strong><em>TextMonkey</em></strong></a>) to capture visual features including spatial layout and structure of each element.</p><p><strong>1.c. Visual and Textual Features Fusion:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0zoyWVZdZfj_du1SCmzkQ.png" /><figcaption>The orange squares in picture indicate visual features and green squares indicate textual features — both are used for predicting answer (value) for given question (key)| Source: <a href="https://arxiv.org/html/2403.04473v1">https://arxiv.org/html/2403.04473v1</a></figcaption></figure><p>These visual and textual features are combined through advanced fusion techniques like <a href="https://www.frontiersin.org/journals/neurorobotics/articles/10.3389/fnbot.2023.1288463/full"><strong><em>cross-modal attention mechanisms</em></strong></a> enabling the model to understand both the prompt and structure of the document holistically.</p><h4>2. Direct Document Processing:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sv5enKlfkDaQq72GxAGMHA.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2106.13884">https://arxiv.org/pdf/2106.13884</a></figcaption></figure><p>VLMs enhance the end-to-end approach by adopting an OCR-free methodology, bypassing traditional character recognition processes to reduce errors. By directly processing raw documents (e.g., text-based PDFs and scanned images) without explicit preprocessing or segmentation, they can accurately interpret layout and content to extract structured information like key-value pairs (e.g., dates, names, monetary values). The output undergoes implicit validation using user prompt to ensure compliance with business rules, making it ready for integration into downstream systems like databases or ERP platforms.</p><h4>3. Pre-Training and Fine-Tuning:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0qh8gBwCFy0wE0iF7PIrzg.png" /><figcaption>Source: <a href="https://huggingface.co/blog/vlms">https://huggingface.co/blog/vlms</a></figcaption></figure><p>Pre-Training and Fine-Tuning are critical phases in developing and deploying Vision-Language Models. These processes enable VLMs to generalize across various document types and adapt to specific domains effectively.</p><p><strong>3.a Pre-Training:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*MS0hJSO4fqP3gh42_E8unA.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2204.08387">https://arxiv.org/pdf/2204.08387</a></figcaption></figure><p>Pre-training exposes VLMs to large multimodal datasets containing various types of documents including books, contracts, invoices, and academic papers. This phase helps the models learn general representations of document layout and content when trained on multiple pre-training tasks. Models like <a href="https://arxiv.org/pdf/2204.08387"><strong><em>LayoutLMv3</em></strong></a><strong><em> </em></strong>and<strong><em> </em></strong><a href="https://arxiv.org/abs/2106.11539"><strong>DocFormer</strong></a> are widely used for pre-training as they can learn both visual and textual features relevant to document parsing.</p><p><strong>3.b Fine-Tuning:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8zP4QNUFySF9_kpYFEFNg.png" /><figcaption>Source: <a href="https://nanonets.com/blog/fine-tuning-vision-language-models-vlms-for-data-extraction/">https://nanonets.com/blog/fine-tuning-vision-language-models-vlms-for-data-extraction/</a></figcaption></figure><p>Fine-tuning further adapts pre-trained models to specific document parsing tasks by training on domain-specific datasets, such as legal documents or medical records. Techniques like <a href="https://www.databricks.com/blog/efficient-fine-tuning-lora-guide-llms"><strong><em>LoRA (Low-Rank Adaptation), QLoRA</em></strong></a><strong><em>, </em></strong><a href="https://nexla.com/ai-infrastructure/prompt-tuning-vs-fine-tuning/#:~:text=While%20fine%2Dtuning%20offers%20deep,and%20constraints%20of%20the%20task."><strong><em>Full Model Fine-Tuning and Prompt Tuning</em></strong></a> are employed, allowing different fine-tuning techniques in different situations while improving parsing accuracy. For example, <a href="https://github.com/NielsRogge/Transformers-Tutorials/blob/master/PaliGemma/Fine_tune_PaliGemma_for_image_%3EJSON.ipynb"><strong><em>PaliGemma</em></strong></a> model is fine-tuned to accurately extract key-value pairs from invoices and bills.</p><blockquote>💡Hands-on article to parse invoice using VLMs approach: <a href="https://medium.com/gopenai/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa"><strong>https://medium.com/gopenai/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa</strong></a></blockquote><h4>Advantages of End-to-End Models:</h4><ul><li><strong>Holistic Processing: </strong>VLMs eliminate the need for multiple modules by offering an integrated, single-pass approach to document parsing.</li><li><strong>Efficiency and Scalability:</strong> The end-to-end nature typically reduces latency and makes these models scalable for real-time applications.</li><li><strong>Reduced Error Propagation:</strong> By bypassing OCR and other intermediate steps, VLMs could minimize errors that typically accumulate in modular pipelines. (However, this depends on keys to be extracted and input documents)</li><li><strong>Domain Adaptation:</strong> Faster adaptation to new domains through fine-tuning.</li><li><strong>Robustness to Complexity:</strong> A viable option for complex layouts and unstructured formats.</li></ul><h4>Limitations of End-to-End Models:</h4><ul><li><strong>Hallucination: </strong>VLMs can sometimes generate incorrect or irrelevant information. This can be problematic in critical applications where accuracy is critical.</li><li><strong>Traceability Issues: </strong>Since end-to-end models rely on OCR-free methods, tracing the origin of a specific answer within the document is often difficult, complicating error debugging and accountability process.</li><li><strong>Inconsistency:</strong> These models sometimes can give different responses to the same prompt when run multiple times due to the architecture and how they interpret prompt each time.</li><li><strong>Data and Resource Intensity for Fine Tuning:</strong> Large training datasets and extensive computing resources are often needed to fine-tune models for complex documents.</li><li><strong>Reduced Transparency:</strong> End-to-end pipelines can be black boxes making error diagnosis and troubleshooting more difficult.</li><li><strong>Latency Concerns:</strong> While Vision Language Models (VLMs) generally excel in speed and versatility, their complex architecture can cause latency, especially with large inputs. Sometimes, for straightforward parsing task — simple documents and keys, pipeline methods may be faster.</li><li><strong>Complex Layouts:</strong> Sometimes, an OCR in pipeline based method performs better than a VLM in detecting text in extremely irregular structures or handwritten notes.</li></ul><h3>Hybrid Solutions: Combining the Best of Both Worlds</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tv7iVYE38RHysPOGyoWw2Q.jpeg" /><figcaption>Picture generated using <a href="https://openai.com/index/dall-e-3/">Dalle</a></figcaption></figure><p>Hybrid solutions in document parsing blend elements of both modular pipelines and end-to-end VLMs. The idea is to leverage each method’s strengths instead of relying on a single one-size-fits-all approach.</p><blockquote><strong>Below are some key reasons for why you need to consider Hybrid Approaches</strong></blockquote><ol><li><strong>Balancing Control and Complexity</strong>: Certain steps (like OCR or post-processing) might be modular, while a VLM tackles more complex tasks (like advanced key-value extraction).</li><li><strong>Flexibility Across Document Types</strong>: Simple and regular-format documents can be handled by a lightweight modular pipeline, while more complex, unstructured documents are routed to a VLM.</li><li><strong>Incremental Upgrades: </strong>Organizations often have existing modular pipelines for certain documents. Rather than discarding them entirely to adopt a brand-new approach, they can incrementally add a VLM for complex tasks or to handle exceptions.</li><li><strong>Optimized Performance and Scalability: </strong>VLMs can be heavy on compute, especially if they’re running for every single page. A hybrid can extract easily recognized fields with lightweight modules, saving the resource-intensive VLM for trickier tasks — leading to more efficient overall system performance.</li></ol><h3>Conclusion</h3><p>Document parsing has come a long way — from manually intensive workflows to sophisticated, AI-driven solutions. Modular pipelines excel with high transparency, flexible module swapping, and easier troubleshooting. End-to-end VLMs offer integrated understanding of text, layout, and visuals, making them especially powerful for complex or variable document environments.</p><p>However, these approaches share a common goal of extracting keys accurately from an ever-growing volume of documents. As the field progresses, hybrid solutions will likely become more prominent, leveraging the best of both worlds. Ongoing research into improved datasets, domain-specific adaptation, and efficient fine-tuning continues to enhance these systems. By staying informed and embracing these innovations, organizations can significantly streamline their operations and maintain a competitive edge in a data-driven world.</p><p><strong>Add me on </strong><a href="https://www.linkedin.com/in/kaushik-shakkari/"><strong>LinkedIn</strong></a><strong>. Thank you!</strong></p><p>If you are interested in document parsing topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/2030d58d5fe9">Understanding Document Parsing</a></p><p>If you are interested in semantic search topic, you are welcome to explore my other articles.</p><p><a href="https://kaushikshakkari.medium.com/list/9df1566193b0">Understanding Semantic Search</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mD_Bi840gE5rxqKZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BLBgQeqnOWm2v3TU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb605c786293" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/understanding-document-parsing-part-2-modern-document-parsing-explained-modular-pipelines-bb605c786293">Understanding Document Parsing — (Part 2: Modern Document Parsing Explained— Modular Pipelines &amp;…</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Document Parsing — (Part 1: The Evolution of Document Parsing from Pre-OCR to LLM)]]></title>
            <link>https://generativeai.pub/understanding-document-parsing-part-1-the-evolution-of-document-parsing-from-pre-ocr-to-llm-51f216f14045?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/51f216f14045</guid>
            <category><![CDATA[document-parser]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Mon, 09 Dec 2024 17:46:33 GMT</pubDate>
            <atom:updated>2025-01-16T06:29:10.941Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Tracing the Evolution of Document Parsing Technologies</em></h4><h3>Understanding Document Parsing — (Part 1: The Evolution of Document Parsing from Pre-OCR to LLMs)</h3><h4><em>How document parsing evolved from manual processes to cutting-edge AI systems transforming industries today.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBTN7-VnLLOcHcQMVXh35A.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>In our <a href="https://medium.com/@kaushikshakkari/understanding-document-parsing-part-0-what-is-it-and-why-it-matters-4939abbbfac3">previous article</a>, we explored document parsing and its applications across industries. Document parsing has undergone a remarkable transformation over the time, evolving from labor-intensive manual processes to sophisticated AI-driven systems that are reshaping industries today.</p><p>In this article, we will explore the fascinating history of document parsing technologies, tracing their evolution from early mechanical tools to modern advancements in machine learning and natural language processing. By looking into some key milestones, we will uncover how each phase has contributed to improving speed, accuracy, and efficiency in handling unstructured data.</p><blockquote>The Evolution of Document Parsing: A Historical Timeline</blockquote><h3>1. The Early Days: Manual Efforts and Mechanical Tools (Late 19th Century–1950s)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F7xJI-O6121o4Fx1rrZJ4Q.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>In the late 19th and early 20th century, document processing relied on manual documenting and organizing information by clerks and librarians. This process required significant effort. As the volume of documents grew, this approach became impractical, causing inefficiencies in data management and retrieval.</p><h4>Key Milestones<strong>:</strong></h4><ul><li><a href="https://www.ibm.com/history/punched-card-tabulator?utm_source=chatgpt.com"><strong>Punch Card Machines (1889)</strong></a><strong>:</strong> Invented by Herman Hollerith in 1889 and first used in the 1890 U.S. Census, punch card machines dominated data processing until the mid-20th century. They became essential in industries for tasks like payroll management and census data handling.</li><li><a href="https://americanprecision.org/learning-resources/machine-of-the-month-photostat-machine/?utm_source=chatgpt.com"><strong>Photostat Machines (1907)</strong></a><strong>:</strong> Introduced in 1907, photostat machines allowed rapid duplication of documents, reducing transcription errors and facilitating data preservation.</li><li><strong>Pattern Recognition in Cryptography (1939):</strong> During World War II (1939–1945), devices like the <a href="https://www.britannica.com/topic/Bombe?utm_source=chatgpt.com">British Bombe</a> and the <a href="https://en.wikipedia.org/wiki/Enigma_machine">German Enigma machine</a> employed early forms of pattern recognition for cryptographic analysis, influencing future text recognition technologies.</li></ul><h4>Advantages</h4><ul><li>Introduced foundational practices for future automation.</li><li>Improved data organization for libraries and businesses.</li></ul><h4>Limitations</h4><ul><li>Labor-intensive and time-consuming.</li><li>Prone to human errors, reducing data accuracy.</li><li>Lacked scalability for large volumes of information.</li></ul><h3>2. The Birth of OCR: Automating Text Recognition (1950–1970)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qW7ytKyCDscScS6aFD_tPQ.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p><a href="https://aws.amazon.com/what-is/ocr/">Optical Character Recognition (OCR</a>) marked a pivotal leap in document processing by automating text recognition. OCR systems scanned documents, identified characters, and translated them into machine-readable text, transforming data entry processes.</p><h4>Key Milestones</h4><ul><li><strong>Development of Early OCR Systems (1950s):</strong> Pioneers like <a href="https://www.historyofinformation.com/detail.php?entryid=885">David H. Shepard developed systems capable of recognizing machine-printed text</a>, laying the groundwork for automated document processing. His machine could recognize numbers and certain characters, used primarily by banks and government agencies for tasks like check sorting and processing.</li><li><strong>Commercialization (1970s):</strong> Companies such as K<a href="https://www.kurzweiltech.com/kcp.html">urzweil Computer Products introduced commercial OCR products</a> and expanded the technology’s accessibility. Their machine was capable of recognizing printed text across many fonts which was a significant leap forward for OCR technology.</li></ul><h4>Advantages</h4><ul><li>Automated conversion of printed text into digital format, reducing manual labor.</li><li>Significantly reduced previous manual data entry requirements.</li></ul><h4>Limitations</h4><ul><li>Struggled with varied fonts and handwriting.</li><li>Required high-quality input for accurate recognition.</li><li>Unable to handle complex layouts, restricting its capabilities compared to human processing.</li></ul><h3>3. Rule-Based Systems: Precision with Rigidity (1970s–1990s)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxTD6FRXwwqmqr-ieLAs3w.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>As OCR matured, developers created rule-based systems designed to extract specific information from specific standardized and layout documents like invoices. These systems relied on predefined rules and templates to identify key fields within documents that had consistent layouts.</p><h4>Key Milestones</h4><ul><li><strong>Template-Based Extraction (1980s)</strong>: <a href="https://parashift.io/template-based-ocr-versus-machine-learning-based-ocr/?utm_source=chatgpt.com">Template-based systems</a> emerged to extract key information from structured documents like invoices and forms using predefined layouts. These systems streamlined data entry for consistent formats but required frequent updates when layouts changed, limiting their flexibility.</li><li><strong>Postal Address Recognition (1980s)</strong>: <a href="https://www.sri.com/press/story/75-years-of-innovation-advanced-postal-address-recognition/?utm_source=chatgpt.com">Postal systems adopted OCR</a> technology to automate mail sorting by recognizing patterns in addresses and ZIP codes. This innovation significantly accelerated sorting processes and improved accuracy for postal services worldwide.</li><li><strong>Enterprise Document Parsing (1990s)</strong>: Businesses used <a href="https://parashift.io/template-based-ocr-versus-machine-learning-based-ocr/?utm_source=chatgpt.com">template-based document parsing systems</a> to validate invoices and purchase orders, reducing manual effort and errors. These systems improved efficiency but struggled with unstructured or inconsistent formats.</li></ul><h4>Advantages</h4><ul><li>Effective for documents with consistent layouts, building on OCR capabilities.</li><li>Provided granular and reliable results for improving accuracy.</li><li>Reduced processing time for standardized documents.</li></ul><h4>Limitations</h4><ul><li>Still unable to handle complex layouts, limiting its application compared to earlier manual human based methods.</li><li>Frequent manual updates were needed for new document types or layouts, increasing maintenance</li></ul><h3>4. Machine Learning: A Paradigm Shift (1990s–2000s)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BSLIBFYDCftL6H-fCFYTgg.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>The rise of machine learning marked a transformative shift. Unlike rigid rules, machine learning models learned patterns from examples, offering adaptability to varied layouts and formats. This shift allowed for greater adaptability to different layouts and formats.</p><h4>Key Milestones</h4><ul><li><strong>Natural Language Processing Tools (1995)</strong>: The release of <a href="https://www.nltk.org/index?utm_source=chatgpt.com">tools like the NLTK</a> marked a turning point for text analysis, introducing accessible machine learning methods for language understanding and processing tasks.</li><li><strong>Adaptive Models (2000s)</strong>: The 2000s saw the rise of adaptive models like <a href="https://link.springer.com/article/10.1023/A:1009715923555?utm_source=chatgpt.com">Support Vector Machines (SVMs)</a> and <a href="https://www.geeksforgeeks.org/decision-tree/">Decision Trees</a>, enabling systems to recognize patterns in documents without relying on predefined rules, enhancing flexibility and accuracy.</li><li><strong>Statistical Learning (2001)</strong>: Statistical methods such as <a href="https://michael.hahsler.net/SMU/CS8331/S2015/tutorials/LDA_Tutorial.pdf?utm_source=chatgpt.com">Latent Dirichlet Allocation (LDA)</a> and <a href="https://web.stanford.edu/~jurafsky/slp3/A.pdf">Hidden Markov Models (HMMs)</a> revolutionized text parsing, allowing automated keyword extraction and topic modeling for large-scale text datasets.</li></ul><h4>Advantages</h4><ul><li>Better at diverse layouts compared to rule-based systems.</li><li>Reduced manual rule creation, lowering maintenance efforts.</li></ul><h4>Limitations</h4><ul><li>Required large datasets for training.</li><li>Struggled with more complex layouts like tables and images in documents.</li><li>These models which are trained on specific datasets often struggled to generalize to unseen or highly variable layouts. This led to low accuracy in real-world scenarios.</li></ul><h3>5. The Deep Learning Revolution (2010s)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0Sgkhsspmi-IMTIxWgb6g.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>Deep learning brought profound advancements in document parsing capabilities through technologies like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). These models enabled machines to process complex layouts while understanding context within documents, significantly improving accuracy in parsing tasks.</p><h4>Key Milestones</h4><ul><li><strong>Convolutional Neural Networks (2012)</strong>: The introduction of CNNs like <a href="https://en.wikipedia.org/wiki/AlexNet?utm_source=chatgpt.com">AlexNet in 2012 revolutionized image recognition</a>, enabling document parsing systems to handle complex layouts, including images, tables, and mixed-content formats.</li><li><strong>Word Embeddings (2013): </strong>Word embeddings like <a href="https://code.google.com/archive/p/word2vec/">Word2Vec</a> revolutionized document parsing by representing text as dense vectors that capture semantic relationships. This advancement enhanced models’ ability to interpret textual elements better in unstructured documents and extract content with higher accuracy.</li><li><strong>Recurrent Neural Networks (2014)</strong>: The development of <a href="https://www.youtube.com/watch?v=AsNTP8Kwu80">RNNs</a>, particularly <a href="https://www.analyticsvidhya.com/blog/2021/03/introduction-to-long-short-term-memory-lstm/">Long Short-Term Memory (LSTM) networks</a>, in 2014 enhanced sequential data parsing, improving systems’ ability to understand contextual relationships in unstructured data.</li><li><strong>Open-Source Frameworks (2015)</strong>: The release of open-source frameworks like <a href="https://www.tensorflow.org/">TensorFlow</a> and <a href="https://pytorch.org/">PyTorch</a> in 2015 democratized AI development, providing tools that made building and deploying advanced document parsing systems more accessible to developers.</li></ul><h4>Advantages</h4><ul><li>Improved ability to process complex layouts including tables and images, surpassing earlier machine learning approaches.</li><li>Improved contextual understanding of document content, enhancing parsing performance.</li><li>Increased accuracy in processing complex documents.</li></ul><h4>Limitations</h4><ul><li>Required significant computational resources for training, deploying and utilizing models.</li><li>Needed large, diverse datasets for optimal performance, increasing data collection efforts.</li><li>Reduced transparency and interpretability due to model complexity, compared to simpler rule-based systems.</li></ul><h3>6. Large Language Models and Multimodal Processing (2018–Present)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5P34HSGZqsARnpLQkJqktQ.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>The emergence of Large Language Models (LLMs) like BERT and GPT-3 has transformed document parsing by adding sophisticated capabilities for understanding meaning and context at a deeper level than previous technologies allowed. These models can process both visual data and textual information simultaneously, facilitating more comprehensive analysis of documents.</p><h4>Key Milestones</h4><ul><li><strong>LayoutLM (2019)</strong>: The introduction of <a href="https://arxiv.org/abs/1912.13318?utm_source=chatgpt.com">LayoutLM</a> in 2019, a transformer-based model combining text and layout information, significantly advanced parsing for complex formats like receipts, forms, and invoices.</li><li><strong>Fine-Tuned BERT Models (2020s)</strong>: <a href="https://link.springer.com/article/10.1007/s40745-024-00524-5?utm_source=chatgpt.com">Fine-tuned BERT models</a> became widely adopted in the 2020s for specialized tasks such as contract analysis and resume parsing, showcasing the power of pre-trained language models for domain-specific applications.</li><li><strong>Multimodal Integration (2021)</strong>: The release of tools like <a href="https://openai.com/index/clip/?utm_source=chatgpt.com">OpenAI’s CLIP</a> in 2021 pioneered multimodal models that combine image and text understanding, enabling holistic analysis of documents with both visual and textual elements.</li><li><strong>GPT-4 (2023):</strong> The release of <a href="https://openai.com/index/gpt-4/?utm_source=chatgpt.com">GPT-4 introduced multimodal capabilities</a>, allowing seamless integration of text, images, and structured data for tasks such as document understanding, complex table extraction, and cross-modal reasoning, setting new benchmarks for document parsing systems.</li></ul><h4>Advantages</h4><ul><li>Handles complex documents with both text and visuals, combining earlier techniques.</li><li>Offers better accuracy across various document types compared to specialized models.</li><li>Provides advanced understanding of document content beyond previous deep learning models.</li></ul><h4>Limitations</h4><ul><li>May produce incorrect or nonsensical outputs (“hallucination”), raising reliability concerns.</li><li>Computationally intensive. Required significant resources for training and deployment which also increases cost.</li><li>Potential biases present in training data can affect output quality, introducing new challenges not present in rule-based systems.</li><li>Complexity reduced transparency and interpretability.</li></ul><h3>Bridging the Old and New: The Rise of Hybrid Solutions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6YCETOupsZ-ti_-4Eqf4Uw.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><p>While advanced AI models like Large Language Models dominate today’s landscape, they do not entirely replace traditional methods. Instead, hybrid solutions have emerged, combining the precision of OCR with the adaptability of LLMs. These systems leverage the strengths of both approaches, ensuring flexibility in handling unstructured documents. <strong>This collaborative evolution of document parsing demonstrates that no single approach solves every challenge.</strong></p><h3>Conclusion</h3><p>The evolution of document parsing has been a dynamic and non-linear journey, transitioning from manual processes and mechanical tools to rule-based systems, machine learning, and advanced AI-driven models. Each phase has not only introduced new capabilities but also incorporated elements of earlier technologies. This led to hybrid solutions that combine the precision of traditional methods with the adaptability of AI. <strong>These hybrid systems are particularly important today, offering the flexibility and robustness needed to manage complex unstructured documents effectively.</strong></p><blockquote>The coexistence of legacy systems with modern innovations shows the versatility of document parsing, where no single approach can address all challenges.</blockquote><p>Kudos on completing and understanding the history of document parsing! The next article in this series will focus on <strong>Document Parsing Methodologies</strong> <strong>. </strong>We will be also delving deeper on how modern approaches like foundational models and modular parser pipeline systems (hybrid solution) shaping the future of document parsing.</p><p>Stay tuned!</p><h4>Add me on <a href="https://www.linkedin.com/in/kaushik-shakkari/">LinkedIn</a>. Thank you!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mD_Bi840gE5rxqKZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BLBgQeqnOWm2v3TU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51f216f14045" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/understanding-document-parsing-part-1-the-evolution-of-document-parsing-from-pre-ocr-to-llm-51f216f14045">Understanding Document Parsing — (Part 1: The Evolution of Document Parsing from Pre-OCR to LLM)</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Document Parsing — (Part 0: What Is It and Why It Matters?)]]></title>
            <link>https://generativeai.pub/understanding-document-parsing-part-0-what-is-it-and-why-it-matters-4939abbbfac3?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4939abbbfac3</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[document-parser]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 18:18:05 GMT</pubDate>
            <atom:updated>2025-01-16T06:29:04.242Z</atom:updated>
            <content:encoded><![CDATA[<h4>Bridging Technology and Industry</h4><h3>Understanding Document Parsing — (Part 0: What Is It and Why It Matters?)</h3><h4>Discover how document parsing is transforming industries with 10 powerful applications and why it’s a cornerstone of modern data processing.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fgUkDbev9zErBKGVB1g86g.jpeg" /><figcaption>Image generated using <a href="https://openai.com/">OpenAI’s DALL-E</a></figcaption></figure><h3>The Challenge of Unstructured Data:</h3><p>Every day, individuals and organizations deal with vast amounts of unstructured data buried in documents like invoices, reports, contracts, insurance forms, resumes, and more. <a href="https://www.forbes.com/councils/forbestechcouncil/2019/01/29/the-80-blind-spot-are-you-ignoring-unstructured-organizational-data/">Studies</a> show that <strong>80% of a company’s data is unstructured</strong>, making it difficult to analyze and act upon directly.</p><p>The challenge has given rise to <strong>document parsing</strong>, a transformative technology that bridges the gap between unstructured data and structured information, reshaping how industries operate.</p><h3>What is Document Parsing?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N1T3lzbWpzqb47_p-5KvYA.png" /><figcaption>Picture by the author</figcaption></figure><p>In simple terms, document parsing is the process of converting unstructured data from formats like PDFs, images, or scanned documents into structured, machine-readable formats.</p><h3><strong>How It Differs from OCR?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIQVxrNJpSd46LPjbJdkwg.png" /><figcaption>Picture from author: A sample <strong>OCR</strong> code to extract raw text from image</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Optical_character_recognition">Optical Character Recognition (OCR)</a> translates images into raw text (as shown in above picture). However, document parsing goes further. It recognizes layouts, extracts meaningful text, and organizes the information into usable formats, such as JSON.</p><h4><strong>Example:</strong></h4><ul><li>OCR might extract plain text from an invoice.</li><li>Document parsing identifies fields like <strong>invoice number</strong>, <strong>due date</strong>, and <strong>total amount</strong>, arranging this data into structured formats suitable for databases, apis or analytics systems for direct data ingestion.</li></ul><h3>Why Document Parsing Matters?</h3><p>Document parsing is more than a technical innovation — it’s a key driver of business transformation.</p><h4><strong>Here are its key benefits:</strong></h4><ol><li><strong>Time and Cost Savings</strong>: Automates repetitive manual data extraction tasks allowing teams to focus on more important work.</li><li><strong>Improved Accuracy</strong>: Reduces errors associated with manual data entry.</li><li><strong>High Scalability</strong>: Handles high volumes of documents efficiently, making it ideal for scaling businesses.</li></ol><h3>What are some Applications of Document Parser?</h3><p>Document parsing is already transforming industries by automating data extraction and organization. Here are some key applications:</p><h4><strong>1. Identity Verification:</strong></h4><p><a href="https://www.id.me/"><strong>ID.me</strong></a> extracts critical data from driver’s licenses and passports for automated identity verification, improving speed and accuracy while minimizing manual effort.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u9iLcug-UheUddm8Vfw_dg.png" /><figcaption>Source: <a href="https://network.id.me/wp-content/uploads/Document-Verification-Use-Machine-Vision-and-AI-to-Extract-Content-and-Verify-the-Authenticity-1.pdf">ID.me</a></figcaption></figure><h4><strong>2. Education &amp; Recruiting:</strong></h4><p><a href="https://www.astrumu.com/"><strong>AstrumU</strong></a> parses transcripts, syllabi, and resumes to identify skills, enabling smarter job matching and bridging education-to-employment gaps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aa3BJHNzsNwlzXl7j1iIKQ.png" /><figcaption>Source: <a href="https://www.astrumu.com/technology">AstrumU</a></figcaption></figure><h4><strong>3. Contract Management</strong>:</h4><p><a href="https://www.icertis.com/"><strong>Icertis</strong></a> parses and organizes contract data, helping businesses track compliance, manage obligations, and quickly find critical details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzcKLy45vcAh0m8NZjQ_cw.png" /><figcaption>Source: <a href="https://www.icertis.com/products/platform/">Icertis</a></figcaption></figure><h4><strong>4. Insurance</strong>:</h4><p><a href="https://www.lemonade.com/"><strong>Lemonade</strong></a> automates claims processing by extracting key details from policy applications and claims, enabling near-instant responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cB2-XgaLmURZRTKh0MzVZg.png" /><figcaption>Source: <a href="https://www.lemonade.com/claims">Lemonade</a></figcaption></figure><h4><strong>5. Banking</strong>:</h4><p><a href="https://www.miteksystems.com/"><strong>Mitek</strong></a><strong>’s</strong> Mobile Deposit uses document parsing to extract information from checks captured via phone cameras, making deposits seamless and user-friendly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKEpjSYDaU2_ZIxBrWUPxA.png" /><figcaption>Source: <a href="https://www.miteksystems.com/mobile-deposit">Miket</a></figcaption></figure><h4><strong>6. Healthcare</strong>:</h4><p><a href="https://www.cdphp.com/"><strong>CDPHP</strong></a> leverages Amazon Comprehend Medical to extract and structure data from patient records and transcripts, improving analytics and organizational access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rz87EmZZ4X0ZjOUeReQz5Q.png" /><figcaption>Source: <a href="https://aws.amazon.com/comprehend/medical/">Amazon</a></figcaption></figure><h4><strong>7. Retail</strong>:</h4><p><a href="https://www.emaar.com/"><strong>Emaar Properties</strong></a> integrates Microsoft Azure Cognitive Services to scan receipts for loyalty programs, enabling efficient point earning and redemption for mall customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4T4Xe_PMDGO2os7df_Z5cw.png" /><figcaption>Source: <a href="https://cloud.google.com/document-ai#customers">Azure</a></figcaption></figure><h4>8. <strong>Real Estate:</strong></h4><p><a href="https://blooma.ai/"><strong>Blooma</strong></a> parses financial documents such as personal financial statements and schedules of real estate to streamline the underwriting process for commercial real estate loans, reducing manual data entry and increasing efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oPm9hT0oD0MTk4-P_xnMgg.png" /><figcaption>Source: <a href="https://blooma.ai/about-us/">Blooma.ai</a></figcaption></figure><h4><strong>9. Credit Card Processing</strong>:</h4><p><a href="https://www.veryfi.com/"><strong>Veryfi</strong></a> provides real-time credit card data extraction with on-device processing, ensuring security and speeding up everyday transactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WW0EbaDb7dNSQ_NnC6gDQg.png" /><figcaption>Source: <a href="https://www.veryfi.com/lens/credit-card-extraction/">Veryfi</a></figcaption></figure><h4>10. <strong>Logistics:</strong></h4><p><a href="https://super.ai/case-studies/automating-straight-bill-of-lading-document-processing"><strong>Super.AI</strong></a> helped a global medical products company automate the processing of freight Bills of Lading (BoL). They claim, their AI-powered solution improved data accuracy by 95%, processed 26,000+ documents weekly, and saved the company $5 million by reducing errors and streamlining cost management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1g94AadohZUqV2_AjjLOYQ.png" /><figcaption>Source: <a href="https://super.ai/case-studies/automating-straight-bill-of-lading-document-processing">Super.AI</a></figcaption></figure><h3>Conclusion</h3><p>Document parsing is transforming how businesses manage unstructured data, bringing automation, efficiency, and precision to previously manual and error-prone processes. By automating the extraction and organization of information, it enables organizations to streamline workflows and scale operations seamlessly.</p><h4><strong>Key Takeaways</strong>:</h4><ul><li><strong>Unstructured Data Challenge</strong>: With 80% of business data being unstructured, document parsing is crucial for unlocking its potential.</li><li><strong>Beyond OCR</strong>: It doesn’t just extract text — it organizes data into structured formats for streamlined use.</li><li><strong>Versatile Applications</strong>: From healthcare to retail, document parsing automates processes, boosting efficiency and reducing human error across industries.</li></ul><p>This article is just the beginning. In the following parts of this series, we’ll delve deeper into parsing techniques, challenges, cutting-edge applications, and the future of document parsing.</p><p>Stay tuned!</p><h4><strong>Add me on </strong><a href="https://www.linkedin.com/in/kaushik-shakkari/"><strong>LinkedIn</strong></a><strong>. Thank you!</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*mD_Bi840gE5rxqKZ.png" /></figure><p>This story is published on <a href="https://generativeai.pub/">Generative AI</a>. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories.</p><p>Subscribe to our <a href="https://www.generativeaipub.com/">newsletter</a> and <a href="https://www.youtube.com/@generativeaipub">YouTube</a> channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*BLBgQeqnOWm2v3TU.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4939abbbfac3" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/understanding-document-parsing-part-0-what-is-it-and-why-it-matters-4939abbbfac3">Understanding Document Parsing — (Part 0: What Is It and Why It Matters?)</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the Best Structured Output Parser Approach | 3 Ways To Generate Structured Output]]></title>
            <link>https://blog.gopenai.com/choosing-the-best-structured-output-parser-approach-3-ways-to-generate-structured-output-d9686482729c?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9686482729c</guid>
            <category><![CDATA[parsing]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[output-parser]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[structured-data]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Fri, 19 Jul 2024 15:51:27 GMT</pubDate>
            <atom:updated>2024-07-24T12:18:59.816Z</atom:updated>
            <content:encoded><![CDATA[<h4>HARNESSING THE POWER OF LARGE LANGUAGE MODELS FOR STRUCTURED DATA GENERATION</h4><h4><strong>A Detailed Comparison of Structured Output Extraction </strong>Methodologies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m5zxcjkR7UIf8c3whYxonQ.jpeg" /><figcaption>Photo by the author: A serene evening with a stunning sunset on Coronado Island, San Diego (May 2024)</figcaption></figure><h3>Introduction:</h3><p>Large Language Models (LLMs) are designed to generate free form text. However, many practical data extraction applications like <a href="https://medium.com/gopenai/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa">invoice parsing</a> require output in structured format. “Structured” signifies that the output will consistently follow a specified format. To address this need, different approaches and Python libraries have been developed and choosing the best one has been challenging. In the article, I will explain different methods for structured extraction and summarize my findings.</p><blockquote>There are three main approaches to perform structured data extraction:<strong><br></strong>1. Prompt Engineering<br>2. Constrained Token Generation<br>3. Fixing Bad JSON</blockquote><h3>1. Prompt Engineering:</h3><p>Prompt engineering involves designing specific prompts to direct an LLM to generate the desired structured output. This method leverages the LLM’s natural language understanding to produce text that fits a particular format.</p><h4>Sample Code: (Using <a href="https://github.com/openai/openai-python">OpenAI</a> library)</h4><pre>from openai import OpenAI<br>import os<br># Load OpenAI Client (Create and view your API key - https://platform.openai.com/account/api-keys)<br>client = OpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br># Define Prompt<br>prompt = &quot;&quot;&quot;Extract JSON about PII: { &quot;name&quot;: &quot;name of person&quot;, &quot;ssn&quot;: &quot;Social security number of the person&quot;, &quot;email&quot;: &quot;Email address of the person, must have @&quot;, &quot;age&quot;: Age of the person} in &quot;&quot;&quot;<br># Sample Input Text <br>input_text = &quot;&quot;&quot;Tony Stark<br>                Age 32<br>                <br>                Education<br>                - University of Southern California, Los Angeles<br>                  - M.S. in Computer Science (Specialization in Data Science)<br>                  - 2020<br>          <br>                IronMan@Avengers.ai&quot;&quot;&quot;<br># Generate JSON Output<br>json_response = client.chat.completions.create(<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;You are a helpful assistant designed to output JSON.&quot;,<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: prompt + input_text,<br>        }<br>    ],<br>    model=&quot;gpt-3.5-turbo&quot;<br>)<br># Output<br>print(json_response.choices[0].message.content)<br>&quot;&quot;&quot;<br>{<br>  &quot;name&quot;: &quot;Tony Stark&quot;,<br>  &quot;ssn&quot;: &quot;&quot;,<br>  &quot;email&quot;: &quot;IronMan@Avengers.ai&quot;,<br>  &quot;age&quot;: 32<br>}<br>&quot;&quot;&quot;</pre><h4>Advantages:</h4><ol><li><strong>Direct Schema Embedding: </strong>Embedding the schema directly in the prompt can make it easier to define the output format.</li><li><strong>Broad LLM Models Compatibility: </strong>This approach can be used with any LLM that can understand and generate JSON-formatted text.</li><li><strong>Simple</strong>: Requires no deep knowledge or complicated setup.</li></ol><h4>Disadvantages:</h4><ol><li><strong>Trial and Error:</strong> Will need many tries, each with uncertain success to fix JSON, often the issue when expecting <a href="https://blog.gopenai.com/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa#:~:text=Nested%20Objects%20Schema%3A">a complex and nested json structure</a>.</li><li><strong>Malfunctioned JSON: </strong>When prompt engineering, it’s very hard to limit the model to produce correct JSON. LLMs often make mistakes like adding unescaped quotes, single quotes, newlines, comments, or text around the JSON, which can basically break parsing JSON.</li><li><strong>Scalability Issues:</strong> Less reliable for large-scale applications where modularity and consistency of code is crucial.</li><li><strong>Slow and Costly: </strong>Some libraries using this method might need to re-try calling LLM to fix bad JSON making it expensive and slow solution.</li><li><strong>Need Examples:</strong> It works better when few-shot JSON formated examples added in the prompt.</li><li><strong>Performance: </strong>Less effective because the schema constraints are less strictly enforced during generation process.</li></ol><h3><strong>2. Constrained Token Generation:</strong></h3><p>Constrained token generation is a process of controlling the token generation process for LLMs. This is achieved by applying specific rules and restrictions that fit predefined schema while models generate tokens. For example, the first token to be generate is “<strong>{</strong>”, the following token must be key in defined json and next token must be “<strong>:</strong>” etc.</p><p>JSON mode, Function Calling and Finite State Machine Constraint Generation are some specific implementations within this broader technique.</p><h4>2.1. JSON Mode:</h4><p>LLMs can be instructed to explicitly generate valid JSON output. JSON Mode approach involves specifying the desired JSON structure as part of the input given to the model and explicitly enforce to generate JSON. Some libraries like OpenAI enables users to explicitly mention <em>“json_object”</em> as response format type in code.</p><h4>Sample Code: (Using <a href="https://github.com/openai/openai-python">OpenAI</a> library)</h4><blockquote>Note: Below code is almost similar to prompt engineering code, except I added <em>response_format={ “type”: “json_object” } </em>in<em> client.chat.completions.create </em>function.</blockquote><pre>from openai import OpenAI<br>import os<br><br># Load OpenAI Client (Create and view your API key - https://platform.openai.com/account/api-keys)<br>client = OpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br># Define Prompt<br>prompt = &quot;&quot;&quot;Extract JSON about PII: { &quot;name&quot;: &quot;name of person&quot;, &quot;ssn&quot;: &quot;Social security number of the person&quot;, &quot;email&quot;: &quot;Email address of the person&quot;, &quot;age&quot;: Age of the person} in &quot;&quot;&quot;<br># Sample Input Text <br>input_text = &quot;&quot;&quot;Tony Stark<br>                Age 32<br>                <br>                Education<br>                - University of Southern California, Los Angeles<br>                  - M.S. in Computer Science (Specialization in Data Science)<br>                  - 2020<br>          <br>                IronMan@Avengers.ai&quot;&quot;&quot;<br># Generate JSON Output<br>json_response = client.chat.completions.create(<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: &quot;You are a helpful assistant designed to output JSON.&quot;,<br>        },<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: prompt + input_text,<br>        }<br>    ],<br>    model=&quot;gpt-3.5-turbo&quot;,<br>    response_format={ &quot;type&quot;: &quot;json_object&quot; }<br>)<br># Output<br>print(json_response.choices[0].message.content)<br>&quot;&quot;&quot;<br>{<br>  &quot;name&quot;: &quot;Tony Stark&quot;,<br>  &quot;ssn&quot;: &quot;&quot;,<br>  &quot;email&quot;: &quot;IronMan@Avengers.ai&quot;,<br>  &quot;age&quot;: 32<br>}<br>&quot;&quot;&quot;</pre><h4>Advantages:</h4><ol><li><strong>Transparent Prompt: </strong>Embedding the schema directly in the prompt can make it easier to understand and have complete control on what prompt is sent to LLM to define the output format.</li><li><strong>Avoid simple JSON errors: </strong>Typically,<strong> </strong>libraries using JSON constrained generation process avoids generating some non sensical output.</li><li><strong>Simple</strong>: Requires no deep knowledge or complicated setup.</li></ol><h4>Disadvantages:</h4><ol><li><strong>Limited LLM Compatibility: </strong>Not all LLMs support JSON Mode, which limits the flexibility in choosing different models for various tasks.</li><li><strong>Potential for Errors: </strong>The model may still generate outputs that do not fully conform to the schema, especially if the JSON schema is complex or the LLM is not fine-tuned for this task. For example, it might have generated <strong><em>{ “items”: 5}</em></strong> but you might wanted <strong><em>{ “items”: [5] }</em></strong>.</li><li><strong>Performance: </strong>Might be less effective as the schema constraints are less strictly enforced during generation.</li></ol><h4>2.2. Function Calling:</h4><p>In June 2023, OpenAI announced a new feature called “<a href="https://openai.com/index/function-calling-and-other-api-updates/">function calling</a>”. This allows users to give the model a list of functions and the needed details for each. In case of our problem, the function can be defined to get the structured output using JSON schema with necessary information in list of dictionaries form. This method was highly successful and other LLMs like <a href="https://ai.google.dev/gemini-api/tutorials/extract_structured_data#:~:text=While%20the%20API%20can%20handle,potentially%20error-prone%20parsing%20step.">Google’s Gemini followed and introduced this process to their API</a>.</p><h4>Sample Code: (Using <a href="https://github.com/openai/openai-python">OpenAI</a> library)</h4><pre>from openai import OpenAI<br>import os<br><br># Load OpenAI Client (Create and view your API key - https://platform.openai.com/account/api-keys)<br>client = OpenAI(api_key=os.environ[&#39;OPENAI_API_KEY&#39;])<br># Define Custom Function<br>pii_data_parsing_custom_functions = [ { &#39;name&#39;: &#39;extract_pii_data&#39;, <br>    &#39;description&#39;: &#39;Extract personal identifiable information from provided data&#39;, <br>    &#39;parameters&#39;: { &#39;type&#39;: &#39;object&#39;, <br>                    &#39;properties&#39;: <br>                      { &#39;name&#39;: { &#39;type&#39;: &#39;string&#39;, &#39;description&#39;: &#39;Complete name of the person&#39; }, <br>                        &#39;ssn&#39;: { &#39;type&#39;: &#39;string&#39;, &#39;description&#39;: &#39;Social security number of the person&#39; }, <br>                        &#39;email&#39;: { &#39;type&#39;: &#39;string&#39;, &#39;description&#39;: &#39;Email address of the person&#39; }, <br>                        &#39;age&#39;: { &#39;type&#39;: &#39;integer&#39;, &#39;description&#39;: &#39;Age of the person&#39; } <br>                       } } }]<br># Sample Input Text <br>input_text = &quot;&quot;&quot;Tony Stark<br>                Age 32<br>                <br>                Education<br>                - University of Southern California, Los Angeles<br>                  - M.S. in Computer Science (Specialization in Data Science)<br>                  - 2020<br>          <br>                IronMan@Avengers.ai&quot;&quot;&quot;<br># Generate JSON Output<br>json_response = client.chat.completions.create(<br>    model = &#39;gpt-3.5-turbo&#39;,<br>    messages = [{&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: input_text}],<br>    functions = pii_data_parsing_custom_functions,<br>    function_call = &#39;auto&#39;<br>)<br># Output<br>print(json_response.choices[0].message.function_call.arguments)<br>&quot;&quot;&quot;<br>{<br>  &quot;name&quot;: &quot;Tony Stark&quot;,<br>  &quot;ssn&quot;: &quot;&quot;,<br>  &quot;email&quot;: &quot;IronMan@Avengers.ai&quot;,<br>  &quot;age&quot;: 32<br>}<br>&quot;&quot;&quot;</pre><h4>Advantages:</h4><ol><li><strong>Flexibility: </strong>Allows defining complex and nested structures easily with function definition. This is particularly useful for applications requiring detailed and hierarchical data formats​.</li><li><strong>Automatic prompt handling: </strong>LLMs automatically generates the prompt for provided function.</li><li><strong>Avoid simple JSON errors: </strong>Typically,<strong> </strong>libraries using JSON constrained generation process avoids generating some non sensical output.</li><li><strong>Simple Setup</strong>: Requires no deep knowledge or complicated setup.</li></ol><h4>Disadvantages:</h4><blockquote>Note: Function Calling has similar disadvantages as JSON Mode</blockquote><blockquote><a href="https://community.openai.com/t/json-mode-vs-function-calling/476994/4">We enable JSON Mode for Function Calling</a> — Open AI</blockquote><blockquote>From my understanding both JSON Mode and Function Calling are very similar for structured data parsing. Both methods are simple but limited to some LLMs (mostly commercial ones) and still can generate malfunctioning JSON (with extra quotes, free-text, missing keys, wrong data types, wrong nesting, etc. &amp; newlines etc.)</blockquote><h4>2.3. Finite State Machine Constraint Generation:</h4><p>Finite State Machine Constraint Generation is a technique used to strictly regulate the token generation process for LLMs. It involves implementing specific rules and restrictions to ensure that the generated tokens fit a predefined format. Essentially, during each token generation (inference), the model is limited to selecting from a specific set of tokens that align with the given JSON schema and grammar rules, while excluding any tokens that do not conform to these constraints.</p><h4>Sample Code: (using <a href="https://github.com/outlines-dev/outlines">Outlines</a> library)</h4><pre>from enum import Enum<br>from pydantic import BaseModel, constr, EmailStr<br>import outlines<br>import torch<br><br># Define PII model using Pydantic<br>class PII(BaseModel):<br>    name: constr(max_length=100)<br>    ssn: constr(pattern=r&#39;^\d{3}-\d{2}-\d{4}$&#39;)<br>    email: EmailStr<br>    age: int<br><br># Load the model from outlines<br>model = outlines.models.transformers(&quot;microsoft/Phi-3-mini-4k-instruct&quot;)<br># Construct the structured sequence generator<br>generator = outlines.generate.json(model, PII)<br># Sample Input Text <br>input_text = &quot;&quot;&quot;Tony Stark<br>                Age 32<br>                <br>                Education<br>                - University of Southern California, Los Angeles<br>                  - M.S. in Computer Science (Specialization in Data Science)<br>                  - 2020<br>          <br>                IronMan@Avengers.ai&quot;&quot;&quot;<br># Generate JSON Output<br>json_response = generator(input_text)<br># Output <br>print(json_response<br>&quot;&quot;&quot;<br>{<br>  &quot;name&quot;: &quot;Tony Stark&quot;,<br>  &quot;ssn&quot;: &quot;&quot;,<br>  &quot;email&quot;: &quot;IronMan@Avengers.ai&quot;,<br>  &quot;age&quot;: 32<br>}<br>&quot;&quot;&quot;</pre><h4>Advantages:</h4><ol><li><strong>High Efficiency:</strong> This method operates at a deeper level of LLM token generation and doesn’t add overhead to the prompt.</li><li><strong>More control on custom constraints:</strong> Based on use case, one can define and experiment with custom constrains and enforce on LLM for better performance.</li><li><strong>Avoid simple JSON errors: </strong>Typically,<strong> </strong>libraries using Finite State Machine Constraint Generation methods avoids generating some non sensical output.</li></ol><h4>Limitations:</h4><ol><li><strong>Integration Requirements with LLMs:</strong> Often need to have self hosted models as deep integration with specific token generation endpoints is necessary to implement this process.</li><li><strong>Implementation Challenges: </strong>Might requires a deeper understanding of LLMs and topics like CFGs (Context-Free Grammar), FSTs (Finite State Machines) &amp; Regular Expressions etc. However libraries may simplify these details in future.</li><li><strong>Reliability:</strong> May still encounter issues with very complex or deeply nested JSON schemas.</li></ol><blockquote>I believe LLMs with JSON Mode and Functional Calling capabilities might be using basic versions of Finite State Machines similar to constrain the token generation process. However, since the internal workings are not disclosed, it’s challenging to know for sure what’s happening inside!</blockquote><h3>3. Fixing Bad JSON:</h3><p>Often, libraries send malfunctioning JSON back to the LLM to fix it. However, Fixing Bad JSON method takes an algorithmic approach to detects and corrects errors in the malfunctioned JSON generated by the LLM. For example, <a href="https://github.com/BoundaryML/baml">BAML</a> use algorithms like hamming distance and multiple cost functions to check if malfunctioned JSON could be mapped to provided data schema. This approach can be efficient and flexible but has its own set of challenges.</p><h4>Sample Code: (Using <a href="https://github.com/BoundaryML/baml">BAML</a> &amp; <a href="https://github.com/openai/openai-python">OpenAI</a> library)</h4><blockquote>Note: BAML is a new language, you can try code <a href="https://www.promptfiddle.com/New-Project-pdCDV">here</a>.</blockquote><pre>// This is a BAML config file, which extends the Jinja2 templating language to write LLM functions.<br><br>class PII {<br>  name string @alias(full_name)<br>  ssn string? @description(&quot;Social security number of the person&quot;)<br>  email string @description(&quot;Email address of the person, must have @&quot;)<br>  age int @description(&quot;Age of the person&quot;)<br>}<br>function ExtractPII(input_text: string) -&gt; PII {<br>  // see clients.baml<br>  client GPT4<br>  // The prompt uses Jinja syntax. Change the models or this text and watch the prompt preview change!<br>  prompt #&quot;<br>    Parse the following PII info and return a structured representation of the data in the schema below.<br> <br>    Text containing PII:<br>    ---<br>    {{input_text}}<br>    ---<br> <br>    {# special macro to print the output instructions. #}<br>    {{ ctx.output_format }}<br> <br>    JSON:<br>  &quot;#<br>}<br>test Test1 {<br>  functions [ExtractPII]<br>  args {<br>    input_text #&quot;<br>      Tony Stark<br>      Age 32<br>      <br>      Education<br>      - University of Southern California, Los Angeles<br>        - M.S. in Computer Science (Specialization in Data Science)<br>        - 2020<br>      IronMan@Avengers.com <br>    &quot;#<br>  }<br>}<br>// Output<br>#&quot;<br>{<br>  &quot;name&quot;: &quot;Tony Stark&quot;,<br>  &quot;ssn&quot;: &quot;&quot;,<br>  &quot;email&quot;: &quot;IronMan@Avengers.ai&quot;,<br>  &quot;age&quot;: 32<br>}<br>&quot;#</pre><h4>Advantages:</h4><ol><li><strong>Fast and Efficient:</strong> Parsing and fixing JSON errors takes microseconds.</li><li><strong>Flexible:</strong> Works with any LLM.</li><li><strong>Cost Effective:</strong> Doesn’t perform any retries or additional calls to LLM if failed.</li></ol><h4>Disadvantages:</h4><ol><li><strong>Limited Error Handling:</strong> Cannot fix completely bad JSON structures.</li><li><strong>Performance: </strong>Might be less effective as the schema constraints are less strictly enforced during generation.</li><li><strong>Reliability:</strong> May still encounter issues with very complex or deeply nested JSON schemas.</li></ol><h3>Conclusion:</h3><p>Choosing the best technique to extract structured data using Large Language Models highly depends on one’s use case and constraints.</p><ol><li>Prompt engineering is great for simple and straightforward extraction but struggles with complex nested formats.</li><li>In Constrained Token Generation approaches, JSON Mode and Function Calling might parse complex format but doesn’t apply for every LLM and cannot guarantee success. Finite State Machine Constrained Token Generation seems promising but may require technical expertise.</li><li>Fixing Bad JSON is quick and works across many LLMs, but struggles with complex errors.</li></ol><blockquote>Ultimately, the best approach depends on project requirements and resources — time, expertise, scalability, accuracy, cost, efficiency and latency etc. One might also combine different methods for their specific requirement. As technology advances, there will be more methods and models which could make structured data parsing efficient.</blockquote><h3>References:</h3><ol><li><a href="https://arxiv.org/abs/2307.09702">https://arxiv.org/abs/2307.09702</a></li><li><a href="https://github.com/boundaryml/baml">https://github.com/boundaryml/baml</a></li><li><a href="https://platform.openai.com/docs/guides/text-generation/json-mode">https://platform.openai.com/docs/guides/text-generation/json-mode</a></li><li><a href="https://platform.openai.com/docs/guides/function-calling">https://platform.openai.com/docs/guides/function-calling</a></li><li><a href="https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#bfcl">https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#bfcl</a></li><li><a href="https://github.com/openai">https://github.com/openai</a></li><li><a href="https://www.lepton.ai/blog/structural-decoding-function-calling-for-all-open-llms">https://www.lepton.ai/blog/structural-decoding-function-calling-for-all-open-llms</a></li><li><a href="https://github.com/outlines-dev/outlines">https://github.com/outlines-dev/outlines</a></li><li><a href="https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor">https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor</a></li><li><a href="https://blog.dottxt.co/coalescence.html">https://blog.dottxt.co/coalescence.html</a></li></ol><p>Kudos, you have learned different approaches to parse structured data. You might be also interested in my previous <a href="https://medium.com/@kaushikshakkari/list/structured-data-extraction-output-parsing-using-llms-51f62e725554">articles</a> for structured data extraction using <a href="https://medium.com/p/7133193358fa">Kor / Langchain</a> and <a href="https://medium.com/p/11e5b7df13be">Instructor</a> libraries.</p><p>Stay tuned for more articles on Large Language Model (LLM) usecases!</p><p><strong>Add me on </strong><a href="https://www.linkedin.com/in/kaushik-shakkari/"><strong>LinkedIn</strong></a><strong>. Thank you!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9686482729c" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/choosing-the-best-structured-output-parser-approach-3-ways-to-generate-structured-output-d9686482729c">Choosing the Best Structured Output Parser Approach | 3 Ways To Generate Structured Output</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Invoice or Bill Custom Parsing using Instructor (Pydantic Extension), Open AI’s GPT-4o & Prompt…]]></title>
            <link>https://blog.gopenai.com/invoice-or-bill-custom-parsing-using-instruction-pydantic-extension-open-ais-gpt-4o-prompt-11e5b7df13be?source=rss-8b97eb1e4ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/11e5b7df13be</guid>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[gpt-40]]></category>
            <category><![CDATA[data-extraction]]></category>
            <dc:creator><![CDATA[Kaushik Shakkari]]></dc:creator>
            <pubDate>Tue, 04 Jun 2024 16:14:30 GMT</pubDate>
            <atom:updated>2024-06-05T16:59:51.113Z</atom:updated>
            <content:encoded><![CDATA[<h4>Multimodal Language Models in Action</h4><h3>Invoice or Bill Custom Parsing using Instructor (Pydantic Extension), Open AI’s GPT-4o &amp; Prompt Engineering</h3><h4>Structured Data Extraction for Semi-Structured Image</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mNvzK5wPDEQBSxxv1xiHOQ.jpeg" /><figcaption>Picture by the author: A pretty sunset near Golden Gate Park, Seattle, Washington (April 2023)</figcaption></figure><p>In my <a href="https://medium.com/gopenai/invoice-or-bill-custom-parsing-using-kor-langchain-extension-generative-language-models-prompt-7133193358fa">last</a> article, I demonstrated how OpenAI’s GPT-3.5 model with the Kor library can be utilized for structured data extraction from unstructured or semi-structured data like PDFs. While this approach is powerful, it is limited to processing textual data alone. Recently, multimodal models, such as OpenAI’s <a href="https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/">GPT-4o</a>, have emerged, capable of processing both text and visual data. These models, trained on large datasets of image-text pairs, understand the correlations between visual cues and textual descriptions. This allows them to comprehend semantic content, interpret visual layouts, and reason about the relationships between visual elements and text, thereby broadening the scope of data extraction tasks. In this article, I will showcase how <a href="https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/">GPT-4o</a> can be used to extract structured data from images and documents, effectively handling complex layouts, handwritten text, and low-quality images without the need for explicit optical character recognition (OCR).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I04oEZmesxRmH65e8hqOWg.png" /><figcaption>Source: <a href="https://openai.com/index/hello-gpt-4o/">https://openai.com/index/hello-gpt-4o/</a></figcaption></figure><h4>Code in Action:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyM5gU6QKBYndZNFpv3quw.png" /><figcaption>Image invoice sample created by the author: <a href="https://drive.google.com/file/d/1yNb2Pbcqlrbqc8C_m9JULxsTzjAc3AJG/view?usp=sharing">download here</a></figcaption></figure><p>Let’s use the sample invoice provided. We aim to extract information such as the invoice number, item details (including description, quantity, unit price, and total), total balance, and various addresses (company, billing, and shipping).</p><p>Link to execute the below code: <a href="https://colab.research.google.com/drive/1s-5GsJ6AXur9HkwPDPRzGCGY5E_3j4cF?usp=sharing">colab notebook</a></p><blockquote><strong>Step 1: Load invoice image file and encode as base64 Image</strong></blockquote><p>Base64 is a widely used encoding scheme that allows binary data like images to be represented as text so the language model can take it as input.</p><pre>import base64<br><br>resume_image_path = &#39;sample_invoice.png&#39;<br><br>with open(resume_image_path, &quot;rb&quot;) as image_file:<br>    encoded_resume_image_path = base64.b64encode(image_file.read()).decode(&#39;utf-8&#39;)</pre><blockquote><strong>Step 2: Install and import relevant packages for extraction</strong></blockquote><p>We mainly use OpenAI and Instructor packages. <a href="https://github.com/jxnl/instructor">Instructor</a> is a Python library designed to simplify working with structured outputs from large language models (LLMs). It is built on Pydantic and offers a straightforward, transparent, and user-friendly API for handling validation, retries, and streaming responses. This enhances the LLM workflows effortlessly.</p><pre>!pip install openai==0.27.8<br><br># instructor is used <br>!pip install instructor==1.3.2<br><br>#replace &#39;&#39; with your OpenAI API key<br>#you can create and view your API keys at https://platform.openai.com/account/api-keys<br>openai_api_key = &#39;&#39;<br><br>from typing import List<br>import instructor<br>from openai import OpenAI<br>from pydantic import BaseModel, Field</pre><blockquote><strong>Step 3: Define PyDantic classes for extraction</strong></blockquote><p>These classes will help in structuring the extracted data from the invoice.</p><pre>class Address(BaseModel):<br>  name: str = Field(description=&quot;the name of person and organization&quot;)<br>  address_line: str = Field(description=&quot;the local delivery information such as street, building number, PO box, or apartment portion of a postal address&quot;)<br>  city: str = Field(description=&quot;the city portion of the address&quot;)<br>  state_province_code: str = Field(description=&quot;the code for address US states&quot;)<br>  postal_code: int = Field(description=&quot;the postal code portion of the address&quot;)<br><br>class Product(BaseModel):<br>  product_description: str = Field(description=&quot;the description of the product or service&quot;)<br>  count: int = Field(description=&quot;number of units bought for the product&quot;)<br>  unit_item_price: float = Field(description=&quot;price per unit&quot;)<br>  product_total_price: float = Field(description=&quot;the total price, which is number of units * unit_price&quot;)<br><br>class TotalBill(BaseModel):<br>  total: float = Field(description=&quot;the total amount before tax and delivery charges&quot;)<br>  discount_amount: float = Field(description=&quot;discount amount is total cost * discount %&quot;)<br>  tax_amount: float = Field(description=&quot;tax amount is tax_percentage * (total - discount_amount). If discount_amount is 0, then its tax_percentage * total&quot;)<br>  delivery_charges: float = Field(description=&quot;the cost of shipping products&quot;)<br>  final_total: float = Field(description=&quot;the total price or balance after removing tax, adding delivery and tax from total&quot;)<br><br>class Invoice(BaseModel):<br>  invoice_number: str = Field(description=&quot;extraction of relevant information from invoice&quot;)<br>  billing_address: Address = Field(description=&quot;where the bill for a product or service is sent so it can be paid by the recipient&quot;)<br>  product: List[Product] = Field(description=&quot;the details of bill&quot;)<br>  total_bill: TotalBill = Field(description=&quot;the details of total amount, discounts and tax&quot;)</pre><blockquote><strong>Step 4: Define chat message and call GPT-4o</strong></blockquote><p>Here, we define the chat messages and call the GPT-4o model to extract data from the invoice image.</p><pre>messages = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Your goal is to extract structured information from the provided invoice&quot;<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>            {<br>                &quot;type&quot;: &quot;image_url&quot;,<br>                &quot;image_url&quot;: {<br>                    &quot;url&quot;: f&quot;data:image/png;base64,{encoded_resume_image_path}&quot;<br>                }<br>            }<br>        ]<br>    }<br>]<br><br><br><br>response = instructor.from_openai(OpenAI(api_key=openai_api_key)).chat.completions.create(<br>    model=&#39;gpt-4o&#39;,<br>    response_model=Invoice,<br>    messages=messages<br>)<br><br>result = response.model_dump_json(indent=2)<br><br>print(result)</pre><blockquote><strong>Sample Output</strong></blockquote><pre>{<br>  &quot;invoice_number&quot;: &quot;INV-28913&quot;,<br>  &quot;billing_address&quot;: {<br>    &quot;name&quot;: &quot;Pepper Potts&quot;,<br>    &quot;address_line&quot;: &quot;Robo Street&quot;,<br>    &quot;city&quot;: &quot;Malibu&quot;,<br>    &quot;state_province_code&quot;: &quot;CA&quot;,<br>    &quot;postal_code&quot;: 90265<br>  },<br>  &quot;product&quot;: [<br>    {<br>      &quot;product_description&quot;: &quot;Lambda Scalar 4U AMD GLU&quot;,<br>      &quot;count&quot;: 1,<br>      &quot;unit_item_price&quot;: 160090.0,<br>      &quot;product_total_price&quot;: 160090.0<br>    },<br>    {<br>      &quot;product_description&quot;: &quot;16-inch MacBook Pro - Space Gray&quot;,<br>      &quot;count&quot;: 2,<br>      &quot;unit_item_price&quot;: 3500.0,<br>      &quot;product_total_price&quot;: 7000.0<br>    },<br>    {<br>      &quot;product_description&quot;: &quot;12.99 inch iPad Pro&quot;,<br>      &quot;count&quot;: 2,<br>      &quot;unit_item_price&quot;: 1200.0,<br>      &quot;product_total_price&quot;: 2400.0<br>    },<br>    {<br>      &quot;product_description&quot;: &quot;2nd generation Apple Pencil&quot;,<br>      &quot;count&quot;: 1,<br>      &quot;unit_item_price&quot;: 130.0,<br>      &quot;product_total_price&quot;: 130.0<br>    },<br>    {<br>      &quot;product_description&quot;: &quot;Space Gray AirPods Max&quot;,<br>      &quot;count&quot;: 1,<br>      &quot;unit_item_price&quot;: 550.0,<br>      &quot;product_total_price&quot;: 550.0<br>    },<br>    {<br>      &quot;product_description&quot;: &quot;Service Fee&quot;,<br>      &quot;count&quot;: 1,<br>      &quot;unit_item_price&quot;: 250.0,<br>      &quot;product_total_price&quot;: 250.0<br>    }<br>  ],<br>  &quot;total_bill&quot;: {<br>    &quot;total&quot;: 170420.0,<br>    &quot;discount_amount&quot;: 0.0,<br>    &quot;tax_amount&quot;: 15378.0,<br>    &quot;delivery_charges&quot;: 100.0,<br>    &quot;final_total&quot;: 153478.0<br>  }<br>}</pre><h4>Conclusion:</h4><p>In this article, we explored how multimodal language models like GPT-4o can be effectively used to extract structured data from images and documents. The integration of text and visual data processing capabilities enables these models to handle complex layouts and low-quality images without the need for OCR. By leveraging tools like OpenAI and the Instructor library, we can streamline the extraction process, enhancing efficiency and accuracy. This approach opens up new possibilities for automating data extraction tasks across various industries, providing a powerful solution for handling semi-structured data.</p><p>Kudos, you have learned how to custom parse!<br>Stay tuned for more articles on generative language modeling!!!</p><p><strong>Add me on </strong><a href="https://www.linkedin.com/in/kaushik-shakkari/"><strong>LinkedIn</strong></a><strong>. Thank you!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11e5b7df13be" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/invoice-or-bill-custom-parsing-using-instruction-pydantic-extension-open-ais-gpt-4o-prompt-11e5b7df13be">Invoice or Bill Custom Parsing using Instructor (Pydantic Extension), Open AI’s GPT-4o &amp; Prompt…</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>