<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mahasri Seenivasan on Medium]]></title>
        <description><![CDATA[Stories by Mahasri Seenivasan on Medium]]></description>
        <link>https://medium.com/@mahasris0304?source=rss-52aea5bc4db7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EBkehjy7dLkdqtC6sW9z1w.png</url>
            <title>Stories by Mahasri Seenivasan on Medium</title>
            <link>https://medium.com/@mahasris0304?source=rss-52aea5bc4db7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:24:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mahasris0304/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DSPy: The Future of AI Programming — A Complete Guide]]></title>
            <link>https://medium.com/@mahasris0304/dspy-the-future-of-ai-programming-a-complete-guide-9caeb9882c82?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9caeb9882c82</guid>
            <category><![CDATA[dspy]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 13 Dec 2025 08:29:51 GMT</pubDate>
            <atom:updated>2025-12-13T09:59:48.965Z</atom:updated>
            <content:encoded><![CDATA[<h3>DSPy: The Future of AI Programming — A Complete Guide</h3><p><em>“The New Way to Build Smarter, Structured, Optimizable AI Workflows”</em></p><h3>Introduction</h3><p>Imagine building AI applications without spending hours tweaking prompts. Imagine your AI systems automatically improving themselves through optimization. This isn’t science fiction — it’s <strong>DSPy</strong> (Declarative Self-improving Python), and it’s revolutionizing how we build AI software.</p><p>I want to share why this framework is a game-changer for AI developers, and how you can leverage it to build production-grade AI applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyEz-VDv-gkxC_pPl-jGxQ.jpeg" /><figcaption>Image created with gemini</figcaption></figure><h3>What Is DSPy?</h3><p>DSPy is a <strong>declarative framework for building LLM-powered systems</strong>.<br> Instead of writing long prompts or ad-hoc chains, you describe the <strong>inputs, outputs, and modules</strong>, and DSPy figures out the best way to run and optimize them.</p><p>Think of it like this:</p><ul><li>LangChain = chain LLM calls</li><li>Prompt engineering = manually write the instructions</li><li><strong>DSPy = define the tasks → build modules → optimize automatically</strong></li></ul><p>It treats LLM workflows as real, structured programs — something we can train, evaluate, test, and improve.</p><blockquote>In simple terms:</blockquote><blockquote>“DSPy is a <strong>framework for programming LLMs</strong> using Python modules instead of raw prompts.</blockquote><blockquote>It replaces this:</blockquote><blockquote>Long prompts</blockquote><blockquote>Trial-and-error tuning</blockquote><blockquote>Copy-pasted instructions</blockquote><blockquote><strong>with this:</strong></blockquote><blockquote><strong>Signatures</strong> (input/output structure)</blockquote><blockquote><strong>Modules</strong> (logical building blocks)</blockquote><blockquote><strong>Compilers</strong> (automatic prompt optimization)</blockquote><h3>Core Philosophy</h3><p>Instead of spending hours crafting and tweaking prompts, DSPy lets you:</p><ul><li>Define <strong>what</strong> you want your AI to do (the task)</li><li>Let the framework figure out <strong>how</strong> to do it (the prompts)</li><li>Automatically optimize prompts and even model weights based on your metrics</li></ul><h3>Key Components</h3><p>DSPy applications are built using three main elements:</p><h4>1. Language Model (LM)</h4><p>The foundation model that powers your application. DSPy supports multiple providers including OpenAI, Anthropic, local models through Ollama, and dozens more through LiteLLM integration.</p><h4>2. Signatures</h4><p>These define the input and output types of a module, ensuring compatibility between different modules <a href="https://www.ibm.com/think/topics/dspy">IBM</a>. Signatures create a contract between your code and the LLM.</p><p>Examples:</p><ul><li>question -&gt; answer (simple Q&amp;A)</li><li>question -&gt; answer: float (Q&amp;A with numeric answer)</li><li>document -&gt; summary (text summarization)</li><li>context, question -&gt; reasoning, answer (complex reasoning)</li></ul><h4>3. Modules</h4><p>These define the strategy for invoking your LM. Common modules include:</p><ul><li>dspy.Predict - Basic prediction</li><li>dspy.ChainOfThought - Step-by-step reasoning</li><li>dspy.ReAct - Reasoning and acting with tools</li><li>dspy.ProgramOfThought - Generates code to solve problems</li></ul><h3>The Three Stages of DSPy Development</h3><h4>Stage 1: Programming</h4><p>Define your task, constraints, and initial pipeline design. Start simple with a single module, then add complexity as needed. Consider whether you need retrieval, tools, or multi-step workflows.</p><h4>Stage 2: Evaluation</h4><p>Collect a development dataset and define metrics to evaluate your system’s performance. This allows you to iterate systematically rather than guessing if changes improve results.</p><h4>Stage 3: Optimization</h4><p>DSPy provides tools to compile high-level code with natural language annotations into low-level computations, prompts, or weight updates that align your LM with your program’s structure and metrics<em>(</em><a href="https://dspy.ai/"><em>DSPy</em></a><em>)</em>.</p><h3>Optimization in DSPy</h3><p>DSPy doesn’t make you tweak prompts manually. You give it 10–50 labeled examples + a metric, and it automatically compiles a better version of your program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8MaEpsfGkm_DRysTp-EDA.png" /><figcaption>Optimizer in DSPy</figcaption></figure><h3>Why Use DSPy?</h3><h4>The Problem with Traditional Prompting</h4><p>When building LLM applications traditionally, you face several challenges:</p><ol><li><strong>Manual prompt crafting</strong>: Hours spent tweaking prompts for each task</li><li><strong>Brittleness</strong>: Prompts that work for GPT-4 might fail on Claude or Llama</li><li><strong>No systematic optimization</strong>: Trial and error instead of data-driven improvement</li><li><strong>Hard to maintain</strong>: Changes in one part break other parts</li><li><strong>Difficult to compose</strong>: Combining multiple LLM calls is messy</li></ol><h4>The DSPy Solution</h4><p>DSPy solves these problems by:</p><ul><li><strong>Separating logic from prompting</strong>: You write what you want, not how to ask for it</li><li><strong>Automatic optimization</strong>: DSPy compiles your program into optimized prompts</li><li><strong>Model portability</strong>: Switch models without rewriting prompts</li><li><strong>Composability</strong>: Build complex pipelines from simple modules</li><li><strong>Data-driven</strong>: Uses examples to optimize performance</li></ul><p>In essence, DSPy makes LLM development more like software engineering: testable, versionable, and scalable.</p><h3>Real-World Use Cases</h3><p>DSPy excels in applications like:</p><ul><li><strong>Multi-hop question answering</strong>: Systems that need to retrieve and reason over multiple documents</li><li><strong>Retrieval-Augmented Generation (RAG)</strong>: Building chatbots that answer from your knowledge base</li><li><strong>Agent systems</strong>: Creating AI agents that use tools and make decisions</li><li><strong>Classification pipelines</strong>: Multi-stage classification with reasoning</li><li><strong>Code generation</strong>: Systems that write and verify code</li><li><strong>Data extraction</strong>: Structured information extraction from unstructured text</li></ul><h3>Learning DSPy: Level by Level</h3><p>To master DSPy, we’ll progress through 7 levels . The 7-level structure builds more gradually, introducing retrieval and agents earlier for broader applicability while maintaining clear improvements per level. Each includes concepts, what’s new/improved, and runnable code examples.</p><p><strong>Level 1: Your First DSPy Program</strong> <strong>Concepts</strong>: Signatures, Modules, Language Models At this level, you’ll learn the absolute basics: how to define what you want the LLM to do using signatures.</p><p><strong>What’s New in Level 1?</strong></p><ul><li>Signatures: A pythonic way to describe input/output behavior</li><li>ChainOfThought: A module that adds reasoning to any task</li><li>Basic LM configuration: Setting up your language model</li><li><strong>Example: Simple Sentiment Analysis</strong></li></ul><pre> ! pip install openai dspy litellm</pre><pre>import os<br>os.environ[&quot;OPENAI_API_KEY&quot;] = &quot;Your_API_key&quot;</pre><pre>import dspy<br><br># Step 1: Configure the language model<br>lm = dspy.LM(&#39;openai/gpt-4o-mini&#39;)<br>dspy.configure(lm=lm)<br><br># Step 2: Define a signature (input -&gt; output specification)<br>class Emotion(dspy.Signature):<br>    &quot;&quot;&quot;Analyze the emotion in the given text.&quot;&quot;&quot;<br>    text: str = dspy.InputField()<br>    emotion: str = dspy.OutputField(desc=&quot;The primary emotion: happy, sad, angry, or neutral&quot;)<br>    rationale: str = dspy.OutputField(desc=&quot;The reasoning for the emotion classification&quot;) # Added rationale to signature<br><br># Step 3: Create a predictor with chain-of-thought reasoning<br>emotion_classifier = dspy.ChainOfThought(Emotion)<br><br># Step 4: Use it!<br>result = emotion_classifier(text=&quot;I just got promoted at work!&quot;)<br>print(f&quot;Emotion: {result.emotion}&quot;)<br>print(f&quot;Reasoning: {result.rationale}&quot;)</pre><p>Output: Emotion: happy<br>Reasoning: The text expresses a positive life event (promotion) which typically evokes happiness…</p><p><strong><em>Key Takeaway</em></strong><em>: You describe what you want (emotion from text) without writing prompts. DSPy handles the rest.</em></p><h3><strong>Level 2: Building Multi-Step Programs</strong> <strong>Concepts</strong>:</h3><p>Modules, Program Composition, Sequential Processing Now you’ll learn to chain multiple LLM calls together into coherent programs. <strong>What’s New in Level 2?</strong></p><ul><li>Custom modules: Creating reusable components</li><li>Sequential reasoning: Breaking complex tasks into steps</li><li>State management: Passing information between steps</li><li><strong>What’s Improved from Level 1?</strong> Instead of single-shot predictions, you can now build multi-stage pipelines where each stage builds on the previous one.</li><li><strong>Example: Question Answering with Context.</strong></li></ul><pre>import dspy<br><br>lm = dspy.LM(&#39;openai/gpt-4o-mini&#39;)<br>dspy.configure(lm=lm)<br><br># Define signatures for each step<br>class GenerateSearchQuery(dspy.Signature):<br>    &quot;&quot;&quot;Generate a search query from a question.&quot;&quot;&quot;<br>    question: str = dspy.InputField()<br>    query: str = dspy.OutputField()<br><br>class AnswerQuestion(dspy.Signature):<br>    &quot;&quot;&quot;Answer a question given context.&quot;&quot;&quot;<br>    context: str = dspy.InputField()<br>    question: str = dspy.InputField()<br>    answer: str = dspy.OutputField()<br><br># Build a multi-step module<br>class SimpleQA(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.generate_query = dspy.ChainOfThought(GenerateSearchQuery)<br>        self.answer = dspy.ChainOfThought(AnswerQuestion)<br>    <br>    def forward(self, question):<br>        # Step 1: Generate search query<br>        query = self.generate_query(question=question).query<br>        <br>        # Step 2: Simulate retrieval (in real app, use a retriever)<br>        context = f&quot;Retrieved documents about: {query}&quot;<br>        <br>        # Step 3: Generate answer<br>        answer = self.answer(context=context, question=question).answer<br>        <br>        return dspy.Prediction(answer=answer, query=query)<br><br># Use the pipeline<br>qa = SimpleQA()<br>result = qa(question=&quot;What is the capital of France?&quot;)<br>print(f&quot;Search query: {result.query}&quot;)<br>print(f&quot;Answer: {result.answer}&quot;)</pre><p><strong>Output:</strong> Search query: capital of France<br>Answer: The capital of France is Paris.</p><p><strong><em>Key Takeaway</em></strong><em>: You can compose complex behaviors from simple, reusable modules.</em></p><h3><strong>Level 3: Adding Retrieval (RAG)</strong> <strong>Concepts</strong>:</h3><p>Retrieval-Augmented Generation, External Knowledge, ColBERTv2 This level introduces retrieval, allowing your program to access external knowledge. <strong>What’s New in Level 3?</strong></p><ul><li>Retrieve module: Built-in retrieval integration</li><li>Knowledge bases: Connecting to vector databases or search indices</li><li>Context management: Handling retrieved documents</li><li><strong>What’s Improved from Level 2?</strong> Your programs can now access real information from databases, documents, or the web instead of relying solely on the model’s knowledge.</li><li><strong>Example: RAG System</strong></li></ul><pre>import dspy<br>import os<br><br><br><br># Configure LM<br>lm = dspy.LM(<br>    provider=&quot;litellm&quot;,  # Required for OpenAI routing<br>    model=&quot;openai/gpt-4o-mini&quot;,  # Or &quot;openai/gpt-4o&quot; for better quality<br>    max_tokens=300<br>)<br>dspy.configure(lm=lm)<br><br># === Knowledge base ===<br>documents = [<br>    &quot;Paris is the capital of France and its largest city.&quot;,<br>    &quot;The Eiffel Tower in Paris was completed in 1889.&quot;,<br>    &quot;France is located in Western Europe.&quot;,<br>    &quot;The population of Paris is about 2.2 million.&quot;,<br>]<br><br># === Fixed Retriever: Returns dspy.Document objects ===<br>class SimpleRetriever(dspy.Retrieve):<br>    def __init__(self, docs):<br>        self.docs = docs<br><br>    def forward(self, query, k=3):<br>        # Simple keyword ranking<br>        ranked = sorted(<br>            self.docs,<br>            key=lambda d: sum(word in d.lower() for word in query.lower().split()),<br>            reverse=True<br>        )<br>        # ← CRITICAL: Wrap each string in dspy.Document<br>        docs = [dspy.Document(text=text) for text in ranked[:k]]<br>        return dspy.Prediction(passages=docs)<br><br># ← Register the retriever (instance, not class!)<br>dspy.configure(rm=SimpleRetriever(documents))<br><br># === RAG Signature ===<br>class RAGSignature(dspy.Signature):<br>    &quot;&quot;&quot;Answer the question using only the provided context.&quot;&quot;&quot;<br>    context: list[str] = dspy.InputField(desc=&quot;Relevant passages&quot;)<br>    question: str = dspy.InputField()<br>    answer: str = dspy.OutputField()<br><br># === RAG Module ===<br>class RAG(dspy.Module):<br>    def __init__(self, k=3):<br>        super().__init__()<br>        self.retrieve = dspy.Retrieve(k=k)<br>        self.generate = dspy.ChainOfThought(RAGSignature)<br><br>    def forward(self, question):<br>        retrieved = self.retrieve(question)<br>        # ← Extract .text from each Document<br>        context = [doc.text for doc in retrieved.passages]<br>        pred = self.generate(context=context, question=question)<br>        return dspy.Prediction(<br>            answer=pred.answer,<br>            passages=context  # return plain strings for display<br>        )<br><br># === Run it! ===<br>rag = RAG(k=2)<br>result = rag(question=&quot;When was the Eiffel Tower completed?&quot;)<br><br>print(&quot;Answer →&quot;, result.answer)<br>print(&quot;\nSources used:&quot;)<br>for i, src in enumerate(result.passages, 1):<br>    print(f&quot;{i}. {src}&quot;)</pre><p><strong><em>Key Takeaway</em></strong><em>: RAG systems become simple to build and reason about with DSPy’s modular approach.</em></p><h3>Level 4: Optimization with Examples</h3><p><strong>Concepts</strong>: Optimizers, Training Data, Metric Functions, Compilation</p><p>This is where DSPy truly shines. Instead of manually tweaking prompts, you provide examples and let DSPy optimize.</p><h4>What’s New in Level 4?</h4><ul><li><strong>Training examples</strong>: Labeled data to guide optimization</li><li><strong>Optimizers</strong>: Algorithms that improve your program</li><li><strong>Metrics</strong>: Functions to measure quality</li><li><strong>Compilation</strong>: Transforming your program into an optimized version</li></ul><h4>What’s Improved from Level 3?</h4><p>Your programs now learn from data. DSPy automatically generates and optimizes prompts, few-shot examples, and reasoning chains based on your examples and metric.</p><h4>Example: Optimizing a Classifier</h4><pre>import dspy<br>from dspy.teleprompt import BootstrapFewShot<br><br>lm = dspy.LM(&#39;openai/gpt-4o-mini&#39;)<br>dspy.configure(lm=lm)<br><br># Define the task<br>class ClassifyIntent(dspy.Signature):<br>    &quot;&quot;&quot;Classify customer support ticket intent.&quot;&quot;&quot;<br>    ticket: str = dspy.InputField()<br>    intent: str = dspy.OutputField(desc=&quot;One of: billing, technical, account, other&quot;)<br><br># Create training examples<br>train_examples = [<br>    dspy.Example(<br>        ticket=&quot;I was charged twice for my subscription&quot;,<br>        intent=&quot;billing&quot;<br>    ).with_inputs(&quot;ticket&quot;),<br>    dspy.Example(<br>        ticket=&quot;The app crashes when I try to log in&quot;,<br>        intent=&quot;technical&quot;<br>    ).with_inputs(&quot;ticket&quot;),<br>    dspy.Example(<br>        ticket=&quot;I forgot my password&quot;,<br>        intent=&quot;account&quot;<br>    ).with_inputs(&quot;ticket&quot;),<br>    dspy.Example(<br>        ticket=&quot;When is your office open?&quot;,<br>        intent=&quot;other&quot;<br>    ).with_inputs(&quot;ticket&quot;),<br>]<br><br># Create validation examples<br>val_examples = [<br>    dspy.Example(<br>        ticket=&quot;Double payment on my credit card&quot;,<br>        intent=&quot;billing&quot;<br>    ).with_inputs(&quot;ticket&quot;),<br>    dspy.Example(<br>        ticket=&quot;Cannot upload files&quot;,<br>        intent=&quot;technical&quot;<br>    ).with_inputs(&quot;ticket&quot;),<br>]<br><br># Define a metric<br>def intent_metric(example, pred, trace=None):<br>    return example.intent.lower() == pred.intent.lower()<br><br># Create the program<br>class IntentClassifier(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.classify = dspy.ChainOfThought(ClassifyIntent)<br>    <br>    def forward(self, ticket):<br>        return self.classify(ticket=ticket)<br><br># Compile (optimize) the program<br>optimizer = BootstrapFewShot(metric=intent_metric, max_bootstrapped_demos=3)<br>classifier = IntentClassifier()<br><br>compiled_classifier = optimizer.compile(<br>    classifier,<br>    trainset=train_examples<br>)<br><br># Compare uncompiled vs compiled<br>test_ticket = &quot;I need to update my billing address&quot;<br><br>print(&quot;=&quot; * 60)<br>print(&quot;UNCOMPILED CLASSIFIER&quot;)<br>print(&quot;=&quot; * 60)<br>result1 = classifier(ticket=test_ticket)<br>print(f&quot;Intent: {result1.intent}&quot;)<br># Check what attributes are available<br>if hasattr(result1, &#39;reasoning&#39;):<br>    print(f&quot;Reasoning: {result1.reasoning}&quot;)<br>elif hasattr(result1, &#39;rationale&#39;):<br>    print(f&quot;Rationale: {result1.rationale}&quot;)<br><br>print(&quot;\n&quot; + &quot;=&quot; * 60)<br>print(&quot;COMPILED (OPTIMIZED) CLASSIFIER&quot;)<br>print(&quot;=&quot; * 60)<br>result2 = compiled_classifier(ticket=test_ticket)<br>print(f&quot;Intent: {result2.intent}&quot;)<br># Check what attributes are available<br>if hasattr(result2, &#39;reasoning&#39;):<br>    print(f&quot;Reasoning: {result2.reasoning}&quot;)<br>elif hasattr(result2, &#39;rationale&#39;):<br>    print(f&quot;Rationale: {result2.rationale}&quot;)<br><br># CORRECT WAY to inspect what DSPy learned<br>print(&quot;\n&quot; + &quot;=&quot; * 60)<br>print(&quot;OPTIMIZED FEW-SHOT EXAMPLES&quot;)<br>print(&quot;=&quot; * 60)<br><br># Method 1: Access the predictor&#39;s demos attribute<br>try:<br>    if hasattr(compiled_classifier.classify, &#39;demos&#39;):<br>        demos = compiled_classifier.classify.demos<br>        print(f&quot;\nFound {len(demos)} demonstration examples:&quot;)<br>        for i, demo in enumerate(demos, 1):<br>            print(f&quot;\nDemo {i}:&quot;)<br>            print(f&quot;  Input: {demo.ticket if hasattr(demo, &#39;ticket&#39;) else demo}&quot;)<br>            if hasattr(demo, &#39;intent&#39;):<br>                print(f&quot;  Output: {demo.intent}&quot;)<br>    else:<br>        print(&quot;Demos not directly accessible via .demos attribute&quot;)<br>except Exception as e:<br>    print(f&quot;Error accessing demos: {e}&quot;)<br><br># Method 2: Inspect the predictor&#39;s signature and extended signature<br>print(&quot;\n&quot; + &quot;=&quot; * 60)<br>print(&quot;PREDICTOR DETAILS&quot;)<br>print(&quot;=&quot; * 60)<br><br>try:<br>    predictor = compiled_classifier.classify<br>    print(f&quot;\nPredictor type: {type(predictor)}&quot;)<br>    print(f&quot;\nPredictor attributes: {[attr for attr in dir(predictor) if not attr.startswith(&#39;_&#39;)]}&quot;)<br>    <br>    # Check for extended_signature which contains the optimized prompt<br>    if hasattr(predictor, &#39;extended_signature&#39;):<br>        print(f&quot;\nExtended signature found!&quot;)<br>        ext_sig = predictor.extended_signature<br>        print(f&quot;Extended signature type: {type(ext_sig)}&quot;)<br>    <br>    # Check for the actual prompt that will be used<br>    if hasattr(predictor, &#39;signature&#39;):<br>        print(f&quot;\nSignature: {predictor.signature}&quot;)<br>    <br>except Exception as e:<br>    print(f&quot;Error inspecting predictor: {e}&quot;)<br><br># Method 3: Test with multiple examples to see the optimization effect<br>print(&quot;\n&quot; + &quot;=&quot; * 60)<br>print(&quot;TESTING ON MULTIPLE EXAMPLES&quot;)<br>print(&quot;=&quot; * 60)<br><br>test_cases = [<br>    &quot;I was billed incorrectly&quot;,<br>    &quot;The system won&#39;t let me reset my credentials&quot;,<br>    &quot;Error 404 when accessing dashboard&quot;,<br>    &quot;What are your business hours?&quot;,<br>]<br><br>print(&quot;\nUncompiled vs Compiled Results:&quot;)<br>print(&quot;-&quot; * 60)<br>for test in test_cases:<br>    r1 = classifier(ticket=test)<br>    r2 = compiled_classifier(ticket=test)<br>    match = &quot;✓&quot; if r1.intent == r2.intent else &quot;✗&quot;<br>    print(f&quot;\nTicket: {test}&quot;)<br>    print(f&quot;  Uncompiled: {r1.intent}&quot;)<br>    print(f&quot;  Compiled:   {r2.intent} {match}&quot;)<br><br># Method 4: Save and inspect the compiled program<br>print(&quot;\n&quot; + &quot;=&quot; * 60)<br>print(&quot;SAVING COMPILED PROGRAM&quot;)<br>print(&quot;=&quot; * 60)<br><br># Save the compiled classifier<br>compiled_classifier.save(&quot;intent_classifier_compiled.json&quot;)<br>print(&quot;✓ Compiled classifier saved to: intent_classifier_compiled.json&quot;)<br>print(&quot;  You can inspect this file to see the optimized prompts and examples&quot;)<br><br># Load it back<br>loaded_classifier = IntentClassifier()<br>loaded_classifier.load(&quot;intent_classifier_compiled.json&quot;)<br>print(&quot;✓ Loaded classifier from file&quot;)<br><br># Test loaded classifier<br>result_loaded = loaded_classifier(ticket=test_ticket)<br>print(f&quot;\nLoaded classifier result: {result_loaded.intent}&quot;)</pre><p><strong><em>Key Takeaway</em></strong><em>: DSPy uses your examples to automatically create optimized prompts with the best few-shot examples, improving accuracy without manual prompt engineering.</em></p><h3>Level 5: Advanced Optimization Strategies</h3><p><strong>Concepts</strong>: Multiple Optimizers, Ensemble Methods, Bootstrap Techniques</p><p>Master different optimization strategies for different scenarios.</p><h4>What’s New in Level 5?</h4><ul><li><strong>MIPRO</strong>: Advanced optimizer using Bayesian optimization</li><li><strong>Signature Optimizer</strong>: Optimizes instruction text itself</li><li><strong>Ensemble methods</strong>: Combining multiple approaches</li></ul><h4>What’s Improved from Level 4?</h4><p>More sophisticated optimization techniques that can handle complex tasks, optimize instruction text, and create ensemble systems.</p><h4>Example: Multi-Stage Optimization</h4><pre>import dspy<br>from dspy.teleprompt import BootstrapFewShot, MIPROv2 as MIPRO<br><br>lm = dspy.LM(&#39;openai/gpt-4o-mini&#39;)<br>dspy.configure(lm=lm)<br><br># Complex multi-hop QA signature<br>class MultiHopQA(dspy.Signature):<br>    &quot;&quot;&quot;Answer complex questions requiring multiple reasoning steps.&quot;&quot;&quot;<br>    question: str = dspy.InputField()<br>    answer: str = dspy.OutputField(desc=&quot;A comprehensive answer with reasoning&quot;)<br>    confidence: str = dspy.OutputField(desc=&quot;high, medium, or low&quot;)<br><br># Advanced program with multiple stages<br>class AdvancedQA(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.decompose = dspy.ChainOfThought(&quot;question -&gt; subquestions: list[str]&quot;)<br>        self.answer_sub = dspy.ChainOfThought(&quot;subquestion -&gt; answer&quot;)<br>        self.synthesize = dspy.ChainOfThought(MultiHopQA)<br><br>    def forward(self, question):<br>        # Decompose into subquestions<br>        subquestions = self.decompose(question=question).subquestions<br><br>        # Answer each subquestion<br>        subanswers = []<br>        for sq in subquestions[:3]:  # Limit to 3<br>            ans = self.answer_sub(subquestion=sq).answer<br>            subanswers.append(f&quot;Q: {sq}\nA: {ans}&quot;)<br><br>        # Synthesize final answer<br>        context = &quot;\n\n&quot;.join(subanswers)<br>        result = self.synthesize(question=f&quot;{context}\n\nOriginal question: {question}&quot;)<br><br>        return result<br><br># Create training data<br>train_data = [<br>    dspy.Example(<br>        question=&quot;If the Eiffel Tower is 330m tall and each level is 110m, how many levels can you visit?&quot;,<br>        answer=&quot;You can visit 3 levels (330m ÷ 110m = 3)&quot;,<br>        confidence=&quot;high&quot;<br>    ).with_inputs(&quot;question&quot;),<br>    dspy.Example(<br>        question=&quot;What&#39;s 15% of the population of Paris?&quot;,<br>        answer=&quot;Approximately 330,000 people (2.2 million × 0.15)&quot;,<br>        confidence=&quot;medium&quot;<br>    ).with_inputs(&quot;question&quot;),<br>]<br><br># Define metric<br>def qa_metric(example, pred, trace=None):<br>    # Check if answer contains key elements<br>    answer_match = any(word in pred.answer.lower() for word in example.answer.lower().split())<br>    confidence_match = pred.confidence.lower() in [&quot;high&quot;, &quot;medium&quot;, &quot;low&quot;]<br>    return answer_match and confidence_match<br><br># Optimize with advanced optimizer<br>program = AdvancedQA()<br><br># MIPRO optimizer - optimizes both prompts and demonstrations<br>optimizer = MIPRO(<br>    metric=qa_metric,<br>    init_temperature=1.0<br>)<br><br># This would take longer in practice, showing concept<br>print(&quot;Optimizing program...&quot;)<br>optimized_program = optimizer.compile(<br>    program,<br>    trainset=train_data,<br>    max_bootstrapped_demos=3,<br>    max_labeled_demos=2,<br>    requires_permission_to_run=False<br>)<br><br># Test the optimized program<br>result = optimized_program(<br>    question=&quot;How many residents of Paris could fit in Eiffel Tower visits if each level holds 500 people?&quot;<br>)<br>print(f&quot;\nAnswer: {result.answer}&quot;)<br>print(f&quot;Confidence: {result.confidence}&quot;)</pre><p><strong><em>Key Takeaway</em></strong><em>: Advanced optimizers can tune multiple aspects of your program simultaneously, finding the best combination of instructions, examples, and reasoning strategies.</em></p><h3>Level 6: Production-Ready Systems</h3><p><strong>Concepts</strong>: Assertions, Error Handling, Retry Logic, Monitoring</p><p>Build robust, production-grade applications with proper error handling and validation.</p><h4>What’s New in Level 6?</h4><ul><li><strong>Assertions</strong>: Validate LLM outputs programmatically</li><li><strong>Retry mechanisms</strong>: Handle failures gracefully</li><li><strong>Suggest</strong>: Backtracking and correction</li><li><strong>Logging and monitoring</strong>: Track performance</li></ul><h4>What’s Improved from Level 5?</h4><p>Your programs become production-ready with robust error handling, validation, and the ability to self-correct.</p><h4>Example: Production RAG with Validation</h4><pre><br><br>import dspy<br><br>lm = dspy.LM(&#39;openai/gpt-4o-mini&#39;)<br>dspy.configure(lm=lm)<br><br>print(&quot;=&quot;*60)<br>print(&quot;DSPy Level 6: Production Systems with Validation&quot;)<br>print(&quot;=&quot;*60)<br><br># ===================================================================<br># METHOD 1: Using dspy.Refine (Modern Approach - DSPy 2.6+)<br># ===================================================================<br><br>print(&quot;\n&quot; + &quot;=&quot;*60)<br>print(&quot;METHOD 1: Using dspy.Refine for Self-Correction&quot;)<br>print(&quot;=&quot;*60)<br><br>class AnswerQuestion(dspy.Signature):<br>    &quot;&quot;&quot;Answer a question concisely.&quot;&quot;&quot;<br>    question: str = dspy.InputField()<br>    answer: str = dspy.OutputField(desc=&quot;A clear, concise answer&quot;)<br><br># Define a reward function (validation function)<br>def answer_quality_reward(args, pred):<br>    &quot;&quot;&quot;<br>    Reward function that scores the prediction quality.<br>    Returns 1.0 if good, 0.0 if bad, or values in between.<br>    &quot;&quot;&quot;<br>    answer = pred.answer.strip()<br>    <br>    # Check minimum length<br>    if len(answer.split()) &lt; 5:<br>        return 0.0  # Too short<br>    <br>    # Check not empty<br>    if not answer or answer.lower() in [&#39;i dont know&#39;, &#39;unknown&#39;, &#39;n/a&#39;]:<br>        return 0.0  # Non-answer<br>    <br>    # Check for reasonable length (not too long)<br>    if len(answer.split()) &gt; 100:<br>        return 0.5  # Too verbose<br>    <br>    # Good answer<br>    return 1.0<br><br># Create base module<br>qa_module = dspy.ChainOfThought(AnswerQuestion)<br><br># Wrap with Refine for automatic retry and improvement<br># Refine will retry up to N times to get a better answer<br>refined_qa = dspy.Refine(<br>    module=qa_module,<br>    N=3,  # Try up to 3 times<br>    reward_fn=answer_quality_reward,<br>    threshold=1.0  # Stop when reward reaches 1.0<br>)<br><br>print(&quot;\nTesting Refine module...&quot;)<br>test_questions = [<br>    &quot;What is AI?&quot;,<br>    &quot;Explain quantum computing&quot;,<br>]<br><br>for question in test_questions:<br>    print(f&quot;\nQuestion: {question}&quot;)<br>    result = refined_qa(question=question)<br>    print(f&quot;Answer: {result.answer}&quot;)<br>    print(f&quot;Quality Score: {answer_quality_reward(None, result)}&quot;)<br><br># ===================================================================<br># METHOD 2: Using dspy.BestOfN (Run Multiple Times, Pick Best)<br># ===================================================================<br><br>print(&quot;\n&quot; + &quot;=&quot;*60)<br>print(&quot;METHOD 2: Using dspy.BestOfN for Multiple Attempts&quot;)<br>print(&quot;=&quot;*60)<br><br>class SummarizeText(dspy.Signature):<br>    &quot;&quot;&quot;Summarize text in one sentence.&quot;&quot;&quot;<br>    text: str = dspy.InputField()<br>    summary: str = dspy.OutputField(desc=&quot;A one-sentence summary&quot;)<br><br>def summary_reward(args, pred):<br>    &quot;&quot;&quot;Reward function for summary quality.&quot;&quot;&quot;<br>    summary = pred.summary.strip()<br>    <br>    # Count sentences (rough estimate)<br>    sentence_count = summary.count(&#39;.&#39;) + summary.count(&#39;!&#39;) + summary.count(&#39;?&#39;)<br>    <br>    # Reward single sentence summaries<br>    if sentence_count &lt;= 1 and len(summary.split()) &gt;= 10:<br>        return 1.0<br>    elif sentence_count &lt;= 2:<br>        return 0.7<br>    else:<br>        return 0.3<br><br>summarizer = dspy.ChainOfThought(SummarizeText)<br><br># BestOfN will run the module N times and return the best result<br>best_of_3 = dspy.BestOfN(<br>    module=summarizer,<br>    N=3,  # Generate 3 summaries<br>    reward_fn=summary_reward,<br>    threshold=1.0  # Return first one that reaches 1.0, or best overall<br>)<br><br>print(&quot;\nTesting BestOfN module...&quot;)<br>long_text = &quot;&quot;&quot;<br>Artificial intelligence (AI) is transforming industries worldwide. <br>From healthcare to finance, AI systems are automating complex tasks. <br>Machine learning algorithms can now predict diseases, detect fraud, <br>and even create art. The future of AI holds immense potential.<br>&quot;&quot;&quot;<br><br>result = best_of_3(text=long_text)<br>print(f&quot;Best Summary: {result.summary}&quot;)<br>print(f&quot;Quality Score: {summary_reward(None, result)}&quot;)<br><br># ===================================================================<br># METHOD 3: Manual Validation with Error Handling (Legacy Support)<br># ===================================================================<br><br>print(&quot;\n&quot; + &quot;=&quot;*60)<br>print(&quot;METHOD 3: Manual Validation with Retry Logic&quot;)<br>print(&quot;=&quot;*60)<br><br>class ClassifyIntent(dspy.Signature):<br>    &quot;&quot;&quot;Classify customer support intent.&quot;&quot;&quot;<br>    message: str = dspy.InputField()<br>    intent: str = dspy.OutputField(desc=&quot;billing, technical, account, or general&quot;)<br><br>def validate_intent(intent):<br>    &quot;&quot;&quot;Validate intent is one of the allowed values.&quot;&quot;&quot;<br>    valid_intents = [&#39;billing&#39;, &#39;technical&#39;, &#39;account&#39;, &#39;general&#39;]<br>    return intent.lower().strip() in valid_intents<br><br>class IntentClassifierWithValidation(dspy.Module):<br>    def __init__(self, max_retries=3):<br>        super().__init__()<br>        self.classify = dspy.ChainOfThought(ClassifyIntent)<br>        self.max_retries = max_retries<br>    <br>    def forward(self, message):<br>        &quot;&quot;&quot;Forward with manual retry logic.&quot;&quot;&quot;<br>        for attempt in range(self.max_retries):<br>            try:<br>                # Get prediction<br>                pred = self.classify(message=message)<br>                <br>                # Validate<br>                if validate_intent(pred.intent):<br>                    return dspy.Prediction(<br>                        intent=pred.intent.lower().strip(),<br>                        attempts=attempt + 1,<br>                        success=True<br>                    )<br>                else:<br>                    print(f&quot;  Attempt {attempt + 1}: Invalid intent &#39;{pred.intent}&#39;, retrying...&quot;)<br>                    <br>            except Exception as e:<br>                print(f&quot;  Attempt {attempt + 1}: Error - {e}&quot;)<br>        <br>        # Failed after all retries<br>        return dspy.Prediction(<br>            intent=&quot;general&quot;,  # Fallback<br>            attempts=self.max_retries,<br>            success=False<br>        )<br><br>print(&quot;\nTesting manual validation with retries...&quot;)<br>classifier = IntentClassifierWithValidation(max_retries=3)<br><br>test_messages = [<br>    &quot;I was charged twice&quot;,<br>    &quot;The app is not working&quot;,<br>    &quot;I need help with my account&quot;,<br>]<br><br>for msg in test_messages:<br>    print(f&quot;\nMessage: {msg}&quot;)<br>    result = classifier(message=msg)<br>    print(f&quot;Intent: {result.intent} (took {result.attempts} attempt(s))&quot;)<br><br># ===================================================================<br># METHOD 4: Production RAG with Comprehensive Validation<br># ===================================================================<br><br>print(&quot;\n&quot; + &quot;=&quot;*60)<br>print(&quot;METHOD 4: Production RAG System&quot;)<br>print(&quot;=&quot;*60)<br><br># Simulate a document collection<br>documents = [<br>    &quot;DSPy is a framework for programming language models.&quot;,<br>    &quot;DSPy uses automatic optimization to improve prompts.&quot;,<br>    &quot;You can build RAG systems easily with DSPy.&quot;,<br>    &quot;DSPy supports multiple language models like GPT-4 and Claude.&quot;,<br>]<br><br>class SimpleRetriever(dspy.Retrieve):<br>    &quot;&quot;&quot;Simple keyword-based retriever.&quot;&quot;&quot;<br>    def __init__(self, documents, k=3):<br>        super().__init__(k=k)<br>        self.documents = documents<br>    <br>    def forward(self, query_or_queries, k=None):<br>        queries = [query_or_queries] if isinstance(query_or_queries, str) else query_or_queries<br>        k = k or self.k<br>        <br>        all_passages = []<br>        for query in queries:<br>            query_words = set(query.lower().split())<br>            scored_docs = []<br>            <br>            for doc in self.documents:<br>                doc_words = set(doc.lower().split())<br>                score = len(query_words.intersection(doc_words))<br>                scored_docs.append((score, doc))<br>            <br>            scored_docs.sort(reverse=True, key=lambda x: x[0])<br>            passages = [doc for _, doc in scored_docs[:k]]<br>            all_passages.extend(passages)<br>        <br>        # Return Passage objects with long_text attribute<br>        class Passage:<br>            def __init__(self, text):<br>                self.long_text = text<br>        <br>        return [Passage(text) for text in all_passages]<br><br># Configure retriever<br>retriever = SimpleRetriever(documents, k=2)<br>dspy.configure(lm=lm, rm=retriever)<br><br>class RAGAnswer(dspy.Signature):<br>    &quot;&quot;&quot;Answer question using context.&quot;&quot;&quot;<br>    context: list[str] = dspy.InputField()<br>    question: str = dspy.InputField()<br>    answer: str = dspy.OutputField(desc=&quot;Answer based on context&quot;)<br><br>def rag_quality_reward(args, pred):<br>    &quot;&quot;&quot;Validate RAG answer quality.&quot;&quot;&quot;<br>    answer = pred.answer.strip()<br>    context = args.get(&#39;context&#39;, [])<br>    <br>    # Check answer length<br>    if len(answer.split()) &lt; 3:<br>        return 0.0<br>    <br>    # Check if answer references context (simple word overlap)<br>    answer_words = set(answer.lower().split())<br>    context_words = set()<br>    for ctx in context:<br>        context_words.update(ctx.lower().split())<br>    <br>    overlap = len(answer_words.intersection(context_words))<br>    <br>    # Reward based on context usage<br>    if overlap &gt;= 3:<br>        return 1.0<br>    elif overlap &gt;= 1:<br>        return 0.6<br>    else:<br>        return 0.3<br><br>class ProductionRAG(dspy.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.retrieve = dspy.Retrieve(k=2)<br>        self.generate = dspy.ChainOfThought(RAGAnswer)<br>    <br>    def forward(self, question):<br>        # Retrieve<br>        try:<br>            retrieved = self.retrieve(question)<br>            context = retrieved.passages<br>        except Exception as e:<br>            print(f&quot;Retrieval error: {e}&quot;)<br>            context = []<br>        <br>        # Generate answer<br>        pred = self.generate(context=context, question=question)<br>        <br>        return dspy.Prediction(<br>            answer=pred.answer,<br>            context=context<br>        )<br><br># Wrap with Refine for automatic quality improvement<br>rag_system = ProductionRAG()<br>production_rag = dspy.Refine(<br>    module=rag_system,<br>    N=2,<br>    reward_fn=rag_quality_reward,<br>    threshold=0.8<br>)<br><br>print(&quot;\nTesting Production RAG...&quot;)<br>questions = [<br>    &quot;What is DSPy?&quot;,<br>    &quot;How does DSPy optimize prompts?&quot;,<br>]<br><br>for q in questions:<br>    print(f&quot;\nQuestion: {q}&quot;)<br>    result = production_rag(question=q)<br>    print(f&quot;Answer: {result.answer}&quot;)<br>    quality = rag_quality_reward({&#39;context&#39;: result.context}, result)<br>    print(f&quot;Quality: {quality:.1f}&quot;)<br><br></pre><p><strong><em>Key Takeaway</em></strong><em>: Production systems need validation, error handling, and retry logic. DSPy provides tools to build these robustly.</em></p><h4>Main Concept</h4><p>Make your DSPy programs <strong>robust and reliable</strong> for real-world use with automatic validation, retry logic, and quality control.</p><h4>Four Validation Approaches</h4><h4>1. dspy.Refine ⭐ (Recommended)</h4><ul><li><strong>What</strong>: Tries multiple times to improve the output</li><li><strong>How</strong>: Uses a reward function to measure quality, keeps refining until good enough</li><li><strong>When</strong>: Quality matters most (medical advice, legal docs, critical answers)</li><li><strong>Example</strong>: If answer is too short → retry with “make it longer” feedback</li></ul><h4>2. dspy.BestOfN</h4><ul><li><strong>What</strong>: Generates N different outputs, picks the best one</li><li><strong>How</strong>: Creates multiple candidates, scores them, returns highest score</li><li><strong>When</strong>: You want variety and selection (creative writing, multiple valid answers)</li><li><strong>Example</strong>: Generate 5 summaries, pick the most concise one</li></ul><h4>3. Manual Validation</h4><ul><li><strong>What</strong>: Write your own retry and error handling code</li><li><strong>How</strong>: Custom if/else logic, explicit checks, manual retries</li><li><strong>When</strong>: Complex business rules (must match database format, specific regulations)</li><li><strong>Example</strong>: Check if output matches required JSON schema exactly</li></ul><h4>4. Hybrid Approach</h4><ul><li><strong>What</strong>: Combine methods (Refine + manual checks)</li><li><strong>How</strong>: Use Refine for quality, add custom validators for business rules</li><li><strong>When</strong>: Production systems with multiple requirements</li><li><strong>Example</strong>: Refine for good answers + manual check for banned words</li></ul><h3>Level 7: Advanced Architectures</h3><p><strong>Concepts</strong>: ReAct Agents, Multi-Agent Systems, Tool Use, Program Synthesis</p><p>Build sophisticated AI agents that can use tools, make decisions, and coordinate.</p><h4>What’s New in Level 7?</h4><ul><li><strong>ReAct pattern</strong>: Reasoning + Acting agents</li><li><strong>Tool integration</strong>: Giving agents access to external tools</li><li><strong>Multi-turn interactions</strong>: Conversational agents</li><li><strong>Program of Thought</strong>: Mathematical reasoning</li></ul><h4>What’s Improved from Level 6?</h4><p>Move from simple input-output to agents that can reason, plan, use tools, and solve complex problems iteratively.</p><h4>Example: ReAct Agent with Tools</h4><pre>import dspy<br><br>lm = dspy.LM(&#39;openai/gpt-4o-mini&#39;)<br>dspy.configure(lm=lm)<br><br># Define tools<br>class Calculator:<br>    def calculate(self, expression: str) -&gt; str:<br>        &quot;&quot;&quot;Safely evaluate mathematical expressions.&quot;&quot;&quot;<br>        try:<br>            result = eval(expression, {&quot;__builtins__&quot;: {}}, {})<br>            return str(result)<br>        except:<br>            return &quot;Error: Invalid expression&quot;<br><br>class WebSearch:<br>    def search(self, query: str) -&gt; str:<br>        &quot;&quot;&quot;Simulate web search.&quot;&quot;&quot;<br>        # In production, use real search API<br>        mock_results = {<br>            &quot;population paris&quot;: &quot;Paris has approximately 2.2 million inhabitants&quot;,<br>            &quot;eiffel tower height&quot;: &quot;The Eiffel Tower is 330 meters tall&quot;,<br>        }<br>        return mock_results.get(query.lower(), &quot;No results found&quot;)<br><br># ReAct agent signature<br>class ReActStep(dspy.Signature):<br>    &quot;&quot;&quot;Decide next action based on question and history.&quot;&quot;&quot;<br>    question: str = dspy.InputField()<br>    history: str = dspy.InputField()<br>    thought: str = dspy.OutputField(desc=&quot;Reasoning about what to do next&quot;)<br>    action: str = dspy.OutputField(desc=&quot;Action to take: search, calculate, or answer&quot;)<br>    action_input: str = dspy.OutputField(desc=&quot;Input for the action&quot;)<br><br>class FinalAnswer(dspy.Signature):<br>    &quot;&quot;&quot;Generate final answer from collected information.&quot;&quot;&quot;<br>    question: str = dspy.InputField()<br>    history: str = dspy.InputField()<br>    answer: str = dspy.OutputField()<br><br>class ReActAgent(dspy.Module):<br>    def __init__(self, max_steps=5):<br>        super().__init__()<br>        self.max_steps = max_steps<br>        self.react_step = dspy.ChainOfThought(ReActStep)<br>        self.final_answer = dspy.ChainOfThought(FinalAnswer)<br>        <br>        # Tools<br>        self.calculator = Calculator()<br>        self.search = WebSearch()<br>    <br>    def forward(self, question):<br>        history = []<br>        <br>        for step in range(self.max_steps):<br>            # Decide next action<br>            history_str = &quot;\n&quot;.join(history)<br>            step_result = self.react_step(<br>                question=question,<br>                history=history_str<br>            )<br>            <br>            # Log thought<br>            history.append(f&quot;Thought: {step_result.thought}&quot;)<br>            <br>            # Execute action<br>            action = step_result.action.lower()<br>            action_input = step_result.action_input<br>            <br>            if &quot;answer&quot; in action:<br>                # Ready to answer<br>                break<br>            elif &quot;calculate&quot; in action:<br>                result = self.calculator.calculate(action_input)<br>                history.append(f&quot;Action: Calculate({action_input})&quot;)<br>                history.append(f&quot;Observation: {result}&quot;)<br>            elif &quot;search&quot; in action:<br>                result = self.search.search(action_input)<br>                history.append(f&quot;Action: Search({action_input})&quot;)<br>                history.append(f&quot;Observation: {result}&quot;)<br>            else:<br>                history.append(f&quot;Action: Unknown action &#39;{action}&#39;&quot;)<br>        <br>        # Generate final answer<br>        final = self.final_answer(<br>            question=question,<br>            history=&quot;\n&quot;.join(history)<br>        )<br>        <br>        return dspy.Prediction(<br>            answer=final.answer,<br>            history=history,<br>            steps=step + 1<br>        )<br><br># Use the agent<br>agent = ReActAgent()<br>result = agent(<br>    question=&quot;What is 10% of the population of Paris?&quot;<br>)<br><br>print(f&quot;Answer: {result.answer}\n&quot;)<br>print(&quot;Agent&#39;s reasoning process:&quot;)<br>for i, step in enumerate(result.history, 1):<br>    print(f&quot;{i}. {step}&quot;)<br>print(f&quot;\nCompleted in {result.steps} steps&quot;)</pre><p><strong><em>Key Takeaway</em></strong><em>: DSPy enables building sophisticated agents that reason, use tools, and solve multi-step problems while remaining composable and optimizable.</em></p><h3>Summary: Your DSPy Journey</h3><p>Here’s what you’ve learned at each level:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JyDG3muYl5n_uCKwTPcQIg.png" /><figcaption>Key concepts in Each level</figcaption></figure><h3>Best Practices</h3><ol><li><strong>Start simple</strong>: Begin with basic signatures and gradually add complexity</li><li><strong>Use meaningful examples</strong>: Quality over quantity in training data</li><li><strong>Define clear metrics</strong>: Good metrics lead to better optimization</li><li><strong>Iterate incrementally</strong>: Optimize one component at a time</li><li><strong>Monitor in production</strong>: Track performance and failures</li><li><strong>Version your programs</strong>: Save compiled programs for reproducibility</li></ol><h3>Next Steps</h3><ol><li><strong>Install DSPy</strong>: pip install dspy-ai</li><li><strong>Try the examples</strong>: Start with Level 1 and progress upward</li><li><strong>Read the docs</strong>: Visit <a href="https://dspy.ai/">dspy.ai</a> for comprehensive documentation</li><li><strong>Join the community</strong>: GitHub discussions and Discord for support</li><li><strong>Build a project</strong>: Apply DSPy to your own use case</li></ol><h3>Conclusion</h3><p>DSPy represents a paradigm shift in building with LLMs. Instead of manually crafting prompts, you write programs that describe what you want, provide examples of good behavior, and let DSPy optimize the rest.</p><p>This approach is more maintainable, more portable across models, and ultimately more powerful than traditional prompting. As you progress through these levels, you’ll find that complex LLM applications become not just possible, but elegant and systematic.</p><p>The future of LLM development is programmatic, and DSPy is leading the way. Happy building!</p><p><strong><em>Let’s Learn Together</em></strong></p><p><em>This guide represents my understanding of DSPy, but I’m constantly learning too. If you notice any mistakes, have questions, or want to share better approaches, please drop a comment. Your insights help improve this resource for the entire community. We’re all in this learning journey together! 💡</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9caeb9882c82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Target Sampling: A Guide to Efficient Data Collection in Research]]></title>
            <link>https://medium.com/@mahasris0304/mastering-target-sampling-a-guide-to-efficient-data-collection-in-research-45cd48f1dcda?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/45cd48f1dcda</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[target-sampling]]></category>
            <category><![CDATA[randam-sampling]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Tue, 07 Oct 2025 05:11:49 GMT</pubDate>
            <atom:updated>2025-10-07T05:11:49.884Z</atom:updated>
            <content:encoded><![CDATA[<p>“Streamlining Data Collection Through Focused Sampling”</p><p>Hey researchers and data lovers! Sampling is key to research — you can’t study everyone, so you choose a representative group. Today, we’re focusing on target sampling, a precise, non-random method perfect for specific insights. We’ll cover its definition, types of sampling, variations of target sampling, info graphic ideas, and real-world uses. Let’s dive in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*6X5vu7h5Z1yRkwQtVJN2aw.png" /></figure><h3>Why Sampling Matters: A Quick Refresher</h3><p>Before we niche down, let’s set the stage. Sampling is the process of selecting a subset of individuals, items, or data points from a larger population to make inferences about the whole. The goal? To gather data that’s accurate, unbiased, and feasible within time and budget constraints.</p><p>Sampling methods fall into two main camps: <strong>probability sampling</strong> (where every member of the population has a known chance of being selected, leading to statistically robust results) and <strong>non-probability sampling</strong> (more flexible but potentially biased, often used in exploratory or qualitative research). Choosing the right one depends on your objectives — do you need generalizability or deep dives into specific groups?</p><h3>Types of Sampling:</h3><p>To appreciate target sampling, it’s helpful to see where it fits. Here’s a rundown of common types:</p><p><strong>Probability Sampling</strong>:</p><ul><li><strong>Simple Random Sampling</strong>: Everyone has an equal shot, like drawing names from a hat. Great for unbiased estimates but can be resource-intensive.</li><li><strong>Stratified Sampling</strong>: Divide the population into subgroups (strata) based on key characteristics (e.g., age or gender), then randomly sample from each. Ensures representation of diverse segments.</li><li><strong>Cluster Sampling</strong>: Group the population into clusters (e.g., neighborhoods), randomly select clusters, and sample everyone within them. Ideal for large, spread-out populations.</li><li><strong>Systematic Sampling</strong>: Pick every nth person from a list. Simple and efficient, but watch out for patterns that could introduce bias.</li></ul><p><strong>Non-Probability Sampling</strong>:</p><ul><li><strong>Convenience Sampling</strong>: Grab whoever’s easiest to reach, like surveying people at a mall. Quick and cheap, but not very representative.</li><li><strong>Quota Sampling</strong>: Set quotas for subgroups (e.g., 50 men, 50 women) and fill them non-randomly. Balances representation without full randomization.</li><li><strong>Snowball Sampling</strong>: Start with a few participants and ask them to refer others. Useful for hidden or hard-to-reach groups, like rare disease patients.</li><li><strong>Purposive (Judgmental) Sampling</strong>: The researcher handpicks participants based on expertise or relevance. Subjective, but targeted.</li></ul><p>Target sampling often overlaps with non-probability methods like purposive or quota sampling, emphasizing intentional selection over chance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kVO2lTcn5beLYb57QvA5PA.png" /></figure><h3>What is Target Sampling?</h3><p>Target sampling — sometimes called targeted or criterion sampling — is a non-probability technique where you deliberately select participants who meet predefined criteria relevant to your research goals. Think of it as “fishing with a spear” instead of a net: you’re not casting wide; you’re aiming for specific “targets” that align with your study’s focus.</p><p>The key here is relevance. Rather than hoping randomness yields useful data, you define your targets upfront (e.g., “experienced software developers aged 25–40”) and seek them out. This method shines in scenarios where the population is niche, time is limited, or you need expert insights. It’s efficient and cost-effective but trades off some generalizability for depth.</p><p><strong>Pros</strong>: High relevance, faster recruitment, lower costs.</p><p><strong>Cons</strong>: Potential for bias (researcher’s judgment plays a big role), limited scalability to the broader population.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ty_P7olv7GWJiw6XG5Dtog.png" /><figcaption>Random sampling Vs Target Sampling</figcaption></figure><h3>Types of Target Sampling</h3><p>Target sampling isn’t one-size-fits-all; it adapts based on your needs. Here are the main subtypes:</p><ul><li><strong>Expert Sampling</strong>: Focus on individuals with specialized knowledge or experience. For instance, interviewing top economists for a study on market trends. It’s like consulting the pros — ensures high-quality, informed data.</li><li><strong>Criterion Sampling</strong>: Select based on specific traits or conditions. If you’re researching post-pandemic mental health, you might target only those who’ve experienced job loss. This keeps your sample laser-focused on the phenomenon.</li><li><strong>Maximum Variation Sampling</strong>: Intentionally choose a diverse range of targets within your criteria to capture variations. Useful for exploring how different perspectives (e.g., urban vs. rural teachers) shape opinions on education policy.</li><li><strong>Homogeneous Sampling</strong>: The opposite — pick similar individuals to study a uniform group in depth. Say, all first-time marathon runners, to understand shared challenges without external noise.</li><li><strong>Extreme Case Sampling</strong>: Target outliers or unusual cases for insights into boundaries. For example, studying the most successful startups to uncover what sets them apart.</li></ul><p>These types can blend; often, you’ll combine them with tools like online panels, social media recruitment, or professional networks to find your targets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6ob1erhjbOcaZ1iWRylsQ.png" /><figcaption>Types of Target Sampling</figcaption></figure><h3>Target Sampling in Action</h3><ul><li><strong>Market Research</strong>: Target gym-goers for fitness app feedback via forums.</li><li><strong>Healthcare</strong>: Use criterion sampling for rare disease trials, adding snowball for small groups.</li><li><strong>Social Science</strong>: Focus on immigrants with varied backgrounds for nuanced insights.</li><li><strong>UX Design</strong>: Test apps with expert users for quick feedback (e.g., Google, Airbnb).</li><li><strong>Policy Evaluation</strong>: Study program beneficiaries for direct impact evidence.</li></ul><p>Example: COVID-19 vaccine hesitancy studies targeted skeptics online, speeding up public health responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2uJx3YI-xnZnTQzMgw79Gg.png" /><figcaption>Examples of use cases.</figcaption></figure><h3>Conclusion</h3><p>Target sampling is your ally when precision trumps breadth — perfect for exploratory work, resource-strapped projects, or niche topics. Just remember to document your criteria transparently to counter bias concerns, and consider supplementing with probability methods for validation if possible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45cd48f1dcda" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Semantic Search and Clustering: A Complete Guide]]></title>
            <link>https://medium.com/@mahasris0304/understanding-semantic-search-and-clustering-a-complete-guide-76e51cf6313d?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/76e51cf6313d</guid>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[semantic-clustering]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 27 Sep 2025 05:44:58 GMT</pubDate>
            <atom:updated>2025-09-27T05:44:58.633Z</atom:updated>
            <content:encoded><![CDATA[<p>“Building Intelligent Search Systems That Actually Understand What You’re Looking For”</p><p>In today’s data-driven world, the ability to search and organize information based on meaning rather than just keywords has become increasingly valuable. This is where semantic search and clustering come into play, revolutionizing how we interact with and understand large datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I1mm1X7vOhE-qIjg" /></figure><h3>What is Semantic Search?</h3><p>Semantic search goes beyond traditional keyword matching to understand the intent and contextual meaning behind search queries. Instead of simply looking for exact word matches, semantic search leverages machine learning models to understand the relationships between words, concepts, and their meanings.</p><h3>How Traditional Search Falls Short</h3><p>Traditional keyword-based search systems have limitations:</p><ul><li>They miss synonyms and related terms</li><li>They can’t understand context or intent</li><li>They often return irrelevant results that happen to contain the search terms</li><li>They struggle with natural language queries</li></ul><h3>The Power of Semantic Understanding</h3><p>Semantic search addresses these limitations by:</p><ul><li>Understanding the meaning behind words and phrases</li><li>Recognizing synonyms and related concepts</li><li>Considering context and user intent</li><li>Handling natural language queries effectively</li></ul><h3>Text Embeddings: The Foundation of Semantic Search</h3><p>At the heart of semantic search lies the concept of text embeddings. These are numerical representations of text that capture semantic meaning in high-dimensional vector spaces.</p><h3>What Are Text Embeddings?</h3><p>Text embeddings convert words, phrases, or entire documents into dense numerical vectors. Words or phrases with similar meanings are positioned closer together in this vector space, while dissimilar concepts are farther apart.</p><h3>Popular Embedding Models</h3><p>Several powerful embedding models are available today:</p><ul><li><strong>OpenAI’s text-embedding-ada-002</strong>: High-quality embeddings suitable for most applications</li><li><strong>BERT and its variants</strong>: Bidirectional encoder representations from transformers</li><li><strong>Sentence-BERT</strong>: Optimized for sentence-level embeddings</li><li><strong>Universal Sentence Encoder</strong>: Google’s model for various NLP tasks</li></ul><h3>Semantic Clustering: Organizing Data by Meaning</h3><p>Semantic clustering groups similar documents or text snippets based on their semantic content rather than superficial characteristics. This approach reveals natural patterns and relationships within data that might not be apparent through traditional clustering methods.</p><h3>Types of Clustering Algorithms</h3><h4>1. K-Means Clustering</h4><p>Partitions data into k predefined clusters by minimizing within-cluster sum of squares.</p><h4>2. Hierarchical Clustering</h4><p>Creates tree-like cluster structures through agglomerative (bottom-up) or divisive (top-down) approaches.</p><h4>3. DBSCAN (Density-Based Spatial Clustering)</h4><p>Groups densely packed points together while identifying outliers in sparse regions.</p><h4>4. Gaussian Mixture Models (GMM)</h4><p>Models data as a mixture of Gaussian distributions with probabilistic cluster assignments.</p><h4>5. Spectral Clustering</h4><p>Uses eigenvalues of similarity matrices to perform dimensionality reduction before clustering.</p><h3>Visualization Techniques for High-Dimensional Data</h3><p>Since embeddings typically exist in high-dimensional spaces (often 512, 768, or 1536 dimensions), visualization becomes crucial for understanding patterns and clusters.</p><h4>t-SNE (t-Distributed Stochastic Neighbor Embedding)</h4><ul><li>Excellent for visualizing local structure and clusters</li><li>Non-linear dimensionality reduction technique</li><li>Preserves local relationships well</li><li>Can be computationally expensive for large datasets</li></ul><h4>UMAP (Uniform Manifold Approximation and Projection)</h4><ul><li>Faster than t-SNE while preserving both local and global structure</li><li>Better for large datasets</li><li>More stable across different runs</li></ul><h4>PCA (Principal Component Analysis)</h4><ul><li>Linear dimensionality reduction</li><li>Fast and interpretable</li><li>Good for initial exploratory analysis</li><li>May not capture complex non-linear relationships</li></ul><h3>Building a Semantic Search System: Implementation Guide</h3><p>Here’s a comprehensive approach to building a semantic search system with clustering capabilities:</p><h4>Step 1: Data Preparation</h4><pre>import pandas as pd<br>import numpy as np<br>from sklearn.metrics.pairwise import cosine_similarity<br><br># Load your dataset<br>df = pd.read_csv(&#39;your_dataset.csv&#39;)<br><br># Prepare text data for embedding<br># Clean and preprocess text as needed</pre><h4>Step 2: Generate Embeddings</h4><pre>import openai<br><br>def get_embedding(text, model=&quot;text-embedding-ada-002&quot;):<br>    &quot;&quot;&quot;Generate embeddings using OpenAI&#39;s API&quot;&quot;&quot;<br>    text = text.replace(&quot;\n&quot;, &quot; &quot;)<br>    return openai.embeddings.create(<br>        input=[text], <br>        model=model<br>    ).data[0].embedding<br><br># Generate embeddings for all texts<br>df[&#39;embeddings&#39;] = df[&#39;text_column&#39;].apply(get_embedding)</pre><h4>Step 3: Perform Clustering</h4><pre>from sklearn.cluster import KMeans<br>from sklearn.manifold import TSNE<br><br># Convert embeddings to numpy array<br>embeddings = np.array(df[&#39;embeddings&#39;].tolist())<br><br># Perform clustering<br>n_clusters = 5<br>kmeans = KMeans(n_clusters=n_clusters, random_state=42)<br>df[&#39;cluster&#39;] = kmeans.fit_predict(embeddings)<br><br># Reduce dimensionality for visualization<br>tsne = TSNE(n_components=2, random_state=42)<br>embeddings_2d = tsne.fit_transform(embeddings)</pre><h4>Step 4: Implement Search Functionality</h4><pre>def semantic_search(query, embeddings, df, top_k=5):<br>    &quot;&quot;&quot;Perform semantic search using cosine similarity&quot;&quot;&quot;<br>    # Get query embedding<br>    query_embedding = np.array(get_embedding(query))<br>    <br>    # Calculate similarities<br>    similarities = cosine_similarity(<br>        query_embedding.reshape(1, -1), <br>        embeddings<br>    ).flatten()<br>    <br>    # Get top results<br>    top_indices = np.argsort(similarities)[::-1][:top_k]<br>    <br>    return df.iloc[top_indices]</pre><h4>Step 5: Create Interactive Visualization</h4><pre>import plotly.express as px<br>import plotly.graph_objects as go<br><br>def create_interactive_plot(embeddings_2d, df, query_results=None):<br>    &quot;&quot;&quot;Create interactive scatter plot with search results&quot;&quot;&quot;<br>    fig = px.scatter(<br>        x=embeddings_2d[:, 0],<br>        y=embeddings_2d[:, 1],<br>        color=df[&#39;cluster&#39;],<br>        hover_data=[&#39;text_column&#39;],<br>        title=&#39;Semantic Clusters Visualization&#39;<br>    )<br>    <br>    # Highlight search results if provided<br>    if query_results is not None:<br>        result_indices = query_results.index<br>        fig.add_trace(go.Scatter(<br>            x=embeddings_2d[result_indices, 0],<br>            y=embeddings_2d[result_indices, 1],<br>            mode=&#39;markers&#39;,<br>            marker=dict(color=&#39;red&#39;, size=12, symbol=&#39;diamond&#39;),<br>            name=&#39;Search Results&#39;<br>        ))<br>    <br>    return fig</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xLgYzvx9LaijE9le" /><figcaption>Eg image for semantic search and clustering with search result highlighted.</figcaption></figure><h3>Applications and Use Cases</h3><p>1. Document Management Systems</p><p>2. Customer Support</p><p>3. Content Recommendation</p><p>4. Research and Academic Applications</p><p>5. E-commerce and Product Search</p><h3>Challenges and Limitations</h3><h4>1. Computational Costs</h4><ul><li>Embedding generation can be expensive with large datasets</li><li>High-dimensional similarity calculations can be slow</li></ul><h4>2. Quality of Embeddings</h4><ul><li>Domain-specific terminology might not be well-represented</li><li>Model biases can affect results</li></ul><h4>3. Interpretability</h4><ul><li>High-dimensional embeddings are difficult to interpret</li><li>Cluster meanings might not always be clear</li></ul><h4>4. Dynamic Content</h4><ul><li>Embeddings need to be regenerated for new content</li><li>Cluster boundaries might shift with new data.</li></ul><h3>Conclusion</h3><p>Semantic search and clustering represent a major leap forward from traditional keyword-based approaches, enabling more intuitive and effective information discovery through meaning and context understanding.</p><p>The combination of embedding models, clustering algorithms, and visualization tools creates intelligent systems that truly comprehend content. As these technologies mature, we can expect applications that better bridge human understanding and machine processing.</p><p>Whether building document systems, improving customer support, or conducting research, semantic technologies provide the foundation for smarter, user-friendly applications. Success requires understanding your use case, selecting appropriate tools, and iterating based on feedback and performance metrics.</p><p>By embracing semantic technologies, organizations unlock new data insights and deliver superior user experiences, moving beyond keyword matching to genuine content and intent understanding.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76e51cf6313d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everything You Need to Know About Data Ingestion]]></title>
            <link>https://medium.com/@mahasris0304/everything-you-need-to-know-about-data-ingestion-ddb4df8dc5e3?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddb4df8dc5e3</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[database-pipeline]]></category>
            <category><![CDATA[data-ingestion]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 13 Sep 2025 10:12:28 GMT</pubDate>
            <atom:updated>2025-09-13T10:12:28.128Z</atom:updated>
            <content:encoded><![CDATA[<p>“The Role of Data Ingestion in a Modern Data Pipeline”</p><p>Data is everywhere — clicks on a website, transactions in a store, temperature readings from IoT sensors, or logs from applications. But before organizations can unlock insights from this data, it must be collected and delivered to the right systems. This process is known as <strong>data ingestion</strong>. In this blog, we’ll explore what data ingestion is, how it differs from simply saving cleaned data, and why it’s a core responsibility for data engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RjGW2o8NnnoivBnLJMDtBg.png" /></figure><h3>What is Data Ingestion?</h3><p><strong>Data ingestion</strong> is the process of moving data from multiple sources into a storage or processing system such as a data warehouse, data lake, or database. It is the <strong>first step in a data pipeline</strong>, ensuring raw data is available for downstream cleaning, transformation, and analysis.</p><p><strong>Data Sources</strong> may include:</p><ul><li>Relational databases (PostgreSQL, MySQL)</li><li>NoSQL databases (MongoDB, Cassandra)</li><li>APIs (REST, GraphQL)</li><li>Files (CSV, JSON, Excel, Parquet)</li><li>Logs, events, or clickstreams</li><li>IoT devices and sensors</li></ul><p><strong>Destinations</strong> usually include:</p><ul><li><strong>Data Lakes</strong> for raw, unstructured storage (e.g., AWS S3, Azure Data Lake)</li><li><strong>Data Warehouses</strong> for structured, analytics-ready data (e.g., Snowflake, BigQuery)</li><li><strong>Operational Databases</strong> for smaller-scale projects (e.g., PostgreSQL).</li></ul><h3>Types of Data Ingestion</h3><ol><li><strong>Batch Ingestion</strong></li></ol><ul><li>Data is collected and loaded at scheduled intervals (e.g., hourly, daily).</li><li>Example: Loading yesterday’s sales transactions into a warehouse.</li><li>Efficient for large historical data.</li><li>Not real-time.</li></ul><p><strong>2. Real-Time (Streaming) Ingestion</strong></p><ul><li>Data is ingested continuously as it is generated.</li><li>Example: Streaming transactions for fraud detection or sensor data for monitoring.</li><li>Low latency, ideal for real-time insights.</li><li>Requires complex infrastructure.</li></ul><p><strong>3. Hybrid Approach</strong></p><ul><li>A combination of batch and streaming.</li><li>Example: Stream online orders in real time, but ingest bulk POS data nightly.</li><li>Balances cost, complexity, and data freshness.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ePmDQXJrvIhHF1a4" /><figcaption>Types of Data ingestion</figcaption></figure><h3>Why Data Ingestion Matters</h3><p>Without ingestion, organizations cannot:</p><ul><li><strong>Centralize data</strong> from different sources.</li><li><strong>Enable analytics</strong> for decision-making.</li><li><strong>Train ML models</strong> with fresh, reliable data.</li><li><strong>Monitor operations</strong> in real time.</li></ul><p>Ingestion is the <strong>foundation of a data pipeline</strong>, ensuring all downstream tasks (cleaning, transformation, analysis, reporting) can even begin.</p><h3>How Is Data Ingestion Different From Just Saving Clean Data?</h3><p>Many beginners confuse <strong>data ingestion</strong> with <strong>saving cleaned data after EDA (Exploratory Data Analysis)</strong> into a database. While related, they are <strong>not the same</strong>:</p><p><strong>🔹Data Ingestion</strong></p><ul><li>Purpose: Move raw/semi-structured data into storage for future processing</li><li>Stage in Pipeline: Happens at the very beginning</li><li>Data Type: Raw, unprocessed, possibly messy</li><li>Tools: Kafka, NiFi, Airbyte, AWS Glue, SQL COPY, Pandas .to_sql()</li><li>Engineering Focus: Scalability, auditability, traceability</li></ul><p><strong>🔹 Saving Cleaned Data (EDA)</strong></p><ul><li>Purpose: Store already-cleaned, analysis-ready data</li><li>Stage in Pipeline: Happens after data wrangling/EDA</li><li>Data Type: Structured, filtered, enriched</li><li>Tools: Pandas, SQL, ETL scripts</li><li>Engineering Focus: Usability for analytics or ML.</li></ul><h3>Challenges in Data Ingestion</h3><p>Building ingestion pipelines is not always straightforward. Data engineers often face:</p><ol><li><strong>Scalability Issues</strong> → Handling growing volumes of data.</li><li><strong>Data Quality Problems</strong> → Missing values, duplicates, schema drift.</li><li><strong>Latency</strong> → Delivering data fast enough for real-time use cases.</li><li><strong>Schema &amp; Format Variety</strong> → JSON, CSV, XML, binary formats.</li><li><strong>Error Handling &amp; Reliability</strong> → Ensuring retries, avoiding data loss.</li><li><strong>Security &amp; Compliance</strong> → Protecting sensitive data (e.g., GDPR, HIPAA).</li></ol><h3>Tools &amp; Frameworks for Data Ingestion</h3><p>Modern ingestion relies on specialized tools. Some popular options:</p><ul><li><strong>Streaming Platforms</strong>: Apache Kafka, Apache Pulsar, AWS Kinesis</li><li><strong>ETL/ELT Tools</strong>: Apache NiFi, Airbyte, Talend, Fivetran, AWS Glue</li><li><strong>Workflow Orchestration</strong>: Apache Airflow, Prefect</li><li><strong>Cloud Services</strong>: Google Dataflow, Azure Data Factory</li></ul><p>Choice of tool depends on the use case: batch, real-time, or hybrid.</p><h3>Best Practices for Data Ingestion</h3><ol><li><strong>Ingest Raw Data First</strong> → Always keep an untouched copy for auditing.</li><li><strong>Use Metadata &amp; Lineage</strong> → Track where data came from and how it changes.</li><li><strong>Automate Pipelines</strong> → Use orchestration tools for scheduling and monitoring.</li><li><strong>Ensure Fault Tolerance</strong> → Handle retries, failures, and partial loads.</li><li><strong>Apply Data Quality Checks</strong> → Validate schema, handle nulls, remove duplicates.</li><li><strong>Secure Data Flows</strong> → Encrypt data in motion and at rest.</li><li><strong>Design for Scalability</strong> → Pipelines should grow with business needs.</li></ol><h3>Real-World Example: Ingesting CSV Files into PostgreSQL</h3><p>Imagine you have a CSV file of e-commerce transactions:</p><ul><li><strong>Step 1:</strong> Ingest the CSV into PostgreSQL using COPY or Pandas .to_sql().</li><li><strong>Step 2:</strong> Store it as raw (Bronze layer).</li><li><strong>Step 3:</strong> Later, apply cleaning and transformations (Silver/Gold layers).</li></ul><p>Here, ingestion is simply <strong>getting the file into the database</strong> — further cleaning comes afterward.</p><h3>✅ Conclusion</h3><p>Data ingestion is much more than just moving data — it’s the backbone of modern data systems. By collecting raw data from diverse sources and delivering it reliably into storage or processing systems, ingestion ensures that organizations can analyze, transform, and use data effectively.</p><p>For data engineers, mastering ingestion means building pipelines that are <strong>scalable, fault-tolerant, secure, and auditable</strong>. Whether using batch, real-time, or hybrid methods, data ingestion is the first and most critical step in turning raw data into business insights.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddb4df8dc5e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Secret Language of AI: Understanding Embeddings and Their Models]]></title>
            <link>https://medium.com/@mahasris0304/the-secret-language-of-ai-understanding-embeddings-and-their-models-6a195978c1d5?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a195978c1d5</guid>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[embedding-model]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 30 Aug 2025 11:18:22 GMT</pubDate>
            <atom:updated>2025-08-30T11:18:22.394Z</atom:updated>
            <content:encoded><![CDATA[<h4>Unlocking Data’s Meaning for Intelligent AI Systems</h4><p>Have you ever wondered how search engines find exactly what you’re looking for, even when you use slightly different words? Or how recommendation systems magically suggest products you’ll love? The answer, in large part, lies in a powerful concept called <strong>embeddings</strong>.</p><p>Embeddings are the unsung heroes behind much of today’s intelligent AI applications. They bridge the gap between human language and computer understanding, allowing machines to grasp the <em>meaning</em> and <em>relationships</em> within data, rather than just processing symbols.</p><p>Let’s dive in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uxv8vkNs9qtWAK0XqdTFLQ.png" /><figcaption>Image created with Nano Banana (Gemini)</figcaption></figure><h3>What is an Embedding?</h3><p>Imagine you have a map, and on this map, every city is represented by a set of coordinates (latitude and longitude). Cities that are geographically close to each other will have similar coordinates.</p><p>An <strong>embedding</strong> is essentially the same concept, but for data like words, sentences, images, or even entire documents. It’s a numerical representation — a <strong>vector</strong> (a list of numbers) — that captures the semantic meaning or inherent characteristics of that data.</p><ul><li><strong>For text:</strong> “King” might be a vector like [0.2, -0.5, 0.8, …], while “Queen” could be [0.1, -0.4, 0.7, …]. Notice how similar they are? The numbers themselves don’t mean anything in isolation, but their relationship to other vectors is key.</li><li><strong>Key Idea:</strong> The magic of embeddings is that items with similar meanings or characteristics are placed <strong>closer together</strong> in this multi-dimensional numerical space. Conversely, items with very different meanings will be far apart.</li></ul><p>Think of it as giving computers a “sense” of meaning and context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ep1ZAmTpsVfcQ__xkrpvIA.png" /><figcaption>Numerical representation — a <strong>vector</strong> (a list of numbers) ,(Image created with Nano Banana (Gemini))</figcaption></figure><h3>What is an Embedding Model?</h3><p>If an embedding is the numerical representation, then an <strong>embedding model</strong> is the sophisticated engine that <em>generates</em> these representations.</p><p>An embedding model is typically a type of <strong>neural network</strong> that has been trained on a massive dataset. Its goal during training is to learn how to map complex inputs (like raw text, pixels from an image, or audio waveforms) into concise, meaningful vectors in a high-dimensional space.</p><p><strong>How it generally works:</strong></p><ul><li><strong>Input:</strong> You feed the model your data (e.g., the word “apple”).</li><li><strong>Transformation:</strong> The model’s layers process this input, learning patterns and relationships from its training.</li><li><strong>Output:</strong> It spits out a fixed-size vector (e.g., 768 or 1536 numbers) that encapsulates the semantic information of “apple.”</li></ul><p>The training process is crucial. The model learns to arrange these vectors in such a way that semantically related concepts are near each other. For example, a good embedding model will ensure that the embedding for “dog” is closer to “cat” than it is to “airplane.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNlvPGf84QgTiHklgBstVw.png" /><figcaption>Visual representation of embeddings: words, images, and audio mapped into a shared vector space where similar concepts are closer together.</figcaption></figure><h3>Why Are Embeddings Useful? (Key Use Cases)</h3><ol><li><strong>Semantic Search</strong> — Instead of keyword matching, embeddings let us find documents, FAQs, or code snippets that <em>mean</em> the same thing.</li><li><strong>Recommendation Systems</strong> — Suggest products, movies, or songs by finding similar embeddings.</li><li><strong>Clustering &amp; Categorization</strong> — Group documents, users, or customers into meaningful clusters.</li><li><strong>Anomaly Detection</strong> — Spot outliers in transactions, logs, or user behavior.</li><li><strong>Multimodal Applications</strong> — Connect different data types (e.g., search images using text).</li><li><strong>Retrieval-Augmented Generation (RAG)</strong> — Use embeddings to fetch context for LLMs like GPT, making them more accurate and up-to-date.</li></ol><h3>Types of Embedding Models</h3><p>Depending on the data type, embedding models come in different flavors:</p><h4>1. Text Embeddings</h4><ul><li>Represent words, sentences, or documents as vectors.</li><li>Examples: Word2Vec, GloVe, FastText, BERT, Sentence Transformers.</li><li>📌 Use case: search engines, chatbots, document clustering.</li></ul><h4>2. Image Embeddings</h4><ul><li>Convert images into vectors where visually similar images are close.</li><li>Examples: ResNet, CLIP, DINO.</li><li>📌 Use case: reverse image search, content moderation, duplicate detection.</li></ul><h4>3. Audio/Speech Embeddings</h4><ul><li>Represent audio or speech clips as vectors.</li><li>Examples: wav2vec2, YAMNet, SpeakerNet.</li><li>📌 Use case: speech recognition, speaker verification, music classification.</li></ul><h4>4. Graph Embeddings</h4><ul><li>Represent nodes, edges, or entire graphs in vector space.</li><li>Examples: Node2Vec, DeepWalk, GraphSAGE.</li><li>📌 Use case: fraud detection, social network analysis, knowledge graphs.</li></ul><h4>5. Multimodal Embeddings</h4><ul><li>Combine multiple data types (e.g., text + image).</li><li>Examples: CLIP (OpenAI), ALIGN (Google), Florence (Microsoft).</li><li>📌 Use case: text-to-image search, cross-modal AI applications.</li></ul><h3>Embedding Workflow</h3><p>Here’s how embeddings are used in practice, step by step:</p><p><strong>Step 1: Identify Your Data Type</strong></p><p>Start by defining the kind of data you want to represent:</p><ul><li><strong>Text</strong> — documents, queries, code</li><li><strong>Images</strong> — photos, medical scans, memes</li><li><strong>Audio/Speech</strong> — voice recordings, music, sound events</li><li><strong>Graphs</strong> — networks of users, entities, or transactions</li><li><strong>Multimodal</strong> — combinations, like text + image</li></ul><h4>Step 2: Choose the Right Type of Embedding Model</h4><p>Embedding models are designed for different modalities:</p><ul><li><strong>Text Embeddings</strong> → words, sentences, documents</li><li><strong>Image Embeddings</strong> → photos, designs, visual content</li><li><strong>Audio/Speech Embeddings</strong> → speech, music, sound</li><li><strong>Graph Embeddings</strong> → nodes, edges, relationships</li><li><strong>Multimodal Embeddings</strong> → combination of text + image + audio.</li></ul><h4>Step 3: Generate Embeddings</h4><ul><li>Feed your raw data into the embedding model.</li><li>Get a fixed-size <strong>vector</strong> output (e.g., [0.21, -0.78, 0.56, …]).</li><li>Similar data = nearby vectors.</li></ul><p>It’s like plotting every piece of data on a massive 3D map where meaning decides the position.</p><h4>Step 4: Store &amp; Manage Embeddings</h4><p>For real-world systems, embeddings are stored in <strong>vector databases</strong> for fast retrieval:</p><ul><li>PostgreSQL</li><li>Pinecone</li><li>Weaviate</li><li>FAISS (Facebook AI)</li><li>Milvus</li></ul><p>These allow <strong>nearest-neighbor search</strong>, which means “find the most similar vectors to this one.”</p><h4>Step 5: Apply Embeddings in Applications</h4><p>Once embeddings are generated and stored, you can power countless applications:</p><ul><li>Semantic search engines</li><li>Personalized product or content recommendations</li><li>Customer segmentation</li><li>Fraud and anomaly detection</li><li>Cross-modal search (e.g., text-to-image retrieval)</li><li>Retrieval-Augmented Generation (RAG) for LLMs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*emMEFLQRlTKd7ib8fvxzhw.png" /><figcaption>Workflow architecture of Embedding and Embedding Model</figcaption></figure><h3>Open-Source Embedding Models</h3><ol><li><strong>Word2Vec</strong> — classic word embeddings for text (Google).</li><li><strong>GloVe</strong> — statistical word embeddings (Stanford).</li><li><strong>FastText</strong> — subword-aware embeddings (Meta).</li><li><strong>Sentence Transformers (SBERT)</strong> — high-quality sentence/document embeddings.</li><li><strong>CLIP</strong> — joint text–image embeddings (OpenAI/LAION).</li></ol><h3>Conclusion</h3><p>Embeddings are the <strong>bridge between raw data and machine understanding.</strong> They power modern applications like semantic search, recommendations, and knowledge retrieval for LLMs.</p><p>With the growing ecosystem of <strong>open-source embedding models</strong>, developers can easily integrate them into real-world systems — whether for text, images, audio, graphs, or multimodal data.</p><p>As AI continues to evolve, embeddings will remain at the <strong>core of intelligent information retrieval and understanding.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a195978c1d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Websites Know It’s You: The Role of IP, Cookies, and Sessions]]></title>
            <link>https://medium.com/@mahasris0304/how-websites-know-its-you-the-role-of-ip-cookies-and-sessions-1c64017fd3bd?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c64017fd3bd</guid>
            <category><![CDATA[sessions]]></category>
            <category><![CDATA[passwords]]></category>
            <category><![CDATA[cookies]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ip-address]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 23 Aug 2025 11:15:50 GMT</pubDate>
            <atom:updated>2025-08-23T11:15:50.907Z</atom:updated>
            <content:encoded><![CDATA[<p>“Learn how IP addresses, cookies, and sessions work together to protect your digital life.”</p><h3>Introduction</h3><p>Have you ever wondered how a website instantly recognizes you — or why sometimes it asks you to log in again? The answer lies in three essential technologies that power the internet: <strong>IP addresses, cookies, and sessions.</strong></p><p>These are not just technical jargon; they’re the foundation of how websites keep you connected and secure. In this guide, we’ll break down what they are, why they matter, and how they work behind the scenes every time you go online.</p><p>If you’ve ever wondered:</p><ul><li><em>What is an IP address?</em></li><li><em>Why is an IP important?</em></li><li><em>What do cookies do?</em><br> You’re in the right place. Let’s break it down in simple terms.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t9u-3RGTEXQvaP96gb2j7Q.png" /></figure><h3>What is an IP Address?</h3><p>An <strong>IP (Internet Protocol) address</strong> is like the return address on an envelope but for your device on the internet.</p><p><strong>Definition:</strong><br> An IP address is a unique number assigned to your device when you connect to the internet.</p><p><strong>Why is IP important?</strong></p><ul><li>It tells the internet where to send the data you request.</li><li>It helps websites detect your general location (city, country, network).</li></ul><blockquote>It’s vital for security:<br> <em>If you log in from New York and five minutes later from another country, the system flags it as suspicious.</em></blockquote><h3>What is a Cookie and Why Do Websites Use Them?</h3><p>Imagine visiting the same café every day, and the barista remembers your favorite order — that’s what cookies do for websites.</p><p><strong>What is a cookie?</strong><br> A cookie is a small file stored in your browser that saves details like your login status, preferences, and browsing activity.</p><p><strong>Types of cookies:</strong></p><ul><li><strong>Session cookies:</strong> Temporary, deleted when you close the browser.</li><li><strong>Persistent cookies:</strong> Stay on your device until they expire or you delete them.</li></ul><p><strong>Why are cookies important?</strong></p><ul><li>They keep you logged in without re-entering your password on every page.</li><li>They personalize your experience by remembering settings and preferences.</li></ul><h3>What is a Session and How Does It Work?</h3><p>Sessions are like a digital wristband at a concert — proof you’re allowed backstage.</p><p><strong>What is a session?</strong></p><ul><li>When you log in, the website creates a session on the server and gives your browser a session cookie.</li></ul><p><strong>How does it work?</strong></p><ul><li>Every time you click something, your browser sends that cookie to prove you’re authenticated.</li></ul><p><strong>Why is it important?</strong></p><ul><li>Without sessions, you’d have to log in again for every single page — very annoying!</li></ul><h3>The Secret Handshake — Browser-Side Security</h3><p>Security isn’t just about servers; it starts right in your browser.</p><p><strong>What is it?</strong></p><ul><li>Modern websites use JavaScript to scramble (hash or encrypt) your password before sending it across the internet.</li></ul><p><strong>How does it work?</strong></p><ul><li>When you click <strong>“Login,”</strong> your browser instantly converts your password into a long, unreadable string of characters.</li><li>That scrambled string is sent to the server. Even if a hacker intercepts the data, all they see is gibberish.</li></ul><p><strong>Why is it important?</strong></p><ul><li>This extra layer of protection makes it extremely difficult for attackers to steal your real password.</li></ul><h3>How Do These Improve Security?</h3><ul><li><strong>IP addresses</strong> detect unusual login locations.</li><li><strong>Sessions</strong> prevent repeated password entry and limit access after logout.</li><li><strong>Browser encryption</strong> scrambles your password before sending it to the server, making it harder for hackers to steal.</li></ul><h3>Everyday Examples</h3><ul><li><strong>Netflix login:</strong> Why it asks you to sign in again when switching devices.</li><li><strong>Public Wi-Fi alerts:</strong> Your bank warns you when you log in from an unknown network.</li><li><strong>Clearing cookies:</strong> Why you get logged out after deleting them.</li></ul><h3>Quick Tips for Users</h3><p>✔ Clear cookies on shared devices to protect your data.<br> ✔ Enable <strong>two-factor authentication</strong> for stronger security.</p><h3>How Websites Know It’s You — A Visual Breakdown of IP, Cookies &amp; More</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*52WP-DapshISbHpMWjzM5w.png" /><figcaption>A Visual Breakdown of IP, Cookies &amp; More</figcaption></figure><h3>Conclusion</h3><p>Your online identity is more than just a password — it’s a combination of <strong>IP addresses, cookies, and sessions</strong> working together to keep you safe. Next time you log in, remember the invisible security dance happening behind the scenes.</p><p><strong>Want to learn more ?Stay tuned for next article! </strong>👋</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c64017fd3bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Supercharged My Chatbot with RAG Using SERP API]]></title>
            <link>https://medium.com/@mahasris0304/how-i-supercharged-my-chatbot-with-rag-using-serp-api-1453c25315e0?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1453c25315e0</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[serpapi]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Thu, 14 Aug 2025 12:50:18 GMT</pubDate>
            <atom:updated>2025-08-14T12:50:18.327Z</atom:updated>
            <content:encoded><![CDATA[<p>“Blending live search data with AI for smarter, fresher answers”</p><p>In today’s digital landscape, real-time information is gold — especially for chatbots that aim to provide accurate, timely, and context-aware responses. While traditional AI models hold vast knowledge, they cannot account for breaking news, sudden market shifts, or newly published content.</p><p>This is where <strong>SERP APIs</strong> <strong><em>(Search Engine Results Page APIs)</em></strong> prove invaluable. When combined with <strong>Retrieval-Augmented Generation (RAG)</strong>, they enable chatbots to deliver answers that are not only intelligent but also <em>up-to-the-minute</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I32Md26-vFWNv92HnJFxrw.png" /></figure><h3>What is a SERP API?</h3><p>A <strong>SERP API</strong> <strong><em>(Search Engine Results Page APIs) </em></strong>is a service that allows programmatic access to real-time search engine data from platforms such as Google, Bing, and Yahoo. Instead of performing manual searches and copying results, a SERP API returns structured search data — organic listings, ads, featured snippets, images, videos, and map packs — in formats like JSON or HTML.</p><p>Modern SERP APIs go beyond raw scraping, managing <strong>proxy rotation, CAPTCHA solving, and data parsing</strong> to provide clean, ready-to-use outputs without the complexity of building a custom scraping infrastructure.</p><h3>Key Applications of SERP API</h3><p>Businesses, marketers, and developers use SERP APIs for:</p><ul><li><strong>SEO tracking:</strong> Monitor keyword rankings and competitor visibility.</li><li><strong>Market research:</strong> Identify trends and analyze product performance.</li><li><strong>Ad intelligence:</strong> Track ad placements and competitor strategies.</li><li><strong>Brand monitoring:</strong> Detect unauthorized use or negative coverage.</li><li><strong>Price comparison:</strong> Gather ecommerce product and pricing data.</li><li><strong>Content discovery:</strong> Spot trending topics and featured snippet opportunities.</li></ul><h3>Advantages and Challenges</h3><p><strong>Advantages:</strong></p><ul><li><strong>Automation:</strong> Eliminates manual scraping.</li><li><strong>Real-Time Data:</strong> Provides current, accurate search results.</li><li><strong>Scalability:</strong> Handles large query volumes.</li><li><strong>Customization:</strong> Filters by keyword, location, language, and device.</li><li><strong>Quality:</strong> Delivers structured, parsed data ready for analysis.</li></ul><h4><strong>Challenges:</strong></h4><ul><li><strong>Cost:</strong> Robust solutions typically require paid plans.</li><li><strong>Rate Limits:</strong> Query limits may restrict high-frequency use.</li><li><strong>Data Consistency:</strong> Search engine updates can impact API performance.</li><li><strong>Occasional Failures:</strong> Geographic restrictions or CAPTCHAs may still occur.</li></ul><h3>Using SERP API as the Retrieval Layer in RAG</h3><p>In a RAG architecture, the <strong>retrieval</strong> step is crucial — it supplies the AI model with relevant, up-to-date information before a response is generated. SERP API serves as an ideal retrieval mechanism by sourcing the latest web results.</p><blockquote><a href="https://medium.com/@mahasris0304/from-static-to-supercharged-how-i-built-a-chat-bot-that-outsmarts-its-own-memory-83cba8d00c63"><strong>SERP API Was Used as RAG in ChatBot Project</strong></a></blockquote><blockquote>For a detailed walkthrough of a real-world implementation using <strong>Python</strong>, <strong>Gemini API</strong>, and <strong>SERP API</strong> as the retrieval layer in RAG, check out the blog</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9pzinDfje5HLRydJhHSrwg.png" /></figure><ol><li><strong>Retrieval (SERP API)</strong></li></ol><ul><li>The chatbot sends a search query to the SERP API.</li><li>The API returns fresh results, including news articles, statistics, and featured snippets.</li></ul><p>2.<strong>Augmentation</strong></p><ul><li>Relevant snippets and data points are extracted from the API’s structured response.</li><li>These facts augment the chatbot’s existing knowledge.</li></ul><p><strong>3.Generation (AI Model)</strong></p><ul><li>Both the user’s query and the retrieved SERP data are passed into a generative AI model.</li><li>The model synthesizes an informed, context-rich answer.</li></ul><p><strong>4.Response</strong></p><ul><li>The chatbot outputs a polished answer that is accurate, relevant, and backed by current information.</li></ul><h3>Example in Action</h3><p><strong>User Query:</strong> <em>“What’s the latest update on electric vehicle sales?”</em></p><p><strong>RAG Workflow with SERP API:</strong></p><ul><li>SERP API searches for “latest electric vehicle sales.”</li><li>Retrieves up-to-date news, statistics, and market reports.</li><li>AI model synthesizes the retrieved facts into a concise answer.</li><li>Final response includes the most recent data, not outdated figures.</li></ul><h3>Why This Approach Works</h3><p>Pairing <strong>retrieval</strong> (live SERP data) with <strong>generation</strong> (AI synthesis) ensures that chatbot responses are:</p><ul><li><strong>Current</strong> — grounded in the latest information.</li><li><strong>Trustworthy</strong> — supported by verifiable sources.</li><li><strong>Context-aware</strong> — combining static knowledge with live updates.</li></ul><p>This approach transforms a standard AI chatbot into a <strong>real-time knowledge assistant</strong>, making it highly effective in sectors like finance, ecommerce, marketing, and news.</p><h3>Final Thoughts</h3><p>Using SERP API as the retrieval component in a RAG pipeline bridges the gap between static AI training data and the ever-changing nature of the web. The result is a chatbot that not only understands questions but also answers them with the most relevant and recent information available.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1453c25315e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Static to Supercharged: How I Built a Chat bot That Outsmarts Its Own Memory]]></title>
            <link>https://medium.com/@mahasris0304/from-static-to-supercharged-how-i-built-a-chat-bot-that-outsmarts-its-own-memory-83cba8d00c63?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/83cba8d00c63</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai-chatbot]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[serpapi]]></category>
            <category><![CDATA[gemini-api]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 09 Aug 2025 11:18:23 GMT</pubDate>
            <atom:updated>2025-08-09T11:18:23.356Z</atom:updated>
            <content:encoded><![CDATA[<p>There’s a limitation most AI chatbots quietly carry.</p><p>They can be brilliant… <em>but only up to a certain moment in time</em>.</p><p>That “moment” is the <strong>knowledge cutoff</strong> — the day the AI stopped learning from the world. After that date, it’s like asking a genius about current events when they’ve been living in a cave for a year.</p><p>A basic chatbot built with Google’s <strong>Gemini</strong> model revealed this exact problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MlCSfABbJwOVbutcf5_ikA.png" /></figure><h3>Chapter 1 — The Birth of a Chatbot</h3><p>It began with a simple setup:<br> A few lines of Python, an API key, and a lightweight interface.</p><p>Basic Gemini Chatbot (Knowledge Cutoff Limitation)</p><pre>!pip install google-generativeai ipywidgets<br><br>import google.generativeai as genai<br>import ipywidgets as widgets<br>from IPython.display import display, HTML, clear_output<br><br># --- API Key ---<br>API_KEY = &quot;your_api_key&quot;<br><br>if &quot;YOUR_NEW_API_KEY_HERE&quot; in API_KEY:<br>    print(&quot;❌ ERROR: You haven&#39;t replaced the placeholder API key yet!&quot;)<br>    model = None<br>else:<br>    print(f&quot;✅ Key loaded successfully, starting with: {API_KEY[:4]}...{API_KEY[-4:]}&quot;)<br>    try:<br>        genai.configure(api_key=API_KEY)<br>        model = genai.GenerativeModel(model_name=&#39;gemini-2.5-pro&#39;)<br>        chat = model.start_chat(history=[])<br>        print(&quot;✅ Model configured successfully!&quot;)<br>    except Exception as e:<br>        print(f&quot;❌ Configuration Error: {e}&quot;)<br>        model = None<br>        chat = None<br><br>def create_widget_chatbot():<br>    &quot;&quot;&quot;Creates and displays an interactive chatbot with ipywidgets.&quot;&quot;&quot;<br>    if not model or not chat:<br>        print(&quot;Cannot start chatbot due to a configuration error above.&quot;)<br>        return<br><br>    chat_history_output = widgets.HTML(<br>        value=&quot;&lt;div style=&#39;border: 1px solid #ccc; padding: 10px; min-height: 200px; max-height: 400px; overflow-y: auto;&#39;&gt;&lt;/div&gt;&quot;<br>    )<br>    user_input = widgets.Text(<br>        placeholder=&#39;Type your message and press Enter...&#39;,<br>        layout=widgets.Layout(width=&#39;80%&#39;)<br>    )<br><br>    def on_submit(widget):<br>        user_message = user_input.value<br>        if not user_message.strip(): return<br>        user_input.value = &quot;&quot;<br>        <br>        old_html = chat_history_output.value<br>        user_html = f&quot;&lt;div style=&#39;margin-bottom: 5px;&#39;&gt;&lt;b&gt;You:&lt;/b&gt; {user_message}&lt;/div&gt;&quot;<br>        thinking_html = &quot;&lt;div style=&#39;margin-bottom: 5px; color: grey;&#39;&gt;&lt;i&gt;Gemini is thinking...&lt;/i&gt;&lt;/div&gt;&quot;<br>        chat_history_output.value = old_html.replace(&quot;&lt;/div&gt;&quot;, &quot;&quot;) + user_html + thinking_html + &quot;&lt;/div&gt;&quot;<br><br>        try:<br>            response = chat.send_message(user_message)<br>            bot_response = response.text.replace(&#39;\n&#39;, &#39;&lt;br&gt;&#39;)<br>            bot_html = f&quot;&lt;div style=&#39;margin-bottom: 10px;&#39;&gt;&lt;b&gt;Gemini:&lt;/b&gt; {bot_response}&lt;/div&gt;&quot;<br>            chat_history_output.value = old_html.replace(&quot;&lt;/div&gt;&quot;, &quot;&quot;) + user_html + bot_html + &quot;&lt;/div&gt;&quot;<br>        except Exception as e:<br>            error_html = f&quot;&lt;div style=&#39;color: red;&#39;&gt;&lt;b&gt;Error:&lt;/b&gt; {e}&lt;/div&gt;&quot;<br>            chat_history_output.value = old_html.replace(&quot;&lt;/div&gt;&quot;, &quot;&quot;) + user_html + error_html + &quot;&lt;/div&gt;&quot;<br><br>    user_input.on_submit(on_submit)<br><br>    display(widgets.VBox([<br>        widgets.HTML(&quot;&lt;h2&gt;🤖 Gemini Chatbot&lt;/h2&gt;&quot;),<br>        chat_history_output,<br>        user_input<br>    ]))<br><br># --- RUN THE CHATBOT ---<br>create_widget_chatbot()</pre><blockquote>When asked:</blockquote><blockquote>“What’s the latest model from OpenAI?”</blockquote><blockquote>The response was: <strong>GPT-4o, released in May 2024</strong> — accurate according to its training.However, a follow-up statement:</blockquote><blockquote><strong><em>“OpenAI released GPT-5 on August 7, 2025.”</em></strong>Prompted a reply that 2025 had not yet happened, exposing the <strong>knowledge cutoff problem</strong>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlo7ikOFKNyCCAOX_hTzKg.png" /><figcaption>chatbot with knowledge cutoff problem</figcaption></figure><h3>What is Knowledge Cutoff?</h3><p>The <strong>knowledge cutoff</strong> is the point in time when a model’s training data ends.</p><p>For example, if a model’s training stopped in April 2024, it cannot know anything that happened afterward — such as product launches, world events, or scientific discoveries — unless that information is provided in the prompt.</p><p><strong>Why this happens:</strong></p><ul><li>Training large AI models involves processing massive datasets over months.</li><li>By the time the model is deployed, new events have already occurred that it hasn’t seen.</li></ul><p><strong>Why it’s a problem for chatbots:</strong></p><ul><li>Answers can be outdated for time-sensitive queries.</li><li>The model may reject correct information simply because it hasn’t seen it before.</li><li>Users may assume the AI “knows everything,” leading to misinformation risks.</li></ul><p>The only way to bridge this gap is to give the model <strong>access to current information</strong> — which leads directly to the next chapter.</p><h3>Chapter 2 — The Detective Upgrade</h3><p>To escape the time trap of the knowledge cutoff, the chatbot needed more than memory — it needed <strong>current awareness</strong>.</p><p>The solution: <strong>Retrieval-Augmented Generation</strong> (RAG).</p><p>With RAG, the chatbot could:</p><ol><li>Perform a live web search.</li><li>Pull relevant details from trusted sources.</li><li>Feed those results to the Gemini model for accurate, current answers.</li></ol><p>This upgrade was powered using <strong>SerpAPI</strong> for search integration.</p><h3>Chapter 3 — The RAG-Powered Chatbot</h3><p>Gemini + SerpAPI Real-Time Chatbot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9JMnxf5MVSXXkQ6bd2sVlQ.png" /></figure><h3>What is RAG and Why It’s Used Here</h3><p><strong>RAG</strong> stands for <strong>Retrieval-Augmented Generation</strong>.<br> It’s a technique where a chatbot doesn’t just rely on its fixed training data — it first <strong>retrieves</strong> fresh, relevant information from external sources, then <strong>generates</strong> a response using that updated context.</p><p><strong>Why it’s used here:</strong></p><ul><li>The Gemini model has a <strong>knowledge cutoff</strong> and cannot know events after its training date.</li><li>By retrieving recent search results before answering, the chatbot avoids outdated or fabricated replies.</li><li>It ensures that answers are both <strong>current</strong> and <strong>fact-backed</strong>.</li></ul><p><strong>Which tools are used here as RAG:</strong></p><ul><li><strong>Google’s Gemini model</strong> → Generates natural language answers.</li><li><strong>SerpAPI</strong> → Performs real-time Google searches and returns structured results.</li><li>The chatbot’s Python code integrates both, feeding the search results into Gemini as additional context before generating a response.</li></ul><pre>!pip install google-generativeai ipywidgets serpapi<br><br>import google.generativeai as genai<br>import ipywidgets as widgets<br>from IPython.display import display, HTML, clear_output<br>from serpapi import GoogleSearch<br>import re<br>from collections import Counter<br>from datetime import datetime<br><br># --- API Keys ---<br>API_KEY = &quot;your_api_key&quot;<br>SEARCH_API_KEY = &quot;your_search_api_key&quot;<br><br>if &quot;YOUR_GEMINI_API_KEY&quot; in API_KEY or &quot;YOUR_SERPAPI_KEY&quot; in SEARCH_API_KEY:<br>    print(&quot;❌ ERROR: Replace the placeholder API keys!&quot;)<br>    model = None<br>else:<br>    print(f&quot;✅ Gemini Key loaded successfully: {API_KEY[:4]}...{API_KEY[-4:]}&quot;)<br>    print(f&quot;✅ Search Key loaded successfully: {SEARCH_API_KEY[:4]}...{SEARCH_API_KEY[-4:]}&quot;)<br>    try:<br>        genai.configure(api_key=API_KEY)<br>        model = genai.GenerativeModel(model_name=&#39;gemini-2.5-pro&#39;)<br>        chat = model.start_chat(history=[])<br>        print(&quot;✅ Model configured successfully!&quot;)<br>    except Exception as e:<br>        print(f&quot;❌ Configuration Error: {e}&quot;)<br>        model = None<br>        chat = None<br><br>def perform_search(query):<br>    trusted_domains = [&quot;openai.com&quot;, &quot;techcrunch.com&quot;, &quot;cnbc.com&quot;, &quot;usatoday.com&quot;, &quot;sciencenewstoday.org&quot;]<br>    params = {<br>        &quot;api_key&quot;: SEARCH_API_KEY,<br>        &quot;engine&quot;: &quot;google&quot;,<br>        &quot;q&quot;: query,<br>        &quot;num&quot;: 10<br>    }<br>    try:<br>        search = GoogleSearch(params)<br>        results = search.get_dict()<br>        organic_results = results.get(&quot;organic_results&quot;, [])<br>        filtered_results = [r for r in organic_results if any(domain in r.get(&quot;link&quot;, &quot;&quot;) for domain in trusted_domains)]<br>        return filtered_results if filtered_results else organic_results[:5]<br>    except Exception as e:<br>        print(f&quot;❌ Search Error: {e}&quot;)<br>        return []<br><br>def create_widget_chatbot():<br>    if not model or not chat:<br>        print(&quot;Cannot start chatbot due to a configuration error above.&quot;)<br>        return<br><br>    chat_history_output = widgets.HTML(<br>        value=&quot;&lt;div style=&#39;border: 1px solid #ccc; padding: 10px; min-height: 200px; max-height: 400px; overflow-y: auto;&#39;&gt;&lt;/div&gt;&quot;<br>    )<br>    user_input = widgets.Text(placeholder=&#39;Ask me anything...&#39;, layout=widgets.Layout(width=&#39;80%&#39;))<br><br>    def on_submit(widget):<br>        user_message = user_input.value.strip()<br>        if not user_message: return<br>        user_input.value = &quot;&quot;<br><br>        old_html = chat_history_output.value<br>        user_html = f&quot;&lt;div&gt;&lt;b&gt;You:&lt;/b&gt; {user_message}&lt;/div&gt;&quot;<br>        thinking_html = &quot;&lt;div style=&#39;color: grey;&#39;&gt;&lt;i&gt;Gemini is searching and thinking...&lt;/i&gt;&lt;/div&gt;&quot;<br>        chat_history_output.value = old_html.replace(&quot;&lt;/div&gt;&quot;, &quot;&quot;) + user_html + thinking_html + &quot;&lt;/div&gt;&quot;<br><br>        try:<br>            search_results_list = perform_search(user_message)<br>            formatted_results = &quot;\n&quot;.join([<br>                f&quot;Source: {r.get(&#39;title&#39;)}\nURL: {r.get(&#39;link&#39;)}\nSnippet: {r.get(&#39;snippet&#39;)}\n&quot;<br>                for r in search_results_list<br>            ])<br>            augmented_prompt = f&quot;&quot;&quot;<br>            Use only the provided search results to answer the question.<br>            If the answer is not present in the search results, state that the information is not available.<br>            <br>            Search Results:<br>            {formatted_results}<br>            <br>            User Question: {user_message}<br>            &quot;&quot;&quot;<br>            response = chat.send_message(augmented_prompt)<br>            bot_response = response.text.replace(&#39;\n&#39;, &#39;&lt;br&gt;&#39;)<br>            bot_html = f&quot;&lt;div&gt;&lt;b&gt;Gemini:&lt;/b&gt; {bot_response}&lt;/div&gt;&quot;<br>            chat_history_output.value = old_html.replace(&quot;&lt;/div&gt;&quot;, &quot;&quot;) + user_html + bot_html + &quot;&lt;/div&gt;&quot;<br>        except Exception as e:<br>            error_html = f&quot;&lt;div style=&#39;color: red;&#39;&gt;&lt;b&gt;Error:&lt;/b&gt; {e}&lt;/div&gt;&quot;<br>            chat_history_output.value = old_html.replace(&quot;&lt;/div&gt;&quot;, &quot;&quot;) + user_html + error_html + &quot;&lt;/div&gt;&quot;<br><br>    user_input.on_submit(on_submit)<br>    display(widgets.VBox([<br>        widgets.HTML(&quot;&lt;h2&gt;🤖 Gemini Chatbot with RAG&lt;/h2&gt;&quot;),<br>        chat_history_output,<br>        user_input<br>    ]))<br><br># --- RUN THE CHATBOT ---<br>create_widget_chatbot()</pre><h3>Why RAG Changes the Game</h3><p>Without RAG:</p><ul><li>The AI is locked to its training cutoff date.</li><li>Outdated or fabricated answers are more likely.</li></ul><p>With RAG:</p><ul><li>The AI can access the most recent events.</li><li>Answers are backed by credible sources.</li><li>The system adapts instantly to new information.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Ev9K3jZZp3h5G2PYFtleA.png" /><figcaption>Chatbot with RAG implementation</figcaption></figure><h3>Conclusion</h3><blockquote>AI on its own is powerful.<br>AI with <strong>real-time retrieval</strong> is <strong>adaptable, informed, and reliable</strong>.</blockquote><p>By adding RAG capabilities through SerpAPI, a static chatbot transforms into a digital researcher — one that stays in sync with the present, no matter when the question is asked.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83cba8d00c63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ From PDF to DB: Your Pipeline to Smart Data]]></title>
            <link>https://medium.com/@mahasris0304/from-pdf-to-db-your-pipeline-to-smart-data-43ae2486b25d?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/43ae2486b25d</guid>
            <category><![CDATA[psycopg2]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[json-parsing]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 09 Aug 2025 05:21:11 GMT</pubDate>
            <atom:updated>2025-08-09T05:21:11.471Z</atom:updated>
            <content:encoded><![CDATA[<h3>🧠 Part 3: Storing Extracted Data in a Database — Completing the Pipeline</h3><p>In Part 1, we converted PDFs into high-quality images using pdfplumber and PyMuPDF. In Part 2, we used PaddleOCR to extract text from these images and save it as JSON. Now, in <strong>Part 3</strong>, we parse the JSON output and store it in a PostgreSQL database, enabling efficient querying and analysis. This final step completes our PDF-to-database pipeline, turning static documents into actionable data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*Y0NsOivamFrc-WP_rGsE_A.jpeg" /><figcaption>JSON to PostgreSQL database.</figcaption></figure><h3>Tools Used</h3><ul><li><strong>Python</strong>: The core language for our pipeline.</li><li><strong>PostgreSQL</strong>: A robust, open-source database for structured data storage.</li><li><strong>psycopg2</strong>: Python’s PostgreSQL adapter for database operations.</li></ul><blockquote>For a beginner-friendly guide on using<a href="https://medium.com/@mahasris0304/database-made-simple-%EF%B8%8F-psycopg2-for-python-beginners-38a0bcee14d3"><strong><em> psycopg2</em></strong></a></blockquote><ul><li><strong>JSON</strong>: For parsing the structured output from Part 2.</li></ul><h3>Why Store Data in a Database?</h3><p>The JSON output from PaddleOCR (e.g., text strings with coordinates) is unstructured and not easily queryable. To make it useful — say, for searching records or analyzing data — we need to:</p><ul><li>Parse the JSON into structured fields (e.g., name, id, address).</li><li>Store these fields in a PostgreSQL database for fast, reliable queries.</li></ul><p>This step transforms raw OCR output into a structured, actionable dataset.</p><h3>Code Highlights</h3><pre><br>import os<br>import json<br>import psycopg2<br><br>class DatabaseHandler:<br>    def __init__(self, db_config):<br>        &quot;&quot;&quot;Initialize PostgreSQL connection.&quot;&quot;&quot;<br>        try:<br>            self.conn = psycopg2.connect(**db_config)<br>            self.cursor = self.conn.cursor()<br>            print(&quot;✅ Connected to PostgreSQL database&quot;)<br>            self.create_table()<br>        except Exception as e:<br>            print(f&quot;❌ Error connecting to database: {e}&quot;)<br>            raise<br><br>    def create_table(self):<br>        &quot;&quot;&quot;Create a generic table.&quot;&quot;&quot;<br>        try:<br>            self.cursor.execute(&quot;&quot;&quot;<br>                CREATE TABLE IF NOT EXISTS records (<br>                    id SERIAL PRIMARY KEY,<br>                    field1 TEXT,<br>                    field2 TEXT,<br>                    field3 TEXT,<br>                    source_file TEXT,<br>                    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP<br>                )<br>            &quot;&quot;&quot;)<br>            self.conn.commit()<br>            print(&quot;✅ Table created&quot;)<br>        except Exception as e:<br>            print(f&quot;❌ Error creating table: {e}&quot;)<br>            self.conn.rollback()<br>            raise<br><br>    def save_to_db(self, data, source_file):<br>        &quot;&quot;&quot;Save data to PostgreSQL.&quot;&quot;&quot;<br>        try:<br>            insert_query = &quot;&quot;&quot;<br>                INSERT INTO records (field1, field2, field3, source_file)<br>                VALUES (%s, %s, %s, %s)<br>            &quot;&quot;&quot;<br>            records_saved = 0<br>            for item in data:<br>                self.cursor.execute(insert_query, (<br>                    item.get(&quot;field1&quot;, &quot;&quot;),<br>                    item.get(&quot;field2&quot;, &quot;&quot;),<br>                    item.get(&quot;field3&quot;, &quot;&quot;),<br>                    source_file<br>                ))<br>                records_saved += 1<br>            self.conn.commit()<br>            print(f&quot;✅ Saved {records_saved} records for &#39;{source_file}&#39;&quot;)<br>            return records_saved<br>        except Exception as e:<br>            print(f&quot;❌ Error saving to database: {e}&quot;)<br>            self.conn.rollback()<br>            raise<br><br>    def __del__(self):<br>        &quot;&quot;&quot;Close database connection.&quot;&quot;&quot;<br>        if self.cursor:<br>            self.cursor.close()<br>        if self.conn:<br>            self.conn.close()<br>            print(&quot;✅ Database connection closed&quot;)<br><br>def process_json_to_db(input_folder, db_config):<br>    &quot;&quot;&quot;Process JSON files in subfolders and save to PostgreSQL.&quot;&quot;&quot;<br>    result = {<br>        &quot;status&quot;: &quot;success&quot;,<br>        &quot;message&quot;: &quot;&quot;,<br>        &quot;processed&quot;: [],<br>        &quot;failed&quot;: [],<br>        &quot;total_records_saved&quot;: 0<br>    }<br><br>    try:<br>        db_handler = DatabaseHandler(db_config)<br>    except Exception as e:<br>        result[&quot;status&quot;] = &quot;failure&quot;<br>        result[&quot;message&quot;] = f&quot;Database connection failed: {str(e)}&quot;<br>        return result<br><br>    if not os.path.exists(input_folder):<br>        result[&quot;status&quot;] = &quot;failure&quot;<br>        result[&quot;message&quot;] = f&quot;Input folder &#39;{input_folder}&#39; not found&quot;<br>        return result<br><br>    subfolders = [f for f in os.listdir(input_folder) if os.path.isdir(os.path.join(input_folder, f))]<br>    if not subfolders:<br>        result[&quot;status&quot;] = &quot;failure&quot;<br>        result[&quot;message&quot;] = &quot;No subfolders found&quot;<br>        return result<br><br>    for subfolder in subfolders:<br>        subfolder_path = os.path.join(input_folder, subfolder)<br>        json_files = [f for f in os.listdir(subfolder_path) if f.endswith(&#39;.json&#39;)]<br>        if not json_files:<br>            result[&quot;failed&quot;].append({&quot;subfolder&quot;: subfolder, &quot;reason&quot;: &quot;No JSON file&quot;})<br>            continue<br><br>        json_file = json_files[0]<br>        json_path = os.path.join(subfolder_path, json_file)<br><br>        try:<br>            with open(json_path, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>                data = json.load(f)<br>            if not isinstance(data, list) or not data:<br>                result[&quot;failed&quot;].append({&quot;subfolder&quot;: subfolder, &quot;reason&quot;: &quot;Invalid or empty JSON&quot;})<br>                continue<br><br>            records_saved = db_handler.save_to_db(data, json_file)<br>            result[&quot;processed&quot;].append(subfolder)<br>            result[&quot;total_records_saved&quot;] += records_saved<br>        except Exception as e:<br>            result[&quot;failed&quot;].append({&quot;subfolder&quot;: subfolder, &quot;reason&quot;: f&quot;Error: {str(e)}&quot;})<br>            continue<br><br>    result[&quot;message&quot;] = (f&quot;Processed {len(result[&#39;processed&#39;])} subfolders, &quot;<br>                         f&quot;failed {len(result[&#39;failed&#39;])}, &quot;<br>                         f&quot;saved {result[&#39;total_records_saved&#39;]} records&quot;)<br>    print(f&quot;🎉 {result[&#39;message&#39;]}&quot;)<br>    return result<br><br>if __name__ == &quot;__main__&quot;:<br>    db_config = {<br>        &quot;host&quot;: &quot;localhost&quot;,<br>        &quot;database&quot;: &quot;mydb&quot;,<br>        &quot;user&quot;: &quot;user&quot;,<br>        &quot;password&quot;: &quot;pass&quot;,<br>        &quot;port&quot;: &quot;5432&quot;<br>    }<br>    input_folder = &quot;sample_data_folder&quot;  # Placeholder<br>    result = process_json_to_db(input_folder, db_config)<br>    print(&quot;Result:&quot;, result)</pre><h3>Setting Up the Environment</h3><p>Install:</p><pre>pip install psycopg2-binary</pre><p>Set up PostgreSQL (postgresql.org) and create a database (mydb). Ensure PaddleOCR is installed from Part 2 (PaddlePaddle guide).</p><h3>Challenges and Solutions</h3><ul><li><strong>JSON Variability</strong>: Handle missing or malformed fields with .get().</li><li><strong>Database Errors</strong>: Use try-except to rollback failed inserts.</li><li><strong>Credentials</strong>: Store in environment variables, not code.</li></ul><h3>Example Workflow</h3><p>For a PDF (sample_example.pdf):</p><ol><li>Part 2 saves JSON files in sample_data_folder/subfolder1/data.json.</li><li>Parse JSON (e.g., [{“field1”: “value1”, “field2”: “value2”, “field3”: “value3”}, …]).</li><li>Store in PostgreSQL table records.</li><li>Query: SELECT * FROM records WHERE field1 = ‘value1’;.</li></ol><h3>Conclusion</h3><p>Storing JSON data in PostgreSQL completes our pipeline, enabling queryable data from PDFs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43ae2486b25d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Database Made Simple: ️“psycopg2 “for Python Beginners]]></title>
            <link>https://medium.com/@mahasris0304/database-made-simple-%EF%B8%8F-psycopg2-for-python-beginners-38a0bcee14d3?source=rss-52aea5bc4db7------2</link>
            <guid isPermaLink="false">https://medium.com/p/38a0bcee14d3</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[psycopg2]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Mahasri Seenivasan]]></dc:creator>
            <pubDate>Sat, 12 Jul 2025 13:17:26 GMT</pubDate>
            <atom:updated>2025-08-30T05:44:00.924Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how to connect your Python code to PostgreSQL using psycopg2 — a beginner-friendly guide to running SQL queries, fetching data, and managing your database with ease.</p><p>If you’re learning Python and wondering how to connect it to a database, you’re in the right place. In this quick guide, we’ll break down how to use <strong>psycopg2</strong> 🐘 — a powerful library that connects your Python code to a <strong>PostgreSQL</strong> database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9WWH7-KuV3Jn6V_dUHViw.png" /></figure><p>Whether you’re building a small app, automating data tasks, or learning SQL, psycopg2 makes database access simple and smooth. ⚡</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*3sR8wJCONU-DJsnfeQBVAg.png" /><figcaption>psycopg2</figcaption></figure><h3>🔍 What is psycopg2?</h3><p><strong>psycopg2</strong> is a PostgreSQL adapter for Python. <br> It lets your Python code connect to a PostgreSQL database and run SQL commands — like reading data, inserting records, or even modifying tables.</p><p>Think of it as the translator between Python and your database. 🔄</p><h3><strong>Quick Installation</strong></h3><pre>pip install psycopg2-binary</pre><h3>🚀 Why You’ll Need It</h3><p>Here’s what you can do with psycopg2:</p><p>✅ Read from a PostgreSQL table<br> ➕ Insert new data<br> 🔍 Check for conditions (NULL, LIKE, etc.)<br> 🔗 Connect to a database:</p><pre>import psycopg2<br><br>conn = psycopg2.connect(<br>    host=&quot;localhost&quot;,<br>    database=&quot;myproject&quot;,<br>    user=&quot;myuser&quot;,<br>    password=&quot;mypassword&quot;<br>)</pre><p>🧭 Create a cursor to run SQL commands:</p><pre>cursor = conn.cursor()<br>cursor.execute(&quot;SELECT * FROM your_table;&quot;)</pre><h3>📥 Fetching Data</h3><p>After running a query, you can pull results like this:</p><pre>rows = cursor.fetchall()   # fetch all rows  <br>row = cursor.fetchone()    # fetch the next row</pre><h3>💾 Making Changes?</h3><p>If you’re inserting or updating data, don’t forget to <strong>commit</strong> the changes:</p><pre>conn.commit()</pre><p>📝 This step finalizes your changes in the database.</p><h3>📚 Built on Standards</h3><p>psycopg2 follows Python’s <strong>DB-API 2.0</strong> standard, which means it offers a consistent way to interact with databases — no surprises, no hacks.</p><h3>💡 Final Thoughts</h3><p>Whether you’re writing your first SQL query or building a small data pipeline, <strong>psycopg2 is the go-to tool</strong> for connecting Python to PostgreSQL. It’s simple, efficient, and beginner-friendly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38a0bcee14d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>