<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vatsa Joshi on Medium]]></title>
        <description><![CDATA[Stories by Vatsa Joshi on Medium]]></description>
        <link>https://medium.com/@vatsajoshi2?source=rss-d3feb3f33795------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Vatsa Joshi on Medium</title>
            <link>https://medium.com/@vatsajoshi2?source=rss-d3feb3f33795------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:35:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vatsajoshi2/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Self Adaptive Context RAG]]></title>
            <link>https://medium.com/data-and-beyond/self-adaptive-context-rag-d52a5dea6213?source=rss-d3feb3f33795------2</link>
            <guid isPermaLink="false">https://medium.com/p/d52a5dea6213</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Vatsa Joshi]]></dc:creator>
            <pubDate>Sat, 19 Jul 2025 17:20:02 GMT</pubDate>
            <atom:updated>2026-03-09T18:28:06.506Z</atom:updated>
            <content:encoded><![CDATA[<p>In the rapidly evolving landscape of artificial intelligence, Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm that combines the best of both worlds: the vast knowledge retrieval capabilities of search systems and the natural language understanding of large language models.</p><p><strong>Today, I’m excited to share with you an advanced implementation of what I call “Self Adaptive Context RAG” — a system that not only retrieves relevant information but also adapts its context understanding and corrects its own reasoning.</strong></p><p>The complete source code for this implementation is available here — <a href="https://colab.research.google.com/drive/17oeMm-LCJNgjQmYaqNjN10ww1vGqxa8v?usp=sharing#scrollTo=85brsNmdZ2oF">https://colab.research.google.com/drive/17oeMm-LCJNgjQmYaqNjN10ww1vGqxa8v?usp=sharing#scrollTo=85brsNmdZ2oF</a></p><p><strong>The Challenge with Traditional RAG Systems</strong></p><p>Traditional RAG systems often suffer from several limitations:</p><ul><li><strong>Static retrieval mechanisms</strong> that don’t adapt to query complexity</li><li><strong>Limited context understanding</strong> when dealing with real-time information</li><li><strong>No self-correction capabilities</strong> leading to propagated errors</li><li><strong>Poor integration</strong> between web scraping and vector databases</li></ul><p>My Self Adaptive Context RAG system addresses these challenges through an innovative architecture that dynamically adapts to different query types and includes built-in self-correction mechanisms.</p><p><strong>System Architecture Overview</strong></p><p>The system is built around five core components that work together seamlessly:</p><p><strong>1. Intelligent Web Scraper</strong></p><p>The WebScraper class implements a multi-strategy approach to content acquisition:</p><ul><li><strong>Dual scraping methods</strong>: Uses both HTTP requests and Selenium WebDriver</li><li><strong>Smart caching</strong>: Implements MD5-based caching with configurable expiry</li><li><strong>Priority-based URL selection</strong>: Targets reputable news sources first</li><li><strong>Adaptive retry logic</strong>: Falls back to alternative methods when one fails</li></ul><pre>class WebScraper:<br>    def __init__(self):<br>        self.session = requests.Session()<br>        self.driver = None<br>        os.makedirs(config.cache_dir, exist_ok=True)</pre><p><strong>2. Advanced Text Chunker</strong></p><p>The TextChunker handles intelligent document segmentation:</p><ul><li><strong>Token-aware chunking</strong>: Uses proper tokenization for accurate length calculation</li><li><strong>Overlap strategy</strong>: Maintains context continuity between chunks</li><li><strong>Metadata preservation</strong>: Keeps track of source information and timestamps</li></ul><p><strong>3. Embedding Manager</strong></p><p>Powered by SentenceTransformer, this component:</p><ul><li><strong>Batch processing</strong>: Efficiently handles multiple texts simultaneously</li><li><strong>Error handling</strong>: Robust processing of empty or malformed texts</li></ul><p><strong>4. Vector Database Integration</strong></p><p>The system supports both FAISS and ChromaDB:</p><ul><li><strong>Duplicate detection</strong>: Uses text hashing to prevent redundant storage</li><li><strong>Similarity threshold filtering</strong>: Only returns genuinely relevant results</li><li><strong>Persistent storage</strong>: Maintains vector embeddings across sessions</li></ul><p><strong>5. Self-Correcting LLM Interface</strong></p><p>The crown jewel of the system — a two-stage reasoning process:</p><ul><li><strong>Initial reasoning</strong>: Generates first-pass answers using retrieved context</li><li><strong>Self-correction</strong>: Critically evaluates and improves the initial response</li></ul><p><strong>The Self-Correction Innovation</strong></p><p>What sets this system apart is its self-correction mechanism. Here’s how it works:</p><p><strong>Stage 1: Initial Reasoning</strong></p><pre>def _create_reasoning_prompt(self, query: str, context: str) -&gt; str:<br>    return f&quot;&quot;&quot;<br>    You are an expert AI assistant with access to relevant information.<br>    <br>    Context Information:<br>    {context}<br>    <br>    User Question: {query}<br>    <br>    Instructions:<br>    1. Analyze the provided context carefully<br>    2. Identify key information relevant to the question<br>    3. Reason through the problem step by step<br>    4. Provide a well-structured, comprehensive answer<br>    &quot;&quot;&quot;</pre><p><strong>Stage 2: Self-Correction</strong></p><p>The system then critically evaluates its own response:</p><pre>def _create_correction_prompt(self, query: str, context: str, initial_answer: str) -&gt; str:<br>    return f&quot;&quot;&quot;<br>    Self-Correction Instructions:<br>    1. Critically evaluate the initial answer for:<br>       - Factual accuracy<br>       - Completeness<br>       - Logical consistency<br>       - Proper use of context<br>    <br>    2. Provide an improved final answer that:<br>       - Corrects any identified errors<br>       - Adds missing important information<br>       - Improves clarity and structure<br>    &quot;&quot;&quot;</pre><p>This dual-stage approach significantly improves answer quality and reduces hallucinations.</p><p><strong>Real-World Performance</strong></p><p>Let me share some actual results from the system in action:</p><p><strong>Query: “Did a plane crash in Ahmedabad?”</strong></p><p><strong>Performance Metrics:</strong></p><ul><li><strong>Processing Time</strong>: 123.24 seconds</li><li><strong>Pages Scraped</strong>: 9</li><li><strong>Text Chunks Generated</strong>: 54</li><li><strong>Retrieved Relevant Chunks</strong>: 5</li><li><strong>Similarity Scores</strong>: [0.662, 0.662, 0.662]</li></ul><p><strong>System Response:</strong></p><blockquote>“Yes, a plane crash occurred in Ahmedabad. The incident took place moments after departure when the aircraft lost altitude and crashed into Meghani Nagar, a residential area. The flight was scheduled to land at London Gatwick but crashed shortly after takeoff…”</blockquote><p>The system successfully retrieved real-time information about a recent incident, demonstrating its ability to handle current events with high accuracy.</p><p><strong>Query: “What’s going on with Trump and Musk?”</strong></p><p><strong>Performance Metrics:</strong></p><ul><li><strong>Processing Time</strong>: 180.69 seconds</li><li><strong>Pages Scraped</strong>: 5</li><li><strong>Text Chunks Generated</strong>: 22</li><li><strong>Retrieved Relevant Chunks</strong>: 5</li><li><strong>Similarity Scores</strong>: [0.657, 0.657, 0.651]</li></ul><p>The system provided a comprehensive analysis of recent political developments, showcasing its ability to synthesize information from multiple sources.</p><p><strong>Technical Implementation Highlights</strong></p><p><strong>Configuration Management</strong></p><p>The system uses a centralized configuration class:</p><pre>@dataclass<br>class Config:<br>    embedding_model_name: str = &quot;all-MiniLM-L6-v2&quot;<br>    llm_model_name: str = &quot;deepseek-r1:14b&quot;<br>    max_tokens: int = 4096<br>    chunk_size: int = 256<br>    top_k_retrieval: int = 5<br>    similarity_threshold: float = 0.15</pre><p><strong>Asynchronous Processing</strong></p><p>The entire pipeline runs asynchronously for optimal performance:</p><pre>async def process_query(self, query: str, use_web_search: bool = True) -&gt; Dict[str, Any]:<br>    # Scraping<br>    scraped_data = await self.scraper.search_and_scrape(query)<br>    <br>    # Chunking and embedding<br>    embeddings = self.embedder.embed_texts(texts)<br>    <br>    # Vector storage and retrieval<br>    self.vector_db.add_documents(all_chunks, embeddings)<br>    relevant_chunks = self.vector_db.search(query_embedding, config.top_k_retrieval)</pre><p><strong>Error Handling and Resilience</strong></p><p>The system implements comprehensive error handling:</p><ul><li><strong>Timeout management</strong>: Prevents hanging on slow responses</li><li><strong>Retry logic</strong>: Multiple attempts with exponential backoff</li><li><strong>Graceful degradation</strong>: Falls back to cached results when live scraping fails</li><li><strong>Resource cleanup</strong>: Proper disposal of Selenium drivers and connections</li></ul><p><strong>Installation and Usage</strong></p><p>Getting started with the system is straightforward:</p><p><strong>Prerequisites</strong></p><pre># Install required packages<br>pip install torch sentence-transformers chromadb selenium<br>pip install beautifulsoup4 nltk transformers ollama<br>pip install aiohttp tqdm webdriver-manager</pre><p><strong>Setup</strong></p><pre># Initialize the system<br>system = RAGSystem()<br><br># Run interactive mode<br>await system.run_interactive()</pre><p><strong>Basic Usage</strong></p><pre># Process a single query<br>results = await reasoner.process_query(&quot;Your question here&quot;)<br>print(results[&#39;final_answer&#39;])</pre><p><strong>Performance Optimizations</strong></p><p>Several optimizations make this system production-ready:</p><p><strong>1. Intelligent Caching</strong></p><ul><li>MD5-based content hashing prevents duplicate processing</li><li>Configurable cache expiry (48 hours by default)</li><li>Persistent vector storage across sessions</li></ul><p><strong>2. Batch Processing</strong></p><ul><li>Embedding generation in configurable batches (64 by default)</li><li>Parallel web scraping with asyncio</li><li>Efficient memory usage with streaming</li></ul><p><strong>Future Enhancements</strong></p><p>The current implementation provides a solid foundation, but there are several areas for future development:</p><p><strong>1. Multi-Modal Support</strong></p><ul><li>Image and video content processing</li><li>Audio transcription integration</li><li>Document parsing (PDF, DOC, etc.)</li></ul><p><strong>2. Advanced Reasoning</strong></p><ul><li>Chain-of-thought prompting</li><li>Multi-step reasoning for complex queries</li><li>Integration with external APIs</li></ul><p><strong>3. Enterprise Features</strong></p><ul><li>User authentication and session management</li><li>Query history and analytics</li><li>Custom knowledge base integration</li></ul><p><strong>4. Performance Improvements</strong></p><ul><li>GPU acceleration for embedding generation</li><li>Distributed processing for large-scale deployment</li><li>Real-time streaming responses</li></ul><p><strong>Lessons Learned</strong></p><p>Building this system taught me several valuable lessons:</p><p><strong>1. Context Quality Over Quantity</strong></p><p>It’s better to have fewer, highly relevant chunks than many loosely related ones. The similarity threshold of 0.15 was fine-tuned through extensive testing.</p><p><strong>2. Self-Correction is Game-Changing</strong></p><p>The two-stage reasoning process consistently produces better results than single-pass generation, even though it requires more computational resources.</p><p><strong>3. Robustness Matters</strong></p><p>Real-world web scraping is unpredictable. Having multiple fallback strategies (requests → Selenium → cached results) is essential.</p><p><strong>4. Monitoring is Crucial</strong></p><p>Detailed logging at every step makes debugging and optimization much easier. The system logs similarity scores, chunk counts, and processing times for analysis.</p><p><strong>Conclusion</strong></p><p>The Self Adaptive Context RAG system represents a significant advancement in intelligent question-answering technology. By combining adaptive web scraping, sophisticated text processing, vector-based retrieval, and self-correcting reasoning, it delivers accurate, contextual responses to complex queries.</p><p>The system’s ability to handle real-time information while maintaining high accuracy makes it suitable for applications ranging from news analysis to research assistance. Its modular design allows for easy customization and extension based on specific use cases.</p><p>Whether you’re building a customer service bot, a research assistant, or any application that requires intelligent information retrieval and synthesis, this Self Adaptive Context RAG system provides a robust foundation that can adapt and evolve with your needs.</p><p>The future of AI lies not just in larger models, but in smarter systems that can dynamically adapt their approach based on the task at hand. This implementation is a step toward that future, where AI systems can reason, adapt, and correct themselves to provide increasingly accurate and helpful responses.</p><p><em>The complete source code for this implementation is available and demonstrates practical applications of modern NLP techniques in a production-ready system. Feel free to experiment with the configuration parameters and extend the system for your specific use cases.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d52a5dea6213" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-and-beyond/self-adaptive-context-rag-d52a5dea6213">Self Adaptive Context RAG</a> was originally published in <a href="https://medium.com/data-and-beyond">Data And Beyond</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Training of Large Language Models (LLMs)]]></title>
            <link>https://medium.com/@vatsajoshi2/understanding-the-training-of-large-language-models-llms-6be49e884beb?source=rss-d3feb3f33795------2</link>
            <guid isPermaLink="false">https://medium.com/p/6be49e884beb</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Vatsa Joshi]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 17:26:25 GMT</pubDate>
            <atom:updated>2025-01-21T17:26:25.488Z</atom:updated>
            <content:encoded><![CDATA[<p>In order to be ready for the exciting adventures in generative AI applications based on large language models (LLMs) or on multimodal models, it is important to understand how these models are trained in detail. Of course, it is completely impractical to build from scratch yourself due to the orders of magnitude of computational resources involved, but I will lead you through the theoretical steps to train LLMs using some popular models such as OpenAI’s ChatGPT and Meta’s LLaMA.</p><p>Step-by-Step Guide : How to Train LLMs</p><p>As such training a language model is a very detailed and structured process. Let’s break down these steps:</p><p><strong>Phase 1: Generative Pre-Training</strong></p><p>Data Gathering: The first step is to collect a massive amount of text data from sources like websites, articles, books, and open public forums.</p><p>Transformer model: The training model at its heart is the transformer which is an encoder-decoder architecture. Mutual information prediction capabilities have earned Transformers a repute in tasks such as language translation, text summarization, text completion, and sentiment analysis.</p><p>Pre-Training the Base Model: The gathered data is passed into the transformer model to develop a base GPT model. This is a generic model capable of doing the above but needs fine-tuning for application-specific implementations.</p><p><strong>Phase 2: Fine-Tuning with Supervised Learning</strong></p><p>Preparing a Training Corpus: Actual human agents spoofing conversation to create request-response pairs. This mechanism will build a supervised fine-tuning (SFT) training dataset from those interactions.</p><p>Training with Supervised Data: The GPT model is fine-tuned with the SFT corpus, optimizing the model with techniques like stochastic gradient descent, creating a SFT ChatGPT model.</p><p><strong>Phase 3: Reinforcement Learning with Human Feedback (RLHF)</strong></p><p>This is one of the key differences between the ChatGPT model based on SFT: Generating Responses.</p><p>Human Ranking: Human raters rank these responses for relevance and accuracy.</p><p>3, Reward Model: A reward model is created to take the rankings and give a score to responses to help classify good and bad outputs.</p><p>Reinforcement Learning: You then optimize that reward model using something like Proximal Policy Optimization (PPO), which allows the ChatGPT model to keep improving based on human feedback.</p><p><strong>Practical use-case of Training LLMs</strong></p><p>Consider a chef capable of preparing multiple meals. A customer orders dinner — non-vegetarian. The chef gets feedback from a lot of people to choose a dish. The chef ranks these dishes according to the feedback received and uses a reward system to ascertain the best possible choice. This analogy can explain how responses are sorted and used again in training the model.</p><p><strong>Importance of Transformers</strong></p><p>LLMs extensively use transformers, which is the technology behind ChatGPT and BERT. They rely on mechanisms such as “attention is all you need,” which allow these systems to process vast amounts of data efficiently. Knowledge of transformers is fundamental to understanding how powerful AI models work.</p><p><strong>Final Thoughts</strong></p><p>Practically these are implemented at a much larger scale which needs extensive resources but all these concepts lay the groundwork for generative AI applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6be49e884beb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>