<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Indrajit on Medium]]></title>
        <description><![CDATA[Stories by Indrajit on Medium]]></description>
        <link>https://medium.com/@official.indrajit.kar?source=rss-d27657cda394------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MMXxryRZDyQf630CuXb_4w.jpeg</url>
            <title>Stories by Indrajit on Medium</title>
            <link>https://medium.com/@official.indrajit.kar?source=rss-d27657cda394------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:30:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@official.indrajit.kar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multi Modal Multivector  Representations in Qdrant for Reranking powered by Adaptive Refresh]]></title>
            <link>https://medium.com/@official.indrajit.kar/multi-modal-multivector-representations-in-qdrant-for-reranking-powered-by-adaptive-refresh-231fd240d812?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/231fd240d812</guid>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 23:12:14 GMT</pubDate>
            <atom:updated>2025-06-25T23:27:40.336Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multi Modal Multivector Representations in Qdrant for Reranking powered by Adaptive Refresh</h3><h3>Introduction:</h3><p>Retrieval-Augmented Generation (RAG) systems enhance the capabilities of language models by incorporating external knowledge retrieved from vector databases. Traditionally, these systems rely on single-vector representations, wherein each document or input is encoded into a single dense embedding. While effective for short or homogeneous content, this approach often fails to capture fine-grained semantic details necessary for high-accuracy retrieval, particularly in complex or multimodal scenarios.</p><p>Multivector representation addresses this limitation by encoding a single document as a collection of vectors, typically at the token, phrase, or segment level. This allows retrieval systems to perform <strong>late interaction</strong>, where relevance scoring is deferred until query time — enabling precise token-level matching rather than relying on pre-aggregated document vectors. Architectures such as ColBERT exemplify this strategy, using token-wise MaxSim operations for efficient reranking.</p><p>In multimodal RAG systems, multivector representations can be extended beyond text. For example, text documents can be embedded as token-level vectors, while associated images can be embedded separately using models such as CLIP. These multimodal embeddings are jointly stored in a vector database (e.g., Qdrant), allowing the system to support <strong>cross-modal retrieval</strong>, reranking, and grounded response generation.</p><p>By combining dense retrieval, multivector reranking, and multimodal alignment, such architectures enable more expressive, accurate, and contextually grounded RAG pipelines — capable of handling diverse information modalities with precision and scalability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*NtC5vaI8KsixVAW4Sj6FgQ.png" /></figure><p>Figure :Multi Modal Multivector Representations in Qdrant for Reranking powered by Adaptive Refresh</p><p>The end to end code can be found here</p><p>The figure below contrasts traditional single-vector encoding with multivector and multimodal representations. While conventional embedding approaches compress an entire sentence or image into a single dense vector, multivector representations preserve finer-grained semantic structure by retaining multiple vectors — often at the token or phrase level. When extended to multimodal contexts, such as pairing text with image embeddings, this architecture enables more expressive and accurate retrieval by aligning complementary signals across modalities. This foundation supports advanced late-interaction retrieval strategies and enables richer contextual understanding in downstream tasks such as RAG, search, and question answering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*nOJW41KzQrGQd_gZYfXSlA.png" /></figure><p>Figure: Comparision traditional single-vector encoding with multivector and multimodal representations.</p><h3>Vector Embedding Logic (in embedding_pipeline.py)</h3><h3>1. Dense Text Embedding (&quot;dense_text&quot;)</h3><pre>dense_vec = generate_text_embedding(text)</pre><ul><li>Uses BAAI/bge-small-en via SentenceTransformer</li><li>Creates <strong>one single 384-dimensional vector</strong> per text</li><li>Used for <strong>initial fast retrieval</strong></li></ul><h3>2. ColBERT Token-level Multivector Embedding (&quot;colbert_text&quot;)</h3><pre>colbert_vecs = generate_colbert_embedding(text)</pre><ul><li>Uses colbert-ir/colbertv2.0 via LateInteractionTextEmbedding</li><li>Generates <strong>multiple 128-dimensional vectors</strong> per document (token-level)</li><li>Stored in Qdrant as a <strong>multivector</strong> field using MultiVectorConfig(MAX_SIM)</li><li>Enables <strong>late interaction reranking</strong> with query_points()</li></ul><h3>3. Image Embedding (&quot;image&quot;)</h3><pre>image_vec = generate_image_embedding(image_path)</pre><ul><li>Uses openai/clip-vit-base-patch32 to convert image to a <strong>512-dim CLIP vector</strong></li><li>Stored separately in Qdrant under &quot;image&quot; key</li><li>Enables <strong>image-based similarity search</strong></li></ul><h3>4. Inserting All Embeddings Together</h3><pre>points.append(models.PointStruct(<br>    id=str(uuid.uuid4()),<br>    vector={<br>        &quot;dense_text&quot;: dense_vec,<br>        &quot;colbert_text&quot;: colbert_vecs,<br>        &quot;image&quot;: image_vec<br>    },<br>    payload={&quot;filename&quot;: fname, &quot;text&quot;: text}<br>)</pre><p>This means:</p><ul><li>A single .txt + .jpg pair results in <strong>three different vectors</strong></li><li>Each vector gets its own field in Qdrant: dense_text, colbert_text, image</li><li>I have built a <strong>multimodal (image + text)</strong> and <strong>multivector (token-level)</strong> vector store</li></ul><h3>Retrieval and Reranking (in reranker_and_llm.py)</h3><h3>Step 1: Fast retrieval using dense vector</h3><pre>prefetch=models.Prefetch(query=dense_query, using=&quot;dense_text&quot;)</pre><h3>Step 2: Late reranking using multivector field</h3><pre>query=colbert_query,<br>using=&quot;colbert_text&quot;,</pre><h3>Step 3: Optional image-based similarity search</h3><pre>if image_vector:<br>    image_results = client.query_points(<br>        query=image_vector,<br>        using=&quot;image&quot;<br>    )</pre><h3>Step 4: Merging and deduplicating results</h3><pre>all_payloads = [text_results] + [image_results]<br># deduplicated using `filename` or `text`</pre><h3>Step 5: Generating answer with LLM</h3><pre>context = &quot;\n&quot;.join(...)<br>response = chain.run({&quot;query&quot;: ..., &quot;context&quot;: ...})</pre><ul><li>I have send the final context (retrieved payloads) into <strong>Mistral via LangChain + ReAct-style prompt</strong></li><li>The final <strong>natural language response is generated</strong> based on hybrid retrieval</li></ul><p><strong>Reranking is happening using multivector embeddings via the </strong><strong>colbert_text field.</strong><br> Here’s the exact reasoning based on the code:</p><p>In the reranker_and_llm.py function:</p><pre>results = client.query_points(<br>    collection_name=COLLECTION_NAME,<br>    prefetch=models.Prefetch(query=dense_query, using=&quot;dense_text&quot;),  # 🔹 fast initial retrieval<br>    query=colbert_query,                                               # 🔸 token-level query<br>    using=&quot;colbert_text&quot;,                                              # 🔸 reranking on multivector<br>    limit=top_k,<br>    with_payload=True<br>)</pre><h3>Why this is multivector reranking:</h3><p>ComponentDescriptioncolbert_querythe query is passed as <strong>token-level embeddings</strong> using ColBERTusing=&quot;colbert_text&quot;This field in Qdrant is configured as a <strong>multivector</strong> (with MAX_SIM)query_points(...)Executes <strong>Late Interaction reranking</strong>: matches each query token to its best document token (via MaxSim)</p><p><strong>In the current setup, indexing is ON for dense_text and image, but OFF for multivector reranking (colbert_text) — which is exactly what I would want for a 2-stage retrieval system.</strong></p><h3>Breakdown from the create_collection() config:</h3><pre>&quot;colbert_text&quot;: models.VectorParams(<br>    size=128,<br>    distance=models.Distance.COSINE,<br>    multivector_config=models.MultiVectorConfig(<br>        comparator=models.MultiVectorComparator.MAX_SIM<br>    ),<br>    hnsw_config=models.HnswConfigDiff(m=0)  # disables indexing<br>)</pre><p>FieldHNSW IndexingWhy?dense_text✅ ON (default)Used for <strong>initial fast retrieval</strong> (single vector per doc)colbert_text❌ OFFUsed for <strong>late reranking</strong>, doesn’t need approximate searchimage✅ ON (default)Used for <strong>image-based similarity search</strong></p><p>the insertion of all embeddings (text, multivector, and image) <strong>is done into a single collection</strong> in Qdrant.</p><h3>The collection name:</h3><pre>COLLECTION_NAME = &quot;multimodal_multivector&quot;</pre><h3>The unified insertion happens via client.upsert():</h3><p>From the refresh_embeddings() function in embedding_pipeline.py:</p><pre>points.append(models.PointStruct(<br>    id=str(uuid.uuid4()),<br>    vector={<br>        &quot;dense_text&quot;: dense_vec,        # single-vector<br>        &quot;colbert_text&quot;: colbert_vecs,   # multi-vector (token-level)<br>        &quot;image&quot;: image_vec              # CLIP vector<br>    },<br>    payload={<br>        &quot;filename&quot;: fname,<br>        &quot;text&quot;: text<br>    }<br>))</pre><p>And then:</p><pre>client.upsert(collection_name=COLLECTION_NAME, points=points)</pre><blockquote><em>Hoever, ❌ </em><strong><em>No — the image vector is </em>not<em> part of the multivector field.</em></strong></blockquote><p>Let’s break this down to clarify exactly what’s happening:</p><h3>Multivector ≠ Multimodal</h3><p>In the code:</p><pre>vector={<br>    &quot;dense_text&quot;: dense_vec,        # Single vector (BGE)<br>    &quot;colbert_text&quot;: colbert_vecs,   # Multivector (ColBERT tokens)<br>    &quot;image&quot;: image_vec              # Single vector (CLIP)<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*cAjH__nFKr-M1616E7AHKg.png" /></figure><ul><li>colbert_text is a <strong>multivector field</strong>, because it&#39;s a list of token-level vectors.</li><li>✅ image is a <strong>separate vector field</strong>, containing exactly one 512-dim CLIP vector.</li><li>❌ image_vec is <strong>not embedded into</strong> the colbert_text multivector.</li></ul><h3>Adaptive Embedding Refresh Mechanism</h3><p>The system includes an adaptive refresh function that scans a specified text and image folder. It detects valid .txt and .jpg file pairs, generates all necessary embeddings, and upserts them into Qdrant.</p><p>This process is “adaptive” in that:</p><ul><li>It reflects the current contents of the data folders.</li><li>It automatically generates embeddings for new files.</li><li>It avoids reprocessing if a required image file is missing.</li><li>However, it currently does not deduplicate or overwrite based on file names. Each insert uses a new UUID.</li></ul><p>This refresh mechanism ensures that Qdrant stays up-to-date with the latest dataset, making it suitable for environments where documents change regularly (e.g., weekly updates).</p><p><strong>Final Thought</strong></p><p>This system presents a unified architecture that combines dense sentence-level embeddings, token-level multivector embeddings, and image embeddings within a single vector database using Qdrant. By supporting both fast retrieval and precise reranking through late interaction mechanisms, the approach enables high-fidelity multimodal search across text and image modalities. The inclusion of an adaptive refresh pipeline ensures that the vector store remains synchronized with evolving datasets, making the system suitable for dynamic content environments. Overall, the design demonstrates an effective and scalable implementation of multimodal Retrieval-Augmented Generation (RAG), integrating multivector reranking, image-text alignment, and local LLM-based reasoning within a cohesive, production-ready framework.</p><p>Refernce :<a href="https://qdrant.tech/documentation/advanced-tutorials/using-multivector-representations/">https://qdrant.tech/documentation/advanced-tutorials/using-multivector-representations/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=231fd240d812" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: Building a Scalable Text-to-SQL Agentic System with LangChain, Vector DB, and Multi DB…]]></title>
            <link>https://medium.com/@official.indrajit.kar/part-1-building-a-scalable-text-to-sql-agentic-system-with-langchain-vector-db-and-multi-db-50d087a41a41?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/50d087a41a41</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[text-2-sql]]></category>
            <category><![CDATA[retail]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Fri, 07 Mar 2025 09:16:58 GMT</pubDate>
            <atom:updated>2025-03-07T09:19:56.619Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1: Building a Scalable Text-to-SQL Agentic System with LangChain, Vector DB, and Multi DB Federated Queries</h3><h3>Problem Statement: AI-Powered Text-to-SQL System for Retail Analytics</h3><h4>Background</h4><p>Retail businesses generate <strong>massive volumes of data</strong> across multiple databases — <strong>customer profiles, sales transactions, inventory, pricing, and marketing campaigns</strong>. However, <strong>data is often siloed across different databases (PostgreSQL, MySQL, MongoDB, etc.)</strong>, making it difficult for non-technical users to <strong>extract insights quickly</strong>.</p><p>Retail analysts, marketers, and executives <strong>lack SQL expertise</strong> and rely on <strong>data engineers</strong> to manually write queries, leading to <strong>delays in decision-making</strong> and <strong>missed revenue opportunities</strong>.</p><h4>Business Challenge</h4><p>The retail company wants to <strong>enable natural language querying</strong> of its <strong>sales, customer, and inventory data</strong> across multiple databases in real-time.</p><p>Currently, the challenges are:<br> <strong>Slow decision-making</strong>: Business teams need insights on <strong>customer behavior, sales trends, and inventory</strong> but lack SQL expertise.<br> <strong>Siloed Data</strong>: Customer and order data are stored in <strong>different databases</strong> (PostgreSQL for customers, MySQL for orders, MongoDB for marketing).<br> <strong>High Query Latency</strong>: Running SQL queries on <strong>billions of records</strong> in a <strong>245GB+ database</strong> is slow and inefficient.<br> <strong>Missed Sales Opportunities</strong>: Marketers can’t quickly analyze <strong>which customers are likely to churn or which products are in high demand</strong>.</p><h4>Solution: AI-Driven Text-to-SQL System</h4><p>To address these challenges, we implement an <strong>AI-powered, scalable Text-to-SQL system</strong> that:</p><p><strong>Allows business users to query data in natural language</strong> (e.g., “Find top customers who spent more than $1000 in the last 30 days”).<br><strong>Uses Vector DB (FAISS/ChromaDB) for row pre-filtering</strong>, reducing search space before executing SQL queries.<br><strong>Performs schema matching</strong> to dynamically identify the correct table and column names.<br><strong>Executes federated queries</strong> across multiple databases (PostgreSQL for customers, MySQL for orders, MongoDB for marketing data).<br><strong>Stores query results in an In-Memory DB (DuckDB/Redis) for faster analytics</strong>.<br><strong>Enables real-time dashboarding &amp; machine learning inference</strong> for predictive analytics.</p><h4>Expected Business Outcomes</h4><p><strong>90% reduction in query execution time</strong> with <strong>vector pre-filtering + federated queries</strong>.<br><strong>Enables real-time customer analytics</strong>, leading to <strong>better sales &amp; inventory management</strong>.<br><strong>Increases marketing efficiency</strong> by allowing real-time segmentation of <strong>high-value customers</strong>.<br><strong>Empowers business teams with self-service data access</strong>, eliminating dependency on SQL experts.<br><strong>Boosts revenue by identifying top-selling products and optimizing pricing strategies.</strong></p><p><strong>The AI-powered Text-to-SQL system transforms retail analytics, making data instantly accessible and actionable.</strong></p><h3>Why This is a Game-Changer for Retail?</h3><p><strong>Fast insights</strong> → No need for data engineers to manually write SQL.<br><strong>Scalable across databases</strong> → Queries PostgreSQL, MySQL, MongoDB at the same time.<br><strong>AI-driven recommendations</strong> → Helps predict churn, demand, and optimize pricing.<br><strong>Real-time analytics</strong> → Enables instant decision-making for sales &amp; marketing.</p><p><strong>Retailers that adopt this system gain a competitive edge by leveraging AI-driven insights in real time!</strong></p><h3>Next Steps :</h3><p><a href="https://medium.com/@official.indrajit.kar/building-a-scalable-text-to-sql-agentic-system-with-langchain-vector-db-and-multi-db-federated-5656e7115451">Blog 2</a> Part 2: Building a Scalable Text-to-SQL Agentic System with LangChain, Vector DB, and Multi DB Federated Queries</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50d087a41a41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2: Building a Scalable Text-to-SQL Agentic System with LangChain, Vector DB, and Multi DB…]]></title>
            <link>https://medium.com/@official.indrajit.kar/building-a-scalable-text-to-sql-agentic-system-with-langchain-vector-db-and-multi-db-federated-5656e7115451?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/5656e7115451</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[text-2-sql]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Fri, 07 Mar 2025 09:04:08 GMT</pubDate>
            <atom:updated>2025-03-07T10:21:34.829Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2: Building a Scalable Text-to-SQL Agentic System with LangChain, Vector DB, and Multi DB Federated Queries</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DGa8pg1vZ_z4UQKzCuqQNA.png" /><figcaption><strong>System Architecture</strong></figcaption></figure><h3>Introduction</h3><p>Refer to previous <a href="https://medium.com/@official.indrajit.kar/part-1-building-a-scalable-text-to-sql-agentic-system-with-langchain-vector-db-and-multi-db-50d087a41a41">blog</a> to understand the problem Statement</p><p>In this blog, we will walk through a <strong>complete implementation</strong> of a <strong>scalable Text-to-SQL system</strong> using <strong>LangChain for orchestration, a Vector DB (ChromaDB/FAISS) for schema matching &amp; row summarization, and a Federated Query Engine (Trino/Presto) for querying multiple databases efficiently.</strong></p><p>This system allows users to ask <strong>natural language queries</strong>, which are then:</p><ol><li><strong>Converted into embeddings</strong> for schema &amp; row retrieval.</li><li><strong>Pre-filtered using Vector DB</strong> to reduce search space.</li><li><strong>Mapped to the correct schema using metadata embeddings.</strong></li><li><strong>Converted into an optimized SQL query</strong> by LangChain.</li><li><strong>Executed across multiple databases</strong> in parallel using a Federated Query Engine.</li><li><strong>Stored in an In-Memory Database</strong> (DuckDB/Redis) for fast analytics &amp; AI inference.</li></ol><h3>System Architecture</h3><h3>Key Components</h3><p><strong>User Query</strong> → Converts natural language input into embeddings.<br><strong>Embedding Model</strong> → Embeds the query for semantic search.⃣ <strong>Vector DB</strong> → Stores <strong>table metadata</strong> and <strong>summarized row embeddings</strong>.<br><strong>Pre-Filtering</strong> → Reduces search space by retrieving relevant row identifiers.<br><strong>Schema Matching</strong> → Ensures the correct column mappings.<br><strong>LangChain (Orchestration)</strong> → Handles SQL generation, federated query execution, and caching.<br><strong>LLM (GPT-4, 70B Model)</strong> → Generates optimized SQL queries.<br><strong>Federated Query Execution</strong> → Runs SQL across multiple databases simultaneously.<br><strong>In-Memory DB</strong> → Stores query results for quick analytics.<br><strong>Data Visualization &amp; Machine Learning</strong> → Uses query results for dashboards &amp; AI models.</p><h3><strong>Step-by-Step Implementation</strong></h3><h3>Install Dependencies</h3><p>Before starting, install the necessary libraries:</p><pre>pip install langchain openai asyncpg psycopg2 faiss-cpu chromadb redis trino duckdb pandas</pre><h3>Convert User Query into Embeddings</h3><p>We first <strong>convert the user’s natural language query into an embedding</strong> using a <strong>Sentence Transformer</strong> model.</p><pre>from sentence_transformers import SentenceTransformer</pre><pre># Load embedding model<br>model = SentenceTransformer(&quot;all-MiniLM-L6-v2&quot;)</pre><pre># Function to convert user query into embedding<br>def embed_query(query):<br>    return model.encode(query)</pre><pre># Example user query<br>query = &quot;Find active users who joined last month and spent more than $1000.&quot;<br>query_embedding = embed_query(query)<br>print(query_embedding)  # Output: Embedding vector</pre><h3>Pre-Filtering: Retrieve Relevant Rows from Vector DB</h3><p>We <strong>search the Vector DB</strong> (ChromaDB/FAISS) for <strong>relevant row embeddings</strong> before running SQL queries.</p><pre>import chromadb</pre><pre># Initialize ChromaDB<br>client = chromadb.PersistentClient(path=&quot;./chroma_db&quot;)<br>row_collection = client.get_or_create_collection(&quot;row_data&quot;)</pre><pre># Function to retrieve relevant rows based on query embedding<br>def retrieve_relevant_rows(query):<br>    query_embedding = embed_query(query)<br>    results = row_collection.query(<br>        query_embeddings=[query_embedding.tolist()],<br>        n_results=5<br>    )<br>    return [r[&quot;summary&quot;] for r in results[&quot;metadatas&quot;][0]]</pre><pre># Example retrieval<br>filtered_rows = retrieve_relevant_rows(query)<br>print(filtered_rows)</pre><p><strong>This reduces the search space for SQL queries!</strong></p><h3>Schema Matching: Retrieve Correct Table &amp; Column Mappings</h3><p>We now <strong>search the Vector DB</strong> for <strong>matching table/column names</strong>.</p><pre># Initialize Vector DB for schema matching<br>schema_collection = client.get_or_create_collection(&quot;schema_metadata&quot;)</pre><pre># Function to match schema<br>def match_schema(query):<br>    query_embedding = embed_query(query)<br>    results = schema_collection.query(<br>        query_embeddings=[query_embedding.tolist()],<br>        n_results=1<br>    )<br>    return results[&quot;metadatas&quot;][0][&quot;column_name&quot;]</pre><pre># Example Schema Matching<br>print(match_schema(&quot;active users count&quot;))</pre><p><strong>Ensures SQL queries reference correct column names.</strong></p><h3>LangChain for SQL Generation &amp; Federated Query Execution</h3><p>We now <strong>use LangChain to orchestrate SQL generation and execution</strong>.</p><pre>from langchain.chat_models import ChatOpenAI<br>from langchain.tools import Tool<br>from langchain.agents import initialize_agent, AgentType</pre><pre># Initialize LLM<br>llm = ChatOpenAI(model=&quot;gpt-4-0613&quot;, temperature=0)</pre><pre># SQL function generator<br>def generate_sql_query(intent: str, table: str, columns: list, conditions: str = None):<br>    query = f&quot;SELECT {&#39;, &#39;.join(columns)} FROM {table}&quot;<br>    if conditions:<br>        query += f&quot; WHERE {conditions}&quot;<br>    return query + &quot;;&quot;</pre><pre># Register SQL tool in LangChain<br>sql_tool = Tool(<br>    name=&quot;SQL Generator&quot;,<br>    func=generate_sql_query,<br>    description=&quot;Generates SQL queries from user intent.&quot;<br>)</pre><pre># Create LangChain Agent<br>agent = initialize_agent(<br>    tools=[sql_tool],<br>    llm=llm,<br>    agent=AgentType.OPENAI_FUNCTIONS,<br>    verbose=True<br>)</pre><pre># Example Usage<br>generated_sql = agent.run(query)<br>print(generated_sql)  # Output: SQL Query</pre><p><strong>Now, LangChain dynamically generates SQL based on user input.</strong></p><h3>Execute SQL Federated Query Across Multiple Databases</h3><p>Now, we <strong>execute the generated SQL query across multiple databases using Trino/Presto</strong>.</p><pre>from trino.dbapi import connect</pre><pre># Function to execute federated query<br>def execute_federated_query(sql_query):<br>    conn = connect(<br>        host=&quot;trino-coordinator&quot;,<br>        port=8080,<br>        user=&quot;admin&quot;,<br>        catalog=&quot;postgresql_db&quot;,<br>        schema=&quot;public&quot;<br>    )<br>    cursor = conn.cursor()<br>    cursor.execute(sql_query)<br>    return cursor.fetchall()</pre><pre># Run federated query<br>results = execute_federated_query(generated_sql)<br>print(results)</pre><p><strong>Executes queries across multiple databases simultaneously!</strong></p><h3>Store Results in In-Memory DB (DuckDB)</h3><p>To avoid redundant queries, we <strong>store the results in DuckDB</strong>.</p><pre>import duckdb<br>import pandas as pd</pre><pre># Convert results to Pandas DataFrame<br>df = pd.DataFrame(results, columns=[&quot;user_id&quot;, &quot;first_name&quot;, &quot;last_name&quot;, &quot;total_spent&quot;])</pre><pre># Store in DuckDB for fast querying<br>conn = duckdb.connect(&quot;:memory:&quot;)<br>conn.register(&quot;federated_results&quot;, df)</pre><pre># Run fast in-memory queries<br>query_result = conn.execute(&quot;SELECT * FROM federated_results WHERE total_spent &gt; 5000&quot;).fetchdf()<br>print(query_result)</pre><p><strong>DuckDB provides ultra-fast analytics &amp; ML integration!</strong></p><h3>Benefits of the AI-Driven Text-to-SQL System</h3><p>This <strong>AI-powered Text-to-SQL system</strong> brings together <strong>LangChain, Vector Databases, Federated Query Engines, and In-Memory Processing</strong>, offering numerous benefits in <strong>speed, scalability, and accuracy</strong>.</p><h3>High-Performance Query Execution</h3><p><strong>Pre-Filtering with Vector DB</strong> → Reduces the search space before SQL execution, minimizing the need for full-table scans in PostgreSQL/MySQL.<br> <strong>Federated Query Execution</strong> → Runs SQL queries across multiple databases <strong>simultaneously</strong>, reducing query execution time.<br> <strong>In-Memory Processing (DuckDB/Redis)</strong> → Speeds up repeated queries by caching results.</p><p><strong>Example:</strong> Instead of scanning a <strong>245GB PostgreSQL database</strong>, <strong>pre-filtered rows are queried, improving performance by 10x.</strong></p><h3>Accurate Schema Matching &amp; Query Generation</h3><p><strong>Vector DB Schema Matching</strong> → Ensures correct table/column selection, <strong>preventing SQL errors</strong>.<br> <strong>LangChain Function Calling</strong> → Dynamically <strong>converts user input into optimized SQL queries</strong>.</p><p><strong>Example:</strong></p><blockquote><strong><em>User Query:</em></strong><em> “Find users who purchased red shoes in January.”<br> </em><strong><em>Without Schema Matching:</em></strong><em> The system may mistakenly reference </em><em>shoe_color instead of </em><em>product_color.<br> </em><strong><em>With Schema Matching:</em></strong><em> Corrects the query to </em><strong><em>match actual database column names</em></strong><em>.</em></blockquote><p><strong>Eliminates incorrect queries and ensures accurate results!</strong></p><h3>Scalability &amp; Multi-Database Support</h3><p><strong>Works with Multiple Databases</strong> → Supports <strong>PostgreSQL, MySQL, Snowflake, BigQuery, and MongoDB</strong>.<br><strong>Handles Large-Scale Datasets</strong> → Federated queries ensure the system is <strong>scalable for big data workloads</strong>.</p><p><strong>Example:</strong><br>A company using <strong>PostgreSQL for customers, MySQL for orders, and MongoDB for logs</strong> can retrieve unified insights <strong>without data migration</strong>.</p><p><strong>No need to centralize databases — query everything seamlessly!</strong></p><h3>Faster Query Execution with Caching &amp; Precomputed Views</h3><p><strong>Redis for Caching</strong> → Stores frequently queried results, reducing redundant database hits.<br> <strong>Materialized Views in PostgreSQL</strong> → Precomputes expensive queries, making analytics faster.</p><p><strong>Example:</strong><br> If <strong>“total active users per month”</strong> is queried frequently, the result is stored in <strong>Redis/DuckDB</strong> for fast retrieval, instead of recalculating every time.</p><p><strong>Reduces database load by up to 90%!</strong></p><h3>AI &amp; Machine Learning Integration</h3><p><strong>Stored Query Results for AI Models</strong> → Data can be used for <strong>predictive analytics, fraud detection, and recommendation systems</strong>.<br> <strong>In-Memory Processing (DuckDB/Arrow)</strong> → Allows <strong>fast training on precomputed query results</strong>.</p><p><strong>Example:</strong></p><ul><li>A machine learning model predicting <strong>customer churn</strong> can use the retrieved SQL data directly.</li><li>A <strong>real-time fraud detection system</strong> can analyze transactions from multiple databases <strong>in seconds</strong>.</li></ul><p><strong>This system supports AI-powered decision-making!</strong></p><h3>Real-Time Analytics &amp; Dynamic Dashboards</h3><p><strong>Query results can be visualized in real-time dashboards (Streamlit/Power BI).</strong><br> <strong>Supports complex aggregations &amp; joins across multiple databases.</strong></p><p><strong>Example:</strong><br> A financial firm tracking <strong>“high-value transactions from new users”</strong> can view <strong>live reports in a dashboard</strong> without waiting for batch processing.</p><p><strong>Ideal for real-time business intelligence!</strong></p><h3>Fully Automated Workflow with LangChain</h3><p><strong>Handles complete orchestration (query generation, execution, and caching).</strong><br><strong>Integrates seamlessly with AI Agents for continuous improvement.</strong></p><p><strong>Example:</strong></p><ul><li>Users <strong>ask questions in natural language</strong>, and the system <strong>automatically retrieves answers</strong>.</li><li>AI agents continuously <strong>optimize queries for better performance</strong>.</li></ul><p><strong>No need for SQL expertise — automates the entire process!</strong></p><h3>Why Choose This System Over Traditional BI &amp; SQL Approaches?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nqKV4g_p2KfhvhXv22lFAQ.png" /></figure><h3>Final Thoughts</h3><p>By integrating <strong>LangChain, Vector DB, and Federated Queries</strong>, this <strong>AI-driven Text-to-SQL system</strong> provides:</p><p><strong>Understands natural language queries</strong></p><p><strong>10x faster query execution</strong> through pre-filtering &amp; caching.<br><strong>Accurate SQL generation</strong> with metadata schema matching.<br> <strong>Seamless multi-database querying</strong> using a federated engine.<br> <strong>Support for AI &amp; real-time analytics</strong> with in-memory processing.</p><p><strong>Uses vector search for row pre-filtering</strong><br><strong>Performs schema matching for correct SQL generation</strong><br><strong>Executes federated queries across multiple databases</strong><br><strong>Stores results in an in-memory DB for fast analytics</strong></p><p><strong>This architecture is ideal for large-scale AI-driven analytics!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5656e7115451" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design Considerations of Advanced Agentic AI for Real world Applications]]></title>
            <link>https://medium.com/@official.indrajit.kar/design-considerations-of-advanced-agentic-ai-for-real-world-applications-c5ef317ba1e1?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5ef317ba1e1</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Sun, 17 Nov 2024 13:57:55 GMT</pubDate>
            <atom:updated>2024-11-17T14:23:30.362Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>F𝚛𝚘𝚖 𝙲𝚕𝚊𝚜𝚜 𝚋𝚊𝚜𝚎𝚍 𝙰𝚐𝚎𝚗𝚝𝚜 𝚝𝚘 𝚕𝚊𝚗g𝚌𝚑𝚊𝚒𝚗 𝙰𝚐𝚎𝚗𝚝s</strong></p><p>As Artificial Intelligence (AI) systems evolve, the concept of <strong>Agentic AI</strong> — where AI systems are composed of modular, task-specific agents working collaboratively — has become a cornerstone of scalable and adaptable AI solutions. This blog explores the design considerations behind Agentic AI, examining how agents, tools, memory, state, and planning come together to create intelligent workflows. We’ll compare three implementations — <strong>CODE1</strong>, <strong>CODE2</strong>, and <strong>CODE5</strong> — to shed light on the practicalities and complexities involved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmUitsAGa-VCjtPEdCzbDQ.png" /></figure><p><strong>Architecture 1</strong>, <strong>Architecture 2</strong>, and <strong>Architecture 3</strong>, mapped to <strong>CODE1</strong>, <strong>CODE2</strong>, and <strong>CODE5</strong> respectively.</p><p>The code can be found <a href="https://github.com/Karindraj/Agents/blob/main/Advanced%20Multi%20AGent%20system.ipynb">Here</a></p><h3><strong>Objective</strong></h3><p>The main objective of the provided code is to automate and streamline the extraction, transformation, and integration of structured data from unstructured textual reviews into a usable format for further analysis. Specifically, this workflow addresses the problem of entity extraction from customer reviews, associating extracted entities with existing structured datasets, and combining the two for comprehensive data analysis.</p><p>The problem being solved involves three key challenges:</p><ol><li><strong>Unstructured Data Handling</strong>: Customer reviews often contain valuable information buried in natural language text. Extracting structured data like customer names and purchase dates from such reviews is critical for enriching existing datasets.</li><li><strong>Data Integration</strong>: After extracting relevant information, integrating it with structured datasets (e.g., customer transaction records) ensures a unified, comprehensive view of data for analysis.</li><li><strong>Automated Metadata Management</strong>: The solution not only extracts and integrates data but also generates and stores metadata for both structured and unstructured datasets. This facilitates a deeper understanding of the data, such as its statistical properties and potential quality issues.</li></ol><h3>Tasks</h3><ul><li><strong>ColumnNameAgent</strong>: Parses column name and description pairs into a structured dictionary.</li><li><strong>ChainCreationAgent</strong>: Sets up a chain using LangChain and a local LLM to perform Named Entity Recognition (NER) on review texts.</li><li><strong>EntityExtractionAgent</strong>: Applies the chain to extract entities from unstructured text in a dataset.</li><li><strong>DataCombinationAgent</strong>: Combines the extracted entities with an existing dataset to create an enriched dataset.</li><li><strong>DatabaseAgent</strong>: Stores the merged data into an SQLite database for further access.</li><li><strong>MetadataExtractionAgent</strong>: Extracts metadata, such as column details and data statistics, for analysis.</li></ul><p>By leveraging advanced language models, database management, and structured workflows, this code enables businesses to automate and optimize their data processing pipelines, reducing manual effort and ensuring consistent, accurate data integration for decision-making.</p><h3>The Role of Global Agents in Agentic AI</h3><p>Global Agents are the central orchestrators in Agentic AI systems, ensuring smooth execution of workflows, coordinating agents or tools, and adapting dynamically to the state of the system. Below, we explore their role in <strong>CODE1</strong>, <strong>CODE2</strong>, and <strong>CODE5</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*QAp-E3afHxOBmEZDfiVwOA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*8u6wYJ9fHD-I4SWyBlC7WA.png" /><figcaption>Global Agents in <strong>CODE1</strong> focus on static, predictable workflows, while those in <strong>CODE2</strong> embrace dynamic adaptability. <strong>CODE5</strong> offers the best of both worlds, orchestrating structured agents while leveraging the flexibility of LangChain tools</figcaption></figure><h4>Orchestration</h4><p>The primary role of a Global Agent is to manage the sequence of task execution.</p><ul><li><strong>CODE1</strong>: The Global Agent explicitly orchestrates a predefined workflow by sequentially invoking agents like ColumnNameAgent, ChainCreationAgent, and others. Each agent is hardcoded to perform specific tasks in a linear fashion.</li><li><strong>CODE2</strong>: Instead of explicit orchestration, the Global Agent leverages LangChain’s reasoning-driven dynamic execution. Tools are invoked adaptively based on inputs, observations, and the custom prompt (ZERO_SHOT_REACT_DESCRIPTION).</li><li><strong>CODE5</strong>: Combines explicit orchestration for structured workflows with dynamic LangChain tools. The Global Agent coordinates both explicit agents and tools for hybrid workflow execution.</li></ul><h4>Monitoring</h4><p>Monitoring the progress and state of the workflow is essential for reliability and debugging.</p><ul><li><strong>CODE1</strong>: The Global Agent performs basic monitoring by tracking which agent is active and updating the state explicitly after each task.</li><li><strong>CODE2</strong>: Monitoring is less explicit, as LangChain’s agent inherently handles observation and reasoning between tasks. The Global Agent primarily ensures the dynamic flow of tool execution.</li><li><strong>CODE5</strong>: Integrates both explicit and implicit monitoring. Structured agents track progress in explicit workflows, while LangChain tools provide reasoning and feedback in adaptive workflows.</li></ul><h4>Error Handling</h4><p>Global Agents play a crucial role in handling errors and ensuring task completion.</p><ul><li><strong>CODE1</strong>: Basic error handling relies on retry logic embedded in the workflow. If an agent fails, the Global Agent may attempt to re-execute the task up to a specified limit.</li><li><strong>CODE2</strong>: LangChain’s memory and reasoning framework simplify error handling. Persistent state allows the Global Agent to retry tools seamlessly without requiring explicit logic.</li><li><strong>CODE5</strong>: Combines structured retry logic for explicit agents with LangChain’s robust error-handling for tools. This hybrid approach ensures resilience across static and dynamic workflows.</li></ul><h4>Adaptability</h4><p>A Global Agent determines the next step dynamically or based on pre-defined rules.</p><ul><li><strong>CODE1</strong>: Limited adaptability, as the workflow is hardcoded and follows a static sequence.</li><li><strong>CODE2</strong>: Fully adaptable through LangChain’s dynamic reasoning. The Global Agent selects tools based on context and custom prompts.</li><li><strong>CODE5</strong>: Partially adaptable. Structured workflows provide predictable behavior, while LangChain tools add flexibility to handle varied inputs.</li></ul><h3>The Role of Agents and Tools in Agentic AI Systems</h3><p>At the heart of Agentic AI systems are <strong>agents</strong> and <strong>tools</strong>, which modularize functionality to address specific tasks. Agents are independent entities capable of decision-making, whereas tools serve as functional utilities invoked by agents to execute well-defined tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*B8UPcAuKo9BzaX_N1xgSMg.png" /><figcaption>Agent and tool</figcaption></figure><h3>Agents and Tools Across the Codes</h3><p><strong>CODE1</strong>: Features <strong>6 class-based agents</strong>:</p><ul><li>ColumnNameAgent, ChainCreationAgent, EntityExtractionAgent, DataCombinationAgent, DatabaseAgent, and MetadataExtractionAgent.</li><li>These agents work in a predefined, sequential workflow.</li></ul><p><strong>CODE2</strong>: Introduces a <strong>LangChain-based approach</strong>:</p><ul><li><strong>5 tools</strong> replace explicit agents: ColumnNameExtraction, ChainCreation, EntityExtraction, DataCombination, and MetadataExtraction.</li><li>These tools function dynamically under the guidance of LangChain’s ZERO_SHOT_REACT_DESCRIPTION agent.</li></ul><p><strong>CODE5</strong>: Combines <strong>6 explicit agents</strong> and <strong>5 tools</strong>:</p><ul><li>Explicit agents handle workflow orchestration (execute_column_name_agent, etc.).</li><li>Tools are invoked dynamically for task execution, blending structured and adaptive approaches.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*WnBome-XicV1iXrJIiiPWA.png" /><figcaption>Here is the number of <strong>agents</strong> and <strong>tools</strong> in <strong>CODE1</strong>, <strong>CODE2</strong>, and <strong>CODE5</strong>, adesign choice</figcaption></figure><p>The choice between agents and tools often depends on the desired balance between modularity and adaptability.</p><h3>Custom Prompts and ‘ZERO_SHOT_REACT_DESCRIPTION’</h3><p>Custom prompts are essential for reasoning, enabling agents to follow the <strong>“Thought → Action → Action Input → Observation → Final Answer”</strong> framework. The ZERO_SHOT_REACT_DESCRIPTION approach in LangChain leverages this framework to dynamically decide which tool to invoke, based on input context and observations. This allows the system to adapt flexibly to diverse scenarios without retraining.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*0Pkg_QXhUYx3evLM8RyYeQ.png" /><figcaption>Comparison of the usage of <strong>“ZERO_SHOT_REACT_DESCRIPTION”</strong> in <strong>CODE1</strong>, <strong>CODE2</strong>, and <strong>CODE5</strong></figcaption></figure><ul><li><strong>CODE1</strong>: Does not use ZERO_SHOT_REACT_DESCRIPTION and relies on static workflows.</li><li><strong>CODE2</strong>: Fully leverages ZERO_SHOT_REACT_DESCRIPTION for dynamic and reasoning-driven tool selection.</li><li><strong>CODE5</strong>: Combines the best of both worlds, using ZERO_SHOT_REACT_DESCRIPTION for tools while maintaining explicit agent-based workflows.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*P8zLehjDJcHCzP37PnCpvA.png" /></figure><h3>State Management</h3><p>Agents and tools rely on a shared state to track progress, intermediate results, and workflow context. In structured systems like <strong>CODE1</strong>, the state is explicitly defined (e.g., GraphState) and manually updated. Dynamic systems like <strong>CODE2</strong> and <strong>CODE5</strong> use LangChain&#39;s StatefulMemory to handle state implicitly during tool execution.</p><h3>Memory Class</h3><p>A dedicated memory class (e.g., StatefulMemory) provides a persistent and centralized way to store, update, and retrieve state variables. While <strong>CODE1</strong> lacks this, <strong>CODE2</strong> and <strong>CODE5</strong> use StatefulMemory for dynamic updates, ensuring state is consistently maintained across retries and workflows.</p><h3>State Variables</h3><p>Explicit variables in structured systems track predefined workflows (e.g., column_names, chain, metadata in <strong>CODE1</strong>). Dynamic systems allow tools to update variables based on observations, adapting to changing inputs.</p><h3>Dynamicity of State</h3><p>Structured agents follow fixed state transitions, while tools in dynamic systems interact with the state adaptively, as seen in <strong>CODE2</strong> and <strong>CODE5</strong>.</p><h3>Persistence</h3><p>Memory ensures state is persisted beyond tool execution, enabling retries and error recovery.</p><h3>Error Handling</h3><p>Dynamic state systems inherently manage errors better, as persistent memory and retries maintain workflow integrity, while static systems require explicit error-handling logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*QljQDEvsObGe_GXIyguKiw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*GxoNPp5gYWzrCGL7WgkybA.png" /></figure><h3>Actions and Planning in Agentic AI</h3><p>In Agentic AI, <strong>Actions</strong> and <strong>Planning Mechanisms</strong> are the fundamental elements that drive task execution and optimize workflows, making the system efficient, adaptive, and resilient. Let’s explore these in more detail with examples from the codes.</p><h3>Actions: The Core of Agent Behavior</h3><p><strong>What Are Actions?</strong></p><ul><li>Actions are the atomic units of an agent’s behavior. They represent specific operations, such as invoking a tool, running a method, or executing a predefined task.</li><li>Actions bridge decision-making (reasoning by agents) and execution (invoking tools or methods).</li></ul><p><strong>Key Characteristics</strong>:</p><ol><li><strong>Task-Specific</strong>: Actions are designed to handle specific tasks like extracting entities, combining data, or creating chains.</li><li><strong>Dynamic Execution</strong>: In systems like CODE2 and CODE5, actions are dynamically chosen based on the system’s reasoning or state.</li><li><strong>Tool Invocation</strong>: Actions often involve invoking tools, such as the ChainCreation tool, which constructs a Named Entity Recognition (NER) pipeline in CODE2.</li></ol><p><strong>Example from CODE2</strong>:</p><ul><li><strong>Thought</strong>: “I need to create an NER chain for the column names.”</li><li><strong>Action</strong>: Invoking the ChainCreation tool.</li><li><strong>Action Input</strong>: A list of column names with descriptions, such as {&quot;CustomerName&quot;: &quot;&lt;Name of customer&gt;&quot;, &quot;PurchaseDate&quot;: &quot;&lt;Date of purchase&gt;&quot;}.</li><li><strong>Observation</strong>: A successfully created NER chain ready for use.</li></ul><p>This action-based modularity allows flexibility, as agents can focus on “deciding what to do” while tools perform the actual tasks.</p><h3>Planning Mechanisms: Optimizing Workflows</h3><p><strong>What Is Planning in Agentic AI?</strong> Planning refers to the process of sequencing and organizing actions to ensure efficient task execution. Effective planning ensures that agents execute the right actions in the right order, handle errors gracefully, and adapt to changing scenarios.</p><p><strong>Key Goals of Planning</strong>:</p><p><strong>Sequencing for Efficiency</strong>:</p><ul><li>Planning ensures actions are executed in a logical sequence to minimize redundant computations.</li><li>Example: In CODE1, the DataCombinationAgent logically follows the EntityExtractionAgent because it depends on the extracted data.</li></ul><p><strong>Error Recovery</strong>:</p><ul><li>A well-planned system anticipates potential errors and includes retries or alternative paths.</li><li>Example: In CODE5, retries are incorporated both for explicit agents (via manual logic) and tools (via LangChain memory).</li></ul><p><strong>Blending Structured and Adaptive Planning</strong>:</p><ul><li>Structured Planning: Predefined workflows, as seen in CODE1, where the sequence of agents is hardcoded.</li><li>Adaptive Planning: Dynamic decision-making, as seen in CODE2 and CODE5, where actions are chosen based on reasoning.</li><li>Hybrid Planning: CODE5 exemplifies this by using structured workflows for explicit agents and adaptive planning for tools.</li></ul><h3>Example of Planning in CODE5</h3><p><strong>Structured Workflow</strong>:</p><ul><li>The Global Agent invokes explicit agents (execute_column_name_agent, execute_chain_creation_agent, etc.) in a fixed sequence.</li><li>This ensures predictable execution for tasks with dependencies.</li></ul><p><strong>Dynamic Planning</strong>:</p><ul><li>For tools like ChainCreation, planning dynamically selects the appropriate tool based on state and reasoning.</li><li>Example: If the system encounters unexpected data, it might dynamically adjust to invoke MetadataExtraction for additional context.</li></ul><p><strong>Error Handling in Planning</strong>:</p><ul><li>If a task like EntityExtraction fails, the system retries using LangChain’s memory or predefined retry logic.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*FP_PlCtUtV9lSCt9dQh4mg.png" /></figure><h3>Comparing CODE1, CODE2, and CODE5</h3><h3>CODE1: A Structured Approach</h3><ul><li><strong>Design</strong>: Relies on <strong>class-based agents</strong> and a predefined workflow orchestrated by a <strong>Global Agent</strong>.</li><li><strong>Strengths</strong>:</li><li>Transparent and easy to debug.</li><li>Suitable for static workflows with predictable task sequences.</li><li><strong>Limitations</strong>:</li><li>Lacks dynamic adaptability.</li><li>No integration with LangChain for reasoning-based tool selection.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*FP_PlCtUtV9lSCt9dQh4mg.png" /></figure><h3>CODE2: Dynamic Flexibility with LangChain</h3><ul><li><strong>Design</strong>: Uses LangChain’s tools and memory to enable a dynamic, reasoning-driven approach.</li><li><strong>Strengths</strong>:</li><li>Highly flexible, capable of selecting tools dynamically based on inputs and state.</li><li>Employs <strong>custom prompts</strong> for reasoning, leveraging the ZERO_SHOT_REACT_DESCRIPTION agent type.</li><li><strong>Limitations</strong>:</li><li>Heavily dependent on LangChain.</li><li>Slightly abstract, making debugging more complex compared to CODE1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*R5kyDbE7vu_lL_sVHwgshA.png" /></figure><h3>CODE5: Hybrid Design for Scalability</h3><ul><li><strong>Design</strong>: Combines the structured approach of CODE1 with the dynamic flexibility of CODE2.</li><li><strong>Strengths</strong>:</li><li>Explicit agents ensure structured workflows, while LangChain tools offer adaptability.</li><li>Suitable for scalable systems requiring both transparency and flexibility.</li><li><strong>Limitations</strong>:</li><li>Complexity in blending static and dynamic components.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*CcgOyzbu6qgmAJl_1lHzvQ.png" /></figure><p><strong>CODE1</strong>:</p><ul><li>Implements all tasks as explicit class-based agents, executed sequentially via a predefined workflow.</li><li>Agents directly update the shared GraphState.</li></ul><p><strong>CODE2</strong>:</p><ul><li>All tasks are converted into <strong>LangChain tools</strong>, invoked dynamically using the ZERO_SHOT_REACT_DESCRIPTION reasoning framework.</li><li>Relies on memory for maintaining state and seamless tool invocation.</li></ul><p><strong>CODE5</strong>:</p><ul><li>Uses explicit agents for workflow orchestration while relying on LangChain tools for task execution.</li><li>Combines structured workflows from CODE1 and dynamic adaptability from CODE2.</li></ul><p>This table highlights the evolution from the static, structured approach of <strong>CODE1</strong> to the fully dynamic, tool-centric implementation in <strong>CODE2</strong>, and finally, the hybrid model in <strong>CODE5</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*WcN6h2qO7nws2sTI7CgDqA.png" /></figure><h3>Converting Class-Based Agents to LangChain Tools</h3><p>Class-based agents, as seen in structured systems like <strong>CODE1</strong>, are explicitly defined entities that perform specific tasks sequentially. They provide clear, modular logic for workflows, making them ideal for static systems with predictable task flows. However, this approach requires manual updates to state and explicit orchestration, limiting adaptability. Converting these agents into LangChain tools involves modularizing their functionality and leveraging LangChain’s dynamic execution framework. Each task becomes a callable tool with a defined input-output schema, enabling seamless invocation during runtime. For example, a ColumnNameAgent can be converted into a ColumnNameExtraction tool that takes column descriptors as input and returns a structured dictionary.</p><p>LangChain tools add flexibility by integrating with <strong>reasoning agents</strong> like ZERO_SHOT_REACT_DESCRIPTION, which dynamically select and execute tools based on input context and state. These reasoning agents replace the need for hardcoded workflows, enabling dynamic planning and execution. Additionally, LangChain’s memory (e.g., StatefulMemory) simplifies state management, ensuring persistence and consistency without explicit updates. The resulting system, as seen in <strong>CODE2</strong> and <strong>CODE5</strong>, balances modularity and adaptability. By transitioning from class-based agents to LangChain tools and reasoning agents, workflows become more scalable and robust, capable of handling varied and complex inputs with minimal manual intervention.</p><h3>Roles of Structured, Unstructured, and Vector Databases in an Agentic AI System</h3><p>In the context of the diagram and workflow, each type of database plays a distinct but interconnected role:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nizawk5n8yumdqpYzxM4nA.png" /><figcaption>End to End implemenation with LLM</figcaption></figure><h3>1. Structured Data</h3><p><strong>Definition</strong>:</p><ul><li><strong>Structured data</strong> is highly organized and stored in tabular formats such as relational databases (e.g., PostgreSQL, MySQL). Examples include customer records, transaction logs, and inventory tables.</li></ul><p><strong>Role</strong>:</p><ul><li><strong>Query and Retrieval</strong>: The <strong>Global Agent</strong> accesses structured databases to retrieve pre-organized data for tasks like matching, aggregation, or reporting.</li><li><strong>Baseline Knowledge</strong>: Acts as the foundational layer for deterministic queries like “Which customer purchased a specific product?” or “What is the total sales volume?”</li><li><strong>Integration</strong>: This data can feed directly into downstream tasks like merging with extracted entities or being used as reference data during LLM processing.</li></ul><p><strong>Example Use</strong>:</p><ul><li>A structured database might store customer purchase histories (CustomerName, PurchaseDate), which are retrieved and combined with extracted unstructured data for a comprehensive analysis.</li></ul><h3>2. Unstructured Data</h3><p><strong>Definition</strong>:</p><ul><li><strong>Unstructured data</strong> includes textual, image, or audio data stored in document-based or NoSQL databases like Elasticsearch, MongoDB, or file repositories. Examples include customer reviews, emails, and social media posts.</li></ul><p><strong>Role</strong>:</p><ul><li><strong>Input for LLM</strong>: Acts as raw input for the <strong>LLM</strong>, which processes the data for tasks like summarization, Named Entity Recognition (NER), or sentiment analysis.</li><li><strong>Entity Extraction</strong>: The extracted entities (e.g., names, dates, or sentiments) can be structured and merged with the structured database for enriched insights.</li><li><strong>Search and Retrieval</strong>: Provides full-text search capabilities to locate specific documents or phrases that are contextually relevant for the task.</li></ul><p><strong>Example Use</strong>:</p><ul><li>A database of customer reviews is fed into the <strong>LLM</strong>, which extracts structured entities like CustomerName and Sentiment for further processing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*XCcn8QIzeuQ_7567uyftzg.png" /></figure><h3>3. Vector Databases</h3><p><strong>Definition</strong>:</p><ul><li>A <strong>Vector Database</strong> (e.g., Pinecone, Weaviate, or Milvus) stores vectorized representations (embeddings) of data, enabling similarity searches and semantic queries. Examples include embeddings of product descriptions, customer reviews, or FAQ documents.</li></ul><p><strong>Role</strong>:</p><ul><li><strong>Memory for Context Retrieval</strong>: Works as an external memory, allowing agents to retrieve relevant context by performing similarity searches on vectorized data.</li><li><strong>Semantic Matching</strong>: Facilitates advanced queries like “Find reviews similar to this one” or “Retrieve similar transactions for this customer.”</li><li><strong>Integration</strong>: Often complements unstructured data by storing its embeddings, enabling quick lookups during reasoning tasks.</li></ul><p><strong>Example Use</strong>:</p><ul><li>Customer reviews are vectorized, stored in a vector database, and retrieved during LLM queries to provide contextual information for a specific customer or sentiment pattern.</li></ul><h3>Comparing Roles</h3><p><strong>TypeStoragePrimary RoleExampleStructured Data</strong>Relational Databases (SQL)Query precise, pre-organized information.Retrieve all purchases by a customer from Customer and Orders tables for analysis.<strong>Unstructured Data</strong>Document-based (NoSQL, Text)Provide raw inputs for LLM or full-text search capabilities.Analyze customer reviews stored in a NoSQL database to extract insights like sentiment or named entities.<strong>Vector Data</strong>Vectorized Embedding StorageSemantic retrieval and similarity searches.Retrieve reviews with similar sentiment or tone by performing a cosine similarity search on vector embeddings stored in the vector database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*WuAbsY2WyV3ujrP5akOXOw.png" /></figure><h3>How They Work Together</h3><p><strong>Input</strong>:</p><ul><li><strong>Structured Data</strong>: Provides the core reference or relational data.</li><li><strong>Unstructured Data</strong>: Supplies rich, free-form content that complements structured data.</li></ul><p><strong>Processing</strong>:</p><ul><li><strong>Unstructured Data</strong> is passed through the <strong>LLM</strong> for feature extraction or semantic understanding.</li><li>Extracted entities are merged with structured data for enriched datasets.</li></ul><p><strong>Retrieval</strong>:</p><ul><li><strong>Vector Database</strong> enables advanced retrieval by storing embeddings, making contextual and similarity-based searches possible.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*DdPb4CABWa5yOquvwvwZng.png" /></figure><h3>Role of LLMs in Architectures 1, 2, and 3 (formerly CODE1, CODE2, CODE5)</h3><p>Last but not the least, Large Language Models (LLMs) like L<strong>lama</strong> play a pivotal role in enabling reasoning, adaptability, and automation across Architectures 1, 2, and 3. Their integration enhances the ability to process natural language inputs, extract insights, and make decisions dynamically.</p><p><strong>Architecture 1 (CODE1)</strong>:</p><ul><li>The LLM is used explicitly within agents like ChainCreationAgent to set up a Named Entity Recognition (NER) pipeline.</li><li>Agents rely on predefined prompts and workflows to invoke the LLM for extracting entities from unstructured data such as review texts.</li><li>While the LLM performs well in task execution, the integration is rigid, with limited adaptability to changing contexts.</li></ul><p><strong>Architecture 2 (CODE2)</strong>:</p><ul><li>The LLM is leveraged dynamically through LangChain tools like ChainCreation and EntityExtraction.</li><li>Using <strong>ZERO_SHOT_REACT_DESCRIPTION</strong>, the LLM reasons about the task and decides which tool to invoke, adapting to various inputs.</li><li>This approach maximizes flexibility and allows the system to handle unforeseen scenarios without predefined workflows.</li></ul><p><strong>Architecture 3 (CODE5)</strong>:</p><ul><li>Combines explicit agents with tools powered by LLMs, achieving a hybrid model.</li><li>LLMs are dynamically invoked by tools and explicitly integrated into agents for structured workflows, ensuring scalability and flexibility.</li></ul><p>Across all architectures, LLMs empower the system to process natural language, extract insights, and dynamically adapt to real-world tasks.</p><h3>Conclusion</h3><p>Designing robust Agentic AI systems requires balancing structure, adaptability, and scalability to meet the demands of modern workflows. <strong>CODE1</strong>, <strong>CODE2</strong>, and <strong>CODE5</strong> highlight different approaches to implementing these systems, showcasing how agents, tools, memory, and planning mechanisms can be tailored to diverse scenarios.</p><p><strong>CODE1</strong> demonstrates the value of structured workflows using explicit agents. This approach is transparent, predictable, and easy to debug, making it ideal for tasks with well-defined sequences. However, its lack of adaptability limits its application in dynamic environments.</p><p><strong>CODE2</strong>, on the other hand, embraces flexibility and reasoning. By leveraging LangChain’s tools and ZERO_SHOT_REACT_DESCRIPTION, it dynamically selects tools based on context, enabling the system to adapt to varying inputs and scenarios. This adaptability, while powerful, can make debugging more complex due to its abstracted logic.</p><p><strong>CODE5</strong> combines the strengths of both approaches, creating a hybrid system. It uses explicit agents for structured tasks and LangChain tools for dynamic adaptability, providing scalability and transparency in complex workflows.</p><p>As Agentic AI continues to evolve, integrating structured planning, dynamic reasoning, and efficient state management will remain key. Systems like <strong>CODE5</strong> offer a glimpse into how these principles can work together to create intelligent, scalable, and future-ready AI solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5ef317ba1e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Performance Deep learning Part 3:Custom Kernel Optimization for PyTorch, Exploring CuDNN and…]]></title>
            <link>https://medium.com/@official.indrajit.kar/high-performance-deep-learning-part-3-custom-kernel-optimization-for-pytorch-exploring-cudnn-and-95d99ed1422b?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/95d99ed1422b</guid>
            <category><![CDATA[cudnn]]></category>
            <category><![CDATA[kernel]]></category>
            <category><![CDATA[triton]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Mon, 28 Oct 2024 21:18:40 GMT</pubDate>
            <atom:updated>2024-12-03T15:15:27.208Z</atom:updated>
            <content:encoded><![CDATA[<h3>High Performance Deep learning Part 3:Custom Kernel Optimization for PyTorch, Exploring CuDNN and Triton</h3><p><strong>By Indrajit Kar and Zonunfeli Ralte</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3JH0wpQMk2uyOEruLIx5Fg.png" /></figure><h3>Introduction</h3><p>In today’s world of AI, where models are continuously growing in size and complexity, the demand for efficient GPU utilization has reached an all-time high. From natural language processing to computer vision, the computational requirements of deep learning have put unprecedented strain on available hardware resources. Efficient GPU processing is more crucial than ever for developers and researchers, as it directly affects the speed of innovation and the practicality of AI solutions in real-world applications.</p><p>One of the key components of GPU efficiency is the kernel used to perform basic operations like matrix multiplication, convolution, and activation functions. Kernels serve as the underlying computational engines in deep learning frameworks, providing optimized routines for processing large datasets in parallel. However, not all kernels are created equal. CuDNN, NVIDIA’s deep learning library, is the default in frameworks like PyTorch, offering highly optimized kernels for standard operations. On the other hand, Triton — a more recent open-source approach — offers developers the flexibility to create custom kernels, potentially unlocking even greater performance for specific tasks but at the cost of additional programming complexity.</p><p>This blog post compares two implementations of a Convolutional Neural Network (CNN) to explore the performance impacts of using different kernel backends. <strong>Code 1</strong> uses PyTorch’s standard operations with CuDNN acceleration, while <strong>Code 2</strong> leverages Triton kernels for custom GPU operations. Both implementations are designed to train and evaluate an MNIST digit classifier but differ significantly in how core operations are executed. By examining these differences, we aim to shed light on the potential trade-offs between pre-built and custom kernels in the GPU-hungry AI landscape.</p><p>The codes for this Blog can be found <a href="https://github.com/Feli4AI/High-Performance-Deep-learning/blob/main/Triton.ipynb">here</a></p><p><strong>Kernels</strong></p><p>In computing, a <strong>kernel</strong> is a function or small program designed to execute a specific computation on a processing unit, like a CPU or GPU. In the context of GPUs, a kernel is executed by thousands of threads in parallel, each processing data in a highly efficient, parallel manner. Kernels are central to GPU programming in frameworks like CUDA, Triton, and OpenCL, where they handle tasks such as matrix multiplication, image processing, and neural network operations. They allow developers to offload computationally intensive tasks to GPUs, exploiting their massive parallelism for faster execution on large datasets.</p><p><strong>Triton</strong></p><p>Triton is a programming language and compiler for writing high-performance GPU code, particularly for deep learning applications. It relies on the LLVM (Low Level Virtual Machine) compiler infrastructure to generate optimized machine code for GPUs. When you write Triton kernels in Python, Triton’s compiler transforms this code into an intermediate representation (IR). This IR is then lowered to LLVM’s IR, where LLVM’s suite of optimization passes enhances the code. Finally, LLVM’s backend targets generate the machine code specific to the GPU architecture. By leveraging LLVM, Triton benefits from its robust optimization capabilities and hardware support, enabling efficient execution of custom GPU kernels.</p><p><strong>About LLVM</strong></p><p>LLVM (Low Level Virtual Machine) is an open-source compiler infrastructure designed to optimize and generate machine code for various hardware architectures. It originally started as a way to create a “virtual machine” for executing programs in a low-level, language-agnostic form. Now, LLVM is widely used for building compilers, thanks to its modular, flexible design.</p><p>Key components of LLVM include:</p><ol><li><strong>Intermediate Representation (IR)</strong>: A platform-independent, low-level code format that allows optimizations and transformations before converting it to machine code.</li><li><strong>Optimization Passes</strong>: A series of transformations that improve performance and efficiency (e.g., inlining, loop unrolling).</li><li><strong>Backend Code Generators</strong>: LLVM can target many architectures (x86, ARM, GPUs) by converting IR into optimized assembly or machine code for each hardware.</li></ol><p>LLVM powers many language compilers, like Clang (C/C++) and Rust, and is widely used in GPUs and hardware-specific tools, such as Triton.</p><p><strong>CuDNN</strong></p><p>cuDNN (CUDA Deep Neural Network library) does not use LLVM. cuDNN is a highly optimized library developed by NVIDIA specifically for accelerating deep learning operations on NVIDIA GPUs. It is built directly on CUDA, NVIDIA’s parallel computing platform and application programming interface (API) for GPU computing.</p><p>cuDNN’s optimizations are tailored to deep learning primitives (like convolutions, pooling, and activation functions) and leverage CUDA-specific optimizations and low-level GPU instructions directly, bypassing the need for LLVM’s intermediate representation and general-purpose compilation process. Therefore, cuDNN operates closer to the hardware through CUDA, without relying on LLVM for code generation or optimization.</p><h3>Table of Contents</h3><ol><li><a href="#introduction">Introduction</a></li><li><a href="#kernels">Kernels</a></li><li><a href="#triton">Triton</a></li><li><a href="#llvm">LLVM</a></li><li><a href="#cudnn">CuDNN</a></li><li><a href="#comparison-of-pytorch-and-triton-implementations-for-cnns">Comparison of PyTorch and Triton Implementations for CNNs</a></li></ol><p><a href="#code-1-pytorch-cnn-model">Code 1: PyTorch CNN Model</a></p><ul><li><a href="#model-architecture">Model Architecture</a></li><li><a href="#data-loading-and-preprocessing">Data Loading and Preprocessing</a></li><li><a href="#training-loop">Training Loop</a></li></ul><p><a href="#code-2-cnn-with-triton-kernels">Code 2: CNN with Triton Kernels</a></p><ul><li><a href="#using-triton-for-relu-activation">Using Triton for ReLU Activation</a></li><li><a href="#custom-triton-2d-convolution-kernel">Custom Triton 2D Convolution Kernel</a></li><li><a href="#linear-layers-and-forward-pass">Linear Layers and Forward Pass</a></li><li><a href="#training-loop-triton">Training Loop</a></li></ul><p><a href="#cudnn-vs-triton-kernels">7. CuDNN vs. Triton Kernels</a></p><ul><li><a href="#performance-and-efficiency">Performance and Efficiency</a></li><li><a href="#flexibility-and-optimization">Flexibility and Optimization</a></li></ul><p><a href="#conclusion">8. Conclusion</a></p><h3>Comparison of PyTorch and Triton Implementations for CNNs</h3><h3>Code 1: PyTorch CNN Model</h3><h3>Model Architecture</h3><p>Our first implementation is a basic CNN model using PyTorch. This model is structured with two convolutional layers, each followed by a ReLU activation function and a flattening layer before moving into fully connected layers.</p><pre>import torch<br>import torch.nn as nn</pre><pre>class CNNModel(nn.Module):<br>    def __init__(self):<br>        super(CNNModel, self).__init__()<br>        self.conv1 = nn.Conv2d(1, 16, 3, stride=1)<br>        self.relu1 = nn.ReLU()<br>        self.conv2 = nn.Conv2d(16, 32, 3, stride=1)<br>        self.relu2 = nn.ReLU()<br>        <br>        # Flattened size calculation after convolutions<br>        self.flattened_size = self._get_flattened_size()<br>        self.fc1 = nn.Linear(self.flattened_size, 128)<br>        self.relu3 = nn.ReLU()<br>        self.fc2 = nn.Linear(128, 10)<br>    <br>    def _get_flattened_size(self):<br>        with torch.no_grad():<br>            x = torch.zeros(1, 1, 28, 28)<br>            x = self.conv1(x)<br>            x = self.conv2(self.relu1(x))<br>            return x.numel()</pre><pre>    def forward(self, x):<br>        x = self.conv1(x)<br>        x = self.relu1(x)<br>        x = self.conv2(x)<br>        x = self.relu2(x)<br>        x = x.view(x.size(0), -1)<br>        x = self.fc1(x)<br>        x = self.relu3(x)<br>        x = self.fc2(x)<br>        return x</pre><h3>Data Loading and Preprocessing</h3><p>The model uses the MNIST dataset for training and testing. The data is normalized to improve convergence.</p><pre>from torchvision import datasets, transforms<br>from torch.utils.data import DataLoader</pre><pre>transform = transforms.Compose([<br>    transforms.ToTensor(),<br>    transforms.Normalize((0.1307,), (0.3081,))<br>])</pre><pre>train_dataset = datasets.MNIST(&#39;./data&#39;, train=True, transform=transform, download=True)<br>test_dataset = datasets.MNIST(&#39;./data&#39;, train=False, transform=transform)<br>train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)<br>test_loader = DataLoader(test_dataset, batch_size=1000, shuffle=False)</pre><h3>Training Loop</h3><p>Using stochastic gradient descent (SGD), we train the model over several epochs and log the loss and accuracy.</p><pre>import torch.optim as optim</pre><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>model = CNNModel().to(device)<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.SGD(model.parameters(), lr=0.001)</pre><pre>for epoch in range(1):<br>    model.train()<br>    for data, target in train_loader:<br>        data, target = data.to(device), target.to(device)<br>        optimizer.zero_grad()<br>        output = model(data)<br>        loss = criterion(output, target)<br>        loss.backward()<br>        optimizer.step()</pre><h3>Code 2: CNN with Triton Kernels</h3><h3>Using Triton for ReLU Activation</h3><p>In Code 2, we replace the default ReLU activation with a custom Triton kernel, allowing for more low-level control.</p><pre>import triton<br>import triton.language as tl</pre><pre>@triton.jit<br>def relu_kernel(x_ptr, y_ptr, N, BLOCK_SIZE: tl.constexpr):<br>    idx = tl.program_id(0) * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)<br>    mask = idx &lt; N<br>    x = tl.load(x_ptr + idx, mask=mask)<br>    y = tl.where(x &gt; 0, x, 0)<br>    tl.store(y_ptr + idx, y, mask=mask)</pre><pre>def triton_relu(x):<br>    y = torch.empty_like(x)<br>    N = x.numel()<br>    BLOCK_SIZE = 1024<br>    relu_kernel[(N + BLOCK_SIZE - 1) // BLOCK_SIZE](x, y, N, BLOCK_SIZE=BLOCK_SIZE)<br>    return y</pre><h3>Custom Triton 2D Convolution Kernel</h3><p>The Triton convolution kernel is similarly implemented to handle 2D convolution directly, offering further performance tuning for input/output sizes and strides.</p><pre>@triton.jit<br>def conv2d_kernel(input_ptr, weight_ptr, output_ptr, H, W, KH, KW, OH, OW, stride_h, stride_w, padding_h, padding_w, c_out: tl.constexpr, C_in: tl.constexpr, BLOCK_H: tl.constexpr, BLOCK_W: tl.constexpr):<br>    # Compute the block indices<br>    pid_h = tl.program_id(0)<br>    pid_w = tl.program_id(1)<br>    h_out = pid_h * BLOCK_H + tl.arange(0, BLOCK_H)<br>    w_out = pid_w * BLOCK_W + tl.arange(0, BLOCK_W)<br>    acc = tl.zeros((BLOCK_H, BLOCK_W), dtype=tl.float32)<br>    # Convolution loop logic<br>    out_ptrs = output_ptr + c_out * OH * OW + h_out * OW + w_out<br>    tl.store(out_ptrs, acc)</pre><h3>Linear Layers and Forward Pass</h3><p>To handle linear layers, we utilize Triton’s matrix multiplication capabilities to mimic PyTorch’s fully connected layers.</p><pre>def triton_linear(x, weight, bias):<br>    return x @ weight + bias</pre><pre>def forward_pass(x, conv1_weight, conv2_weight, fc1_weight, fc1_bias, fc2_weight, fc2_bias):<br>    x = triton_conv2d(x, conv1_weight)<br>    x = triton_relu(x)<br>    x = triton_conv2d(x, conv2_weight)<br>    x = triton_relu(x)<br>    x = x.view(x.shape[0], -1)<br>    x = triton_linear(x, fc1_weight, fc1_bias)<br>    return x</pre><h3>CuDNN vs. Triton Kernels</h3><h3>Performance and Efficiency</h3><p>CuDNN, widely used by PyTorch, offers optimized and pre-tuned GPU kernels, ideal for most CNN operations. However, Triton offers kernel customization for specific needs, potentially achieving performance gains in particular use cases by controlling memory access patterns and leveraging block-based calculations.</p><h3>Flexibility and Optimization</h3><p>While CuDNN is a black-box solution, Triton allows users to customize kernels, which may lead to more efficient implementations for specific architectures or hardware setups. Triton kernels are ideal for fine-tuning models on hardware beyond what standard libraries cover.</p><h3>Conclusion</h3><p>In conclusion, both CuDNN and Triton offer distinct advantages depending on the needs of your project. CuDNN is a robust choice for high-performance, out-of-the-box CNN operations, leveraging deeply optimized and standardized kernels for reliable results on a wide range of hardware. On the other hand, Triton provides developers with the flexibility to fine-tune kernels for specific tasks, offering a unique edge in performance customization and adaptability. While CuDNN is ideal for most conventional deep learning tasks, Triton shines in scenarios where granular control over memory access patterns and computation flow can yield enhanced performance. Ultimately, the choice between CuDNN and Triton depends on whether efficiency or kernel customization is the priority.</p><p><strong>Check out </strong>our last Blog</p><p>High Performance Deep learning Part 4: Optimizing Deep Learning Training with PyTorch, PyTorch Lightning, and Triton</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95d99ed1422b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Performance Deep learning Part 4: Optimizing Deep Learning Training with PyTorch, PyTorch…]]></title>
            <link>https://medium.com/@official.indrajit.kar/optimizing-deep-learning-training-with-pytorch-pytorch-lightning-and-triton-174bb3e26593?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/174bb3e26593</guid>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[pytorch-lightning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Sun, 27 Oct 2024 13:03:29 GMT</pubDate>
            <atom:updated>2024-10-28T21:19:40.524Z</atom:updated>
            <content:encoded><![CDATA[<h3>High Performance Deep learning Part 4: Optimizing Deep Learning Training with PyTorch, PyTorch Lightning, and Triton</h3><h3><strong>By Indrajit Kar and Zonunfeli Ralte</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z10_vF3OFVLUXjIAiZa1NQ.jpeg" /></figure><p>The code for the Blog can be found <a href="https://github.com/Feli4AI/High-Performance-Deep-learning/blob/main/Quick_and_Dirty_HPC.ipynb">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QVAQTzOeiM7ZECq7" /></figure><p>Deep learning and large language models (LLMs) are transforming industries, enabling breakthroughs in natural language understanding, image recognition, and beyond. However, these models are resource-intensive, requiring significant computational power to train and deploy. GPUs have become essential for handling this demand, providing the parallel processing capabilities necessary to accelerate deep learning tasks. But to fully harness GPU power, it’s crucial to use optimized frameworks and techniques that maximize efficiency and streamline model development.</p><p>Standard deep learning frameworks like PyTorch offer the flexibility to build and train neural networks, but they often come with substantial boilerplate code and can be challenging to optimize at scale. This is where tools like <strong>PyTorch Lightning</strong> and <strong>Triton</strong> come into play. PyTorch Lightning abstracts much of the repetitive setup, enabling cleaner and more scalable code, while Triton allows for writing custom, highly optimized GPU kernels to push hardware to its full potential.</p><p>In this blog, we’ll walk through various code snippets and configurations to optimize a neural network model on the MNIST dataset. By leveraging PyTorch, PyTorch Lightning, and Triton, we can achieve better performance with GPU optimizations, custom kernels, and mixed precision. Each approach includes code examples to illustrate the changes, showing how incremental modifications can improve training speed and reduce memory consumption.</p><p>Whether you’re looking to train complex LLMs or optimize more conventional deep learning models, understanding how to use these tools effectively is crucial. By the end of this guide, you’ll have a solid foundation for building, scaling, and deploying high-performance neural networks that fully utilize GPU resources</p><h3>Table of Contents</h3><ol><li><a href="#introduction">Introduction</a></li><li><a href="#technologies-overview">Technologies Overview</a></li><li><a href="#experiment-setup">Experiment Setup</a></li><li><a href="#approaches">Approaches</a></li></ol><ul><li><a href="#approach-1-native-pytorch-with-standard-relu">Approach 1: Native PyTorch with Standard ReLU</a></li><li><a href="#approach-2-cudnn-verification-with-pytorch">Approach 2: CuDNN Verification with PyTorch</a></li><li><a href="#approach-3-pytorch-lightning-integration">Approach 3: PyTorch Lightning Integration</a></li><li><a href="#approach-4-mixed-precision-training-with-lightning">Approach 4: Mixed Precision Training with Lightning</a></li><li><a href="#approach-5-custom-triton-relu-kernel-with-pytorch">Approach 5: Custom Triton ReLU Kernel with PyTorch</a></li><li><a href="#approach-6-mixed-precision-with-triton-kernel-in-pytorch">Approach 6: Mixed Precision with Triton Kernel in PyTorch</a></li><li><a href="#approach-7-pytorch-lightning-with-triton-kernel">Approach 7: PyTorch Lightning with Triton Kernel</a></li><li><a href="#approach-8-pytorch-lightning-with-mixed-precision-and-triton-kernel">Approach 8: PyTorch Lightning with Mixed Precision and Triton Kernel</a></li><li><a href="#approach-9-mixed-precision-for-forward-pass-with-lightning-precision-for-backward-pass">Approach 9: Mixed Precision for Forward Pass with Lightning Precision for Backward Pass</a></li><li><a href="#approach-10-full-triton-kernel-with-custom-mixed-precision-and-increased-workers">Approach 10: Full Triton Kernel with Custom Mixed Precision and Increased Workers</a></li></ul><p><a href="#conclusion-and-key-takeaways">5. Conclusion and Key Takeaways</a></p><h3>Introduction</h3><p>In deep learning, model training time and efficiency are crucial. This post covers 10 incremental approaches to optimize training, illustrating each with code examples. We’ll use <strong>PyTorch</strong> as our base framework, <strong>PyTorch Lightning</strong> to simplify and streamline training loops, and <strong>Triton</strong> to optimize GPU operations via custom kernels.</p><h3>Technologies Overview</h3><ul><li><strong>PyTorch</strong>: A powerful and flexible framework for building deep learning models.</li><li><strong>PyTorch Lightning</strong>: A wrapper around PyTorch that abstracts away much of the boilerplate code.</li><li><strong>Triton</strong>: A tool that allows for writing custom, highly optimized GPU kernels.</li></ul><h3>1. PyTorch: A Powerful and Flexible Framework for Deep Learning</h3><p>PyTorch is one of the most popular deep learning frameworks, widely known for its flexibility, ease of use, and dynamic computational graph. Developed by Facebook’s AI Research lab, it provides powerful tools for building and training neural networks. Here’s what makes PyTorch a preferred choice for researchers and developers:</p><ul><li><strong>Dynamic Computational Graphs</strong>: PyTorch uses dynamic computation graphs, also known as “define-by-run,” which allows developers to modify the graph on the fly. This makes it easy to experiment with different neural network architectures without the need to recompile the model, a significant advantage in research environments.</li><li><strong>Ease of Debugging</strong>: PyTorch’s code structure and syntax are highly readable and similar to Python, making it easy to debug with tools like pdb (Python’s debugger) or print statements. This simplicity and transparency in debugging make it more accessible to both beginners and advanced users.</li><li><strong>Flexible Tensor Manipulations</strong>: Tensors are the core data structure in PyTorch, similar to NumPy arrays but with GPU support. PyTorch allows seamless manipulation of tensors, making it easy to implement custom operations and processes directly on tensors.</li><li><strong>Efficient GPU Acceleration</strong>: PyTorch supports GPU acceleration, allowing developers to leverage the parallel processing power of GPUs with a simple .to(device) function, which moves the tensors to GPU memory. The integration with CUDA and CuDNN ensures optimized performance for deep learning tasks.</li></ul><h3>2. PyTorch Lightning: Simplifying PyTorch with Organized Code and Abstractions</h3><p>While PyTorch is flexible and powerful, it requires significant boilerplate code to manage training, validation, logging, and other processes. PyTorch Lightning is a lightweight wrapper around PyTorch that abstracts away much of this repetitive code, allowing researchers to focus more on developing and improving their models rather than writing infrastructure code.</p><p>Key features of PyTorch Lightning include:</p><ul><li><strong>Structured Code</strong>: PyTorch Lightning organizes the code into clear sections for defining the model, training step, validation step, and other core components. This structure encourages best practices, improving code readability and maintainability.</li><li><strong>Automatic Hardware Management</strong>: PyTorch Lightning can automatically handle multiple GPUs, TPUs, or CPUs, making it easier to scale models across different hardware configurations. This feature saves significant time, as developers don’t need to write custom code for hardware management.</li><li><strong>Simplified Mixed Precision</strong>: Mixed precision training, which combines float16 and float32 to reduce memory usage and speed up training, is easier to implement in PyTorch Lightning. A simple flag, precision=&#39;16-mixed&#39;, is all that’s needed to enable mixed precision.</li><li><strong>Callback Support</strong>: PyTorch Lightning offers a robust callback system, allowing developers to implement functionalities like early stopping, model checkpointing, and logging with minimal code. This simplifies complex tasks, which would otherwise require significant custom code.</li><li><strong>Enhanced Productivity</strong>: By reducing boilerplate and handling complex training setups (e.g., distributed training, 16-bit precision), PyTorch Lightning speeds up development cycles and makes PyTorch even more accessible for both academic and industry use.</li></ul><h3>3. Triton: High-Performance Custom GPU Kernels</h3><p>Triton is a tool for writing highly optimized GPU code, developed by OpenAI. It’s particularly useful for cases where deep learning models or specific layers need custom operations that aren’t well-supported by PyTorch’s native operations. Triton allows developers to write custom GPU kernels with optimized performance, often achieving significant speedups.</p><p>Here’s how Triton enhances the deep learning workflow:</p><ul><li><strong>Custom GPU Kernels</strong>: Triton enables developers to write custom GPU kernels that go beyond the operations provided by libraries like CuDNN and PyTorch. This is particularly useful for researchers experimenting with new operations or for applications where specific optimizations can significantly improve performance.</li><li><strong>Automatic Performance Optimization</strong>: Triton is designed to automatically optimize kernel performance, including memory access patterns and parallel execution. This feature saves developers from manually tuning GPU code, which is a complex and time-consuming process.</li><li><strong>High-Level API for Low-Level Control</strong>: While Triton allows for low-level control over GPU operations, its API is designed to be as accessible as possible. It abstracts away some of the complexities of GPU programming, making it more accessible to developers who may not be experts in CUDA programming.</li><li><strong>Efficient Integration with PyTorch</strong>: Triton can be integrated with PyTorch, allowing developers to replace specific operations with Triton kernels. This enables significant speed improvements in certain tasks, especially in use cases where PyTorch’s built-in functions may be suboptimal.</li></ul><h3>Bringing it All Together</h3><p>Together, PyTorch, PyTorch Lightning, and Triton provide a powerful toolkit for building, scaling, and optimizing deep learning models:</p><ul><li><strong>PyTorch</strong> is the foundation, offering flexibility, ease of use, and a dynamic computation graph that makes it ideal for research and development.</li><li><strong>PyTorch Lightning</strong> builds on PyTorch’s flexibility and removes boilerplate code, allowing for clean and scalable codebases, which are essential for both academic research and production-level applications.</li><li><strong>Triton</strong> offers additional performance optimization by enabling custom GPU kernel programming, allowing users to achieve fine-grained control over hardware utilization.</li></ul><p>Using these tools, developers can focus on designing high-quality models without worrying as much about repetitive code, performance bottlenecks, or complex GPU optimization. Each tool complements the others, providing an ecosystem that balances flexibility, productivity, and performance.</p><h3>Experiment Setup</h3><p>Each approach trains a two-layer neural network on the MNIST dataset. We start with a standard PyTorch setup, and then progressively add PyTorch Lightning, mixed precision, and Triton optimizations.</p><h3>Approaches</h3><h4>Approach 1: Native PyTorch with Standard ReLU</h4><p>Our starting point is a simple PyTorch network using a basic training loop.</p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim</pre><pre># Define a simple neural network<br>class SimpleNet(nn.Module):<br>    def __init__(self):<br>        super(SimpleNet, self).__init__()<br>        self.fc1 = nn.Linear(784, 500)<br>        self.relu = nn.ReLU()<br>        self.fc2 = nn.Linear(500, 10)<br>    <br>    def forward(self, x):<br>        x = self.fc1(x)<br>        x = self.relu(x)<br>        x = self.fc2(x)<br>        return x</pre><pre># Initialize model, loss, and optimizer<br>model = SimpleNet()<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(model.parameters())</pre><p>This approach provides a baseline performance and accuracy, serving as a reference for our optimizations.</p><h4>Approach 2: CuDNN Verification with PyTorch</h4><p>Before continuing, we can check if CuDNN is enabled, as it’s essential for GPU optimization.</p><pre>print(&quot;CuDNN Enabled:&quot;, torch.backends.cudnn.enabled)</pre><p>If this returns True, PyTorch will utilize GPU-optimized operations, which is important for later steps.</p><h4>Approach 3: PyTorch Lightning Integration</h4><p>Switching to PyTorch Lightning simplifies our training loop by abstracting away some of the boilerplate code.</p><pre>import pytorch_lightning as pl</pre><pre># Define the Lightning model<br>class LightningNet(pl.LightningModule):<br>    def __init__(self):<br>        super(LightningNet, self).__init__()<br>        self.fc1 = nn.Linear(784, 500)<br>        self.relu = nn.ReLU()<br>        self.fc2 = nn.Linear(500, 10)<br>        self.criterion = nn.CrossEntropyLoss()<br>    <br>    def forward(self, x):<br>        x = self.fc1(x)<br>        x = self.relu(x)<br>        x = self.fc2(x)<br>        return x</pre><pre>    def training_step(self, batch, batch_idx):<br>        x, y = batch<br>        y_hat = self(x)<br>        loss = self.criterion(y_hat, y)<br>        return loss</pre><pre>    def configure_optimizers(self):<br>        return optim.Adam(self.parameters())</pre><p>Using PyTorch Lightning makes it easier to scale and manage models, setting up our model for efficient training and testing.</p><h4>Approach 4: Mixed Precision Training with Lightning</h4><p>Adding mixed precision training to the Lightning model helps reduce memory usage and speed up training.</p><pre>trainer = pl.Trainer(precision=16, max_epochs=10)<br>trainer.fit(model, train_loader)</pre><p>This minor change enables mixed-precision training, where some computations use FP16 instead of FP32. This reduces memory load and can accelerate training.</p><h4>Approach 5: Custom Triton ReLU Kernel with PyTorch</h4><p>Next, we introduce a custom Triton kernel for the ReLU activation function. Triton helps optimize this operation for GPU.</p><pre>import triton<br>import triton.language as tl</pre><pre>@triton.jit<br>def relu_kernel(x_ptr, y_ptr, BLOCK_SIZE: tl.constexpr):<br>    offsets = tl.arange(0, BLOCK_SIZE)<br>    x = tl.load(x_ptr + offsets)<br>    y = tl.maximum(x, 0.0)<br>    tl.store(y_ptr + offsets, y)</pre><p>This kernel replaces the ReLU function, making it more efficient on the GPU. We use it in the forward pass of our network:</p><pre>def forward(self, x):<br>    x = self.fc1(x)<br>    x = relu_kernel(x)  # Triton custom ReLU<br>    x = self.fc2(x)<br>    return x</pre><h4>Approach 6: Mixed Precision with Triton Kernel in PyTorch</h4><p>This approach combines mixed precision with the Triton kernel, using autocast for mixed precision in the training loop.</p><pre>from torch.cuda.amp import autocast, GradScaler</pre><pre>scaler = GradScaler()</pre><pre>with autocast():<br>    output = model(input)<br>    loss = criterion(output, target)</pre><pre># Backward pass<br>scaler.scale(loss).backward()<br>scaler.step(optimizer)<br>scaler.update()</pre><p>This approach leverages both Triton’s kernel optimization and the memory efficiency of mixed precision.</p><h4>Approach 7: PyTorch Lightning with Triton Kernel</h4><p>We can now combine PyTorch Lightning with the Triton kernel, simplifying the training process while benefiting from the optimized kernel.</p><pre>class LightningTritonNet(pl.LightningModule):<br>    # Define model as above, replacing standard ReLU with the Triton kernel</pre><pre>trainer = pl.Trainer(max_epochs=10)<br>trainer.fit(LightningTritonNet(), train_loader)</pre><p>PyTorch Lightning manages training steps, and the custom Triton kernel ensures efficient GPU operations.</p><h4>Approach 8: PyTorch Lightning with Mixed Precision and Triton Kernel</h4><p>This approach combines PyTorch Lightning, Triton, and mixed precision for maximum efficiency.</p><pre>trainer = pl.Trainer(precision=&#39;16-mixed&#39;, max_epochs=10)<br>trainer.fit(LightningTritonNet(), train_loader)</pre><p>Using all three together — PyTorch Lightning, Triton, and mixed precision — results in substantial memory and computational efficiency.</p><h4>Approach 9: Mixed Precision for Forward Pass with Lightning Precision for Backward Pass</h4><p>Here, we use mixed precision in the forward pass (using Triton) and PyTorch Lightning’s mixed precision for the backward pass.</p><pre>with autocast():<br>    output = model(input)<br>    loss = criterion(output, target)</pre><p>This configuration is more granular, controlling memory usage and computational precision separately for forward and backward passes.</p><h4>Approach 10: Full Triton Kernel with Custom Mixed Precision and Increased Workers</h4><p>Our final approach uses Triton for the forward pass in mixed precision, while the backward pass leverages GradScaler and autocast. We also increase the number of num_workers to optimize data loading.</p><pre>train_loader = DataLoader(dataset, batch_size=32, num_workers=4)</pre><p>Increasing num_workers improves data throughput. Triton, mixed precision, and PyTorch Lightning’s managed training give us the best performance.</p><h3>Conclusion and Key Takeaways</h3><h3>Conclusion and Key Takeaways</h3><p>Throughout these 10 approaches, we demonstrated how incremental changes in code and configuration can yield significant performance gains for neural network training. Here’s a recap of the key learnings:</p><ol><li><strong>Baseline Performance</strong>: Starting with a standard PyTorch implementation, we establish a baseline for training time and accuracy, which helps us measure the impact of each optimization.</li><li><strong>Optimizing with PyTorch Lightning</strong>: PyTorch Lightning simplifies the code, manages boilerplate functions, and offers flexible configurations, making it easier to scale and integrate additional optimizations.</li><li><strong>Leveraging Mixed Precision</strong>: Mixed precision training allows models to use FP16 precision where possible, reducing memory usage and accelerating computation without sacrificing accuracy. This optimization shines when combined with GPU accelerators like CUDA.</li><li><strong>Triton for Custom Kernel Optimization</strong>: Using Triton, we can write highly optimized custom kernels for specific operations like ReLU, improving GPU efficiency. The Triton ReLU kernel serves as a drop-in replacement, making it a seamless yet impactful optimization.</li><li><strong>Combining Optimizations for Maximum Efficiency</strong>: The most efficient approach (Approach 10) combines PyTorch Lightning, Triton, and mixed precision while adjusting num_workers to optimize data loading. This setup leverages each tool’s strengths to achieve the best possible balance of speed and resource use.</li></ol><p><strong>Key Takeaway</strong>: For deep learning developers, combining <strong>PyTorch Lightning</strong> for streamlined training, <strong>Triton</strong> for GPU-optimized kernels, and <strong>mixed precision</strong> training offers a powerful and flexible strategy for efficient model training. These tools together allow us to scale models, minimize memory usage, and maximize GPU utilization, which is particularly valuable in research and production environments with resource constraints.</p><p>By following these approaches, you can create models that train faster, use fewer resources, and ultimately deliver better performance on real-world tasks.</p><p><strong>Happy Optimizing!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=174bb3e26593" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crafting Intelligent Agents for Dynamic Query Resolution: Multi-Approach Workflows with Web and…]]></title>
            <link>https://medium.com/@official.indrajit.kar/crafting-intelligent-agents-for-dynamic-query-resolution-multi-approach-workflows-with-web-and-bf6d389bb2d4?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf6d389bb2d4</guid>
            <category><![CDATA[generative-ai-use-cases]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[llama-3]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Sun, 27 Oct 2024 09:15:31 GMT</pubDate>
            <atom:updated>2024-10-27T09:18:33.435Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crafting Intelligent Agents for Dynamic Query Resolution: Multi-Approach Workflows with Web and LinkedIn Search: LangGraph, Local LLama and LangChain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q_2XVPEveH2JFUj6BFnknw.png" /></figure><p>With the rapid advancements in AI, the demand for robust, intelligent question-answering (QA) systems has grown. This blog will guide you through building a flexible QA system using LangChain, LangGraph, and vector databases. Our design dynamically routes questions to relevant sources, retrieves documents, and generates accurate responses. We’ll also explore three different architectural approaches to execute these workflows, which are visualized in the image below.</p><p>Each workflow approach provides unique advantages in handling various types of queries. Let’s explore how this system was built, step-by-step. The code can be found <a href="https://github.com/Karindraj/Agents.git">here</a></p><h3>Prerequisites and Installation</h3><p>To set up the environment for our QA system, install the following libraries:</p><pre># Install required libraries<br>!pip install langgraph<br>!pip install langchain-nomic<br>!pip install -U langchain-ollama<br>!pip install tiktoken<br>!pip install langchain-community<br>!pip install gpt4all</pre><p>These libraries enable document retrieval, vector embedding, workflow management, and interaction with pre-trained models.</p><h3>Architectural Overview and Approaches</h3><p>Our QA system is designed with three distinct approaches for routing, retrieving, and generating answers. The workflow diagram above illustrates these three approaches:</p><ol><li><strong>Approach 1</strong> (Left) — <strong>Dynamic Routing with Maximum Retries</strong>: This approach handles queries by first checking LinkedIn (for profile-related queries), then accessing a vector store, followed by a fallback to web search if documents are insufficient. It dynamically retries the answer generation step a set number of times to maximize response accuracy.</li><li><strong>Approach 2</strong> (Center) — <strong>Streamlined Retrieval with Conditional Generation</strong>: This streamlined workflow reduces redundancy by focusing on retrieving and grading documents from the vector store, followed by generating an answer. It only falls back to web search if no useful documents are found, minimizing the number of retries and enhancing efficiency.</li><li><strong>Approach 3</strong> (Right) — <strong>Minimal Retry with LinkedIn Integration</strong>: This approach focuses on retrieving documents with minimal retries, providing a straightforward path from retrieval to answer generation. It integrates LinkedIn redirection for profile queries but emphasizes a faster conclusion if no relevant information is found.</li></ol><p>Each approach has unique benefits depending on the complexity of the query and desired response accuracy.</p><h3>Step 1: Setting Environment Variables</h3><p>Our system relies on several APIs, such as Tavily for web searches. Setting up these environment variables securely with getpass helps protect sensitive information:</p><pre>import os<br>import getpass</pre><pre>def _set_env(var: str):<br>    if not os.environ.get(var):<br>        os.environ[var] = getpass.getpass(f&quot;{var}: &quot;)</pre><pre>_set_env(&quot;TAVILY_API_KEY&quot;)<br>_set_env(&quot;LANGCHAIN_API_KEY&quot;)</pre><h3>Step 2: Document Retrieval with Vector Store</h3><p>To retrieve relevant documents, we use a vector database setup with NomicEmbeddings. This allows for efficient semantic search on topics relevant to the user’s question.</p><pre>from langchain_community.vectorstores import SKLearnVectorStore<br>from langchain_nomic.embeddings import NomicEmbeddings<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain_community.document_loaders import WebBaseLoader</pre><pre>urls = [<br>    &quot;https://lilianweng.github.io/posts/2023-06-23-agent/&quot;,<br>    &quot;https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/&quot;,<br>    &quot;https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/&quot;,<br>]</pre><pre># Load documents and split text<br>docs = [WebBaseLoader(url).load() for url in urls]<br>docs_list = [item for sublist in docs for item in sublist]<br>text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000, chunk_overlap=200)<br>doc_splits = text_splitter.split_documents(docs_list)</pre><pre># Create vector store and retriever<br>vectorstore = SKLearnVectorStore.from_documents(<br>    documents=doc_splits,<br>    embedding=NomicEmbeddings(model=&quot;nomic-embed-text-v1.5&quot;, inference_mode=&quot;local&quot;),<br>)<br>retriever = vectorstore.as_retriever(k=3)</pre><h3>Step 3: Query Routing</h3><p>Our routing system directs queries to the appropriate source (vector store, web search, or LinkedIn). We’ve designed a custom routing function that identifies profile queries and redirects them to LinkedIn.</p><pre>def route_question(state):<br>    &quot;&quot;&quot;Route question to vectorstore, web search, or LinkedIn based on content.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    person_keywords = re.compile(r&quot;(who is|tell me about|profile of|bio of|biography of) (.+)&quot;, re.IGNORECASE)<br>    match = person_keywords.match(question)</pre><pre>    if match:<br>        state[&quot;person_name&quot;] = match.group(2).strip()<br>        return &quot;linkedin_redirect&quot;<br>    else:<br>        return &quot;vectorstore&quot;</pre><p>This ensures that biography-related queries get a direct LinkedIn profile link, while other questions are handled by the vector store or web search.</p><h3>Step 4: Grading Document Relevance</h3><p>Each document retrieved is graded for relevance. If the retrieved documents lack relevant information, the system initiates a web search to improve the answer quality.</p><pre>def grade_documents(state):<br>    &quot;&quot;&quot;Check if retrieved documents are relevant.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    documents = state[&quot;documents&quot;]<br>    filtered_docs = []<br>    web_search = &quot;No&quot;</pre><pre>    for d in documents:<br>        if d.page_content and question in d.page_content:<br>            filtered_docs.append(d)<br>        else:<br>            web_search = &quot;Yes&quot;<br>    return {&quot;documents&quot;: filtered_docs, &quot;web_search&quot;: web_search}</pre><h3>Step 5: Answer Generation</h3><p>After gathering relevant documents, we generate an answer using a pre-trained model. The documents are formatted into a prompt, and the response is generated based on this context.</p><pre>def generate(state):<br>    &quot;&quot;&quot;Generate answer from retrieved documents.&quot;&quot;&quot;<br>    question = state[&quot;question&quot;]<br>    documents = state[&quot;documents&quot;]<br>    docs_txt = &quot;\n\n&quot;.join([doc.page_content for doc in documents])<br>    <br>    rag_prompt_formatted = f&quot;Answer based on context:\n{docs_txt}\n\nQuestion: {question}\nAnswer:&quot;<br>    generation = llm.invoke([HumanMessage(content=rag_prompt_formatted)])<br>    return {&quot;answer&quot;: generation.content}</pre><h3>Step 6: Handling LinkedIn Redirects</h3><p>For biography-related queries, we redirect the user to the LinkedIn profile of the person in question</p><pre>def handle_linkedin_redirect(state):<br>    &quot;&quot;&quot;Redirect to LinkedIn for profile-related queries.&quot;&quot;&quot;<br>    person_name = state[&quot;person_name&quot;].replace(&quot; &quot;, &quot;-&quot;)<br>    linkedin_url = f&quot;https://in.linkedin.com/in/{person_name}&quot;<br>    print(f&quot;LinkedIn Profile: {linkedin_url}&quot;)<br>    return linkedin_url</pre><h3>Executing the Workflow with Different Approaches</h3><p>The final step is to define and execute the workflow, connecting all nodes for routing, retrieving, grading, generating, and handling redirects. Here’s the setup for each approach:</p><h4>Approach 1: Dynamic Routing with Maximum Retries</h4><p>This approach prioritizes dynamic retrieval and maximum retries for complex queries, optimizing for accuracy and completeness.</p><h4>Approach 2: Streamlined Retrieval with Conditional Generation</h4><p>In this streamlined workflow, document retrieval is prioritized with conditional generation based on document relevance, allowing for a more efficient response.</p><h4>Approach 3: Minimal Retry with LinkedIn Integration</h4><p>This approach minimizes retries, focusing on a straightforward retrieval-to-response pipeline with LinkedIn integration for profile queries.</p><p>Here’s the code snippet to compile the workflow graph:</p><pre>from langgraph.graph import StateGraph</pre><pre>workflow = StateGraph(GraphState)</pre><pre># Define nodes for workflow<br>workflow.add_node(&quot;retrieve&quot;, retrieve)<br>workflow.add_node(&quot;grade_documents&quot;, grade_documents)<br>workflow.add_node(&quot;generate&quot;, generate)<br>workflow.add_node(&quot;linkedin_redirect&quot;, handle_linkedin_redirect)<br>workflow.add_node(&quot;websearch&quot;, web_search)</pre><pre># Build workflow with conditional routing<br>workflow.set_conditional_entry_point(<br>    route_question,<br>    {<br>        &quot;linkedin_redirect&quot;: &quot;linkedin_redirect&quot;,<br>        &quot;vectorstore&quot;: &quot;retrieve&quot;,<br>        &quot;websearch&quot;: &quot;websearch&quot;,<br>    }<br>)</pre><pre># Add workflow edges<br>workflow.add_edge(&quot;retrieve&quot;, &quot;grade_documents&quot;)<br>workflow.add_conditional_edges(<br>    &quot;grade_documents&quot;,<br>    lambda state: &quot;generate&quot; if state[&quot;web_search&quot;] == &quot;No&quot; else &quot;websearch&quot;,<br>    {<br>        &quot;generate&quot;: &quot;generate&quot;,<br>        &quot;websearch&quot;: &quot;websearch&quot;,<br>    },<br>)</pre><pre># Compile and display workflow graph<br>graph = workflow.compile()<br>display(Image(graph.get_graph().draw_mermaid_png()))</pre><h3>Testing the System</h3><p>You can test the system with different types of questions to see how each approach handles them:</p><pre>question = input(&quot;Please enter your question: &quot;)<br>process_question(question)</pre><p>The system behavior varies based on the approach used, balancing response time and accuracy for each question type.</p><h3>Conclusion</h3><p>This QA system demonstrates how different architectural approaches can be employed to answer various types of queries. By leveraging LangChain, LangGraph, and other NLP tools, we’ve created a robust and flexible QA framework that provides accurate, concise responses, adapting its workflow based on the type of question.</p><p>The three approaches discussed here — dynamic routing with maximum retries, streamlined retrieval with conditional generation, and minimal retry with LinkedIn integration — offer versatility for diverse use cases, from customer support to research assistance. This modular design can be further customized with additional data sources or tailored workflows to meet the specific needs of any information retrieval task.</p><h3>Reference</h3><p><a href="https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag_local/">https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag_local/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf6d389bb2d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Multi-Agent Q-A System with Local Llama3.2 “Swarm” and LangChain]]></title>
            <link>https://medium.com/@official.indrajit.kar/building-a-multi-agent-q-a-system-with-local-llama3-2-swarm-and-langchain-bee7faa18c9f?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/bee7faa18c9f</guid>
            <category><![CDATA[genai-agents]]></category>
            <category><![CDATA[langchain-agents]]></category>
            <category><![CDATA[openai-swarm]]></category>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[llama-3-2]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Fri, 18 Oct 2024 13:30:30 GMT</pubDate>
            <atom:updated>2024-10-27T09:20:49.072Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ccAXi_J86Llud5oiVaFYUg.jpeg" /></figure><p><strong>Introduction</strong></p><p>OpenAI’s Swarm project. Swarm is an experimental framework developed by OpenAI that allows you to coordinate multiple AI agents to collaborate on complex tasks. It facilitates the creation of systems where agents can communicate, share information, and make decisions collectively.</p><p>In Today’s world the integration of multiple specialized agents to solve complex tasks is becoming increasingly prevalent. This blog post delves into the creation of a sophisticated question-answering (QA) system that leverages a combination of local language models, vector stores, web search capabilities, and a multi-agent framework called Swarm.</p><p>We’ll explore how to:</p><ul><li>Set up a local language model using ChatOllama.</li><li>Utilize LangChain for document retrieval and processing.</li><li>Implement a multi-agent system with Swarm to handle task delegation and agent handoffs Using OpenAI Swarm</li><li>Ensure accurate and relevant answers through grading agents.</li></ul><p>By the end of this post, you’ll have a comprehensive understanding of how to build a multi-agent QA system capable of handling diverse queries efficiently and accurately.</p><p>The code is found <a href="https://github.com/Karindraj/Llama-Swarm.git">here</a></p><h3>Table of Contents</h3><ol><li><a href="#overview-of-the-system">Overview of the System</a></li><li><a href="#installation-and-setup">Installation and Setup</a></li><li><a href="#language-model-setup">Language Model Setup</a></li><li><a href="#document-retrieval-and-vector-store">Document Retrieval and Vector Store</a></li><li><a href="#swarm-framework-and-agent-definitions">Swarm Framework and Agent Definitions</a></li><li><a href="#implementing-agent-handoffs">Implementing Agent Handoffs</a></li><li><a href="#process-flow-of-the-system">Process Flow of the System</a></li><li><a href="#testing-the-system">Testing the System</a></li><li><a href="#conclusion">Conclusion</a></li></ol><h3>Overview of the System</h3><p>Our QA system is designed to process user questions by coordinating multiple specialized agents, each responsible for a specific task. The system employs a local language model for generating responses and uses LangChain for document processing and retrieval. The Swarm framework facilitates agent communication and task delegation, enabling dynamic decision-making and efficient problem-solving.</p><p><strong>Key Components:</strong></p><ul><li><strong>Language Model:</strong> ChatOllama using llama3.2:3b-instruct-fp16.</li><li><strong>Document Retrieval:</strong> Utilizing LangChain’s WebBaseLoader, RecursiveCharacterTextSplitter, and SKLearnVectorStore.</li><li><strong>Agents:</strong></li><li><strong>Router Agent:</strong> Routes questions to the appropriate data source.</li><li><strong>Retrieval Agent:</strong> Fetches documents from the vector store.</li><li><strong>Web Search Agent:</strong> Performs web searches for up-to-date information.</li><li><strong>Answer Generation Agent:</strong> Generates answers based on retrieved information.</li><li><strong>Hallucination Grader Agent:</strong> Checks answers for factual accuracy.</li><li><strong>Answer Grader Agent:</strong> Ensures answers sufficiently address the questions.</li><li><strong>Swarm Framework:</strong> Manages agent interactions and handoffs.</li></ul><h3>Installation and Setup</h3><p>Before diving into the code, ensure you have the necessary packages installed. You can install them using pip:</p><pre>pip install langchain langchain-ollama langchain-communities<br>pip install git+https://github.com/openai/swarm.git</pre><p><strong>Import Required Libraries:</strong></p><pre>import os<br>import json<br>from swarm import Swarm, Agent<br>from langchain_ollama import ChatOllama<br>from langchain.schema import SystemMessage, HumanMessage, AIMessage<br>from langchain_community.document_loaders import WebBaseLoader<br>from langchain.text_splitter import RecursiveCharacterTextSplitter<br>from langchain_community.vectorstores import SKLearnVectorStore<br>from langchain_nomic.embeddings import NomicEmbeddings<br>from langchain_community.tools.tavily_search import TavilySearchResults</pre><p><strong>Set Up Environment Variables:</strong></p><pre>def _set_env(var: str):<br>    if not os.environ.get(var):<br>        os.environ[var] = input(f&quot;{var}: &quot;)</pre><pre>_set_env(&quot;TAVILY_API_KEY&quot;)<br>os.environ[&quot;TOKENIZERS_PARALLELISM&quot;] = &quot;true&quot;</pre><h3>Language Model Setup</h3><p>We use ChatOllama, a wrapper around local Llama models, to handle language generation tasks.</p><p><strong>Initialize the Language Model:</strong></p><pre>local_llm = &quot;llama3.2:3b-instruct-fp16&quot;<br>llm = ChatOllama(model=local_llm, temperature=0)</pre><h3>Document Retrieval and Vector Store</h3><p>We utilize LangChain’s tools to load, process, and store documents for efficient retrieval.</p><p><strong>Load Documents from URLs:</strong></p><pre>urls = [<br>    &quot;&quot;,<br>    &quot;&quot;,<br>    &quot;&quot;,<br>]</pre><pre>docs = [WebBaseLoader(url).load() for url in urls]<br>docs_list = [item for sublist in docs for item in sublist]</pre><p><strong>Split Documents into Chunks:</strong></p><pre>text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(<br>    chunk_size=1000, chunk_overlap=200<br>)<br>doc_splits = text_splitter.split_documents(docs_list)</pre><p><strong>Create Vector Store with Embeddings:</strong></p><pre>embedding = NomicEmbeddings(<br>    model=&quot;nomic-embed-text-v1.5&quot;, inference_mode=&quot;local&quot;<br>)<br>vectorstore = SKLearnVectorStore.from_documents(<br>    documents=doc_splits,<br>    embedding=embedding,<br>)</pre><p><strong>Initialize Retriever:</strong></p><pre>retriever = vectorstore.as_retriever(k=3)</pre><h3>Swarm Framework and Agent Definitions</h3><p>The Swarm framework facilitates the coordination between agents, allowing them to hand off tasks to one another dynamically.</p><h3>Swarm Class with Handoff Handling</h3><pre>class Swarm:<br>    def __init__(self, model, temperature=0, agents=None):<br>        self.llm = ChatOllama(model=model, temperature=temperature)<br>        self.agents = agents or {}</pre><pre>    def run(self, agent_name, messages, context_variables=None, max_turns=float(&quot;inf&quot;)):<br>        context_variables = context_variables or {}<br>        current_agent = self.agents.get(agent_name)<br>        response_messages = messages[:]<br>        turn_count = 0</pre><pre>        while turn_count &lt; max_turns and current_agent is not None:<br>            # Prepare messages<br>            messages_for_llm = self._prepare_messages(current_agent, response_messages, context_variables)<br>            # Generate response<br>            response = self.llm.invoke([SystemMessage(content=messages_for_llm)])<br>            # Add response<br>            response_message = AIMessage(content=response.content)<br>            response_messages.append(response_message)<br>            # Parse response<br>            response_data = self._parse_response(response.content, current_agent.name)<br>            if response_data is None:<br>                break<br>            # Update context variables and get next agent<br>            context_variables.update(response_data)<br>            next_agent_name = response_data.get(&#39;handoff&#39;)<br>            if next_agent_name == &#39;None&#39; or next_agent_name is None:<br>                break<br>            else:<br>                current_agent = self.agents.get(next_agent_name)<br>            turn_count += 1</pre><pre>        return {<br>            &quot;messages&quot;: response_messages,<br>            &quot;agent&quot;: current_agent,<br>            &quot;context_variables&quot;: context_variables,<br>        }</pre><pre>    def _prepare_messages(self, agent, messages, context_variables):<br>        instructions = agent.instructions.format(**context_variables)<br>        return instructions</pre><pre>    def _parse_response(self, content, agent_name):<br>        try:<br>            response_data = json.loads(content)<br>            return response_data<br>        except json.JSONDecodeError:<br>            print(f&quot;Failed to parse response from agent {agent_name}&quot;)<br>            return None</pre><h3>Agent Definitions</h3><p>Types of Agents</p><ul><li><strong>Router Agent</strong> decides where to route the question.</li><li><strong>Retrieval Agent</strong> or <strong>Web Search Agent</strong> retrieves relevant information.</li><li><strong>Answer Generation Agent</strong> creates an answer using the retrieved information.</li><li><strong>Hallucination Grader Agent</strong> checks the answer for factual accuracy.</li><li><strong>Answer Grader Agent</strong> evaluates if the answer sufficiently addresses the question.</li></ul><p><strong>Router Agent:</strong></p><p>Decides whether to use the vector store or web search based on the question.</p><pre>router_agent = Agent(<br>    name=&quot;Router Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>You are an expert at routing a user question to a vectorstore or web search.</pre><pre>The vectorstore contains documents related to agents, prompt engineering, and adversarial attacks.</pre><pre>Use the vectorstore for questions on these topics. For all else, and especially for current events, use web-search.</pre><pre>Return ONLY a JSON object:</pre><pre>{&quot;handoff&quot;: &quot;Retrieval Agent&quot;}</pre><pre>or</pre><pre>{&quot;handoff&quot;: &quot;Web Search Agent&quot;}</pre><pre>Do not include any other text in your response.</pre><pre>Question: {question}<br>&quot;&quot;&quot;<br>)</pre><p><strong>Retrieval Agent:</strong></p><p>Retrieves documents from the vector store.</p><pre>retrieval_agent = Agent(<br>    name=&quot;Retrieval Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>You are responsible for retrieving documents from the vectorstore based on the user&#39;s question.</pre><pre>Retrieve relevant documents and decide if they are sufficiently relevant.</pre><pre>If the documents are relevant, proceed to the next step by handing off to the &#39;Answer Generation Agent&#39;.</pre><pre>If the documents are not relevant, hand off to the &#39;Web Search Agent&#39;.</pre><pre>Return JSON with keys:<br>- &quot;documents&quot;: a list of retrieved documents.<br>- &quot;handoff&quot;: the next agent&#39;s name (&quot;Answer Generation Agent&quot; or &quot;Web Search Agent&quot;).</pre><pre>Question: {question}<br>&quot;&quot;&quot;<br>)</pre><p><strong>Web Search Agent:</strong></p><p>Performs a web search for up-to-date information.</p><pre>web_search_agent = Agent(<br>    name=&quot;Web Search Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>You are responsible for performing a web search based on the user&#39;s question.</pre><pre>Perform a web search and return the top results.</pre><pre>Proceed to the next step by handing off to the &#39;Answer Generation Agent&#39;.</pre><pre>Return JSON with keys:<br>- &quot;documents&quot;: a list of search result contents.<br>- &quot;handoff&quot;: &quot;Answer Generation Agent&quot;.</pre><pre>Question: {question}<br>&quot;&quot;&quot;<br>)</pre><p><strong>Answer Generation Agent:</strong></p><p>Generates an answer using the retrieved information.</p><pre>answer_generation_agent = Agent(<br>    name=&quot;Answer Generation Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>You are an assistant for question-answering tasks.</pre><pre>Here is the context to use to answer the question:</pre><pre>{documents}</pre><pre>Think carefully about the above context.</pre><pre>Now, review the user question:</pre><pre>{question}</pre><pre>Provide an answer to this question using only the above context.</pre><pre>Use three sentences maximum and keep the answer concise.</pre><pre>Return JSON with keys:<br>- &quot;answer&quot;: the generated answer.<br>- &quot;handoff&quot;: &quot;Hallucination Grader Agent&quot;.<br>&quot;&quot;&quot;<br>)</pre><p><strong>Hallucination Grader Agent:</strong></p><p>Checks if the generated answer is grounded in facts.</p><pre>hallucination_grader_agent = Agent(<br>    name=&quot;Hallucination Grader Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>You are a teacher grading a quiz.</pre><pre>You will be given FACTS and a STUDENT ANSWER.</pre><pre>Ensure the STUDENT ANSWER is grounded in the FACTS and does not contain &quot;hallucinated&quot; information outside the scope of the FACTS.</pre><pre>Score:<br>- A score of &quot;yes&quot; means that the student&#39;s answer meets all of the criteria.<br>- A score of &quot;no&quot; means that the student&#39;s answer does not meet all of the criteria.</pre><pre>Explain your reasoning in a step-by-step manner.</pre><pre>Return JSON with keys:<br>- &quot;binary_score&quot;: &quot;yes&quot; or &quot;no&quot;.<br>- &quot;explanation&quot;: your reasoning.<br>- &quot;handoff&quot;: &quot;Answer Grader Agent&quot; if &quot;yes&quot;, or &quot;None&quot; if &quot;no&quot;.</pre><pre>FACTS:</pre><pre>{documents}</pre><pre>STUDENT ANSWER:</pre><pre>{answer}<br>&quot;&quot;&quot;<br>)</pre><p><strong>Answer Grader Agent:</strong></p><p>Evaluates if the answer sufficiently addresses the question.</p><pre>answer_grader_agent = Agent(<br>    name=&quot;Answer Grader Agent&quot;,<br>    instructions=&quot;&quot;&quot;<br>You are a teacher grading a quiz.</pre><pre>You will be given a QUESTION and a STUDENT ANSWER.</pre><pre>Ensure the STUDENT ANSWER helps to answer the QUESTION.</pre><pre>Score:<br>- A score of &quot;yes&quot; means that the student&#39;s answer meets the criteria.<br>- A score of &quot;no&quot; means that the student&#39;s answer does not meet the criteria.</pre><pre>Explain your reasoning in a step-by-step manner.</pre><pre>Return JSON with keys:<br>- &quot;binary_score&quot;: &quot;yes&quot; or &quot;no&quot;.<br>- &quot;explanation&quot;: your reasoning.<br>- &quot;handoff&quot;: &quot;None&quot;.</pre><pre>QUESTION:</pre><pre>{question}</pre><pre>STUDENT ANSWER:</pre><pre>{answer}<br>&quot;&quot;&quot;<br>)</pre><h3>Implementing Agent Handoffs</h3><p>The Swarm framework allows agents to hand off tasks to one another based on the outcome of their processing. Each agent’s response includes a “handoff” key, indicating the next agent.</p><p><strong>Example Handoff Flow:</strong></p><ol><li><strong>Router Agent</strong> decides to hand off to <strong>Retrieval Agent</strong> or <strong>Web Search Agent</strong>.</li><li><strong>Retrieval Agent</strong> retrieves documents and hands off to <strong>Answer Generation Agent</strong>.</li><li><strong>Answer Generation Agent</strong> creates an answer and hands off to <strong>Hallucination Grader Agent</strong>.</li><li><strong>Hallucination Grader Agent</strong> validates the answer and hands off to <strong>Answer Grader Agent</strong> if acceptable.</li><li><strong>Answer Grader Agent</strong> evaluates the answer and concludes the process.</li></ol><h3>Process Flow of the System</h3><p>The process_question function orchestrates the entire workflow, starting with the user&#39;s question and progressing through the agents.</p><pre>def process_question(question):<br>    # Initialize context variables<br>    context_variables = {<br>        &#39;question&#39;: question,<br>        &#39;documents&#39;: &#39;&#39;,<br>        &#39;answer&#39;: &#39;&#39;<br>    }</pre><pre>    # Initialize messages<br>    messages = []</pre><pre>    # Define agents<br>    agents = {<br>        &#39;Router Agent&#39;: router_agent,<br>        &#39;Retrieval Agent&#39;: retrieval_agent,<br>        &#39;Web Search Agent&#39;: web_search_agent,<br>        &#39;Answer Generation Agent&#39;: answer_generation_agent,<br>        &#39;Hallucination Grader Agent&#39;: hallucination_grader_agent,<br>        &#39;Answer Grader Agent&#39;: answer_grader_agent,<br>    }</pre><pre>    # Instantiate Swarm client with agents<br>    client = Swarm(model=local_llm, agents=agents)</pre><pre>    # Run the Swarm starting with the Router Agent<br>    response = client.run(agent_name=&#39;Router Agent&#39;, messages=messages, context_variables=context_variables)</pre><pre>    # Extract final answer from context_variables<br>    final_answer = context_variables.get(&#39;answer&#39;)<br>    print(f&quot;Final Answer: {final_answer}&quot;)</pre><pre>    # Print grading results<br>    hallucination_grade = context_variables.get(&#39;binary_score&#39;)<br>    explanation = context_variables.get(&#39;explanation&#39;)</pre><pre>    print(f&quot;Hallucination Grading Result: {hallucination_grade}&quot;)<br>    print(f&quot;Explanation: {explanation}&quot;)</pre><h3>Testing the System</h3><p>Let’s test the system with a sample question:</p><pre>process_question(&quot;What is Chain of Thought prompting?&quot;)</pre><p><strong>Expected Flow:</strong></p><ol><li><strong>Router Agent</strong> determines the question relates to prompt engineering and hands off to the <strong>Retrieval Agent</strong>.</li><li><strong>Retrieval Agent</strong> fetches relevant documents from the vector store and hands off to the <strong>Answer Generation Agent</strong>.</li><li><strong>Answer Generation Agent</strong> generates an answer and hands off to the <strong>Hallucination Grader Agent</strong>.</li><li><strong>Hallucination Grader Agent</strong> validates the answer and, if acceptable, hands off to the <strong>Answer Grader Agent</strong>.</li><li><strong>Answer Grader Agent</strong> evaluates the answer’s relevance to the question.</li><li><strong>Final Answer</strong> is printed along with grading results.</li></ol><p><strong>Sample Output:</strong></p><pre>Document not relevant, performing web search...<br>Search Results: [&#39;OpenAI Swarm is an open-source framework designed to simplify the creation and coordination of AI agents, focusing on agent coordination and execution through agents and handoffs.&#39;, &quot;Swarm is OpenAI&#39;s innovative approach to creating a flexible and efficient framework for orchestrating multiple AI agents. Described as an ergonomic, lightweight, multi-agent orchestration framework , Swarm is currently in its experimental phase, aiming to explore user-friendly interfaces for managing complex multi-agent systems.&quot;, &quot;OpenAI has announced and launched a new AI product called Swarm that showcases advances encompassing agentic AI. This is the future of AI. Here&#39;s what you need to know.&quot;, &quot;OpenAI unveiled a new open-source framework last week designed to simplify the development and management of multi-agent AI systems that can collaborate autonomously to perform tasks. Dubbed &#39;Swarm,&#39; the experimental release hints at a future where artificial intelligence will not only answer questions but also complete complex tasks on behalf of users, both online and in real-world applications.&quot;, &#39;An open-source project called &quot;OpenAI Agent Swarm Project: Hierarchical Autonomous Agent Swarms (HOS)&quot; demonstrates a possible implementation, including a hierarchy of AI agents with distinct&#39;]<br>Hallucination detected: {&#39;binary_score&#39;: &#39;no&#39;, &#39;explanation&#39;: &quot;The student answer does not match any of the facts provided. The facts are about OpenAI Swarm and its features, while the student answer is about Meta&#39;s Llama models and their multimodal capabilities.&quot;}<br>&#39;The generated answer contains hallucinated information. Please verify the details.</pre><h3>Conclusion</h3><p>In this blog post, we’ve built a sophisticated multi-agent QA system that efficiently processes user questions by leveraging:</p><ul><li><strong>Local Language Models:</strong> For generating responses without relying on external APIs.</li><li><strong>LangChain Tools:</strong> For document loading, processing, and retrieval.</li><li><strong>Swarm Framework:</strong> To manage agent interactions and dynamic task delegation.</li></ul><p>By integrating specialized agents with clear responsibilities and a robust handoff mechanism, the system ensures accurate and relevant answers while maintaining efficiency.</p><p><strong>Key Takeaways:</strong></p><ul><li><strong>Modular Design:</strong> Breaking down the system into specialized agents simplifies development and maintenance.</li><li><strong>Local Models with Swarm :</strong> Using local LLMs with OpenAI Swarm</li><li><strong>Agent Coordination:</strong> Effective communication and task delegation between agents are crucial for complex workflows.</li></ul><h3>Reference</h3><p><a href="https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag_local/">https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_adaptive_rag_local/</a></p><p><a href="https://github.com/openai/swarm">https://github.com/openai/swarm</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bee7faa18c9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Performance Deep learning Part 2]]></title>
            <link>https://medium.com/@official.indrajit.kar/high-performance-deep-learning-part-2-4bd35147b25c?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bd35147b25c</guid>
            <category><![CDATA[pytorch-lightning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[code-like-a-pro]]></category>
            <category><![CDATA[high-performace-computing]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 19:12:19 GMT</pubDate>
            <atom:updated>2024-10-28T21:23:28.118Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z10_vF3OFVLUXjIAiZa1NQ.jpeg" /></figure><h3>From Naive to Ninja with PyTorch Lightning: Boom</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KCmPbUCo1WQ99vN22hu6qA.png" /><figcaption>Comparison of Naive vs Optimized PyTorch Models</figcaption></figure><h3>Optimizing PyTorch Lightning in Just 48 Lines</h3><p>In the journey of developing deep learning models, efficiency is the key to standing out. This blog is a deep dive into the step-by-step evolution of a PyTorch Lightning model. We’ll start with a simple, naive implementation, and progressively optimize it — cutting down complexity, improving readability, and reducing execution time.</p><p>First, we’ll build a naive version of a PyTorch Lightning model, laying the foundational concepts in a straightforward manner. From there, we’ll transition to native PyTorch Lightning optimizations that enhance speed and reduce boilerplate code. But the story doesn’t end there. We’ll apply Python’s powerful lambda functions to achieve even more concise and functional code.</p><p>Finally, we’ll push the boundaries to the extreme — shrinking our entire model down to just 37 lines of code. Along the way, I’ll benchmark each version, tracking execution time, and highlighting the trade-offs and improvements. This exercise isn’t just about compact code — it’s about learning to make models more efficient while preserving readability and flexibility.</p><p>By the end of this blog, you’ll gain insights into how to balance elegance, performance, and simplicity when developing deep learning models in PyTorch Lightning. Let’s begin the transformation — one optimization at a time!</p><p>The entire code can be found here , <a href="https://github.com/Karindraj/High-Performance-Deep-learning/blob/main/boom.ipynb">Boom</a></p><h3>Recap: Optimizing PyTorch Deep Learning Models</h3><p>In this blog, we explored how to optimize a deep learning model from a naive PyTorch implementation to a highly efficient, concise version. We started by building a model using PyTorch’s nn.Module, providing complete control over the training loop, which, while educational, was verbose and complex. This approach made implementing features like validation and model checkpointing challenging, often leading to errors.</p><p>We then transitioned to better PyTorch abtractions, which significantly simplified the process by abstracting away boilerplate code, such as the training loop and checkpointing. This enabled us to focus more on improving the model rather than dealing with the intricacies of implementation.</p><p>Through further optimizations, we shortened the code from 108 lines to 64, leveraging lambda functions, list comprehensions, and grouping similar logic to reduce redundancy. These improvements led to better performance and a more manageable codebase, demonstrating how functional programming techniques can make deep learning code both shorter and more efficient.</p><p>The journey culminated in showcasing how to build a deep learning model with as few as 64 lines of code, focusing on striking the perfect balance between performance and readability.</p><p>We pick up where we left off in the previous chapter, continuing with the same code, which can be found <a href="https://github.com/Karindraj/High-Performance-Deep-learning/blob/main/Write%20Less%20code.ipynb">here</a>. This code is quite verbose, and we’ve made several key changes to improve it:</p><ol><li><strong>Moved Flattening of Inputs</strong>: Flattening (using view) has been moved to the forward() method to simplify the training_step, validation_step, and test_step.</li><li><strong>Reused Cross Entropy Loss</strong>: Instead of defining a separate criterion, we directly used F.cross_entropy().</li><li><strong>Reduced Redundant Operations</strong>: Similar operations were combined, and the explicit dataset definition for testing was removed.</li><li><strong>Used ‘auto’ for Accelerator and Devices</strong>: To simplify GPU usage, we used ‘auto’ for both the accelerator and devices in the trainer.</li><li><strong>Removed Explicit Timing Code</strong>: Since PyTorch Lightning provides internal profiling and logging, explicit timing code was removed for simplicity.</li></ol><p><strong>Execution Time </strong>:Training Time: 114.35 seconds, Testing Time: 1.42 seconds</p><p><strong>Lines of Code</strong>: 128</p><p><strong>Next, we</strong></p><ul><li><strong>Imports and Class Definition:</strong></li><li>It uses PyTorch Lightning to simplify the training loop and reduce verbosity.</li><li>SimpleNet is defined as a subclass of pl.LightningModule. The model is built using torch.nn.Sequential, making the code cleaner and more concise.</li><li>The forward() method uses view() to flatten input directly within the method.</li><li>The training_step, validation_step, and test_step methods use PyTorch&#39;s F.cross_entropy function and efficient tensor operations to compute loss and accuracy.</li><li><strong>Logging:</strong></li><li>Logging is performed using the self.log() function in each training, validation, and test step to keep track of metrics.</li><li><strong>Trainer and Training:</strong></li><li>A PyTorch Lightning Trainer is used for training, with accelerator and devices set to &#39;auto&#39; for flexibility.</li><li>The code also includes simple timing logic for training and testing.</li></ul><p><strong>Let’s do a Comparison</strong></p><ol><li><strong>Code Conciseness:</strong></li></ol><ul><li>The optimized version uses torch.nn.Sequential for defining the model layers, reducing verbosity. It also avoids repeated operations by consolidating common actions within the forward() method.</li><li>The naive version has more explicit definitions, which makes it longer but perhaps more suitable for beginners who need to see all details clearly.</li></ul><ol><li><strong>Logging and Loss Computation:</strong></li></ol><ul><li>In the optimized version, F.cross_entropy() is used directly in the training, validation, and test steps, removing the need for a separate loss function.</li><li>In the naive version, CrossEntropyLoss is defined separately, leading to more explicit (but longer) code.</li></ul><ol><li><strong>Training Logic:</strong></li></ol><ul><li>The optimized version uses log() for logging relevant metrics like loss and accuracy in each step, providing a streamlined approach.</li><li>The naive version explicitly flattens input images multiple times, while the optimized version moves this to the forward() method, reducing redundancy.</li></ul><ol><li><strong>Hardware Usage:</strong></li></ol><ul><li>The optimized version uses ‘auto’ settings for accelerator and devices, making it adaptable to the available hardware.</li><li>The naive version explicitly checks for GPU availability, which requires more code and lacks the same flexibility.</li></ul><p><strong>Execution Time </strong>:Training Time: 112.91 secondsTesting Time: 1.41 seconds</p><p><strong>Lines of Code:61 lines</strong> Voila!!!!</p><p><strong>Next , Further Changes are Made:</strong></p><ol><li><strong>Used </strong><strong>torch.nn.Sequential</strong>: Combined the linear layers and activation into a single model using Sequential to make it more concise.</li><li><strong>Replaced Method Definitions with Lambdas</strong>: Converted forward, training_step, validation_step, test_step, and configure_optimizers into lambda functions to reduce verbosity.</li><li><strong>Shared Logic in </strong><strong>_shared_step</strong>: Created a _shared_step function to handle the common logic between training_step, validation_step, and test_step.</li></ol><p><strong>Execution Time </strong>:Training Time: 116.01 seconds, esting Time: 1.38 seconds</p><p><strong>Lines of Code</strong>: 51 lines okayyyy!!!</p><p><strong>Note: </strong>PyTorch Native is often faster than PyTorch Lightning for several reasons:</p><ol><li><strong>Abstraction Overhead</strong>: PyTorch Lightning provides a higher-level abstraction over PyTorch, which aims to simplify the training loop by handling tasks like logging, validation, checkpointing, and more. This abstraction adds some computational overhead, as it involves more steps and generalized structures compared to the more direct implementation in PyTorch Native.</li><li><strong>Customization Flexibility</strong>: In Native PyTorch, developers have complete control over every aspect of the model and training loop, allowing for optimization tailored specifically to the problem, hardware, and dataset. PyTorch Lightning, while convenient, follows a standardized framework that can be less optimized for specific scenarios, leading to potentially slower execution.</li><li><strong>Execution Path Complexity</strong>: Lightning handles multiple features like multi-GPU training, precision adjustment, and integration of callbacks. This additional logic in the execution path can introduce minor latencies compared to a lightweight, manually controlled PyTorch implementation, which focuses solely on core training without additional monitoring.</li><li><strong>Fine-Tuning of CUDA Operations</strong>: In PyTorch Native, operations like setting manual seeds, optimizing CUDA streams, and controlling synchronization points can be more finely tuned. PyTorch Lightning abstracts this, which might not always result in optimal performance.</li></ol><p><strong>Finally, t</strong>he code achieves a smaller and potentially faster implementation compared to more verbose versions by utilizing concise, functional programming techniques and eliminating redundant operations. Let’s break down the key features and optimizations that contribute to the reduced line count and efficiency:</p><h3>1. Use of Lambda Functions for Conciseness</h3><ul><li>The forward, shared_step, training_step, validation_step, and test_step methods are all defined using lambda functions. This approach removes the need for explicit, multi-line method definitions, making the code significantly more compact.</li><li>forward = lambda self, x: self.model(x.view(x.size(0), -1)): This lambda function flattens the input and passes it through the model in a single line, which simplifies the forward() definition.</li><li>The training_step, validation_step, and test_step methods are defined as lambdas that call a shared step function. This keeps the logic unified and reduces duplication.</li></ul><h3>2. Shared Step Logic with shared_step Method</h3><ul><li>A shared_step function handles both loss calculation and accuracy measurement depending on the step (train_loss, val_acc, or test_acc), which reduces the amount of repeated code.</li><li>For the training step, cross-entropy loss is calculated and logged, while for validation and test steps, accuracy is computed using torchmetrics.functional.accuracy. This reuse of logic results in a more compact and readable codebase.</li></ul><h3>3. Use of torchmetrics for Accuracy Calculation</h3><ul><li>The accuracy function from torchmetrics.functional is used directly, allowing for a concise calculation of the accuracy metric. This removes the need for manual accuracy calculation, reducing code complexity and improving clarity.</li></ul><h3>4. Efficient DataLoader Initialization</h3><ul><li>The loaders dictionary is defined using a dictionary comprehension to manage both training and validation data loaders in a compact manner. This reduces redundancy by combining multiple DataLoader initializations into a single line.</li><li>The use of for name in [&#39;train&#39;, &#39;val&#39;] allows efficient reuse of code for loading datasets, avoiding the need for multiple distinct lines.</li></ul><h3>5. Single Trainer Loop for Both Training and Testing</h3><ul><li>Instead of defining separate blocks for training and testing, the main() function uses a loop (for stage in [&#39;fit&#39;, &#39;test&#39;]) to handle both stages. The getattr() function is used to dynamically call either trainer.fit() or trainer.test() based on the current stage, reducing repetitive lines.</li><li>This also makes the code more flexible if additional training stages are added in the future.</li></ul><h3>6. Automatic Device Selection and Configuration</h3><ul><li>The trainer is configured with &#39;auto&#39; for both accelerator and devices, which automatically detects and uses the available hardware (e.g., GPU if available). This removes the need for manual checks for GPU availability and reduces extra lines.</li></ul><h3>7. Reduction of Explicit Logic and Redundancy</h3><ul><li>The use of lambda functions and shared logic helps remove repeated code blocks, making the code more concise and efficient.</li><li>Explicit operations like flattening the input and defining separate loss functions are combined into fewer, multifunctional lines, reducing verbosity.</li></ul><h3>Why This Code is Smaller and Potentially Faster</h3><ol><li><strong>Smaller Code</strong>:</li></ol><ul><li>By using lambda functions, dictionary comprehensions, and shared logic for common tasks, the code size is greatly reduced. Fewer lines mean less redundancy and better readability for those comfortable with functional programming styles.</li><li>The lambda functions eliminate the need for defining separate methods for simple operations, resulting in concise method definitions.</li></ul><ol><li><strong>Potentially Faster Code</strong>:</li></ol><ul><li>While the reduction in lines of code itself does not directly make the execution faster, the refactored approach leads to fewer function calls and better management of shared operations, which can have a slight impact on efficiency.</li><li>The shared_step method reduces the repeated calls to similar logic, which makes each training or validation step more efficient.</li><li>Using torchmetrics for accuracy is more optimized for PyTorch workflows compared to manual tensor operations, which could also contribute to faster execution times.</li></ul><p><strong>Execution Time </strong>:Training Time: 88.85 seconds, esting Time: 2.01 seconds</p><p><strong>Lines of Code</strong>: 48</p><p><strong>Important : </strong>The “Boom” model focus is on <strong>reducing training time </strong>and the <strong>number of lines of code</strong>, which is apparent from the faster training time. The optimizations done for training might not translate well to testing, causing the test time to be slightly longer. The operations used during training, such as more aggressive parameter updates or specific caching mechanisms, may not benefit the inference phase as much.</p><h3>Conclusion</h3><p>This blog covered the evolution of a PyTorch deep learning model from a naive implementation to a highly optimized version. Starting with 128 lines of code and a training time of 114.35 seconds, we demonstrated how different techniques like PyTorch Lightning and lambda functions could significantly reduce code complexity and training time. Each model version achieved progressively fewer lines of code, with the “Boom” model being the most compact at 48 lines and demonstrating the fastest training time at 88.85 seconds. Despite these improvements, we observed that the testing time slightly increased in the “Boom” version, likely due to trade-offs in optimization that focused mainly on training efficiency.</p><p>The results show that reducing code verbosity and focusing on efficient training can lead to better overall performance, but some careful adjustments are needed to maintain testing efficiency. Optimizing for compactness should always be balanced with execution considerations to avoid impacting performance during evaluation.</p><p>In the next section, we will focus on execution time in more depth and introduce a comprehensive comparison study across various code versions. We’ll explore the integration of CuDNN and even consider a new programming language. Stay tuned!</p><h3>High Performance Deep learning Part 3</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bd35147b25c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Performance Deep learning Part 1]]></title>
            <link>https://medium.com/@official.indrajit.kar/high-performance-deep-learning-part-1-5858a24cfe96?source=rss-d27657cda394------2</link>
            <guid isPermaLink="false">https://medium.com/p/5858a24cfe96</guid>
            <category><![CDATA[high-performace-computing]]></category>
            <category><![CDATA[code-like-a-pro]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[code-optimization]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Indrajit]]></dc:creator>
            <pubDate>Sun, 29 Sep 2024 13:40:57 GMT</pubDate>
            <atom:updated>2024-12-03T15:15:55.185Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Mastering Deep Learning Optimization: Speed Up Your Pytoch Code and Write less code</strong></p><p><strong>By Indrajit Kar and Zonunfeli Ralte</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hz7w8QFNDNYsjvhL" /><figcaption>Reduced Lines of code faster execution</figcaption></figure><h3>Introduction</h3><p>Deep learning has become an essential tool in machine learning, revolutionizing industries ranging from image recognition to natural language processing. However, building effective and efficient deep learning models is often a challenging task, especially when striving for optimal performance. If you’ve ever started coding a deep learning model using PyTorch’s nn.Module and realized it’s not fast or efficient enough, this blog is for you. Here, we will go through the journey of improving our deep learning code, making it faster and more optimized while also reducing the lines of code—ultimately, simplifying the process.</p><p>We will start with a naive implementation using PyTorch’s standard nn.Module, proceed to adopt PyTorch Lightning, and then optimize our model further to make it smaller and faster. The goal is to demonstrate how to maximize performance while minimizing code complexity.</p><h3>Blog Structure</h3><ol><li><strong>Introduction</strong> — Setting the stage for deep learning optimization.</li><li><strong>Naive PyTorch Implementation</strong> — Initial approach with the traditional PyTorch nn.Module.</li><li><strong>Naive PyTorch Lightning Implementation</strong> — Transition to PyTorch Lightning for easier experimentation and more built-in functionality.</li><li><strong>Optimized PyTorch Lightning Code</strong> — Leveraging PyTorch Lightning abstractions to make the code faster and shorter.</li><li><strong>Conclusion</strong> — Summarizing the impact of optimizations on performance and code quality.</li></ol><p>All codes downloads MNIST dataset and keeps the Hyperparameters constant. you can use Google Colab</p><p>input_size = 784 # 28x28 images</p><p>hidden_size = 500</p><p>num_classes = 10</p><p>batch_size = 100</p><p>learning_rate = 0.001</p><p>num_epochs = 10</p><p>All code downloads the MNIST dataset and keeps the hyperparameters constant. You can use Google Colab (T4) if you want to follow along .</p><h3>Naive PyTorch Implementation</h3><p>The starting point for any deep learning project is usually a simple approach: a deep learning model built with PyTorch’s nn.Module. This traditional approach offers complete control over the training loop and model implementation. However, this level of control comes with increased code complexity. The training loop has to be defined manually, with explicit code for optimizers, loss functions, and iteration over batches.</p><p>This approach is good for learning purposes, as it provides deep insights into how the training loop works. However, as the complexity of the model grows, so does the code, making it harder to manage. This is where moving to a more structured framework like PyTorch Lightning becomes beneficial.</p><p><strong>Challenges</strong>:</p><ul><li><strong>Verbosity</strong>: Writing everything manually results in verbose code.</li><li><strong>Complexity</strong>: Implementing features like validation, model checkpointing, and logging requires more custom code.</li><li><strong>Error-prone</strong>: Handling every detail manually can lead to more bugs and less focus on model improvements.</li></ul><h3>Code Implementation of Naive PyTorch</h3><p>The code of this blog can be found<a href="https://github.com/Karindraj/High-Performance-Deep-learning"> here</a></p><ul><li><strong>Importing libraries</strong>:<br>The necessary Python libraries are imported:</li><li>time is used for tracking the time taken for training and testing.</li><li>torch, the core PyTorch library, is used for tensor computations and GPU acceleration.</li><li>torch.nn contains neural network building blocks.</li><li>torch.optim handles optimization algorithms like Adam.</li><li>DataLoader is used for managing datasets.</li><li>transforms handles image transformations.</li><li>MNIST dataset from torchvision is used for digit classification.</li><li><strong>Defining the neural network (</strong><strong>SimpleNet)</strong>:<br>A basic neural network class is defined using PyTorch&#39;s nn.Module. It consists of:</li><li>An input layer that flattens the MNIST 28x28 images into a 784-dimensional vector.</li><li>A hidden layer with 500 neurons and a ReLU activation function for non-linearity.</li><li>An output layer with 10 neurons, corresponding to the 10 possible digit classes.</li><li><strong>Training function (</strong><strong>train)</strong>:<br>This function trains the network:</li><li>The model is set to training mode.</li><li>For each batch of images and labels, the images are flattened and passed through the model.</li><li>The loss is calculated using the model’s output and actual labels.</li><li>The optimizer updates the model’s weights by performing backpropagation.</li><li>Every 100 batches, the current batch loss is printed.</li><li><strong>Testing function (</strong><strong>test)</strong>:<br>This function evaluates the model:</li><li>The model is set to evaluation mode, and gradient calculations are turned off.</li><li>Each test image is flattened and passed through the network.</li><li>The model’s predictions are compared to the actual labels to calculate the number of correct predictions.</li><li>The accuracy of the model on the test dataset is printed.</li><li><strong>Main function (</strong><strong>main)</strong>:</li><li><strong>Hyperparameters</strong> like input size, hidden size, batch size, learning rate, and number of epochs are defined.</li><li><strong>Transformations</strong> are set up to convert images to tensors.</li><li><strong>Datasets</strong> are loaded for training and testing using MNIST.</li><li><strong>Data loaders</strong> are created to manage batches of data for training and testing.</li><li>The SimpleNet model is initialized and moved to the GPU if available.</li><li>The loss function (cross-entropy loss) and optimizer (Adam) are defined.</li><li>The time for training is tracked, and the model is trained over several epochs.</li><li>After training, the time is recorded, and the model is tested on the test set, with the testing time also recorded.</li><li><strong>Program entry point</strong>:<br>The main() function is called when the script is executed. This serves as the entry point for the program.</li></ul><p><strong>Execution Time </strong>:Training Time: 80.16 seconds , Testing Time: 1.22 seconds</p><p><strong>Lines of Code</strong>: 108</p><p><strong>Let’s reduce the number of Lines to 64 by :</strong></p><ul><li>Shortening the code which involved <strong>lambda functions</strong>, <strong>list comprehensions</strong>, and grouping similar logic together. This minimized the use of explicit loops and repetitive variable declarations, while maintaining the structure and logic of the original code.</li><li>The <strong>functional approach</strong> (like replacing explicit methods with inline lambda functions) was a key strategy to reduce the verbosity.</li></ul><h3><strong>Details</strong></h3><h3>1. Module Imports</h3><ul><li>The import statements were simplified. Instead of importing multiple components from torch individually, they were grouped: from torch import nn, optim. This reduces redundancy in import statements.</li></ul><h3>2. Neural Network Definition</h3><ul><li>In the original code, a separate method (forward) defined how the input flows through the layers. This was rewritten using nn.Sequential for simplicity and conciseness.</li><li>The forward method was replaced with a <strong>lambda function</strong> to directly reshape the input and pass it through the model, reducing boilerplate code.</li></ul><h3>3. Training Loop</h3><ul><li>The original training loop, which included separate forward, backward, and optimization steps, was condensed into a single <strong>lambda function</strong> (train_step). This function handles the forward pass, loss calculation, backpropagation, and optimizer steps all at once, eliminating the need for explicit loops and making the code more compact.</li><li>The batch loop inside train_step is implemented as a <strong>list comprehension</strong> to further minimize lines while keeping the logic the same.</li></ul><h3>4. Testing Loop</h3><ul><li>The testing loop was also reduced using a <strong>lambda function</strong> (test_step) and a <strong>list comprehension</strong> to evaluate model accuracy. The original for-loop to check the predictions and labels was condensed into a single line, where the correct predictions are accumulated.</li></ul><h3>5. Parameter Handling</h3><ul><li>Hyperparameters (input size, hidden size, batch size, etc.) were moved into a <strong>dictionary</strong> (params). This allows for easier modification and reduces the number of individual variable declarations.</li></ul><h3>6. DataLoader Initialization</h3><ul><li>The train and test DataLoader initialization was merged into a <strong>dictionary comprehension</strong> (loaders), where both the training and test loaders are created in a more compact way. This removes repetitive code and combines the data loading logic into fewer lines.</li></ul><h3>7. Time Measurement</h3><ul><li>Timing for both training and testing was done similarly, but the code was shortened by reusing the start_time variable and eliminating redundant variable declarations.</li></ul><h3>8. Print Statements</h3><ul><li>The code for calculating and printing training time, accuracy, and testing time was simplified by combining related calculations in fewer lines, avoiding the need for intermediate variables and repeated code.</li></ul><p><strong>Execution Time </strong>:Training Time: 77.87 seconds , Testing Time: 1.17 seconds</p><p><strong>Lines of Code</strong>: 64</p><p><strong>Note:</strong></p><p>I didn’t use from torchmetrics.functional import accuracy because the aim was to keep the code purely PyTorch native, which aligns with the original requirement for optimizing the naive PyTorch code. Since torchmetrics is an external library designed to integrate seamlessly with PyTorch Lightning, it would add an additional dependency that isn&#39;t strictly necessary when using PyTorch natively.</p><p>Instead, I used the PyTorch native approach to calculate accuracy directly within the test loop, which keeps the code straightforward and consistent with the original pure PyTorch implementation. Using torch.argmax to determine predictions and then comparing them with the labels achieves the same result without introducing additional libraries.</p><p>If you prefer to use torchmetrics for a cleaner and more modular accuracy calculation, we could integrate it later blogs .</p><h3>Next :<strong>Naive PyTorch Lightning Implementation</strong></h3><p>We just scratched the surface. Optimizing deep learning models isn’t only about tweaking hyperparameters or using better hardware; it also involves reducing code complexity and improving code efficiency. Starting with the naive PyTorch implementation, we manually defined every aspect of our training loop, which provided control but at the cost of verbosity and complexity. By moving to PyTorch Lightning, we greatly simplified our workflow, enabling more efficient experimentation.</p><h3><a href="https://medium.com/@official.indrajit.kar/high-performance-deep-learning-part-2-4bd35147b25c?source=your_stories_page-------------------------------------">High Performance Deep learning Part 2</a></h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5858a24cfe96" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>