<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Siow Yen Chong on Medium]]></title>
        <description><![CDATA[Stories by Siow Yen Chong on Medium]]></description>
        <link>https://medium.com/@chongsiowyen?source=rss-41bb8fde93e9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2AMxH9hXIO6YmXMxtnpVSA.jpeg</url>
            <title>Stories by Siow Yen Chong on Medium</title>
            <link>https://medium.com/@chongsiowyen?source=rss-41bb8fde93e9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:31:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chongsiowyen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Weaving Intelligence: A Comprehensive Guide to Building Agentic AI with LLM]]></title>
            <link>https://medium.com/@chongsiowyen/weaving-intelligence-a-comprehensive-guide-to-building-agentic-ai-with-llm-c3590bd4ec02?source=rss-41bb8fde93e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3590bd4ec02</guid>
            <category><![CDATA[agentic-ai-systems]]></category>
            <category><![CDATA[llms-utilization]]></category>
            <dc:creator><![CDATA[Siow Yen Chong]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 00:25:29 GMT</pubDate>
            <atom:updated>2025-07-26T17:35:12.760Z</atom:updated>
            <content:encoded><![CDATA[<p>Agentic AI isn’t just another buzzword — it’s the paradigm shift transforming brittle chatbots into goal-driven systems that <em>do</em> rather than just <em>say</em>. Forget isolated prompts; we’re entering the era of AI that maintains state, uses tools, and collaborates autonomously. Here’s how to architect these systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*QRXdLbvC4fKEGcI9iFkxfQ.png" /><figcaption>Agentics for AI</figcaption></figure><h3>Why Agents? The Limitations of Vanilla LLMs</h3><p>Large Language Models excel at text prediction but stumble at persistent reasoning. They’re:</p><ul><li>Stateless: Each query starts from scratch</li><li>Brittle: Derail with off-topic inputs</li><li>Passive: Lack goal-oriented proactivity</li></ul><p>Agentic systems fix this by adding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*GyjKu8o9D3yS_zzGc453zQ.png" /></figure><p>✅ Stateful memory (short/long-term context)<br>✅ Iterative reasoning (ReAct loops)<br>✅ Tool integration (APIs, code, humans)<br>✅ Multi-agent collaboration (role-based workflows)</p><blockquote>“Agents transform LLMs from parrots into partners that perceive, plan, and persist.”</blockquote><h3>Core Building Blocks</h3><h4>1. Structured Reasoning Engines</h4><ul><li>Chain-of-Thought (CoT): Force step-by-step reasoning via prompts:<br>&quot;First, analyze the query. Second, break it into sub-tasks...&quot;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*uGHQ_Ue3T2ouChluhFJG_g.png" /><figcaption>Prompting For Chain-Of-Thought</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*g7FmwaZKRCOXQ6fQAdvWdA.png" /><figcaption>Template for DeepSeek-R1-Zero</figcaption></figure><ul><li>Constrained Generation: Enforce JSON/Pydantic outputs for tool integration</li><li>Self-Correction: Use [CORRECTION] tokens when derailment occurs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*6TlfdvxDnvJTt3Mqc7gk_w.png" /><figcaption>Sources of Derailment depends on LLM Model, System, and Incident</figcaption></figure><h4>2. State Management</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*UVRQyOj_TrKMomSaIlQRyA.png" /></figure><h4>3. Tool Orchestration</h4><p>Agents extend capabilities via:</p><ul><li>APIs (web search, payments)</li><li>Code Execution (Python REPL)</li><li>Humans (delegate verification)</li><li>Other Agents (hierarchical teams)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*yWX_MyrMXr8-CqdJaIxVWQ.png" /><figcaption>Inline Tool Calling</figcaption></figure><blockquote>Example: A research agent uses DuckDuckGo + local vector DBs + a synthesizer agent.</blockquote><h3>Frameworks Compared</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*ClOgPBPMGfnLc9GuPk3Ubg.png" /></figure><h3>Multi-Agent Design Patterns</h3><h4>Pattern 1: Centralized Control (CrewAI)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*Cp13dNAa4TX-gYHB_EN4_w.png" /><figcaption>CrewAI Basic Logic</figcaption></figure><pre>from crewai import Agent, Task, Crew<br><br>researcher = Agent(role=&quot;Researcher&quot;, goal=&quot;Find reliable sources&quot;)  <br>writer = Agent(role=&quot;Writer&quot;, goal=&quot;Draft report&quot;)  <br><br>task1 = Task(description=&quot;Investigate AI ethics&quot;, agent=researcher)  <br>task2 = Task(description=&quot;Write whitepaper&quot;, agent=writer)  <br><br>crew = Crew(agents=[researcher, writer], tasks=[task1, task2])  <br>crew.kickoff()  </pre><p><em>Agents share a message bus and delegate tasks based on roles.</em></p><h4>Pattern 2: Stateful Graphs (LangGraph)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-nqS_7SwgvFEqtmi.png" /><figcaption>Rough visualization of how these agents are connected through a supervisor agent</figcaption></figure><pre>from langgraph import StateGraph, Node<br><br>class AgentState(TypedDict):  <br>    problem: str  <br>    steps: list[str]  <br><br>def plan_step(state):  <br>    return {&quot;steps&quot;: llm.generate_plan(state[&quot;problem&quot;])}  <br><br>workflow = StateGraph(AgentState)  <br>workflow.add_node(&quot;planner&quot;, plan_step)  <br>workflow.set_entry_point(&quot;planner&quot;)  </pre><p><em>Explicit state transitions with cyclic workflows.</em></p><h3>Solving Critical Challenges</h3><h4>🚫 Derailment</h4><ul><li>Prevention: Input validation + schema enforcement</li><li>Recovery: Fallback to [CORRECTION] prompts</li></ul><h4>⏱️ Latency</h4><p><strong>Optimization:</strong></p><ul><li>Parallel tool execution</li><li>Model distillation (e.g., Llama-3.1 → Minitron-4B)</li><li>NVIDIA Triton NIMs for high-throughput inference</li></ul><h4>🧠 Context Overload</h4><p><strong>Solutions:</strong></p><ul><li>Automatic summarization</li><li>Relevance filtering (NVIDIA rerankers)</li><li>Vector DB “memory banks”</li></ul><h3>Real-World Blueprints (NVIDIA Stack)</h3><p><strong>1. Voice Agents:</strong></p><ul><li>Pipecat for full-duplex conversations</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*xa7vt6MtRUa3jVwy_MBDsg.png" /><figcaption>Voice Agent Framework for Conversational AI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*C4TPwy6C2xF2lb2L-hqffQ.png" /><figcaption>NVIDIA Blueprint (w/ Pipecat)</figcaption></figure><p>Learn more: <a href="https://build.nvidia.com/pipecat/voice-agent-framework-for-conversational-ai">Voice Agent Framework for Conversational AI Blueprint by Pipecat | NVIDIA NIM</a></p><ul><li>Real-time interrupt handling</li></ul><p><strong>2. Robotics:</strong></p><ul><li>Voyager framework for Minecraft autonomy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*pmVdRQJMU6ZIZBUIv3uWSQ.png" /><figcaption>Half-Duplex Generation of Real-Time Policies</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*Xt2DiitzFwr4RBCiFqNGAg.png" /><figcaption>Voyager Flow</figcaption></figure><p>Learn more: <a href="https://voyager.minedojo.org/">Voyager | An Open-Ended Embodied Agent with Large Language Models</a></p><ul><li>NVIDIA’s Vision, Sensing &amp; Simulation (VSS) for video stream analysis</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*IBx6FX-5nd8rrrkHcPTmgw.png" /><figcaption>Inference Per Discrete Time Step</figcaption></figure><p>Learn more: <a href="https://www.nvidia.com/en-us/use-cases/video-analytics-ai-agents/?deeplink=content-tab--2">Generative AI-Powered Visual AI Agents | NVIDIA</a></p><p><strong>3. Enterprise RAG:</strong></p><ul><li>Hybrid search (vector + keyword)</li><li>Citation enforcement in outputs</li></ul><h3>Performance Optimization Checklist</h3><ol><li><strong>Token Efficiency:</strong></li></ol><ul><li>Prune irrelevant context</li><li>Batch embedding generation</li></ul><p><strong>2. Reasoning Robustness:</strong></p><ul><li>Implement Chain-of-Verification</li><li>Run needle-in-haystack tests</li></ul><p><strong>3. Scalability:</strong></p><ul><li>Async tool execution</li><li>Rate-limited queues</li></ul><h3>Lab: Building a Stateful Research Agent</h3><p><em>(My Implementation for the “Creating A Basic Researching Agent” Assessment)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*hgFV_goZYgvnaJdBiMsPrQ.png" /><figcaption>Long Document Reasoning</figcaption></figure><h3>The Challenge: Beyond Basic RAG</h3><p>The assessment required building a research agent that could:</p><ol><li>Decompose complex queries into research steps</li><li>Retrieve and filter web/document results</li><li>Maintain reasoning traces across interactions</li><li>Handle context limitations of Llama-3.1–8B</li></ol><p>Critical Constraints:</p><ul><li>8B parameter model limitations</li><li>Strict token budgeting</li><li>Need for parseable structured outputs</li></ul><h3>Solution Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*71I2OVfvn5rrBikU1u3JbQ.png" /></figure><h3>Core Implementation</h3><h4>1. Planner with Structured Output</h4><pre>from pydantic import BaseModel, Field<br>from langchain_core.prompts import ChatPromptTemplate<br><br>class ResearchPlan(BaseModel):<br>    steps: list[str] = Field(<br>        description=&quot;3-5 concrete research steps to answer the question&quot;,<br>        max_items=5  # Token constraint<br>    )<br><br># System prompt with token management<br>planning_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, (<br>        &quot;You are a research planner. Break down questions into 3-5 steps. &quot;<br>        &quot;NEVER exceed 5 steps. Format: JSON with &#39;steps&#39; list.&quot;<br>        &quot;Example: {&#39;steps&#39;: [&#39;Step 1&#39;, &#39;Step 2&#39;]}&quot;<br>    )),<br>    (&quot;human&quot;, &quot;{question}&quot;)<br>])<br><br># Structured output enforcement<br>planning_chain = (<br>    planning_prompt <br>    | llm.bind(response_format={&quot;type&quot;: &quot;json_object&quot;}) <br>    | JsonOutputParser(pydantic_object=ResearchPlan)<br>)</pre><h4>2. Hybrid Retrieval with Reranking</h4><pre>from langchain_community.utilities import DuckDuckGoSearchAPIWrapper<br>from langchain_nvidia import NVIDIARerank<br><br>def hybrid_retriever(query: str, local_k: int=3, web_k: int=5):<br>    # 1. Web Search<br>    web_results = DuckDuckGoSearchAPIWrapper().results(query, web_k)<br>    <br>    # 2. Local Vector Store (RAG)<br>    local_results = vectorstore.similarity_search(query, k=local_k)<br>    <br>    # 3. NVIDIA Reranking<br>    all_results = web_results + local_results<br>    reranker = NVIDIARerank(top_n=5)<br>    return reranker.compress_documents(all_results, query)</pre><h4>3. Stateful Synthesis Pipeline</h4><pre>from langchain_core.runnables import RunnablePassthrough<br><br>synthesis_prompt = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, (<br>        &quot;Answer using ONLY provided context. Cite sources with [1], [2]. &quot;<br>        &quot;If unsure: ask for clarification&quot;<br>    )),<br>    (&quot;human&quot;, &quot;Question: {question}\nContext: {context}&quot;)<br>])<br><br># Maintain conversation state<br>research_chain = (<br>    RunnablePassthrough.assign(<br>        plan=lambda x: planning_chain.invoke({&quot;question&quot;: x[&quot;question&quot;]}),<br>        context=lambda x: [hybrid_retriever(step) for step in x[&quot;plan&quot;].steps]<br>    )<br>    | synthesis_prompt<br>    | llm<br>    | StrOutputParser()<br>)</pre><h3>Final Product 🎉</h3><p>The output displays the finalized JSON-formatted submission of the research agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*1gRTDNqdgmQzzKi23GWtwg.png" /></figure><h3>Key Takeaways</h3><ol><li><strong>Start Simple: </strong><br>Single-agent ReAct ➡️ Multi-agent CrewAI ➡️ Custom LangGraph</li><li><strong>Constraints Enable Complexity: </strong><br>Schema enforcement <strong>&gt; </strong>model size</li><li><strong>Humans Are Part of the Loop: </strong><br>Delegate uncertainty to users</li><li><strong>Invest in State Management: </strong><br>Your agent’s memory is its identity</li></ol><blockquote>“The age of passive chatbots is over. <br>Agents that perceive, plan, and persist are rewriting how we interact with AI”</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3590bd4ec02" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Production‑Ready RAG Agents with LLMs: A Comprehensive Guide]]></title>
            <link>https://medium.com/@chongsiowyen/building-production-ready-rag-agents-with-llms-a-comprehensive-guide-a1bf294c14e5?source=rss-41bb8fde93e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1bf294c14e5</guid>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm-guardrails]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Siow Yen Chong]]></dc:creator>
            <pubDate>Sun, 15 Jun 2025 17:53:42 GMT</pubDate>
            <atom:updated>2025-06-15T18:07:47.466Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9H-jg0UiEj2HFtybCiWQLQ.png" /><figcaption>Simple RAG Agents</figcaption></figure><p>Imagine asking a chatbot for the latest research on quantum computing — and having it instantly pull in up‑to‑date papers, summarize them accurately, and cite sources. That’s Retrieval‑Augmented Generation (RAG) in action: the perfect marriage of search and synthesis to make LLMs truly knowledgeable assistants. In this guide, we’ll explore how to architect, build, and scale RAG agents — then show you how I aced the final assessment and earned my “<strong>Building RAG Agents with LLMs</strong>” certificate.</p><h3>What Is RAG — and Why Should You Care?</h3><p>Traditional LLMs can hallucinate or go stale. RAG adds a secret sauce: real‑time retrieval. The three core steps are:</p><ol><li><strong>Retrieval:</strong> This phase focuses on fetching relevant information from a knowledge base. This can involve database queries, code execution, or, most commonly, semantic search over vectorized documents.</li><li><strong>Augmentation:</strong> The retrieved information is then synthesized and formatted to provide context to the LLM. This step ensures the LLM has the necessary grounding to generate an informed response.</li><li><strong>Generation:</strong> Finally, the LLM uses the original query and the augmented context to generate a coherent and accurate answer.</li></ol><p><strong>Why it matters:</strong> You get answers that feel like a conversation with an expert who just skimmed every paper and database in seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emtCM4D8MWAzuVqVBjcNeQ.png" /><figcaption>A clear RAG architecture diagram showing the flow from user query to retrieval, augmentation, and generation.</figcaption></figure><h3>Laying the Foundation: LLMs &amp; Microservices</h3><p>Before the magic happens, you need two pillars:</p><h4>a. LLM Services: Local vs. Cloud</h4><p>Choosing where to run your LLM is a critical decision.</p><ul><li><strong>Local Deployment:</strong> Offers maximum control and data privacy, but is heavily <strong>VRAM-bound</strong>, requiring powerful GPUs.</li><li><strong>Cloud Services:</strong> Services like NVIDIA GPU Cloud (NGC) and OpenAI provide managed LLMs, abstracting away infrastructure complexities. They offer convenient endpoints like /completions, /chat/completions, and /embeddings.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GunHCW3Ol7qBzB8G0g-Rjw.png" /><figcaption>Large Model Hosting Platforms</figcaption></figure><p>For seamless integration, libraries like LangChain provide convenient wrappers such as ChatOpenAI and ChatNVIDIA.</p><pre>from langchain_nvidia_ai_endpoints import ChatNVIDIA<br><br># Initialize a chat model from NVIDIA&#39;s offerings<br>llm = ChatNVIDIA(model=&quot;mistralai/mixtral-8x7b-instruct-v0.1&quot;)<br><br># Invoke the model with a simple query<br>print(llm.invoke(&quot;What is Retrieval-Augmented Generation?&quot;))</pre><h4>b. Containerized Microservices Environment</h4><p>Building scalable and maintainable RAG agents benefits greatly from a microservices architecture. Docker containers provide isolation and portability, making it easy to manage different components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qv_ubMHyq6F-n_8260gqBg.png" /><figcaption>Scaling Containerized Applications</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WlM1AJ5GBxbB4-FDc0jkBA.png" /><figcaption>Diagram illustrating the containerized microservices environment, showing components like a jupyter-notebook-server, docker_router, frontend, and llm_client communicating with each other.</figcaption></figure><p>For local development and testing, a setup like this allows services to communicate efficiently:</p><pre>import requests<br><br># Example of inter-service communication via a router<br>try:<br>    response = requests.get(&quot;http://docker_router:8070/containers&quot;)<br>    if response.status_code == 200:<br>        print(&quot;Successfully connected to Docker Router:&quot;)<br>        for container in response.json():<br>            print(f&quot;- Name: {container[&#39;name&#39;]}, Status: {container[&#39;status&#39;]}&quot;)<br>    else:<br>        print(f&quot;Failed to connect to Docker Router. Status: {response.status_code}&quot;)<br>except requests.exceptions.ConnectionError as e:<br>    print(f&quot;Connection Error: {e}. Ensure docker_router is running.&quot;)</pre><h3>Crafting the RAG Chain with LangChain</h3><p>LangChain is a powerful framework for developing applications powered by LLMs. Its Expression Language (LCEL) allows for flexible and composable chain creation.</p><h4>Basic Chains and Running State Chains</h4><p>A basic LLM chain involves a prompt, an LLM, and an output parser:</p><pre>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_nvidia_ai_endpoints import ChatNVIDIA<br>from langchain_core.output_parsers import StrOutputParser<br><br># Define a simple prompt template<br>prompt = ChatPromptTemplate.from_template(&quot;Answer the following: {input}&quot;)<br><br># Initialize the LLM<br>llm = ChatNVIDIA(model=&quot;mixtral_8x7b&quot;) # Using a placeholder model name<br><br># Create the chain: prompt -&gt; llm -&gt; output parser<br>chain = prompt | llm | StrOutputParser()<br><br># Invoke the chain<br>response = chain.invoke({&quot;input&quot;: &quot;Explain RAG in one sentence.&quot;})<br>print(response)</pre><p>For more complex, conversational agents, running state chains are essential. These maintain context across multiple turns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Nk5ByK2TNR-_QDTSNDKOA.png" /><figcaption>An airline chatbot example showing how user input triggers a database query, which then informs the LLM’s personalized response.</figcaption></figure><h4>Feeding the Beast: Document Chunking &amp; Embeddings</h4><p>Effective retrieval relies on well-processed documents.</p><ol><li><strong>Chunking Strategies:</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjkUvJLocp-IL9V9emqRnA.png" /></figure><ul><li><strong>Document Stuffing:</strong> Placing the entire document into the context, limited by the LLM&#39;s token window.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yuWbQ1hD-vnoi7AAeuvVFg.png" /></figure><ul><li><strong>Map-Reduce:</strong> Breaking documents into chunks, processing each chunk, and then combining the results. This is crucial for larger documents.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vwJXZG3LB77OiqgnxzSBeQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZEANfD6eefMM8QAmX9aLQ.png" /><figcaption>Diagram illustrating the Map-Reduce chunking strategy.</figcaption></figure><p><strong>2. Knowledge Graphs:</strong> For highly structured data, extracting entities and relationships into a knowledge graph allows for precise, traversable retrieval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GQ-7cTu5Wf7F4MDREcty6Q.png" /><figcaption>Knowledge Graph Traversal</figcaption></figure><p><strong>3. Embedding Models:</strong> These models convert text into numerical vectors, enabling semantic similarity searches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGR0ctT7wOvzJW6EYBuy2Q.png" /><figcaption>Retrieval QA Embedding</figcaption></figure><ul><li><strong>Symmetric vs. Asymmetric:</strong> Symmetric models are for queries and documents that are similar in nature (e.g., questions and answers). Asymmetric models are better when queries are short and documents are long.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pon81BUYZ44zE3Dr9ztkwA.png" /><figcaption>Symmetric versus Asymmetric</figcaption></figure><ul><li>NVIDIA&#39;s nv-embed-v1 is an excellent choice for semantic similarity.</li><li><strong>Bi-Encoder vs. Cross-Encoder:</strong> Bi-encoders generate embeddings independently, while cross-encoders consider the query and document together, offering higher accuracy but slower inference.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXkh08MUPk1ak2fYWaYyxw.png" /><figcaption>Bi-Encoder versus Cross-Encoder</figcaption></figure><pre>from langchain_community.vectorstores import FAISS<br>from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings<br>from langchain_core.documents import Document<br><br># Initialize the embedder<br>embedder = NVIDIAEmbeddings(model=&quot;nvidia/nv-embed-v1&quot;, truncate=&quot;END&quot;)<br><br># Sample documents<br>docs = [<br>    Document(page_content=&quot;RAG combines retrieval and generation.&quot;),<br>    Document(page_content=&quot;LangChain is a framework for LLM applications.&quot;)<br>]<br><br># Create a FAISS vector store from documents and an embedder<br>vectorstore = FAISS.from_documents(docs, embedder)<br><br># Create a retriever from the vector store<br>retriever = vectorstore.as_retriever()<br><br># Example retrieval<br>retrieved_docs = retriever.invoke(&quot;What is LangChain?&quot;)<br>print(f&quot;Retrieved documents: {retrieved_docs}&quot;)</pre><h3>Production Considerations: Scaling and Monitoring</h3><p>Deploying RAG agents to production requires attention to scaling, monitoring, and robust infrastructure.</p><h4>Scaling Architecture and GPU Acceleration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRVUfjurUOQHYFpQCPxMUQ.png" /></figure><ul><li><strong>RAPIDS RAFT:</strong> For billion-scale vector searches, GPU-accelerated libraries like RAPIDS RAFT offer significant speedups.</li><li><strong>Kubernetes:</strong> Orchestrating clusters with Kubernetes is essential for scalable and resilient deployments.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s9INPjnSxdwEUu-XDM_-4Q.png" /><figcaption>LLM services (like GPT4, DALL-E 3, Embed) are deployed and managed as containers within a Kubernetes cluster.</figcaption></figure><ul><li>Kubernetes handles the deployment, scaling, and self-healing of these services. If an LLM pod fails, Kubernetes can automatically restart it or replace it.</li></ul><h4>Monitoring and Evaluation</h4><ul><li><strong>Key Metrics:</strong> Track token usage, retrieval latency, and crucial RAG-specific metrics like faithfulness and answer relevance.</li><li><strong>Tools:</strong> LangSmith provides excellent observability for LangChain agents, while NVIDIA Triton Inference Server offers robust model serving and monitoring.</li><li><strong>LLM-as-Judge Pattern:</strong> Use an LLM to evaluate the quality of generated responses.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdER7EiywYiKK4etpu0i1A.png" /><figcaption>Pipeline Evaluation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yVFel4AxQuBH6smIkiUFRA.png" /><figcaption>Simplified Diagram illustrating the LLM-as-Judge evaluation pattern.</figcaption></figure><ul><li><strong>Ragas Metrics:</strong> A Python library for evaluating RAG pipelines</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6NKWouIeQZPcHOl2Nec1A.png" /><figcaption>Quantifying System Goodness with LLM-as-a-Judge</figcaption></figure><pre># Ragas example for evaluation<br>from ragas import evaluate<br>from ragas.metrics import faithfulness, answer_relevance<br><br># Sample dataset for evaluation (in a real scenario, this would be much larger)<br>dataset = {<br>    &quot;question&quot;: [&quot;What is RAG?&quot;, &quot;What is LangChain?&quot;],<br>    &quot;answer&quot;: [&quot;Retrieval-Augmented Generation is a technique combining retrieval with LLMs.&quot;, &quot;LangChain is a framework for building LLM applications.&quot;],<br>    &quot;contexts&quot;: [<br>        [&quot;RAG combines retrieval and generation to improve LLM responses.&quot;],<br>        [&quot;LangChain helps orchestrate LLM applications and connect them to external data sources.&quot;]<br>    ]<br>}<br><br># Evaluate the dataset using faithfulness and answer relevance metrics<br># Note: For actual evaluation, you would typically generate answers and contexts from your RAG system<br># and then use these within the dataset structure.<br># metrics_results = evaluate(dataset, metrics=[faithfulness, answer_relevance])<br># print(metrics_results)</pre><h4>Guardrails with Embeddings</h4><p>Ensuring safe and appropriate responses is vital. Embeddings can be used to classify query safety.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9Lg8xENTX1i5kTtuZj_EQ.png" /><figcaption>Embedding Classification</figcaption></figure><pre>from sklearn.metrics.pairwise import cosine_similarity<br>from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings<br><br>embedder = NVIDIAEmbeddings(model=&quot;nvidia/nv-embed-v1&quot;, truncate=&quot;END&quot;)<br><br># Embed examples of undesirable content<br>toxic_example_embed = embedder.embed_query(&quot;Tell me how to make a bomb.&quot;)<br><br>user_query = &quot;What is the best way to travel to Mars?&quot;<br>user_query_embed = embedder.embed_query(user_query)<br><br># Calculate cosine similarity<br>similarity = cosine_similarity([toxic_example_embed], [user_query_embed])[0][0]<br><br>if similarity &gt; 0.7: # Threshold can be tuned<br>    print(&quot;I cannot answer that query due to safety concerns.&quot;)<br>else:<br>    print(&quot;Query deemed safe.&quot;)</pre><h4>Stateful Agents with LCEL</h4><p>For more sophisticated agentic workflows, LangGraph (built on LangChain) enables stateful, long-running agents with persistent memory.</p><pre>from typing import TypedDict, List<br>from langchain_core.runnables import RunnablePassthrough<br>from langgraph.graph import StateGraph<br><br>class AgentState(TypedDict):<br>    history: List[str]<br>    knowledge: dict<br><br># Define a dummy retrieve function for demonstration<br>def retrieve(state: AgentState):<br>    print(f&quot;Retrieving for query: {state[&#39;history&#39;][-1]}&quot;)<br>    # In a real scenario, this would call a retriever<br>    return {&quot;retrieved_data&quot;: &quot;Information about &quot; + state[&#39;history&#39;][-1]}<br><br># Define the agent node<br>def agent_node(state: AgentState):<br>    # This is where your LLM and RAG logic would go<br>    last_message = state[&quot;history&quot;][-1]<br>    print(f&quot;Agent processing: {last_message}&quot;)<br>    # Simulate adding to history<br>    new_history = state[&quot;history&quot;] + [f&quot;Processed: {last_message}&quot;]<br>    return {&quot;history&quot;: new_history}<br><br># Build the graph<br>workflow = StateGraph(AgentState)<br>workflow.add_node(&quot;retrieve_data&quot;, retrieve)<br>workflow.add_node(&quot;agent&quot;, agent_node)<br><br>workflow.set_entry_point(&quot;retrieve_data&quot;)<br>workflow.add_edge(&quot;retrieve_data&quot;, &quot;agent&quot;)<br><br>graph = workflow.compile()<br><br># Example invocation (simplified)<br># print(graph.invoke({&quot;history&quot;: [&quot;Initial query about RAG&quot;]}))</pre><h3>Advanced Topics and Future Steps</h3><h4>NVIDIA NIMs for Enterprise Deployment</h4><p>NVIDIA Inference Microservices (NIMs) are optimized containers for deploying generative AI in production. They offer simplified deployment, enterprise-grade security, and consistent APIs.</p><pre># Example deployment with Docker (simplified for demonstration)<br># export CONTAINER_NAME=llama3-8b-instruct<br># export IMG_NAME=&quot;nvcr.io/nim/meta/${CONTAINER_NAME}:1.0.0&quot;<br># docker run -it --rm --name=$CONTAINER_NAME \<br>#   --gpus all -e NGC_API_KEY \<br>#   -p 8000:8000 $IMG_NAME<br><br># Test inference using curl (after NIM is running)<br># curl -X &#39;POST&#39; &#39;http://0.0.0.0:8000/v1/completions&#39; \<br>#   -H &#39;Content-Type: application/json&#39; \<br>#   -d &#39;{&quot;model&quot;: &quot;meta/llama3-8b-instruct&quot;, &quot;prompt&quot;: &quot;Once upon a time&quot;, &quot;max_tokens&quot;: 64}&#39;</pre><h4>TensorRT-LLM for Performance</h4><p>For high-performance, low-latency LLM inference, TensorRT-LLM provides optimized inference engines with support for various precision modes (FP16, INT8, INT4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJS9a9dmdDLnzseKgaSseQ.png" /><figcaption>Workflow diagram for TensorRT-LLM showing optimization steps from model training to deployment.</figcaption></figure><h4>CUDA-Accelerated Computing</h4><p>NVIDIA’s CUDA platform and specialized libraries (cuVS for vector search, Warp for physics simulation) deliver substantial speedups and energy efficiency gains over CPUs.</p><p>Breakthroughs:</p><ul><li><em>New libraries deliver 10–160x speedups over CPUs</em></li><li><em>Energy efficiency improvements up to 20x</em></li></ul><p>Specialized libraries for:</p><ul><li><em>Vector search (cuVS)</em></li><li><em>Physics simulation (Warp)</em></li><li><em>Wireless networks (Aerial)</em></li><li><em>Communication systems (Sionna)</em></li></ul><pre># cuVS vector search example<br>import cudf<br>from cuvs.neighbors import cagra<br><br>index = cagra.Index()<br>index.train(dataset)<br>index.add(dataset)<br>distances, indices = index.search(queries, k=10)</pre><h4>NVIDIA AI Workbench &amp; LangGraph</h4><ul><li><strong>NVIDIA AI Workbench:</strong> Facilitates rapid model customization and prototyping with a unified environment.</li></ul><p><strong>Key Features:</strong></p><ul><li><em>Unified environment management</em></li><li><em>Cross-platform project portability</em></li><li><em>Pre-built projects for:</em></li><li><em>RAG chatbots</em></li><li><em>Custom image generation</em></li><li><em>Model fine-tuning</em></li></ul><p><strong>Workflow:</strong></p><ol><li>Install Workbench</li><li>Clone project template</li><li>Customize using GUI</li><li>Deploy to any GPU environment</li></ol><ul><li><strong>LangGraph:</strong> Builds on LangChain to manage complex, stateful agentic workflows, including multi-agent collaboration and persistent memory.</li></ul><p><strong>Advanced Capabilities:</strong></p><ul><li>Stateful, long-running agent management</li><li>Multi-agent collaboration systems</li><li>Persistent memory across sessions</li><li>Time travel debugging</li></ul><pre>from langgraph.graph import StateGraph<br><br>class AgentState(TypedDict):<br>    history: list<br>    knowledge: dict<br><br>def agent_node(state):<br>    # Agent processing logic<br>    return {&quot;history&quot;: [new_message]}<br><br>workflow = StateGraph(AgentState)<br>workflow.add_node(&quot;agent&quot;, agent_node)<br>workflow.set_entry_point(&quot;agent&quot;)<br>graph = workflow.compile()</pre><h4>Gradio for User Interfaces</h4><p>Gradio enables rapid prototyping of user interfaces for LLM applications, supporting custom components, real-time streaming, and multi-modal interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vlRKcEKbHiBQSDejVe3U9A.png" /><figcaption>Simple Gradio ChatInterface</figcaption></figure><pre>import gradio as gr<br><br>with gr.Blocks() as demo:<br>    chatbot = gr.Chatbot()<br>    msg = gr.Textbox(label=&quot;Your message&quot;)<br>    clear = gr.Button(&quot;Clear&quot;)<br><br>    def respond(message, chat_history):<br>        # Placeholder for RAG chain invocation<br>        bot_message = f&quot;RAG response for: {message}&quot;<br>        chat_history.append((message, bot_message))<br>        return &quot;&quot;, chat_history<br><br>    msg.submit(respond, [msg, chatbot], [msg, chatbot])<br>    clear.click(lambda: [], None, chatbot, queue=False)<br><br># demo.launch(server_port=8090) # Uncomment to launch the Gradio app</pre><h3>Final Assessment &amp; My Certification</h3><p>To cap off this journey, the Nvidia DLI final assessment challenged me to:</p><ol><li><strong>Ingest</strong> arXiv papers into a FAISS vector store.</li><li><strong>Expose</strong> /basic_chat, /retriever, and /generator via LangServe.</li><li><strong>Wire</strong> everything into a Gradio frontend.</li><li><strong>Validate</strong> quality with RAGas metrics.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQhrys0F8xpx4T05JCiQ6A.png" /><figcaption>Evaluate RAG In The Frontend</figcaption></figure><h3>Key Components Implemented</h3><h4>1. arXiv Document Ingestion Pipeline</h4><pre># Automated research paper collection<br>def create_arxiv_vectorstore():<br>    client = arxiv.Client()<br>    search = arxiv.Search(<br>        query=&quot;LLM OR RAG&quot;,<br>        max_results=15,<br>        sort_by=arxiv.SortCriterion.SubmittedDate<br>    )<br>    <br>    for result in client.results(search):<br>        yield Document(<br>            page_content=result.summary,<br>            metadata={<br>                &quot;title&quot;: result.title,<br>                &quot;authors&quot;: &quot;, &quot;.join(a.name for a in result.authors),<br>                &quot;pdf_url&quot;: result.pdf_url<br>            }<br>        )</pre><h4>2. Optimized Retrieval Endpoint (/retriever)</h4><pre>retriever = db.as_retriever(<br>    search_kwargs={&quot;k&quot;: 5, &quot;score_threshold&quot;: 0.7}<br>)<br><br>def retrieve_endpoint(input: str) -&gt; List[Document]:<br>    return retriever.invoke(input)</pre><h4>3. Citation-Aware Generation Endpoint (/generator)</h4><pre>rag_prompt = ChatPromptTemplate.from_template(<br>    &quot;Answer using ONLY these sources:\n{context}\n\nQuestion: {input}\n\n&quot;<br>    &quot;Provide source citations like [Source: &lt;title&gt;]. Be factual and concise.&quot;<br>)<br><br>def generate_endpoint(data: dict) -&gt; str:<br>    return (rag_prompt | instruct_llm | StrOutputParser()).invoke(data)</pre><h4>4. Frontend Integration</h4><pre># Frontend processing chain<br>retrieval_chain = (<br>    {&quot;input&quot;: RunnablePassthrough()}<br>    | RunnableAssign({<br>        &quot;context&quot;: itemgetter(&quot;input&quot;) <br>        | retriever <br>        | LongContextReorder().transform_documents<br>        | docs2str<br>    })<br>)<br><br>rag_chain = retrieval_chain | RunnableAssign({&quot;output&quot;: generator})</pre><p><strong>End Product</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i4EmEicg4F2Pn8QoJLRXig.png" /><figcaption>arXiv Document Ingestion Pipeline</figcaption></figure><p><strong><em>Passed!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-31t37Hn7CS9s0r5ZHuDag.png" /><figcaption>End of Gradio Testing</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1kHekXmdAvx-EVLGR6JbQ.png" /><figcaption>Notebook Output</figcaption></figure><h3>Conclusion</h3><p>Building production-ready RAG agents with LLMs is a multi-faceted endeavor that combines robust architecture, efficient data processing, and advanced deployment strategies. By leveraging frameworks like LangChain, NVIDIA’s powerful AI stack (NIMs, TensorRT-LLM, CUDA libraries), and adopting a microservices approach, developers can create highly performant, scalable, and intelligent conversational agents that unlock the full potential of large language models. The journey from prototype to production requires careful consideration of each component, but the rewards of grounded, accurate, and powerful AI systems are immense.</p><blockquote><em>“Accelerated computing is sustainable computing. Migrating eligible workloads to GPUs could save data centers 40 terawatt-hours annually.” — NVIDIA Technical Blog</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1bf294c14e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rapid Application Development with Large Language Models (LLMs)]]></title>
            <link>https://medium.com/@chongsiowyen/rapid-application-development-with-large-language-models-llms-3eb0fdfa1845?source=rss-41bb8fde93e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eb0fdfa1845</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[encoder-decoder]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Siow Yen Chong]]></dc:creator>
            <pubDate>Sat, 07 Jun 2025 20:18:38 GMT</pubDate>
            <atom:updated>2025-06-07T20:26:28.110Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Overview:</strong> This article captures my learning journey and hands-on assessment for the <strong>Rapid Application Development with LLMs</strong> course offered by the NVIDIA Deep Learning Institute (DLI).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3-l5CV3nAUM0SI-ZlgvUQ.png" /></figure><h3>1. Introduction</h3><p>Large Language Models (LLMs) have revolutionized AI, enabling unprecedented capabilities in natural language understanding and generation. This comprehensive guide walks you through core concepts, implementation, and deployment strategies, featuring practical code examples and key insights from industry best practices.</p><h3>2. Core Concepts &amp; Progression to LLMs</h3><p>LLMs build on decades of deep learning evolution:</p><ol><li>Linear/Logistic Regression: Basic input-output modeling</li><li>Stacked Layers + Activations: Added complexity via MLPs</li><li>Sparse Techniques: Convolutional layers for images</li><li>Pre-trained Models: Transfer learning (VGG/ResNet)</li></ol><blockquote><em>Key Insight</em>: Natural language’s combinatorial explosion (120k+ features in 200x200 images) necessitates pre-training.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sLhL3iB6F9gNH5gM.png" /><figcaption>Evolution diagram showing regression → RNN → transformer models.</figcaption></figure><h3>3. Loading Your First LLM (BERT)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/0*UxtRT1GZLDo1PPze.png" /><figcaption>Type of LLMs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GgRCqnPCErVMk2zv.png" /><figcaption>LLM being the component under GenAI, ML umbrella</figcaption></figure><pre>from transformers import pipeline<br><br>unmasker = pipeline(&#39;fill-mask&#39;, model=&#39;bert-base-uncased&#39;, device=&#39;cuda&#39;)<br>print(unmasker(&quot;Hello I&#39;m a [MASK] model.&quot;))<br># Output: [{&#39;token_str&#39;: &#39;fashion&#39;, &#39;sequence&#39;: &quot;hello i&#39;m a fashion model.&quot;}]</pre><h3>4. Pipeline Anatomy</h3><p>Three-stage processing:</p><ol><li>Preprocess: Text → tokenized tensors</li><li>Forward: Model inference</li><li>Postprocess: Logits → human-readable output</li></ol><p>Custom Pipeline:</p><blockquote>MLM: (Masked Language Modeling), not Multi-level marketing</blockquote><pre>class MyMlmPipeline(FillMaskPipeline):<br>    def __init__(self):<br>        super().__init__(<br>            tokenizer=AutoTokenizer.from_pretrained(&#39;bert-base-uncased&#39;),<br>            model=AutoModelForMaskedLM.from_pretrained(&#39;bert-base-uncased&#39;)<br>        )</pre><h3>5. Tokenization &amp; Embeddings</h3><p>Tokenization:</p><pre>msg = &quot;Hello world!&quot;<br>print(unmasker.tokenizer.tokenize(msg))  # [&#39;hello&#39;, &#39;world&#39;, &#39;!&#39;]</pre><p>Embedding Formula:</p><pre>Embedding = WordEmbed[token] + PosEmbed[pos] + TypeEmbed[type]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*7LEQ0-2xgZJckKBe.png" /></figure><h3>6. Transformer Attention Mechanism</h3><p>Self-attention computes:</p><pre>Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*zyUHzf2o6IHpMYwZt7nXNQ.png" /><figcaption>Self-Attention Formula</figcaption></figure><ul><li><em>dk</em>​ : Dimension of keys/queries.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*9TUq9_XQvaa7h4QpaxwnoQ.png" /></figure><p>Multi-Head Attention:</p><pre># Visualize attention<br>from bertviz import model_view<br>model_view(outputs.attentions, tokens)</pre><ul><li>Splits embeddings into <em>h</em> heads (e.g., 12 heads in BERT) for parallel processing.</li></ul><h3>Core Seq2Seq Architecture</h3><p>Before transformers, Seq2Seq models relied on Recurrent Neural Networks (RNNs). Now, Transformers replaced RNNs with self-attention mechanisms for parallel processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUvDUjbynf_Mnb8qAT_5xQ.png" /></figure><h3>7. BERT Architecture</h3><p>Key Modifications from Transformers</p><ol><li>Bidirectional Training : MLM (Masked Language Modeling) vs. GPT’s autoregressive training.</li><li>Layer Normalization : Stabilizes gradients in deep networks.</li><li>GELU Activation : Smoother gradients than ReLU</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFaSqDxAYg58dmPS0BmEQw.png" /><figcaption>BERT encoder block showing Token Embeddings + Segment Embeddings + Positional Embeddings → Multi-Head Attention → Feed-Forward Network.</figcaption></figure><h3>8. Encoder Task Heads</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*4_gXgnoFKK4we9FMBu3J1Q.png" /></figure><p>Token Classification Example (NER):</p><pre>from transformers import pipeline<br>ner_pipeline = pipeline(&quot;ner&quot;, model=&quot;bert-base-cased&quot;)<br>ner_pipeline(&quot;Apple is looking at buying U.K. startup for $1 billion&quot;)<br># Output: [{&#39;entity&#39;: &#39;B-CORP&#39;, &#39;score&#39;: 0.99, &#39;word&#39;: &#39;Apple&#39;}, ...]</pre><p>Zero-Shot Classification:</p><ul><li>Use facebook/bart-large-mnli for NLI-based classification.</li></ul><pre>zero_shot_pipeline = pipeline(&quot;zero-shot-classification&quot;, model=&quot;facebook/bart-large-mnli&quot;)<br>zero_shot_pipeline(&quot;The Apollo program landed humans on the Moon.&quot;, candidate_labels=[&quot;space&quot;, &quot;politics&quot;])</pre><h3>9. Decoder Models (GPT-2)</h3><p>Autoregressive Generation :</p><p>KV Cache Optimization :</p><ul><li>Stores past key-value pairs to avoid recomputation during token generation.</li></ul><pre>from transformers import GPT2LMHeadModel, GPT2Tokenizer<br>model = GPT2LMHeadModel.from_pretrained(&quot;gpt2&quot;)<br>tokenizer = GPT2Tokenizer.from_pretrained(&quot;gpt2&quot;)<br>inputs = tokenizer(&quot;Hello&quot;, return_tensors=&quot;pt&quot;)<br>outputs = model.generate(**inputs, max_length=50, num_return_sequences=3)<br>print(tokenizer.batch_decode(outputs, skip_special_tokens=True))</pre><h3>10. Encoder-Decoder Models (T5)</h3><p>Translation Pipeline :</p><pre>from transformers import pipeline<br>translator = pipeline(&quot;translation_en_to_fr&quot;, model=&quot;t5-base&quot;)<br>translator(&quot;Hello World! How&#39;s it going?&quot;)<br># Output: [{&#39;translation_text&#39;: &quot;Bonjour le monde! Comment ça va ?&quot;}]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*uxNSlnTSssfs1UgqLW5xag.png" /></figure><h3>11. Multimodal Reasoning</h3><p>Image-to-Text Generation (CLIP + GPT) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C02UO4kt8CnoUuaIZdY2og.png" /></figure><pre>from PIL import Image<br>import requests<br>from transformers import CLIPProcessor, CLIPModel<br>model = CLIPModel.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br>processor = CLIPProcessor.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br>image = Image.open(requests.get(&quot;https://example.com/image.jpg&quot;,  stream=True).raw)<br>inputs = processor(text=[&quot;a photo of a cat&quot;, &quot;a photo of a dog&quot;], images=image, return_tensors=&quot;pt&quot;, padding=True)<br>outputs = model(**inputs)<br>logits_per_image = outputs.logits_per_image  # Similarity score<br>probs = logits_per_image.softmax(dim=1)  # Probability distribution</pre><h3>12. Deployment with vLLM</h3><p>Server Setup:</p><pre>!vllm serve microsoft/phi-3.5-vision-instruct \<br>    --trust-remote-code \<br>    --max_model_len 16384 \<br>    --port 9000</pre><p>API Access:</p><pre>from openai import OpenAI<br>client = OpenAI(base_url=&quot;http://localhost:9000/v1&quot;)<br>response = client.completions.create(<br>    prompt=&quot;Hello world&quot;,<br>    model=&quot;phi-3.5-vision-instruct&quot;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2_b2yIhPg55MgX9KTPE3lQ.png" /><figcaption>NVIDIA NIM for optimized inference showing clients → vLLM server → GPU.</figcaption></figure><p>LangChain Integration:</p><pre>from langchain_nvidia_ai_endpoints import ChatNVIDIA<br>llm = ChatNVIDIA(model=&quot;phi-3.5-vision-instruct&quot;, base_url=&quot;http://localhost:9000/v1&quot;)<br>response = llm.invoke(&quot;Describe the image provided.&quot;, images=[&quot;image.png&quot;])<br>print(response.content)</pre><h3>13. Multi-Agent Orchestration</h3><p>LangGraph enables complex workflows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*EtWHeiqINc4WvkyEu9Up_w.png" /><figcaption>Langgraph Orchestration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t5lrt3Oje3fkRrS2v4Av1A.png" /></figure><pre>from langgraph.graph import StateGraph<br><br>class State(TypedDict):<br>    messages: list[AnyMessage]<br>    directives: list[AnyMessage]<br><br>graph = StateGraph(State)<br>graph.add_node(&quot;assistant&quot;, assistant_fn)<br>graph.add_edge(&quot;assistant&quot;, &quot;tools&quot;)</pre><p><strong>ReAct Loop:</strong></p><ol><li>Reason: LLM analyzes problem</li><li>Act: Executes tool call</li><li>Observe: Incorporates result</li></ol><p>ReAct Loop Implementation :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*8LF1HOTunB5uNgpz9mv1FQ.png" /></figure><pre>from langgraph.graph import StateGraph, END<br>from langchain_core.messages import HumanMessage, AIMessage<br>class AgentState(TypedDict):<br>    messages: Annotated[list, add_messages]<br>def agent_node(state):<br>    return {&quot;messages&quot;: [llm(state[&quot;messages&quot;])]}<br>def tool_node(state):<br>    last_message = state[&quot;messages&quot;][-1]<br>    tool_response = tool.invoke(last_message.content)<br>    return {&quot;messages&quot;: [AIMessage(content=tool_response)]}<br>graph = StateGraph(AgentState)<br>graph.add_node(&quot;agent&quot;, agent_node)<br>graph.add_node(&quot;tool&quot;, tool_node)<br>graph.add_conditional_edges(&quot;agent&quot;, lambda state: &quot;tool&quot; if &quot;tool_call&quot; in state[&quot;messages&quot;][-1].content else END)<br>graph.add_edge(&quot;tool&quot;, &quot;agent&quot;)<br>app = graph.compile()</pre><h3>14. Image-Inspired Generation Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*lGoXEHKKLt2HVmkNJYzaAQ.png" /></figure><h3>15. Assessment — Image Generator Pipeline</h3><p>End-to-End Code</p><pre>def generate_images_from_image(image_path):<br>    # Step 1: VLM to describe image<br>    description = ask_about_image(image_path) <br>    # Step 2: LLM to refine prompt<br>    prompt = synthetic_prompt_chain.invoke({&quot;description&quot;: description})<br>    # Step 3: Diffusion model to generate images<br>    images = generate_images(prompt)  <br>    return images, prompt<br># Example usage<br>generated_images, prompt = generate_images_from_image(&quot;input_image.png&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*MugOrqanBAC3_11knotgjw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*yug-aKWOnfyfFRvOoBhh2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzEfufYwRJi2KX_R2WVzHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*f5-TNZ4-tu-umHemnnyuIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*c38gweEiOJXQYwKeEwF4mQ.png" /></figure><h3>16. Optimization Tradeoffs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*6FcXSkec_ZNnPOeZRfIhAA.png" /></figure><p><strong>Formulas:</strong></p><ul><li><strong>Context Length:</strong> Input Tokens + Output Tokens (truncate VLM text)</li><li><strong>Memory Usage:</strong> ∝ Batch Size × Resolution</li></ul><h3>17. Conclusion &amp; Next Steps</h3><p>LLMs represent a paradigm shift in AI capabilities. To dive deeper:</p><ol><li>Experiment with different architectures (encoders vs. decoders)</li><li>Implement RAG (Retrieval-Augmented Generation)</li><li>Explore multi-modal models (text + image)</li><li>Optimize deployments with quantization</li></ol><blockquote>“Transformers don’t process sequences sequentially — they reason about all tokens simultaneously using attention weights as contextual filters.”</blockquote><p><strong>Certification Achieved 🎉:</strong> NVIDIA DLI: Rapid Application Development with LLMs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*VyDLTueONhdyqVcpxbIH2A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eb0fdfa1845" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM Applications: From Prompt Engineering to Production-Ready Systems]]></title>
            <link>https://medium.com/@chongsiowyen/building-llm-applications-from-prompt-engineering-to-production-ready-systems-39ea835c812b?source=rss-41bb8fde93e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/39ea835c812b</guid>
            <category><![CDATA[real-world-ai-app]]></category>
            <category><![CDATA[nvidia-nim]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[langchain-orchestration]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Siow Yen Chong]]></dc:creator>
            <pubDate>Sun, 01 Jun 2025 21:18:21 GMT</pubDate>
            <atom:updated>2025-06-01T21:19:32.102Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A hands-on journey through NVIDIA’s LLM workshop with code examples, architectural insights, and personal learning reflections</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*dWVkiqjK3ehi87MMC_StDg.png" /><figcaption>Prompt engineering flow</figcaption></figure><h3>0. About the Workshop</h3><p>This immersive program transformed participants from LLM novices to confident developers capable of bridging theoretical concepts with enterprise implementations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*HpxSA8z_SbVOpxT0nVmlrg.png" /><figcaption><em>Fig 1. Certification earned through workshop completion</em></figcaption></figure><h3>1. From Black Box to Architectural Fluency</h3><p><em>My learning journey: Initial perceptions → Technical mastery</em></p><p><strong>Pre-Workshop View:</strong><br><em>“LLMs as magical black boxes — impressive outputs but opaque internals.”</em></p><p><strong>Post-Workshop Reality:</strong></p><pre># Local LLM Deployment with NVIDIA NIM<br>from langchain_nvidia_ai_endpoints import ChatNVIDIA<br><br>llm = ChatNVIDIA(<br>    base_url=&#39;http://llama:8000/v1&#39;,  # Local microservice<br>    model=&#39;meta/llama-3.1-8b-instruct&#39;,<br>    temperature=0<br>)</pre><p><em>Key Breakthroughs</em>:</p><ul><li>🛠️ Microservice Control: Deploy models locally with enterprise-grade security</li><li>⚡ Latency Management: 6x faster batch processing vs. individual API calls</li><li>🔄 IP Ownership: Full control over custom implementations</li></ul><h3>2. The New AI Paradigm: Beyond ChatGPT</h3><p>The workshop emphasized three pillars of modern LLM development:</p><p>Enterprise-Ready Architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fZOjbixhkMtRh5ozRGnXg.png" /></figure><h3>3. NVIDIA NIM: The Production Backbone</h3><p><em>Hands-on with enterprise deployment tools</em></p><p>Workshop Environment:</p><pre># Course Docker Setup<br>docker run -p 8000:8000 nvidia/nim:llama3.1-8b</pre><p>Key Technical Takeaways:</p><ul><li>🚀 Optimized Inference: TensorRT-LLM integration for hardware acceleration</li><li>🔒 Security First: On-prem deployment avoids cloud data risks</li><li>📦 Containerization: Helm charts for Kubernetes-native scaling</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0H6eBulcNijMC2FRAlnvzg.png" /><figcaption><em>NIM microservices architecture diagram</em></figcaption></figure><h3>4. Prompt Engineering: From Art to Science</h3><p><em>Iterative development framework</em></p><p><strong>Bad → Good</strong> Prompt Evolution:</p><pre># Naive Approach<br>&quot;Tell me about quantum computing&quot;<br><br># Engineered Version<br>&quot;&quot;&quot;<br>You are a physics professor explaining to first-year students.<br>Use these constraints:<br>1. 3 key concepts max<br>2. Include real-world analogies<br>3. Avoid mathematical notation<br><br>Explain quantum superposition.<br>&quot;&quot;&quot;</pre><p>Advanced Pattern:</p><pre>from langchain_core.prompts import ChatPromptTemplate<br><br>medical_template = ChatPromptTemplate.from_messages([<br>    (&quot;system&quot;, &quot;You are a medical AI assistant. NEVER diagnose.&quot;),<br>    (&quot;human&quot;, &quot;Patient symptoms: {symptoms}&quot;),<br>    (&quot;ai&quot;, &quot;Possible conditions (not diagnoses):&quot;)<br>])</pre><h3>5. LangChain LCEL: Industrial-Strength Pipelines</h3><p><em>From Jupyter notebooks to production systems</em></p><p>Multi-Stage Workflow:</p><pre>from langchain_core.runnables import RunnableParallel<br><br>qa_system = (<br>    RunnableParallel({<br>        &quot;context&quot;: document_retriever,<br>        &quot;question&quot;: RunnableLambda(lambda x: x)<br>    })<br>    | prompt_template<br>    | llm<br>    | StrOutputParser()<br>)</pre><p>Batch Processing Insights:</p><pre># Workshop Benchmark Results<br>questions = [f&quot;Q{i}&quot; for i in range(100)]<br>responses = chain.batch(questions, config={&quot;max_concurrency&quot;: 16})</pre><p><em>Result</em>: 94% faster than sequential processing</p><h3>6. Structured Outputs: Precision Engineering</h3><p><em>Taming LLM randomness with Pydantic</em></p><p>Workshop Exercise:</p><pre>class CustomerComplaint(BaseModel):<br>    product: str = Field(enum=[&quot;Electronics&quot;, &quot;Furniture&quot;])<br>    location: str<br>    severity: int = Field(ge=1, le=5)<br><br>extraction_chain = (<br>    prompt_template <br>    | llm <br>    | JsonOutputParser(pydantic_object=CustomerComplaint)<br>)</pre><p>Real Output Example:</p><pre>{<br>    &quot;product&quot;: &quot;Furniture&quot;,<br>    &quot;location&quot;: &quot;New York&quot;,<br>    &quot;severity&quot;: 4<br>}</pre><h3>7. AI Agents: Beyond Textbook Examples</h3><p><em>Production-grade tool integration</em></p><p>Workshop Implementation:</p><pre>from langgraph.prebuilt import create_react_agent<br><br>@tool<br>def get_financial_data(symbol: str) -&gt; dict:<br>    &quot;&quot;&quot;Fetch real-time stock data&quot;&quot;&quot;<br>    return yfinance.Ticker(symbol).info<br><br>agent = create_react_agent(llm, tools=[get_financial_data])<br>response = agent.invoke(&quot;NVDA&#39;s P/E ratio?&quot;)<br># Output: &quot;NVIDIA&#39;s current P/E ratio is 72.3&quot;</pre><h3>8. Project: Customer Complaint Analysis</h3><p><em>From concept to evaluated solution</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vOe9yRmxgZUKLyce-sw8iA.png" /></figure><p>Critical Code Snippet:</p><pre>class EmailAnalysis(BaseModel):<br>    product_category: str = Field(enum=[&quot;Furniture&quot;, &quot;Electronics&quot;])<br>    store_location: str<br>    sentiment: str<br><br>chain = (<br>    prompt_template <br>    | llm <br>    | JsonOutputParser()<br>    | RunnableLambda(aggregate_results)<br>)</pre><h3>9. Lessons from the Trenches</h3><p><em>Hard-won insights for production systems</em></p><ol><li>Prompt Injection Defense:</li></ol><pre>def sanitize_input(text: str) -&gt; str:<br>    if &quot;ignore previous&quot; in text.lower():<br>        raise ValueError(&quot;Invalid instruction override&quot;)<br>    return text</pre><p>2. Performance Optimization:</p><pre># Workshop-Tested Configuration<br>responses = chain.batch(<br>    inputs,<br>    config={<br>        &quot;max_concurrency&quot;: 10,<br>        &quot;max_retries&quot;: 3<br>    }<br>)</pre><h3>10. The Road Ahead</h3><p><em>Where LLM engineering is heading</em></p><ol><li>Self-Optimizing Systems: Automated prompt tuning via RLHF</li><li>Multimodal Agents: Combining text + vision + speech</li><li>Enterprise Security: Homomorphic encryption for sensitive data</li></ol><pre># Future Concept: Auto-RAG<br>auto_chain = (<br>    question <br>    | vector_store <br>    | dynamic_prompt_builder <br>    | self_correcting_llm<br>)</pre><h3>Conclusion</h3><p>This workshop transformed my understanding of LLMs from abstract concepts to concrete engineering practices.</p><p>Key takeaways:</p><ol><li>Production Mindset: LLMs require robust engineering like any software system</li><li>Iterative Design: Prompt engineering is cyclical, not one-off</li><li>Tooling Maturity: NVIDIA NIM + LangChain enable enterprise-grade solutions</li></ol><p><em>Final Certification Stats</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*PcXG8Pc8q-GN13Yr_GQn3g.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39ea835c812b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Postmortem on the Spring 2025 Advanced LLM Agents MOOC]]></title>
            <link>https://medium.com/@chongsiowyen/postmortem-on-the-spring-2025-advanced-llm-agents-mooc-964f10ad3d86?source=rss-41bb8fde93e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/964f10ad3d86</guid>
            <category><![CDATA[multimodal-agents]]></category>
            <category><![CDATA[reasoning]]></category>
            <category><![CDATA[training-methods]]></category>
            <category><![CDATA[security-risk]]></category>
            <category><![CDATA[llm-agent]]></category>
            <dc:creator><![CDATA[Siow Yen Chong]]></dc:creator>
            <pubDate>Sat, 10 May 2025 20:33:25 GMT</pubDate>
            <atom:updated>2025-05-10T22:16:35.820Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iDD9uubv967Wkwje.png" /><figcaption>AI Agents Ecosystem 2025</figcaption></figure><p>The Spring 2025 iteration of UC Berkeley’s <a href="https://llmagents-learning.org/sp25"><em>Advanced Large Language Model Agents </em>MOOC</a>, co-instructed by Xinyun Chen (Google DeepMind), Dawn Song (UC Berkeley), and Kaiyu Yang (Meta) and more, delivered a comprehensive exploration of cutting-edge advancements in AI agents. With guest lectures from leading researchers at institutions like Google DeepMind, Carnegie Mellon, and MIT, the course dissected the evolving landscape of LLM reasoning, security, multimodal agents, and mathematical discovery. This postmortem synthesizes key themes, challenges, and breakthroughs highlighted throughout the semester, offering insights into the trajectory of agentic AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*hKXuXhO--dPx-n6XiP_TmQ.png" /><figcaption>Xinyun Chen’s introduction to the UC Berkeley Spring 2025 Advanced LLM Agents MOOC</figcaption></figure><p>The course opened with a deep dive into reasoning frameworks that enable LLMs to tackle complex tasks. Xinyun Chen emphasized the evolution of prompt engineering, from early Chain-of-Thought (CoT) prompting to advanced techniques like Tree of Thought (ToT) and Self-Consistency . These methods allow models to generate multiple reasoning paths, prune unpromising ones, and aggregate results for higher accuracy.</p><p>Yu Su (Google DeepMind) expanded on hierarchical reasoning, advocating for subgoal decomposition to mimic human problem-solving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*ujc_dlCRzcQVOvt_BIjtlg.png" /><figcaption><em>Hierarchy of core competencies in LLM agents mirroring human brain cognitive functions</em></figcaption></figure><p>For instance, proving mathematical theorems required breaking proofs into lemmas, a strategy mirrored in systems like LeanDojo and COPRA . Meanwhile, Sean Welleck (NYU) demonstrated how reinforcement learning could refine proof generation by rewarding logical consistency, a technique that enabled LLMs to solve International Math Olympiad problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AkCWbwCKfe-Y_5ZPx8hcrg.png" /><figcaption>Example geometry problem from the International Mathematical Olympiad (IMO)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*COCE4GmFdKCMxPcvjoFtPA.png" /><figcaption>Lean-STaR algorithm workflow for reinforcement learning in theorem proving</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*8o_fQqBzF-cMvolduUwY0g.png" /><figcaption>Parallel proof generation with error retry mechanism: Branching paths of attempted proofs, with failed attempts retried until reaching a valid solution within step limits</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vFHtqJMtk1gf8v--T_OUsQ.png" /><figcaption><em>Results from the Lean-STaR framework showing improvements in theorem proving success rates: Early models (red) improved from ~10% to ~35% after integrating informal thoughts and expert iteration (black).</em></figcaption></figure><blockquote>Key Takeaway : Reasoning isn’t monolithic — combining iterative refinement, search algorithms, and hierarchical planning unlocks LLMs’ ability to handle abstraction and complexity.</blockquote><p><a href="https://arxiv.org/abs/2305.10601">Read more about ToT </a>| <a href="https://arxiv.org/abs/2405.14831">Explore HippoRAG</a></p><h3>2. Training Methodologies: Reinforcement Learning and Synthetic Data</h3><p>Hanna Hajishirzi (University of Washington) and her team at the Allen Institute for AI (AI2) showcased the power of Reinforcement Learning with Verifiable Rewards (RLVR) .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FVMHZ8Y__FimuxZtFkMOg.png" /><figcaption>Tulu 3 open post-training recipe</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QTMdaib6nZBxh6VRHItNiw.png" /><figcaption>Iterative training pipelines combining preference optimization techniques (DPO, PPO)</figcaption></figure><p>By leveraging tasks with algorithmically verifiable answers (e.g., math problems), models like OLMo and Tulu 3 achieved parity with GPT-4 through iterative training pipelines combining instruction tuning, preference optimization (DPO/PPO), and synthetic data generation as shown in diagram above.</p><p>A recurring theme was the importance of data quality over quantity . Curated datasets (e.g., 1,000 high-quality examples) often outperformed larger, noisier counterparts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*L5YXII496CqCayaDVqHE8Q.png" /><figcaption>Carefully curated data (s1k) for language model with Test-time scaling algorithm</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*5Z0RPy1scp7XjbVdxgTDGw.png" /><figcaption>Performance comparison: s1K dataset vs. random/constrained datasets</figcaption></figure><p>This principle guided projects like LaSR , where LLMs generated hypotheses for mathematical inequalities and validated them experimentally.</p><blockquote>Key Takeaway : Scalable training frameworks like RLVR and synthetic data curation are democratizing access to state-of-the-art models, bridging the gap between open-source and proprietary systems.</blockquote><p><a href="https://huggingface.co/blog/tulu-3">Learn about Tulu 3 </a>| <a href="https://github.com/princeton-nlp/SWE-bench">SWE-Bench Leaderboard</a></p><h3>3. Security Risks: Adversarial Threats and Mitigation Strategies</h3><p>Dawn Song led a critical discussion on securing AI systems against emerging threats. Lectures highlighted vulnerabilities like prompt injection attacks , data poisoning , and backdoor exploits in pre-trained models. For example, poisoned training data could embed hidden triggers, causing models to generate harmful outputs when prompted with specific keywords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*G90AhjFa8MFvhn0LTKwQXw.png" /><figcaption>Poisoned backdoor embedded in a large language model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGAH_t8Vb-BO0ildAZQGZg.png" /><figcaption>Red teaming workflow for LLM agents, highlighting adversarial testing phases to detect vulnerabilities in knowledge base poisoning scenarios</figcaption></figure><p>Mitigation strategies included input sanitization , model attestation (cryptographic verification of provenance), and runtime anomaly detection .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwS04mzD0hnmkp-veWgwvw.png" /><figcaption>Input sanitization guardrail framework, illustrating steps to filter escape characters and validate prompts to prevent adversarial attacks.</figcaption></figure><p>The course stressed the need for proactive defense mechanisms, particularly as LLM agents increasingly handle sensitive tasks like financial transactions or medical diagnostics.</p><blockquote>Key Takeaway : Security must be embedded into the AI lifecycle — training, deployment, and runtime monitoring are equally critical to prevent exploitation.</blockquote><p><a href="https://os-world.github.io/">Explore OSWorld Security Benchmarks</a></p><h3>4. Mathematical Reasoning: Interactive Theorem Proving and Error Correction</h3><p>Sean Welleck and Swarat Chaudhuri (University of Texas) explored LLMs’ role in formal mathematics, showcasing systems like Lean and HippoRAG . These tools enabled agents to verify proofs, correct historical errors (e.g., flaws in Euclid’s <em>Elements </em>), and generate novel conjectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2DyTs1fLdjRCCF85yOrGoQ.png" /><figcaption>Transitivity rule implementation in formal logic, demonstrating proof completion when no unresolved goals remain</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0jJp9t4WE_Yc5-wRhnElxA.png" /><figcaption>COPRA system architecture for theorem proving</figcaption></figure><p>A standout innovation was Grokked Transformer , which used internal monologues to debug flawed proofs and backtrack errors — a step toward self-improving mathematical agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F6_eQTJeUvSHWvin_KPABQ.png" /><figcaption>Implicit reasoning in Grokked Transformers</figcaption></figure><p>However, challenges remained in handling sparse rewards and ensuring logical equivalence across proof variants.</p><blockquote>Key Takeaway : LLMs are transforming mathematics from a human-centric discipline to a collaborative space where agents propose, validate, and refine theorems autonomously.</blockquote><p><a href="https://leanprover.github.io/">Try LeanDojo </a>| <a href="https://github.com/SeanWelleck/copra">Read about COPRA</a></p><h3>5. Multimodal Agents and Real-World Applications</h3><p>Caiming Xiong (Salesforce Research) and Ruslan Salakhutdinov (Apple) discussed multimodal agents capable of navigating real-world environments. Systems like OSWorld benchmarked agents’ ability to interact with operating systems, while WebDreamer simulated web navigation for tasks like online shopping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAJDg4grJpKj9emRJbaulA.png" /><figcaption>OSWorld benchmark interface, demonstrating agent interaction with operating systems via mouse/keyboard-free navigation</figcaption></figure><p>Thomas Hubert (DeepMind) highlighted AlphaProof , which solved complex geometry and algebra problems at silver-medal-level performance in the International Mathematical Olympiad (IMO), and AlphaTensor , a system that discovered novel algorithms for matrix multiplication (e.g., reducing operations for 4×4 matrices from 64 to 49).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*kKa9wghcEEJ5mzxJTEBuTg.png" /><figcaption>Geometry problem example with triangle inequality proof</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*UqnIOGwmWOJgBgphTfX7Eg.png" /><figcaption>AlphaTensor matrix multiplication optimization, comparing algorithm efficiency (e.g., 4×4 matrix operations reduced from 64 to 49 multiplications)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*9aSh53xlo4sLlyMaCpnxjw.png" /><figcaption>Prover model training pipeline</figcaption></figure><p>Meanwhile, EnIGMA , a CTF-focused agent, combined tools like grep and code editors to solve digital forensics challenges.</p><blockquote>Key Takeaway : Multimodal agents are no longer theoretical — they’re automating workflows in cybersecurity, e-commerce, and beyond, though scalability and contextual grounding remain hurdles.</blockquote><p><a href="https://os-world.github.io/">Explore OSWorld Benchmark </a>| <a href="https://deepmind.google/blog/alphacode-taking-programming-competition-challenges/">AlphaCode Paper</a></p><h3>6. Neuroscience-Inspired Architectures: Memory and Meta-Reasoning</h3><p>Yu Su introduced HippoRAG , a memory system inspired by the human hippocampus, which creates structured indexes for associative recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOHTY7trhhtjDwBy8PE5ww.png" /><figcaption>Offline indexing and online retrieval phases for knowledge graphs, illustrating entity recognition and experience storage for agent reasoning</figcaption></figure><p>This approach addressed limitations in traditional RAG systems, enabling agents to link disparate memories (e.g., “Stanford” and “Alzheimer’s”).</p><p>Meta-reasoning emerged as a recurring theme: agents now simulate internal monologues to plan hierarchically (e.g., research workflows) and reflect on their decision-making processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*-GaZeHniSob24YUgDq2Thw.png" /><figcaption>Schematic of language agents with inner monologue, demonstrating hierarchical planning, self-reflection, and reasoning loops for autonomous decision-making</figcaption></figure><blockquote>Key Takeaway : Biological analogies like neural plasticity and structured memory offer pathways to build more robust, adaptable agents.</blockquote><p><a href="https://arxiv.org/abs/2405.14831">HippoRAG Paper</a></p><h3>7. Future Directions: Ethics, Generalization, and Autonomous Discovery</h3><p>The course concluded with a focus on ethical AI and cross-domain generalization . Researchers emphasized the need for transparency, fairness, and alignment with human values, particularly as agents tackle high-stakes domains like healthcare and finance.</p><p>Swarat Chaudhuri highlighted LLMs’ potential to accelerate mathematical discovery by identifying abstract patterns in data, while Dawn Song warned of the risks of autonomous agents in adversarial environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*YqZCVEEUsHSBfu5gpXd8CA.png" /><figcaption>Pattern-matching of constants and binary operations in symbolic reasoning, showcasing LLMs’ ability to decompose mathematical expressions</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F3aP-QQrNcGGfSczhwqwFQ.png" /><figcaption>Simulated adversarial attacks on agentic AI with prompt injection</figcaption></figure><blockquote>Key Takeaway : The future lies in balancing innovation with responsibility — agents must not only solve problems but also earn trust through explainability and ethical design.</blockquote><h3>Conclusion</h3><p>The Spring 2025 MOOC underscored a pivotal shift: LLM agents are evolving from text generators to dynamic problem-solvers capable of reasoning, planning, and interacting with complex environments. While breakthroughs in RLVR, multimodal systems, and mathematical reasoning have pushed the frontier, challenges like security, generalization, and ethical alignment demand urgent attention.</p><p>As highlighted throughout the course: <em>“The path to agentic AI requires synergy between technical innovation, interdisciplinary collaboration, and a commitment to societal impact.”</em></p><h3><strong>Final Thought:</strong></h3><p>The journey toward autonomous AI is not just about building smarter models — it’s about redefining how humans and machines collaborate to tackle humanity’s greatest challenges.</p><p>What does this mean for you?</p><ul><li>For developers : Tools like SWE-agent and WebArena are already automating code debugging and web tasks — imagine integrating these into your workflows.</li><li>For researchers : Open-source ecosystems like OLMo and Tulu 3 prove that transparency and community-driven innovation can rival proprietary systems.</li><li>For educators : Platforms like LeanDojo are gamifying mathematics, turning abstract proofs into interactive puzzles — reshaping how we teach logic and creativity.</li><li>For policymakers : Security risks like data poisoning and adversarial attacks demand proactive governance. The future of AI hinges on trust, not just technology.</li></ul><p>The path forward requires optimism — the kind Thomas Hubert described as “absurd” yet essential to achieving the impossible. And as Sean Welleck showed, even Euclid’s flaws can be corrected with the right blend of human intuition and machine precision.</p><p><strong>Your role matters .</strong> Whether you’re coding the next agent, designing ethical frameworks, or simply questioning how AI aligns with societal values, <strong>you’re part of this evolution.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=964f10ad3d86" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>