<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Avikumar Talaviya on Medium]]></title>
        <description><![CDATA[Stories by Avikumar Talaviya on Medium]]></description>
        <link>https://medium.com/@avikumart_?source=rss-13fcacf52e4f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NlPZPIHNXYq4ZHvVogmMZw.png</url>
            <title>Stories by Avikumar Talaviya on Medium</title>
            <link>https://medium.com/@avikumart_?source=rss-13fcacf52e4f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:34:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@avikumart_/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Thoughts and learnings from the AI Dev 2026 by the DeepLearningAI]]></title>
            <link>https://medium.com/@avikumart_/thoughts-and-learnings-from-the-ai-dev-2026-by-the-deeplearningai-fd9bea6f017e?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd9bea6f017e</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Thu, 07 May 2026 03:43:44 GMT</pubDate>
            <atom:updated>2026-05-07T03:43:44.091Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osom4l5hc-N8J5PAOQEAiA.png" /><figcaption>AI developer conference representative image</figcaption></figure><h3><strong>Introduction</strong></h3><p>The AI developer summit in San Francisco was one of the most outstanding events I attended when it comes to tech in the past few years. I was so amazed and mindblown by the amount of energy, people and technologies that were being discussed over the two days of the summit. Everyone was involved in some sort of conversation and trying to solve the latest tech problems. Industry leaders were talking about their latest products and innovations involving AI agents and agentic AI developments. The most common theme was how AI agents are going to develop the next generation of technology, as well as the products of the future.</p><p>There are many new learnings that I have had over the two days that I wouldn’t have been able to learn without this summit. The first one is AI harness — AI is really just a fancy term when it comes to marketing the AI tools, but real innovation comes with orchestrating multitudes of tools, databases, APIs, and user feedback to harness the maximum potential of the AI that aligns with the user needs and privacy measures put in place.</p><p>Another interesting conversation that I had was with a representative from Docker who demonstrated the AI sandboxes to test packaged AI applications against possible vulnerabilities and security threats from malicious prompt injections and user requests. Docker sandboxes can monitor all allowed and unidentified requests and block them before they can penetrate the systems.</p><p>Other aspects that I believe are really critical to any AI applications are the context engine or the memory layer of AI tools. Most practical and enterprise AI needs the context engine to run, as well as aligning AI to the needs of the organisation. I had a brief discussion with the senior engineer from Oracle who developed the context layer for the Oracle AI databases. It comprises the distributed memory layer that can store the database context across the machines and take external tools as and when necessary to do tasks that are required for the users.</p><p>Observability was another topic that caught my attention over two days, as many startups are building around the simulation of AI-native products, testing the application using a team of agents to make sure the AI applications work in production as intended, without severe consequences. Particularly, I attended a session by Harrison Chase of Langchain, who talked about the observability flywheel that deep agents are highly effective in the AI harness, which is the latest product from Langchain. These agents are used in a loop test to evaluate multiple responses from the AI tools and ensure the reliability, guardrails and oversight of the Agents in production.</p><p>The conference started with some of the exciting keynotes from the founders and chief executives of the leading organisations, including the Andrew Ng, founder of DeepLearningAI and organiser of the AI Dev Conference, 2026.</p><p><strong>Andrew Ng’s vision for the future of software engineering</strong></p><p>Andrew Ng, founder of DeepLearning.AI, delivered a keynote that challenged the traditional necessity of human-in-the-loop code reviews. Ng argued that for “frontier teams” — those operating at the cutting edge of AI implementation — the trend is moving toward 100 per cent AI-written code. This perspective is rooted in the belief that human review processes have become the primary friction point in the development cycle. Ng suggested that if a developer must manually inspect every line of generated code, they effectively remain the bottleneck, neutralising the velocity gains provided by agentic systems.</p><p>This vision implies a radical reorganisation of development teams. The emerging model favours small teams of generalists who oversee fleets of AI agents. These generalists must possess a broad understanding of system architecture, product management, and user needs, as their primary role is to provide the “intent” and “oversight” rather than the “execution”.</p><p>A panel discussion moderated by Marina Mogilko featured leaders from Replit, LandingAI, Oracle, and Practical Data Media, providing a quantitative assessment of industry optimism. When asked to rate the future of software development on a scale of one to ten, the responses reflected a high degree of confidence in the AI-native shift:</p><p>The high rating from Michele Catasta aligns with Replit’s aggressive pursuit of “vibe coding” — a paradigm where developers focus on high-level orchestration and the “vibe” of the application while the AI manages the underlying syntax. Richmond Alake’s more conservative rating of seven was accompanied by the prediction that roles will continue to blur, with software engineers increasingly required to speak directly to customers and understand market needs, effectively becoming hybrid product managers.</p><p>AMD’s commitment to the ROCm open software stack was presented as a critical enabler for developers seeking to avoid vendor lock-in. Elangovan detailed several projects aimed at optimising the AI-hardware interface, such as “HotSwap,” which allows for the interception of GPU kernel workloads and the retargeting of the Instruction Set Architecture (ISA) at runtime. Additionally, the introduction of a new native HIP backend for llama.cpp and a high-performance IREE C tokeniser demonstrates the industry’s push toward lower-level optimisations to squeeze maximum performance from available silicon.</p><h3>Red Teaming and Failure Finding</h3><p>Jean-Marie John-Mathews of Giskard addressed the reliability gap through the lens of systematic red teaming. His presentation focused on finding failures in agents, RAG (Retrieval-Augmented Generation), and chatbots before they reach production. The core insight is that as agents become more autonomous, traditional unit testing is insufficient; developers must adopt adversarial testing to ensure that agents do not fail in unpredictable ways when exposed to edge cases in the wild.</p><h3>Context Engineering: Beyond Simple Retrieval</h3><p>The conference signalled a shift from Retrieval-Augmented Generation to a more sophisticated discipline known as “Context Engineering”. Jeff Huber, CEO of Chroma, argued that “RAG is dead, context engineering is king”. The challenge in 2026 is no longer about finding any relevant data, but about precisely managing what occupies the LLM’s context window at any given moment to minimize “context bloat” and prevent hallucinations.</p><h3>Deterministic Memory and Context Graphs</h3><p>Nyah Macklin of Neo4j and Andrew K. Davies of Harmony8 explored the intersection of graph databases and agent memory. Macklin’s talk on “auditable AI agents” explained how context graphs allow for stateful reasoning that can be traced back to specific data points. This is particularly critical in regulated industries like finance, where “the AI said so” is not an acceptable justification for a decision.</p><p>Harmony8’s approach to “deterministic memory” seeks to build AI systems that “cannot lie” by anchoring their reasoning in fixed, verifiable data structures rather than probabilistic weights alone. This requires an architectural separation between the reasoning engine (the LLM) and the memory engine (the context graph).</p><h3>Vector Databases Across the Continuum</h3><p>Emma McGrattan of Actian discussed the engineering of the context layer across cloud, edge, and on-premises AI. Her analysis highlighted that technical innovation must account for the “political reality” of data sovereignty. As European governments and companies become increasingly uneasy about housing data on US soil, the context layer must be designed to be distributed and hybrid by default.</p><h3><strong>Final Thoughts</strong></h3><p>Overall<strong>,</strong> the conference was full of learning, networking, problem-solving, and building the future of software engineering with some of the best developers in the world. I really enjoyed my time in SF, especially the two days at the conference venue, engaging with multiple people and learning all things agentic AI and the future it holds, which, of course, is very exciting.</p><p><strong>References:</strong></p><ol><li><a href="https://ai-dev.deeplearning.ai/">https://ai-dev.deeplearning.ai/</a></li><li><a href="https://www.theregister.com/2026/04/28/software_development_ai_dev25xsf/">https://www.theregister.com/2026/04/28/software_development_ai_dev25xsf/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd9bea6f017e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reflection on My Experience Working on a Capstone Project This Spring Semester (2026)]]></title>
            <link>https://medium.com/@avikumart_/reflection-on-my-experience-working-on-a-capstone-project-this-spring-semester-2026-e03caf95afb2?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e03caf95afb2</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Mon, 04 May 2026 01:24:10 GMT</pubDate>
            <atom:updated>2026-05-04T04:07:14.377Z</atom:updated>
            <content:encoded><![CDATA[<p>A brief note on designing and developing the Agentic RAG system for the clinical trials</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20nZQ8ygCuA5Dcc0IPlF0Q.png" /><figcaption>iShowcase poster presentation</figcaption></figure><h3>Background</h3><p>This was the last semester of my MS in Information Science programme at UofA. I had only a capstone project this semester to earn the final 3 credits in the College of Information Science at the University of Arizona. As we started this semester, we had to choose what project to take on and develop or research over this semester. We were given the choice of selecting a project from a pool of ongoing projects at UofA or coming up with our own idea. I was really not sure what to do because I had multiple ideas that I was pondering. Given my previous experience with the research lab, I prefer not to pursue a research project. Instead, I plan to develop a project that combines language model technologies and software system design. I was also not so keen on building on my scratchy idea, so I chose to work on Mentor’s own project that can have an actual audience. In hindsight, this turned out to be the right approach, and I could learn much more than I could have done the other way around. So here’s more thoughts on this project and how it went.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2arYZdiccmgJ7No7ofIg8g.jpeg" /><figcaption>Me with the project poster presentation</figcaption></figure><h3>February 2026</h3><p>So, in mid-February, I decided to work on Dr Maggie O’Haire’s project on an “AI chatbot and protocol management system for clinical trials”. The motivation was to contribute to the latest models with the RAG pipeline and have real, impactful work that has a direct impact on users who will use this product. We had our first meeting with our mentor’s team around late February, and they kicked off the project with a brief introduction to the background information on the same. The main aim was to develop a chatbot system that helps lab researchers search through complicated protocol documents and answer the relevant questions with brief answers, as well as provide citations, contradictions within the documents and any relevant documents in other documents if exists.</p><p>We decided to use the LangChain ecosystem tools and tech stack, which turned out to be the best in the business, as we could build the end-to-end agentic pipeline without depending on other tools beyond LangChain. We used the langchain, langgraph and langsmith for the development of the AI agentic pipeline, orchestration and monitoring the metrics. This was one of the best decisions among others because other platforms are highly reliant on multiple different tools, which would have required a specific configuration and an increase in the settings of the project.</p><h3>March 2026</h3><p>As we entered March 2026, we had decided on a core tech stack, including the Redis vector stores and Redis in-memory stores for caching and storing the chat history. We started implementing the core system design, which we initially thought of using the langchain and langgraph stategraph templates to orchestrate the pipeline. We were four members in the team, so we divided tasks such as the ingestion pipeline, vector DB implementation, and RAG pipeline setup, as well as the flagger and linker agent design.</p><p>For the specific case, Redis vector databases and in-memory stores, I would consider this the best decision we made since Redis databases are highly optimised key value stores, and they have incredible memory storage capacity. Particularly, caching memories makes Redis a go-to database for low-latency applications where data is critical to the software systems. Once we had the initial version ready with a complete end-to-end data ingestion pipeline along with the RAG agents implemented, we quickly decided to do the initial demo to the lab team members who are going to be the actual users, so that we can have early feedback and we can improve the project performance at the earliest opportunity.</p><p>One of the most challenging tasks was to identify the right retrieval pipeline strategy, such as whether we need the reranking agent or not. Do we need the reflection agent? Or should we keep the direct retrieval with flagger and linker agents to produce the most efficient answers that give as much information as possible to the users?</p><p>Our data ingestion pipeline also had a lot of challenges along the way, because we wanted to extract a lot of headers, subheaders, tables, paragraphs and even the comments from the document files, which was quite complicated in many ways. But with team efforts and the right feedback front he mentors and lab team members, we could achieve the demoable prototype version, which is shipped via the Docker compose file so that it can be easily installed on the users’ systems.</p><p>For the frontend part, we implemented the Streamlit chat ui and sidebar admin dashboard, which can be minimal to produce as well as reliable enough to use by the users on their systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XzhJ-iigRnwFR-5J" /><figcaption>Agentic RAG pipeline workflow</figcaption></figure><h3>April 2026</h3><p>As we entered April 2026, we were giving final touches to the system and incorporating the final feedback in the product so that it could be shipped to the users. We refined the reranker and reflection agent to take past chat history into account so that the retrieval pipeline has more information while producing output to the user query. I added the rewriter agent that takes the user query and makes sure it’s relevant enough to retrieve correct output from the vector database system.</p><p>With all such improvements, we finally had an end product that was ready to demo and ship. It received incredible feedback from our mentors and lab members. It was really satisfying to see this whole exercise over the past 2–3 months work out, and we could come up with a final version in time with the report and poster presentation being prepared by mid to end of April.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nvaht28-qV4EdL46" /><figcaption>Screenshot of the frontend user interface</figcaption></figure><h3>Final thoughts!</h3><p>As we ended this project on the 15th of April 2026, we prepared the project report, which needs to be submitted as a requirement for the credits, along with the poster artefact and poster presentation on the 1st of May. Overall, it was a great experience with the loads of learning and development by actually implementing the Agentic RAG system, for which I will write more technical articles later on, and if all goes well, I will publish a book or a paper in this direction.</p><p><strong><em>Check out more information about my work at my site: </em></strong><a href="https://avikumart.github.io/"><strong><em>https://avikumart.github.io/</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e03caf95afb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Agentic AI Applications: What the Course Actually Taught Me]]></title>
            <link>https://medium.com/@avikumart_/building-agentic-ai-applications-what-the-course-actually-taught-me-0e06a200f43d?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e06a200f43d</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 21:36:21 GMT</pubDate>
            <atom:updated>2026-04-20T21:36:21.685Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQJGF-RDlRMoWkoZZCGBiA.png" /><figcaption>Snapshot from the Agentic AI course from Deeplearning.AI</figcaption></figure><p>There’s a lot of buzz on the agentic AI applications, AI agents, and so on. While building a project for the capstone project (More on the project will be published later) at UofA this Spring (Spring 2026), I pursued the Agentic AI course by <a href="http://deeplearning.ai/">Deeplearning.ai</a>, taught by Andrew Ng. AI is no longer about giving the queries to the models and producing an answer anymore — it’s about building <em>workflows</em> where models reason, plan, use tools, reflect on their outputs, and coordinate with other agents to get complex tasks done. As I was building alongside going through the course, this was a hands-on experience implementing some of the techniques in my project work as well. Below are some of my course notes that helped build agentic AI applications. I hope it is helpful to you as well.</p><h3><strong>What is an Agentic AI workflow?</strong></h3><p>At its core, an agentic workflow is about decomposing a task into a sequence of smaller, manageable steps — and letting the AI drive through them. Think of it like converting a vague, end-to-end problem into a structured plan that a system can actually execute.</p><p>A simple email responder is agentic. So is a research agent that searches the web, synthesises findings, and generates a report. What makes these “agentic” isn’t just that they use AI — it’s that the AI is making decisions across multiple steps, calling tools, and often checking its own work.</p><p>The building blocks are pretty consistent across these systems: models (obviously), other AI models, APIs, information retrieval systems, and code execution environments. These components get orchestrated together into workflows that can do real work.</p><h3><strong>Design pattern to build Agentic AI applications</strong></h3><p>The course covers several design patterns that are highly relevant and useful for building modern AI apps.</p><p><strong>Reflection</strong> is one I’ve thought about a lot. The idea is simple: don’t just generate an output — generate it, reflect on it, get feedback, and regenerate. This works especially well when you can use an external signal, like code execution output,t as the feedback mechanism. Generate code → run it → observe the error → fix and retry. This is systematic and works surprisingly well in practice. Even adding a simple reflection prompt to your workflow before producing a final answer can move the needle. This one I have implemented in my spring capstone project to reflect on the outputs generated by one LLM to improve and refine based on the user feedback.</p><p><strong>Planning</strong> is another core pattern. Instead of sending a complex prompt and hoping the model figures it out linearly, you explicitly ask the model to produce a plan — often formatted as structured JSON — before executing it. The course emphasised this clearly: <em>planning improves performance</em>. Without an explicit plan, the model might wander. With one, it has a roadmap. The customer service agent example illustrated this well — a well-designed plan of steps made the agent’s behaviour far more predictable and auditable.</p><p><strong>Tool use</strong> ties everything together. Modern LLMs are actually quite good at deciding <em>when</em> to call a specific tool based on what the prompt demands — whether that’s a web search, a database query, a math calculator, or a code runner. The key insight is that tools aren’t just bolted on; they’re what let the agent interact with the real world. The course also touched on <strong>MCP (Model Context Protocol)</strong>, initially proposed by Anthropic, which provides a unified interface for connecting models to tools through a single server. Multiple apps can share access to tools via one MCP server, which is a clean architectural move.</p><p><strong>Multi-agent systems</strong> are where things get interesting and also get complicated. The idea is straightforward: decompose the work across multiple specialised agents rather than dumping everything onto one. A researcher agent handles market analysis and web search. A graphic designer agent handles visuals. A writer handles narrative. A reviewer agent reflects and critiques before the final output ships. Each agent does one thing well, and the orchestration layer coordinates between them. The course identified several communication patterns for these systems — linear (agents talk in sequence), cyclical (feedback loops), hierarchical (an orchestrator and sub-agents), and all-to-all (every agent can talk to every other agent). Each pattern makes sense for different problem structures.</p><p>Code files are attached to the GitHub repository <a href="https://github.com/avikumart/LLM-GenAI-Transformers-Notebooks/tree/main/AgenticAIDeepLearningAI">here</a>.</p><h3><strong>The evaluations: A backbone of production-grade applications</strong></h3><p>This section of the course was probably the most practically grounded. Evals are how you actually know if your agentic system is working — and it’s an area most people underinvest in early.</p><p>There are two broad types: <strong>objective evals</strong> (answer correctness, faithfulness, answer relevance, text length checks) and <strong>subjective evals</strong> (LLM-as-Judge approaches, scoring outputs on a 1–5 scale via prompt engineering, A/B testing between outputs). Both matter at different stages.</p><p>The course also drew a useful distinction between <strong>end-to-end evals</strong> and <strong>component-level evals</strong>. End-to-end tells you if the overall system works. Component-level tells you <em>why</em> it doesn’t, and which piece is responsible. When you’re debugging an agentic pipeline, looking at intermediate traces is non-negotiable — you can’t fix what you can’t see.</p><p>Evals help to measure what matters early and detect bugs faster. Let’s be honest, AI applications are complex, and they can generate unpredictable outputs, which, if not detected early, can cost a lot of money. From my experience building the Agentic RAG project, I found that token usage can be quite different for various queries. This is hard to track without a proper eval framework. This is just one of the examples of why evaluations matter the most in AI applications.</p><h3><strong>Some tips to remember while building Agentic apps</strong></h3><p>A few things from the course that stuck with me as genuinely useful:</p><p><strong>Develop model intuition by playing with multiple models.</strong> Different models behave differently on the same prompt, and using the right model for each component of your agentic system matters. This also helps in optimising the cost early. Do not use expensive reasoning models for every task or tool call.</p><p><strong>Focus on quality first, then optimise for cost.</strong> The instinct to optimise cost early is usually wrong. Get the system working at high quality, then — when cost actually becomes a bottleneck — profile which components are most expensive and optimise them.</p><p><strong>Iterate on prompts before reaching for bigger guns.</strong> When a component isn’t performing well, the first move should usually be prompt engineering — better instructions, more examples, decomposing complex instructions into simpler ones. Simple prompt modifications can generate vastly different outputs. We really do not know how powerful models can be. Be precise in the instructions to generate better outputs.</p><p><strong>Error analysis is the loop, not a one-time exercise.</strong> The course framed this as a continuous back-and-forth: build → eval → trace → analyse errors → improve components → repeat. This is the more underrated part I mentioned in the previous section.</p><h3><strong>Final thoughts</strong></h3><p>This course was fun because I was implementing most of it in my course project as well. Course it self has a lot of coding exercises, so if you are new to agentic AI applications with relevant knowledge about large language models, this course is right there for you. Agents are clearly more about design, system thinking and architecture modifications than just building with the best models. Identification of which pattern works best matters the most. Learning about key evaluation metrics early helps you keep measuring everything that matters for your project. Don’t just run behind a building with the best model in the industry. Simple models with well-engineered prompts, along with a clear plan, work much better.</p><p><strong>Here’s the course Information to get started: </strong><a href="https://learn.deeplearning.ai/courses/agentic-ai/lesson/pu5xbv/welcome!?startTime=1"><strong>link</strong></a></p><p><strong><em>Check out more at my website: </em></strong><a href="https://avikumart.github.io/"><strong><em>https://avikumart.github.io/</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e06a200f43d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why a 74% Accurate Health Prediction Model Can Be a Complete Failure]]></title>
            <link>https://medium.com/@avikumart_/why-a-74-accurate-health-prediction-model-can-be-a-complete-failure-8370faeabfb8?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8370faeabfb8</guid>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 16:01:46 GMT</pubDate>
            <atom:updated>2026-01-12T16:01:46.724Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-p4mJMs0VDteBV8R.png" /></figure><p>The dream of using AI to predict health risks based on our daily choices is powerful. A student project analysing 100,000 health records set out to do just that, but in the process, it uncovered some surprising lessons not about success, but about the hidden pitfalls of data science. The project’s results reveal why the most common ways we measure AI performance can be dangerously misleading.</p><p><strong>Takeaway 1: “Accuracy” Can Be a Deceptive Mirage</strong></p><p>The project’s most technically advanced model, an XGBoost classifier, achieved a 74.3% accuracy score. On the surface, correctly predicting the outcome for nearly three out of every four people sounds moderately successful.</p><p>However, this number hides a critical flaw in the data. The dataset had a significant class imbalance: 75% of the records were for “Low Risk” individuals, while only 25% were for “High Risk” individuals. This reveals the statistical trap: a useless model that does nothing but guess the majority “Low Risk” category for every single person would still score 75% accuracy. Crucially, the project team was aware of this imbalance and planned to use standard techniques to correct for it, which makes the final results even more surprising. This single fact demonstrates why accuracy alone is an insufficient and often misleading metric for real-world problems.</p><p><strong>Takeaway 2: Advanced Models Were Actually Worse at the Most Important Task</strong></p><p>The central, counterintuitive finding was a battle of simple vs. complex. The project established a simple baseline model, Logistic Regression, chosen for its interpretability. Then, it deployed more advanced contenders — Random Forest and XGBoost — which were expected to perform far better. The results were shocking. The complex models were disastrously bad at the single most important task: identifying high-risk patients.</p><p>The project’s results, particularly the Recall metric, which measures the ability to find “High Risk” cases, tell the real story.</p><p><strong>“Model”</strong> “<strong>Overall</strong> <strong>Accuracy” “Recall (Ability to find High-Risk Patients)”</strong></p><p><strong>XGBoost </strong>74.3% 1.7%</p><p><strong>Random Forest </strong>71.0% 6.5%</p><p><strong>Logistic Regression </strong>49.9% 50.3%</p><p>A Recall score of just 1.7% means the high-accuracy XGBoost model failed to identify over 98% of the high-risk individuals. To put this in perspective, for every 100 high-risk patients the model analysed, it would fail to raise a flag for 98 of them, sending them home with a false sense of security. This is the critical failure: the models that were supposed to be “smarter” almost completely missed the very people they were designed to help.</p><p><strong>Takeaway 3: The Real Goal Isn’t Prediction, It’s Preventing Harm</strong></p><p>While the results were poor, the project’s design was thoughtful. The team correctly identified <strong>Recall</strong> as the most critical evaluation metric from the outset, showing a clear understanding of the problem’s real-world context.</p><p>They prioritized this metric to:</p><p>…minimize false negatives (missing high-risk patients), which is critical in a clinical setting.</p><p>The results table starkly illustrates this trade-off: the Logistic Regression model, while having poor overall accuracy, was <strong>ten times better</strong> at this critical task than Random Forest and nearly <strong>thirty times better</strong> than XGBoost. In medicine, the cost of a “false negative” — telling a sick person they are healthy — is far higher than the cost of a “false positive,” which might involve telling a healthy person to get a follow-up test. This project, even in its failure, provides a powerful lesson in responsible AI development: defining success must always start with understanding the real-world consequences of a model’s errors.</p><p><strong>Conclusion: A More Honest Way to Think About AI</strong></p><p>The spectacular failure of this project’s advanced models offers a crucial lesson for anyone building or relying on AI. Building effective systems isn’t just about chasing high accuracy scores with complex models. It is about deeply understanding the data, choosing metrics that align with human-centric goals, and being transparent about a model’s limitations.</p><p>The next time you hear about a new AI with impressive “accuracy,” what crucial questions will you ask?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8370faeabfb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build a Smarter Book Recommender: 3 Lessons from a Hybrid AI Project]]></title>
            <link>https://medium.com/@avikumart_/how-to-build-a-smarter-book-recommender-3-lessons-from-a-hybrid-ai-project-ba88a6517bb4?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba88a6517bb4</guid>
            <category><![CDATA[book-recommendations]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[recsys]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 05:10:46 GMT</pubDate>
            <atom:updated>2026-01-08T05:10:46.733Z</atom:updated>
            <content:encoded><![CDATA[<p>Building large language models enabled hybrid book recsys</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NjClXmEkfLCJxTYd.png" /><figcaption>Rough system diagram for the book rec sys</figcaption></figure><p>In an age of endless digital bookshelves, finding your next great read can feel like a paradox of choice. We’ve all been there: typing keywords into a search bar only to be met with a flood of irrelevant results. Traditional search engines, built on simple keyword matching, often struggle to understand our true interests, making the hunt for the right book a daunting and inefficient task.</p><p>But what if there was a better way? A new generation of AI-powered tools is emerging, designed not just to match words but to understand intent. This post dives into a fascinating open-source book recommendation system that demonstrates a smarter approach. By blending classic recommendation techniques with the creative power of modern large language models (LLMs), this project offers a glimpse into the future of information discovery.</p><p>After analyzing its design and architecture, I’ve distilled the three most impactful takeaways. These lessons go beyond just recommending books; they reveal a more thoughtful and effective way to build intelligent systems.</p><h4>1. The Real Power Isn’t Just New AI — It’s Blending It With the Old.</h4><p>The most compelling insight from this project is its deeply hybrid philosophy. The system doesn’t just layer a new AI model onto a simple, old method. Instead, it builds upon a sophisticated foundation of multiple, proven techniques. The architecture intelligently combines <strong>content-based filtering</strong> using item-user matrices, <strong>collaborative filtering</strong> to leverage user behaviour, and <strong>text clustering</strong> to group similar books. This multi-faceted engine is already powerful on its own.</p><p>This is where the true innovation lies. The system uses the LLM not as a replacement, but as an enhancement layer on top of this robust, traditional core. This is a critical lesson in modern AI development: the goal isn’t always to throw out what works, but to augment it. By using generative AI to enrich an already-powerful blend of algorithms, the system achieves a level of relevance and precision that no single method could accomplish alone. It’s a powerful demonstration that the most robust solutions come from an elegant fusion of the old and the new.</p><h4>2. Large Language Models Are Used as Creative Partners, Not Just Answer Machines.</h4><p>This project uses generative AI in two distinct and clever ways, treating the LLM less like a simple answer engine and more like a creative collaborator in the search process.</p><p>First, the LLM acts as a <strong>semantic scout</strong>. When a user enters a book’s ISBN, the system uses a generative model to brainstorm and generate a rich constellation of related themes and keywords. This expanded context allows the search to move beyond literal matching and find books that are thematically aligned with the user’s deeper interests, not just their initial query.</p><p>Second, after the system identifies and clusters a set of relevant books, the LLM is used as a <strong>descriptive tool</strong>. It generates a detailed, human-readable summary for each group of recommendations, making it easy for the user to understand <em>why</em> a particular set of books was suggested.</p><p>As the project’s description highlights, a key benefit is the ability to:</p><p>Leverage generative AI model capabilities to describe book content and expand user queries to many keywords</p><p>This dual-role approach — using the LLM for both creative input and descriptive output — is a masterclass in applying the right tool for the right job.</p><h4>3. Building a Powerful Recommendation System is More Accessible Than You Think.</h4><p>Perhaps the most empowering takeaway is that building sophisticated AI systems is no longer the exclusive domain of giant tech corporations. This project was built using a stack of powerful and largely accessible tools, demonstrating that individual developers and small teams can create truly innovative applications.</p><p>The core technologies used in this system include:</p><ul><li>FastAPI</li><li>Pandas</li><li>Singular Vector Decomposition (SVD)</li><li>OpenAI</li><li>K-means clustering</li><li>Streamlit</li><li>Docker</li><li>Numpy</li></ul><p>This stack represents a modern AI development paradigm: using <strong>Streamlit</strong> for rapid UI prototyping, <strong>FastAPI</strong> to serve a robust backend API, <strong>Pandas</strong> and <strong>Numpy</strong> as the backbone for data manipulation, and <strong>OpenAI</strong> as a plug-and-play intelligence layer. This modularity is what makes such a sophisticated system achievable for a small team, proving that with the right architecture and tools, advanced AI is within reach for a new generation of builders.</p><h4>Conclusion: Beyond a Better Bookshelf</h4><p>This book recommendation system is more than just a tool for finding books; it’s a blueprint for a more intelligent and effective approach to building AI. It teaches us that the smartest solutions are often thoughtful hybrids, that large language models are powerful creative partners, and that the tools to build the future are more accessible than ever before.</p><p>Ultimately, this project serves as a powerful case study for the future of applied AI. It proves that the next wave of innovation won’t come from a single, monolithic “super-AI,” but from the intelligent orchestration of specialised tools. The lesson is clear: the smartest path forward is not replacement, but elegant enhancement. What other industries are ready to be transformed by this hybrid philosophy?</p><p><strong><em>Project repository: </em></strong><a href="https://github.com/avikumart/Book-recommendation-system"><strong><em>Click Here</em></strong></a></p><p><strong><em>Check out my website for my work in AI and Data Science: </em></strong><a href="https://avikumart.github.io/"><strong><em>Here</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba88a6517bb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Counterintuitive Ideas That Revolutionised How Machines Understand Language]]></title>
            <link>https://medium.com/@avikumart_/5-counterintuitive-ideas-that-revolutionised-how-machines-understand-language-8660f657133c?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8660f657133c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Wed, 07 Jan 2026 01:45:27 GMT</pubDate>
            <atom:updated>2026-01-07T01:45:27.042Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction: From Clunky Commands to Fluid Conversation</h3><p>We’ve all seen it: a machine translation that’s technically correct but completely misses the nuance, or a chatbot that gets stuck in a loop, unable to grasp a simple request. These moments remind us that teaching a machine to understand language is far more complex than just programming a massive dictionary. Language isn’t just a collection of words; it’s a dynamic, sequential system where context is everything and subtle shifts in word order can change meaning entirely.</p><p>The journey from clunky, command-based systems to the fluid, conversational AI we see today has been anything but straightforward. It wasn’t a simple path of building bigger and bigger models. Instead, progress has been marked by a series of surprising, often counter-intuitive breakthroughs — ideas that seemed backward at first but ultimately unlocked new levels of comprehension. These were the “aha!” moments that allowed machines to move beyond simple pattern matching and begin to grasp the intricate web of relationships that defines human language.</p><p>This article explores five of the most impactful of these revolutionary ideas. From teaching machines to forget in order to remember, to abandoning sequential processing altogether, these concepts fundamentally changed the field of Natural Language Processing (NLP) and paved the way for the powerful language models we interact with every day.</p><h3>1. To Understand the Future, Machines Had to Look at the Past and the Future</h3><p>The most fundamental principle of language is that it’s sequential. Word order is not arbitrary; it’s the bedrock of meaning. Consider these two sentences, which use the exact same words but have completely different meanings:</p><ul><li><em>A student is running machine learning to analyze the data.</em></li><li><em>The machine is learning data to analyze a student running.</em></li></ul><p>Early models for sequence processing, known as Recurrent Neural Networks (RNNs), embraced this sequential nature. They processed text one word at a time, from left to right, building an understanding of a sentence based on the words that came <em>before</em> the current one. This made intuitive sense, as it mimics how we might read a sentence.</p><p>However, this one-way street has a critical flaw. The meaning of a word isn’t just shaped by its past; it’s also profoundly influenced by what comes next. Later words can completely reverse or re-contextualize what you thought you understood. Take this example:</p><p>I truly believe that Paris is the most beautiful city of Texas.</p><p>An RNN reading this one word at a time would have a strong positive sentiment about Paris until the very last word, which forces a complete re-interpretation. The model needs to see the whole picture. The solution was a conceptually simple but powerful innovation: the Bidirectional RNN (Bi-RNN). This architecture runs two separate RNNs — one that processes the text from left-to-right (forward) and another that processes it from right-to-left (backward). By combining the final understanding from the forward pass with the final understanding from the backward pass, the model gains a complete, holistic view of the sentence.</p><p>This simple-sounding idea was a massive leap forward. By looking at both the history and the future of a word, models could finally grasp the full context of a sentence, leading to a much richer and more accurate understanding of its meaning.</p><h3>2. We Taught AI to Intentionally Forget to Improve Its Memory</h3><p>While standard RNNs could handle short sentences, they had a notoriously bad memory. When processing a long paragraph, information from the beginning would often be lost or diluted by the time the model reached the end. This is known as the “long-term dependency” problem — the model struggles to connect words and ideas that are far apart in the text.</p><p>The solution was a clever and counter-intuitive hack called a Gated RNN, most famously implemented in the Long Short-Term Memory (LSTM) unit. LSTMs introduced a separate, parallel track for information to travel along, called the <strong>Cell State</strong>. Think of this as the model’s “long-term memory,” which can carry important context across very long sequences.</p><p>The magic of LSTMs lies in a series of mechanisms called “gates,” which control the flow of information into and out of this long-term memory. The three most important gates are:</p><ul><li><strong>The Forget Gate:</strong> This gate decides what information from the long-term memory is no longer relevant and should be discarded.</li><li><strong>The Input Gate:</strong> This gate decides which pieces of new information from the current word are important enough to be stored in the long-term memory.</li><li><strong>The Output Gate:</strong> This gate decides what part of the newly updated long-term memory is relevant enough to influence the model’s very next step.</li></ul><p>It was a conceptual breakthrough: to build a better memory, engineers first had to build a better system for forgetting. The key to remembering what’s important over long stretches of text was to give the model an explicit mechanism to selectively <em>forget</em> what’s irrelevant. By learning what to keep and what to throw away, LSTMs could maintain a clean, useful “memory” of the context, solving the long-term dependency problem and enabling a much deeper understanding of entire documents.</p><h3>3. The Big Leap Forward Was Ditching Sequence for Pure “Attention”</h3><p>LSTMs were a huge improvement, but they still had a fundamental bottleneck: they had to process text one word at a time. This sequential chain, even with LSTMs, was like a game of telephone; the message from the beginning of a long text could get distorted by the end. The next revolution didn’t just try to improve the chain — it broke it entirely. For decades, sequence was considered the defining, inescapable feature of language processing. The success of the next architecture showed that this foundational assumption was not only wrong but was actually the primary bottleneck holding the field back.</p><p>In 2017, researchers introduced the Transformer, an architecture that was, in their words, “based entirely on attention” and, crucially, operated “Without Recurrent Units.” Instead of a one-by-one process, the Transformer’s core mechanism, called <strong>Self-Attention</strong>, allows it to look at all the words in a sentence simultaneously.</p><p>Consider the sentence: “The bee landed on the flower because <strong>it</strong> was full of nectar.” To understand what “it” refers to, a model needs to weigh the relevance of other words. Self-attention allows the model to instantly calculate high relevance scores between “it” and “flower” and low scores for “bee,” correctly resolving the ambiguity in parallel. This happens for every word at the same time.</p><p>This ability to process all words at once wasn’t just a minor speed-up; it was a fundamental enabler of the “pre-training” paradigm. For the first time, researchers could feasibly train models on nearly the entire internet, a scale unimaginable for sequential RNNs. This innovation is the foundational technology behind almost all modern large language models, from ChatGPT to LLaMA.</p><h3>4. The “One Model to Rule Them All” Trick</h3><p>For decades, the standard approach in NLP was to build highly specialized models for each individual task. A team would design one architecture for sentiment analysis, a completely different one for machine translation, and yet another for summarizing articles. This was effective but inefficient, requiring bespoke engineering for every new problem.</p><p>Then came a surprisingly elegant idea: what if you could solve all of these different tasks within a single, unified framework? This concept of task unification was powerfully demonstrated by the <strong>T5 (Text-to-Text Transfer Transformer)</strong> model. T5 reframes every NLP problem as a “text-to-text” task. The model is given some text as input and is trained to generate some new text as output.</p><p>The key is to use a simple prefix to tell the model which task to perform. For example, to get the sentiment of a movie review, you don’t need a special sentiment model. You just add a prefix to the input text and feed it to the general-purpose T5 model:</p><p>sentiment: It confirms Fincher’s status as a film maker who artfully bends technical know-how to the service of psychological insight.</p><p>The model learns that when it sees the sentiment: prefix, its job is to output a single word representing the sentiment, such as positive. This text-to-text format was remarkably versatile. For a more complex task like Natural Language Inference, the model would be given a premise and a hypothesis with a prefix like nli premise: [text] hypothesis: [text], and it would learn to output a single word like entailment, neutral, or contradiction.</p><p>This powerful simplification allowed for the creation of far more general and versatile models that could be trained on many tasks at once, a crucial step toward the multi-talented AI assistants we use today.</p><h3>5. To Solve Hard Problems, We Tell It: “Let’s Think Step by Step”</h3><p>As language models grew larger, they became remarkably good at tasks like translation and summarization, which rely heavily on recognizing and reproducing patterns in text. However, they still struggled with problems that required multiple steps of logical reasoning, like solving a math word problem. They would often jump to an intuitive but incorrect answer.</p><p>The solution was not a new architecture but a startlingly simple prompting technique: <strong>Chain-of-Thought (CoT)</strong>. Researchers discovered that for very large models, you could dramatically improve their reasoning abilities by simply appending the phrase <strong>“Let’s think step by step”</strong> to the end of the prompt.</p><p>This phrase isn’t a magic incantation. Instead, it prompts the model to externalize its reasoning process. It encourages the model to break down the complex problem into a series of smaller, intermediate steps, write them down, and then arrive at a final answer based on that explicit chain of thought. Showing its work, just as we’re taught in school, makes the model much more likely to get the correct answer than if it tries to solve the problem in a single leap.</p><p>An extension of this idea, known as <strong>Self-Consistency</strong>, makes the process even more robust. The model is asked to generate multiple different reasoning paths for the same problem. The final answer is then chosen by a majority vote among the different paths, significantly increasing the reliability of the result. This concept has been pushed even further with methods like Tree-of-Thoughts, where models don’t just generate multiple answers, but explore multiple <em>reasoning paths</em> simultaneously, pruning dead ends and pursuing promising lines of thought, much like a human expert.</p><h3>Conclusion: More Than Just Words</h3><p>The journey to create machines that understand language has taken us from models that processed words in a rigid, one-way sequence to architectures that can weigh the relevance of every word at once. We’ve moved from specialized, single-task systems to unified models that handle dozens of tasks, and from simple pattern-matching to prompting techniques that elicit step-by-step reasoning.</p><p>Each of these counter-intuitive leaps has brought us closer to artificial intelligence that doesn’t just manipulate text but interacts with information in a way that feels more thoughtful and coherent. As we get better at teaching machines not just to predict text, but to model the very process of reasoning, what essential human quality will be the next frontier for AI to learn?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8660f657133c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Do LLMs Function?]]></title>
            <link>https://medium.com/@avikumart_/how-do-llms-function-9af28aeb7b35?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9af28aeb7b35</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 07:57:13 GMT</pubDate>
            <atom:updated>2025-07-29T07:57:45.460Z</atom:updated>
            <content:encoded><![CDATA[<p>Understanding the mechanisms and fundamentals behind the working of large language models</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZAIXn6qwqjIiqOKXIjVkQ.png" /><figcaption>AI-generated image showing an AI language model in a cyberpunk world</figcaption></figure><h3>Introduction</h3><p>Large Language Models (LLMs) represent a profound shift in artificial intelligence, demonstrating an advanced capacity to comprehend, generate, and interact with human language. Their operational core is rooted in the Transformer architecture, a groundbreaking innovation that revolutionised natural language processing by leveraging sophisticated attention mechanisms for parallel sequence processing.</p><p>The functionality of LLMs stems from a multi-stage developmental process. This begins with extensive pre-training on vast text corpora, enabling models to acquire fundamental language patterns. This is followed by iterative fine-tuning and alignment techniques, such as instruction tuning and Reinforcement Learning from Human Feedback (RLHF), which imbue these models with the ability to follow complex instructions and align with nuanced human preferences. The process of generating text itself is a sophisticated sequence, encompassing initial tokenisation and culminating in advanced decoding strategies.</p><p>Recent advancements have significantly expanded LLM capabilities, introducing multimodal processing for diverse data types, integrating Retrieval-Augmented Generation (RAG) for enhanced factual accuracy, extending context windows to manage longer inputs, and adopting Mixture of Experts (MoE) architectures for improved computational efficiency. Despite these notable strides, LLMs continue to face inherent challenges, including the propensity for hallucination, the presence of biases inherited from training data, and substantial computational resource demands. These persistent issues underscore the ongoing research imperative to develop more robust, reliable, and ethically sound AI systems.</p><h3>The Foundational Architecture: The Transformer</h3><h4>Self Attention: Core Mechanism</h4><p>The foundational strength of the Transformer architecture, and consequently of LLMs, is derived from its self-attention mechanism. This mechanism empowers the model to dynamically assess and weigh the importance of different words within an input sequence relative to each other, particularly when processing a specific word or token. This dynamic weighting facilitates a comprehensive understanding of the contextual relationships within the text.</p><p>Remarkably, even a single self-attention layer, when coupled with a softmax-based function and low-rank weight matrices, has been shown to perfectly capture the context of an entire input sequence.7 Furthermore, theoretical analyses have demonstrated that one-layer and single-head Transformers possess a memorisation capacity for finite samples. When augmented with two feed-forward neural networks, these minimal Transformer configurations can serve as universal approximators for continuous permutation-equivariant functions. This finding indicates that the fundamental computational primitive of the Transformer is inherently powerful. Consequently, the immense complexity and scale of modern LLMs, characterised by numerous layers and multiple attention heads, primarily serve to enhance their expressive power, improve generalisation abilities, and enable them to learn intricate, hierarchical representations of language, rather than merely achieving basic functional capabilities. The architectural scaling, involving stacking many layers and increasing parameters to billions, is not about achieving basic language understanding but about enabling deeper, more abstract, and more generalised learning. Each additional layer allows the model to build upon the representations learned by previous layers, forming a hierarchical understanding of language that captures long-range dependencies, complex semantic relationships, and even emergent reasoning abilities, far beyond what a single layer could achieve.</p><h4>Multi-head Attention for Enhanced Context</h4><p>Building upon the self-attention mechanism, multi-head attention serves as a critical extension within the Transformer architecture. While a single attention head can capture specific relationships between tokens, multi-head attention allows the model to simultaneously attend to information from different “representation subspaces” at various positions within the input sequence.7 This parallel processing of distinct relational aspects enables the model to capture a significantly more diverse and comprehensive set of relationships concurrently. The utilisation of multiple heads has been shown to extend the theoretical performance capabilities of even a one-layer Transformer.</p><p>The concept of “different representation subspaces” implies that LLMs develop multiple, distinct “lenses” or perspectives through which to analyse the input data. For instance, one “lens” might focus on grammatical dependencies, another on semantic similarities, and yet another on coreference resolution. This parallel analysis of various aspects of the input leads to a substantially richer and more robust understanding of context compared to a single, undifferentiated perspective. This multi-faceted analysis directly contributes to the LLM’s enhanced ability to resolve ambiguities, capture subtle meanings, and handle complex linguistic phenomena more effectively. The aggregated, multi-perspective understanding derived from multi-head attention is crucial for the LLM’s capacity to generate coherent, contextually relevant, and grammatically correct text, particularly in complex or ambiguous scenarios.</p><h3>The Encode-Decoder VS. Decoder Only Architecture</h3><p>The original Transformer model, as detailed in the seminal paper “Attention Is All You Need,” was designed with an encoder-decoder configuration.6 In this architecture, the encoder processes the entire input sequence to construct a rich, contextual representation. Subsequently, the decoder generates the output sequence one element at a time. This generation process is inherently “autoregressive,” meaning that each new token is predicted based on the previously generated tokens, which are fed back into the model as additional input.</p><p>Many contemporary Large Language Models, particularly those optimised for generative tasks such as text completion and conversational AI (e.g., GPT models), predominantly employ decoder-only architectures. While the provided sources do not explicitly detail the “decoder-only” aspect, their references to generative GPT models and their wide range of applications strongly suggest this architectural choice.</p><p>The observed prevalence of decoder-only architectures in the latest generative LLMs, compared to the original encoder-decoder design optimised for sequence-to-sequence tasks like machine translation, indicates a strategic architectural specialisation. This shows that for tasks focused on generating new, free-form content from a given prompt, the streamlined ability to process an initial input and then autoregressively extend it is more efficient and effective. The encoder-decoder setup, with its distinct input and output domains, is better suited for transformations between sequences. A decoder-only architecture, however, treats the input prompt as the initial context and then continuously predicts the next token, using its own previously generated outputs as part of the new input. This design aligns perfectly with the causal language modelling objective — predicting the next token based on all preceding tokens — which is central to the pre-training of generative LLMs, making the decoder-only design a more natural and efficient fit for their primary function. The Positional Embeddings and FFNs:</p><p>Beyond the attention mechanisms, two other critical components contribute to the Transformer’s efficacy: positional encodings and feed-forward networks.</p><p><strong>Positional Encodings:</strong> A crucial element of the Transformer architecture, positional encodings are added to the input embeddings, which are the numerical representations of tokens. This addition is essential to inject information about the relative or absolute position of tokens within the sequence. This is necessary because the core self-attention mechanism, by its very design, processes all tokens in parallel and is permutation-equivariant. This means it inherently lacks an understanding of the sequence order without this explicit positional information. The explicit necessity of positional encodings highlights a fundamental design trade-off in the Transformer. While the self-attention mechanism is exceptionally powerful for capturing relationships between tokens regardless of their distance, its permutation equivariance means it loses information about the original sequence order. This implies that the Transformer’s efficiency, derived from parallel processing, comes at the cost of needing an external mechanism to re-inject sequential context. This represents a clever engineering solution to a core architectural challenge. The reliance on positional encodings clarifies that while attention excels at identifying</p><p><em>relationships</em> between tokens (e.g., “it” refers to “dog”), it does not inherently understand <em>sequence</em>. Without positional encodings, sentences like “dog bites man” and “man bites dog” would be processed identically in terms of token relationships, potentially leading to catastrophic semantic errors. Therefore, positional encodings are a crucial complement to attention, allowing the model to leverage the benefits of parallel computation while retaining the vital sequential information necessary for accurate language understanding.</p><p><strong>Feed-Forward Networks (FFNs):</strong> Transformers also incorporate feed-forward neural networks within their architecture. These networks are typically connected both before and after the self-attention mechanisms within each layer.7 FFNs operate independently and identically on each position (token) in the sequence, applying a non-linear transformation to the outputs of the attention layers. These networks significantly contribute to the model’s capacity to learn complex patterns and transformations of the input data.</p><p><a href="https://www.linkedin.com/in/avikumar-talaviya-739153147/">Connect with me on LinkedIn</a></p><p><a href="https://x.com/avikumart_">Connect with me on X</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9af28aeb7b35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Large Language Models Using Hugging Face Transformers and Phi-3-Mini-Instruct…]]></title>
            <link>https://medium.com/@avikumart_/understanding-the-large-language-models-using-hugging-face-transformers-and-phi-3-mini-instruct-6b9886f42f40?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b9886f42f40</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Thu, 19 Jun 2025 12:36:07 GMT</pubDate>
            <atom:updated>2025-06-19T12:36:07.086Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding the Large Language Models Using Hugging Face Transformers and Phi-3-Mini-Instruct Model</h3><p>Overview of the large language model using Microsoft’s Phi3 mini instruct tuned model with code examples</p><h4>Introduction</h4><p>Large language models have transformed the perception of the artificial intelligence field among the general public, extending beyond the typical technical and academic audience. In the early 2010s, when neural networks took off from scholarly papers to real-world applications, nobody would have thought that humble neural network-based architecture would lead to cutting-edge reasoning models within the next decade or so. From early neural network-based architecture to sequential models such as recurrent neural networks or LSTMs, we now have an attention-based model, which is called the transformer architecture, which came out in 2017, written by Vaswani et. al.</p><p>This article particularly focuses on the model features and architecture of Microsoft Phi-3-mini-instruct based on its Hugging Face model page. I describe details of the training process, the dataset used for training and the inference code with benchmarks.</p><h4>Model Overview</h4><p><strong>Phi-3-Mini-4K-Instruct</strong> is a compact yet powerful open-source language model with 3.8 billion parameters, built using the Phi-3 training dataset. This dataset combines high-quality synthetic data and filtered web content, emphasising strong reasoning capabilities. It’s part of the Phi-3 family and comes in two variants based on context length — 4K and 128K tokens.</p><p>The model has gone through post-training that includes supervised fine-tuning and direct preference optimization to improve instruction-following ability and safety. Despite its smaller size (under 13B parameters), Phi-3-Mini-4K-Instruct delivers state-of-the-art performance on tasks involving commonsense reasoning, language understanding, mathematics, code generation, long-context handling, and logical reasoning.</p><p>Model being light-weight and computationally less expensive it can be used for commercial and research applications with following requirements:</p><ol><li>memory/compute-constrained environments;</li><li>latency-bound scenarios;</li><li>strong reasoning (especially math and logic).</li></ol><p>The model is for general-purpose use-cases and has not been particularly evaluated on many downstream tasks for specific use-cases such as finance, medical AI, etc. Developers and users should evaluate the performance on such tasks with safety, privacy, and compliance as per the applicable laws and regulations.</p><p>Hugging Face provides built-in classes for tokenizer and model inference to build model pipeline that handles user inputs in a structured manner. We will look at the inference code as described in the <em>Hands on Large Language Models</em> book by Jay Alammar et. al.</p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline<br><br># Load model and tokenizer<br>tokenizer = AutoTokenizer.from_pretrained(&quot;microsoft/Phi-3-mini-4k-instruct&quot;)<br><br>model = AutoModelForCausalLM.from_pretrained(<br>    &quot;microsoft/Phi-3-mini-4k-instruct&quot;,<br>    device_map=&quot;cuda&quot;,<br>    torch_dtype=&quot;auto&quot;,<br>    trust_remote_code=False,<br>)<br><br># Create a pipeline<br>generator = pipeline(<br>    &quot;text-generation&quot;,<br>    model=model,<br>    tokenizer=tokenizer,<br>    return_full_text=False,<br>    max_new_tokens=50,<br>    do_sample=False,<br>)</pre><p>HF’s AutoTokenizer class tokenises the input queries and passes them through the model class, which generates a suitable output from the model. Let’s look at the example prompt for the model pipeline using the <em>Phi-3-mini-4k-instruct</em> model.</p><pre>prompt = &quot;Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.&quot;<br><br>output = generator(prompt)<br><br>print(output[0][&#39;generated_text&#39;])</pre><p>Model architecture consists of the tokeniser vocabulary size of 32K with an embedding layer and an attention layer to train 4.9T tokens of publicly available data, including synthetic data for better reasoning, general knowledge, and truthfulness. Below is the model architecture representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRq3ykYvQFKa9RO1YzWCvg.png" /><figcaption>architecture layers of the phi3-mini-instruct</figcaption></figure><h4>Model Training</h4><ul><li>Architecture: Phi-3 Mini-4K-Instruct has 3.8B parameters and is a dense decoder-only Transformer model. The model is fine-tuned with Supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) to ensure alignment with human preferences and safety guidlines.</li><li>Inputs: Text. It is best suited for prompts using chat format.</li><li>Context length: 4K tokens</li><li>GPUs: 512 H100–80G</li><li>Training time: 10 days</li><li>Training data: 4.9T tokens</li><li>Outputs: Generated text in response to the input</li><li>Dates: Our models were trained between May and June 2024</li><li>Status: This is a static model trained on an offline dataset with a cutoff date of October 2023. Future versions of the tuned models may be released as we improve the models.</li><li>Release dates: June 2024.</li></ul><h4>Dataset used</h4><p>The Microsoft team has used a wide range of data consisting of 4.9 tokens, which includes:</p><ol><li>Publicly available documents filtered rigorously for quality, selected high-quality educational data, and code;</li><li>Newly created synthetic, “textbook-like” data to teach math, coding, common sense reasoning, general knowledge of the world (science, daily activities, theory of mind, etc.);</li><li>High-quality chat format supervised data covering various topics to reflect human preferences on different aspects such as instruction-following, truthfulness, honesty and helpfulness.</li></ol><p>Despite being smaller in size model achieves greater performance across many benchmarks or similar level of benchmarks in language understanding and reasoning capabilities.</p><p>For more information, check out the Hugging Face model page and follow the GitHub repo to run the model.</p><p>HF model page: <a href="https://huggingface.co/microsoft/Phi-3-mini-4k-instruct"><em>link</em></a></p><p>Github repo: <a href="https://github.com/avikumart/LLM-GenAI-Transformers-Notebooks/tree/main/HandsOnLLMs"><em>link</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b9886f42f40" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build an AI agent using Llama Index and MonsterAPI]]></title>
            <link>https://medium.com/@avikumart_/how-to-build-an-ai-agent-using-llama-index-and-monsterapi-c067006e2726?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c067006e2726</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 08:58:34 GMT</pubDate>
            <atom:updated>2024-09-25T08:58:34.424Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xSECiepUuM85MzVG.jpeg" /><figcaption>An illustration of robot to depict AI agent’s working capabilities</figcaption></figure><h3>Introduction</h3><p>AI agents are the future and they can be the driving force to the future. AI agents are becoming increasingly integral to AI’s growth and new technological advancements. AI agents are applications that mirror human-like attributes to interact, reason, and even make suitable decisions to achieve certain targets with sophisticated autonomy and perform multiple tasks in real time, which was impossible with LLMs.</p><p>In this article, we will look into the details of AI agents and their work with the help of MonsterAPIs and LlamaIndex tools. LlamaIndex is a powerful framework designed for building and managing large-scale language models, offering seamless integration with various data sources. Key features include efficient vector-based retrieval, customizable query engines, and support for complex indexing, making it ideal for developing AI-powered applications. LlamaIndex also provides a suite of tools and abstractions to easily develop AI agents.</p><p>We will also use MonsterAPI for LLM APIs to build agentic applications with real-world examples. MonsterAPI is a platform with a suite of APIs for fine-tuning, testing, evaluating, and deploying the latest large language models such as Llama 3, Gemma, Mistral, Stable diffusion, etc. So let’s get started:</p><p><strong>Table of contents:</strong></p><ol><li>What are AI agents?</li><li>Working with AI agents</li><li>Use-cases of AI agents</li><li>Building Text to SQL agent using llama index and MonsterAPI</li><li>Conclusion</li></ol><h3>What are AI agents?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_WxZUxFejNbZSIo9.png" /><figcaption>AI agents depiction</figcaption></figure><p>AI agents are autonomous systems designed to mimic human behaviours, allowing them to perform tasks that resemble human thinking and observations. Agents act in an environment in conjunction with LLMs, tools and memory to perform various tasks. AI agents differ from large language models in their working and process to generate outputs. We will look at some basic attributes of AI agents in this blog along with differences between LLMs and AI agents to have a detailed idea of AI agents.</p><ol><li><strong>AI agents think like humans:</strong> AI agents use tools to perform specific functions to produce a certain output. For example Search engine, Database search, Calculator, etc.</li><li><strong>AI agents act like humans: </strong>Just like humans, AI agents use a sequence of actions and planning to use tools to perform certain actions and produce outputs.</li><li><strong>AI agents observe like humans:</strong> Using frameworks for planning agents react, reflect and take action suitable for certain inputs. Memory components allow AI agents to retain previous steps and actions so that AI agents can efficiently produce desired outputs.</li></ol><p>Let’s look at the core difference between LLMs and AI agents to clearly distinguish betweeechniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gFTY22ezz1A0-QGY.png" /><figcaption>LLMs Vs AI agents</figcaption></figure><h3>Working with AI agents</h3><p>Agents are developed out of a set of components mainly the memory layer, tools, models and reasoning loops that work in orchestration to achieve a set of tasks or certain specific tasks that the user might want to solve. For example, Using a weather agent to extract real-time weather data with the voice or text command by the user. Let’s learn more about each component to build AI agents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m31SKTKSxe-xk9PC" /></figure><p>AI agents working components</p><ol><li><strong>Reasoning Loop:</strong> The reasoning loop is at the core of AI agents to make the planning of actions and enable decision-making for processing of the inputs, refining outputs to produce desired results at the end of the loop.</li><li><strong>Memory Layer:</strong> Memory is a crucial part of the AI agents to remember planning, thoughts and actions throughout the processing of the user inputs for generating certain outcomes out of it. The memory could be short-term and long-term depending upon a problem.</li><li><strong>Models:</strong> Large language models help to synthesize and generate results in ways humans can interpret and understand.</li><li><strong>Tools:</strong> These are external built-in functions that agents can utilize to perform specific tasks such as a retrieval of data from databases, and APIs, getting real-time weather data or a calculator to perform certain calculations that result in the capabilities of the agents.</li></ol><p><strong>Interaction between components:</strong></p><p>The Reasoning Loop continuously interacts with both the model and the tools. The loop uses the model’s outputs to inform decisions, while the tools are employed to act on those decisions.</p><p>This interaction forms a closed loop where data flows between the components, allowing the agent to process information, make informed decisions, and take appropriate actions seamlessly.</p><p>Let’s look at the use cases of AI agents and then we will look at live code examples of AI agents using MonsterAPI.</p><p><strong>Usage patterns in AI agents:</strong></p><p>LlamaIndex provides high-level tools and classes to develop AI agents without worrying about execution and implementation.</p><p>In the reasoning loop, LlamaIndex provides function-calling agents that integrate well with LLMs, ReAct Agents, Vector stores, and advanced agents to build effective working agentic applications from prototype to production.</p><p>In LlamaIndex agents are developed in the following pattern. We will look at the AI agent’s development later in the blog.</p><pre>from llama_index.agent.openai import OpenAIAgent<br>from llama_index.llms.openai import OpenAI<br><br># import and define tools<br>...<br># initialize llm<br>llm = OpenAI(model=&quot;gpt-3.5-turbo-0613&quot;)<br># initialize openai agent<br>agent = OpenAIAgent.from_tools(tools, llm=llm, verbose=True)</pre><h3>Use-cases of AI agents</h3><p>AI agents have a wide range of use cases in the real world to achieve common tasks and improve time efficiency while enhancing revenue for businesses. Some of the common use cases are as follows:</p><ol><li><strong>Agentic RAG:</strong> Building a context-augmented system to leverage business-specific datasets for enhanced user query response and accuracy of answers for certain input queries.</li><li><strong>SQL Agent:</strong> Text to SQL is another use-case where agents utilize LLMs and databases to generate automated SQL queries and result in a user-friendly output without writing a SQL query.</li><li><strong>Workflow assistant:</strong> Building an agent that can work along with common workflow assistants like weather APIs, calculators, calendars, etc.</li><li><strong>Code assistant:</strong> Assistant to help review, write and enhance code writing experience for the developers.</li><li><strong>Content curation:</strong> AI agents can suggest personalized content such as articles, and blog posts and can also summarize the information for users.</li><li><strong>Automated trading:</strong> AI agents can extract real-time market data including sentiment analysis to trade automatically that maximizes profit for the businesses.</li><li><strong>Threat detection:</strong> AI agents can monitor network traffic, identify potential security threats, and respond to cyber-attacks in real time, enhancing an organization’s cybersecurity posture.</li></ol><h3>Building Text to SQL agent using LlamaIndex and MonsterAPI</h3><p>In this section, we will look at the Text to SQL application with LlamaIndex tools and MonsterAPI for accessing large language models. Before deep diving into code, let’s have a look at the overview of a MonsterAPI platform.</p><h4>Overview of a MonsterAPI</h4><p>MonsterAPI is an easy-to-use no-code/low-code tool that simplifies deployment, fine-tuning, testing, evaluating and error management for large language model-based applications including AI agents. It costs less compared to other cloud platforms and can be used for FREE for personal projects or research work. It supports a wide range of models such as text generation, image generation and code generation models. In our example, MonsterAPI model APIs access the custom dataset stored using LlamaIndex vector store for augmented answers to use query based on new dataset added.</p><h4>1. Install the necessary library and set up an environment</h4><p>Firstly, we will install the necessary libraries and modules including MonsterAPI LLMs, LlamaIndex, <strong>sqlalchemy</strong>, and <strong>llamaindex</strong> embeddings for further development of the agent. Also, sign up on the <a href="https://monsterapi.ai/llm-finetuning">MonsterAPI</a> platform to get the API key to access the large language model.</p><pre>!pip install pymysql<br>!pip install llama_index<br>!pip install --force-reinstall &#39;sqlalchemy&lt;2.0.0&#39;<br>!pip install llama-index-callbacks-aim<br>!pip install sqlalchemy<br>!pip install monsterapi<br>!pip install llama-index-llms-monsterapi<br>!pip install llama-index-embeddings-huggingface<br><br># import logging modules to configure login parameters<br>import logging<br>import sys<br>import pymysql<br>logging.basicConfig(stream=sys.stdout, level=logging.INFO, force=True)<br>logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))<br>from IPython.display import Markdown, display<br># db credentials <br>db_user = &quot;your_username&quot;<br>db_password = &quot;your_password&quot;<br>db_host = &quot;your_host&quot;<br>db_name = &quot;your_name&quot;<br># create_engine, text classes from sqlalchemy<br>from sqlalchemy import create_engine, text<br># Construct the connection string<br>connection_string = f&quot;mysql+pymysql://{db_user}:{db_password}@{db_host}/{db_name}&quot;<br># Create an engine instance<br>engine = create_engine(connection_string)<br># Test the connection using raw SQL<br>with engine.connect() as connection:<br>    result = connection.execute(text(&quot;show tables&quot;))<br>    for row in result:<br>        print(row)<br>---------------------------------[Output]-------------------------------------<br>(&#39;Customers&#39;,)<br>(&#39;OrderDetails&#39;,)<br>(&#39;Orders&#39;,)<br>(&#39;info&#39;,)<br>(&#39;table_36370395d9314572970b325bee42817d&#39;,)<br>(&#39;table_59af3c5a5388482193f88093fc2eaa36&#39;,)</pre><p>In the above code, first, you have to connect with the MySQL database using the database credential available to you. create a connection string object to the database then create an engine that connects database tables. once the connection is established we can fetch table details for querying purposes.</p><h4>2. Llamaindex SQL Database instance</h4><p>Once the database is connected we use the ‘<strong><em>SQLDatabase’</em></strong> class from the llamaindex core library to fetch the table schema of selected tables. after the SQL database is created we can now go ahead with NL2SQL application development.</p><pre>from llama_index.core import SQLDatabase<br># select tables to connect for application<br>tables = [&#39;table_36370395d9314572970b325bee42817d&#39;, &#39;table_59af3c5a5388482193f88093fc2eaa36&#39;]<br>sql_database = SQLDatabase(engine, include_tables=tables, sample_rows_in_table_info=2)</pre><h4>3. Developing Text to SQL application using llama index modules</h4><p>Now, we will set up a token counter and large language models for query handling to the database for response generation using database tables.</p><pre>import os<br>from llama_index.llms.monsterapi import MonsterLLM<br>import tiktoken<br>from llama_index.core.callbacks import CallbackManager, TokenCountingHandler<br><br># token counter using token counting handler<br>token_counter = TokenCountingHandler(<br>    tokenizer=tiktoken.encoding_for_model(&quot;Meta-Llama-3-8B-Instruct&quot;).encode<br>)<br># callback manager for encoding input queries<br>callback_manager = CallbackManager([token_counter])<br># set up monsterapi key<br>os.environ[&quot;MONSTER_API_KEY&quot;] = &quot;YOUR_API_KEY&quot;<br># create an instance of llama-3-8b model<br>model = &quot;meta-llama/Meta-Llama-3-8B-Instruct&quot;<br>llm = MonsterLLM(model=model, temperature=0.75)<br># setting up service context for llm<br>service_context = ServiceContext.from_defaults(<br>  llm=llm,callback_manager=callback_manager<br>)</pre><p>This code sets up a basic infrastructure for an agentic application that uses a LlamaIndex-based model hosted on MonsterAPI. Here’s what it achieves:</p><ul><li><strong>Token Management:</strong> The TokenCountingHandler tracks the number of tokens used, which is crucial for managing costs and performance in applications that interact with large language models.</li><li><strong>Model Initialization:</strong> The MonsterLLM instance is configured with a specific model (“Meta-Llama-3–8B-Instruct”) and a temperature setting to control response variability.</li><li><strong>Service Context:</strong> A ServiceContext is created to manage the model’s operations, including token counting, ensuring smooth and efficient interactions.</li></ul><h4>4. Creating SQL table node mapping</h4><p>Using llama index vectorstore, embeddings and objectindex, we can list tables of the database and create a table schema for the prompt to MonsterAPI model API to generate suitable SQL query for the user’s query.</p><pre>from llama_index.core.embeddings import resolve_embed_model<br>from llama_index.core import VectorStoreIndex, SimpleDirectoryReader<br>from llama_index.core.objects import ObjectIndex, SQLTableNodeMapping, SQLTableSchema<br>import pandas as pd<br><br># list all the tables from database and crate table schema for prompt to LLM<br>tables = list(sql_database._all_tables)<br>table_node_mapping = SQLTableNodeMapping(sql_database)<br>table_schema_objs = []<br>for table in tables:<br>    table_schema_objs.append((SQLTableSchema(table_name = table)))</pre><h4>5. Initializing SQL table query retrieval engine</h4><pre>from llama_index.core.indices.struct_store.sql_query import SQLTableRetrieverQueryEngine<br><br># create an object index to store vectors <br>obj_index = ObjectIndex.from_objects(<br>    table_schema_objs,<br>    table_node_mapping,<br>    VectorStoreIndex<br>)<br># initializing query engine <br>query_engine = SQLTableRetrieverQueryEngine(<br>    sql_database, obj_index.as_retriever(similarity_top_k=3), service_context=service_context<br>)<br># Ask query to query engine and store it in a response variable<br>response = query_engine.query(&quot;How many people have previous work experience?&quot;)</pre><p>This code snippet implements a query engine using LlamaIndex’s SQLTableRetrieverQueryEngine to interact with a database. The process involves several key steps:</p><p>1. <strong>Object Index Creation:</strong> An ObjectIndex is created to store vectors, representing structured data objects such as database tables (table_schema_objs and table_node_mapping). This index is configured to use a VectorStoreIndex to manage and retrieve these vectors based on their similarities.</p><p>2. <strong>Query Engine Initialization:</strong> The SQLTableRetrieverQueryEngine is initialized by passing in a SQL database (sql_database), the object index as a retriever configured to return the top 3 most similar results (similarity_top_k=3), and a service_context to manage interactions with the database.</p><p>3. <strong>Executing a Query:</strong> A query is executed against the database using the query engine, asking for information on the number of people with previous work experience. The response from this query is stored in the response variable.</p><p>With this, we can query the database in a natural language and get the answer without writing explicit queries in SQL to answer business questions.</p><h3>Conclusion</h3><p>In conclusion, AI agents are transforming the way we interact with AI technologies by having AI assistants, or tools that will mimic human-like thinking and behaviour to perform tasks autonomously.</p><p>We learned what are AI agents, how they work and many real-world use cases of such agents. Agents contain mainly memory layers, reasoning loops, models and tools to achieve desired tasks without much human intervention.</p><p>By leveraging powerful frameworks like LlamaIndex and MonsterAPI, we can build capable agents that can connect to a database and extract the database tables schema for SQL query generation. Generated SQL query then shall be executed on SQL database to return required user response. check out MonsterAPI to leverage cheap and efficient LLMs and fine-tuning services for your next Agentic application.</p><p><strong>MonsterAPI Website</strong>: <a href="https://monsterapi.ai/">Click here</a></p><p><strong>MonsterAPI on X:</strong> <a href="https://x.com/monsterapis">Click here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c067006e2726" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A comprehensive guide to fine-tune open-source LLMs using Lamini]]></title>
            <link>https://medium.com/@avikumart_/a-comprehensive-guide-to-fine-tune-open-source-llms-using-lamini-1dde12f51d82?source=rss-13fcacf52e4f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dde12f51d82</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Avikumar Talaviya]]></dc:creator>
            <pubDate>Wed, 18 Sep 2024 05:09:02 GMT</pubDate>
            <atom:updated>2024-09-18T05:09:02.742Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vE8B-EsoqFZuzcI4.png" /><figcaption>Illustration of Lamini’s capabilities</figcaption></figure><h3>Introduction</h3><p>Recently, we have seen innumerable advancements in natural language processing with the rise of large language models and AI in general. Models in domains like text generation, code generation, and image/video generation have archived human-like reason and performance.</p><p>These models perform exceptionally well in general knowledge-based questions. Models like GPT-4o, Llama 2, Claude, and Gemini are trained on many publicly available datasets, so they fail to answer domain or subject-specific questions that may be more useful for a wide range of organisational tasks. Fine-tuning helps developers and businesses adapt and train pre-trained models to a domain-specific dataset that archives high-accuracy, and coherency on queries related to the domain. Fine-tuning enhances the model’s performance without requiring extensive computing resources because pre-trained models have already learned the general text from the vast public data.</p><p>In this blog, we will look at the comprehensive guide on why we need to fine-tune pre-trained models with the help of the Lamini platform which allows us to fine-tune and evaluate models without much computation resources. So let’s get started:</p><p><strong>(Check out the original published article on Analytics Vidhya — </strong><a href="https://www.analyticsvidhya.com/blog/2024/09/fine-tune-open-source-llms-using-lamini/"><strong>Click Here</strong></a><strong>)</strong></p><h4>Learning objectives:</h4><ol><li>To explore the need for fine-tuning large language models using the Lamini platform</li><li>To find out the use of Lamini and understand Instruction fine-tuned models</li><li>To get a hands-on understanding of the end-to-end process of fine-tuning models.</li></ol><h4>Table of contents:</h4><ol><li>Why fine-tune large language models?</li><li>Fine-tuning using Lamini</li><li>Data preparation</li><li>Fine-tuning process</li><li>Conclusion</li></ol><h3>Why fine-tune large language models?</h3><p>Pre-trained models are largely trained on vast general data with a high chance of lack of context or domain-specific knowledge. Pre-trained models also can result in hallucinations, inaccurate outputs and incoherent outputs. Most popular large language models based on chatbots like ChatGPT, Gemini, and BingChat have repeatedly shown that pre-trained models are prone to such inaccuracies. This is where fine-tuning comes to the rescue, which can help to adapt pre-trained LLMs to subject-specific tasks and questions effectively. Other ways to align models to your objectives include prompt engineering and few-shot prompt engineering. Still, fine-tuning remains an outperformer when it comes to performance metrics. Methods such as Parameter efficient fine-tuning and Low adaptive ranking fine-tuning have further improved the model fine-tuning and helped developers generate better models. Let’s look at how fine-tuning fits in a large language model context.</p><pre># Load the fine-tuning dataset<br>filename = &quot;lamini_docs.json&quot;<br>instruction_dataset_df = pd.read_json(filename, lines=True)<br>instruction_dataset_df<br><br># Load it into a python&#39;s dictionary<br>examples = instruction_dataset_df.to_dict()<br><br># prepare a samples for a fine-tuning <br>if &quot;question&quot; in examples and &quot;answer&quot; in examples:<br>  text = examples[&quot;question&quot;][0] + examples[&quot;answer&quot;][0]<br>elif &quot;instruction&quot; in examples and &quot;response&quot; in examples:<br>  text = examples[&quot;instruction&quot;][0] + examples[&quot;response&quot;][0]<br>elif &quot;input&quot; in examples and &quot;output&quot; in examples:<br>  text = examples[&quot;input&quot;][0] + examples[&quot;output&quot;][0]<br>else:<br>  text = examples[&quot;text&quot;][0]<br><br># Using a prompt template to create instruct tuned dataset for fine-tuning<br>prompt_template_qa = &quot;&quot;&quot;### Question:<br>{question}<br><br>### Answer:<br>{answer}&quot;&quot;&quot;</pre><p>In the above code, we can see that instruct tuning uses a prompt template to prepare a dataset for instruction tuning and fine-tune a model for a specific dataset. We can fine-tune the pre-trained model to a specific use case using such a custom dataset. In the next section, we will look at how Lamini can help fine-tune large language models for custom datasets.</p><h3>Fine-tuning using Lamini</h3><p>The Lamini platform enables users to fine-tune models and deploy models seamlessly without much cost and hardware setup requirements. <a href="https://www.lamini.ai/product">Lamini</a> provides an end-to-end stack to develop, train, tune and deploy models at per user’s convenience and model requirements. Lamini provides its own hosted GPU computing network to train models cost-effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lAIHTA9_yJ2ZuWxY.png" /><figcaption>Fine-tuning using Lamini illustration (source: Lamini)</figcaption></figure><p>Lamini memory tuning tools and compute optimisation help to train and tune models with high accuracy while keeping costs in control. Models can be hosted anywhere you want. It can be hosted on a private cloud, or by Lamini’s own GPU network. Next, we will see step by step guide to prepare data for fine-tuning large language models using Lamini platform.</p><h3>Data preparation</h3><p>Generally, we need to select a domain-specific dataset for data cleaning, instruction promoting, tokenisation, and storage to prepare data for any fine-tuning task. Once the dataset is loaded, we pre-process it to make it an instruction-tuned dataset. Each sample from the dataset shall be formatted in Instruction, question and answer format for better fine-tuning for our use cases. Let’s look at the code example of instruction tuning with tokenisation for training using the Lamini platform.</p><pre>import pandas as pd<br><br># load the dataset and store it as an instruction dataset<br>filename = &quot;lamini_docs.json&quot;<br>instruction_dataset_df = pd.read_json(filename, lines=True)<br>examples = instruction_dataset_df.to_dict()<br><br>if &quot;question&quot; in examples and &quot;answer&quot; in examples:<br>  text = examples[&quot;question&quot;][0] + examples[&quot;answer&quot;][0]<br>elif &quot;instruction&quot; in examples and &quot;response&quot; in examples:<br>  text = examples[&quot;instruction&quot;][0] + examples[&quot;response&quot;][0]<br>elif &quot;input&quot; in examples and &quot;output&quot; in examples:<br>  text = examples[&quot;input&quot;][0] + examples[&quot;output&quot;][0]<br>else:<br>  text = examples[&quot;text&quot;][0]<br><br>prompt_template = &quot;&quot;&quot;### Question:<br>{question}<br><br>### Answer:&quot;&quot;&quot;<br><br># Store fine-tuning examples as an instruction format<br>num_examples = len(examples[&quot;question&quot;])<br>finetuning_dataset = []<br>for i in range(num_examples):<br>  question = examples[&quot;question&quot;][i]<br>  answer = examples[&quot;answer&quot;][i]<br>  text_with_prompt_template = prompt_template.format(question=question)<br>  finetuning_dataset.append({&quot;question&quot;: text_with_prompt_template, &quot;answer&quot;: answer})</pre><p>In the above example, using a prompt template, we have formatted <strong>“questions”</strong> and <strong>“answers”</strong> in a prompt template and then stored them in a separate file for tokenisation and padding before training the LLM.</p><h4><strong>Tokenise the dataset</strong></h4><pre># Tokenization of the dataset with padding and truncation<br>def tokenize_function(examples):<br>    if &quot;question&quot; in examples and &quot;answer&quot; in examples:<br>      text = examples[&quot;question&quot;][0] + examples[&quot;answer&quot;][0]<br>    elif &quot;input&quot; in examples and &quot;output&quot; in examples:<br>      text = examples[&quot;input&quot;][0] + examples[&quot;output&quot;][0]<br>    else:<br>      text = examples[&quot;text&quot;][0]<br>    <br>    # padding<br>    tokenizer.pad_token = tokenizer.eos_token<br>    tokenized_inputs = tokenizer(<br>        text,<br>        return_tensors=&quot;np&quot;,<br>        padding=True,<br>    )<br><br>    max_length = min(<br>        tokenized_inputs[&quot;input_ids&quot;].shape[1],<br>        2048<br>    )<br>    # truncation of the text<br>    tokenizer.truncation_side = &quot;left&quot;<br>    tokenized_inputs = tokenizer(<br>        text,<br>        return_tensors=&quot;np&quot;,<br>        truncation=True,<br>        max_length=max_length<br>    )<br><br>    return tokenized_inputs</pre><p>The above code takes the dataset samples as input for padding and truncation with tokenisation to generate pre-processed tokenised dataset samples, which can be used for fine-tuning pre-trained models. Now that the dataset is ready, we will look into the training and evaluation of models using the Lamini platform.</p><h3>Fine-tuning process</h3><p>Now that we have a dataset prepared in an instruction-tuning format, we will load the dataset into the environment and fine-tune the pre-trained model using Lamini&#39;s easy-to-use training techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dEq99mjqJshhCa5M.png" /><figcaption><strong>Credit:</strong> Jose J. Martinez Via Medium</figcaption></figure><h4>Setting up an environment</h4><p>To begin the fine-tuning process, we first need to ensure our code environment has suitable resources and libraries installed. we need to ensure that you have a suitable machine with sufficient GPU resources and install necessary libraries such as transformers, datasets, torch, and pandas. Environment variables like api_url and api_key must be securely loaded, typically from environment files. You can utilize packages like <em>dotenv</em> to load these variables. once the environment is prepared, load the dataset and models for the training process.</p><pre>import os<br>import lamini<br><br>lamini.api_url = os.getenv(&quot;POWERML__PRODUCTION__URL&quot;)<br>lamini.api_key = os.getenv(&quot;POWERML__PRODUCTION__KEY&quot;)<br><br># import necessary library and load the environment files<br>import datasets<br>import tempfile<br>import logging<br>import random<br>import config<br>import os<br>import yaml<br>import time<br>import torch<br>import transformers<br>import pandas as pd<br>import jsonlines<br><br># Loading transformer architecture and [[<br>from utilities import *<br>from transformers import AutoTokenizer<br>from transformers import AutoModelForCausalLM<br>from transformers import TrainingArguments<br>from transformers import AutoModelForCausalLM<br>from llama import BasicModelRunner<br><br><br>logger = logging.getLogger(__name__)<br>global_config = None</pre><h4>Load dataset</h4><p>After setting up logging for monitoring and debugging, prepare your dataset using datasets or other data handling libraries like jsonlines and pandas. after loading the dataset, we will set up a tokenizer and model with training configurations for the training process.</p><pre># load the dataset from you local system or HF cloud<br>dataset_name = &quot;lamini_docs.jsonl&quot;<br>dataset_path = f&quot;/content/{dataset_name}&quot;<br>use_hf = False<br><br># dataset path<br>dataset_path = &quot;lamini/lamini_docs&quot;</pre><h4>Set up model, training config, and tokeniser</h4><p>Then we need to select the model for fine-tuning which is “<strong><em>EleutherAI/pythia-70m</em></strong>”, and its configuration is defined under <strong><em>training_config</em></strong>, which specifies the pre-trained model name and the dataset path. The <strong><em>AutoTokenizer</em></strong> is initialized with the model’s tokenizer, and padding is set to the end-of-sequence token. Data is then tokenized and split into training and testing datasets using a custom function, <strong><em>tokenize_and_split_data</em></strong>. The base model is instantiated using <strong><em>AutoModelForCausalLM</em></strong>, which allows it to perform causal language modelling tasks. Also, the below code sets up compute requirements for our model fine-tuning process.</p><pre># model name<br>model_name = &quot;EleutherAI/pythia-70m&quot;<br><br># training config<br>training_config = {<br>    &quot;model&quot;: {<br>        &quot;pretrained_name&quot;: model_name,<br>        &quot;max_length&quot; : 2048<br>    },<br>    &quot;datasets&quot;: {<br>        &quot;use_hf&quot;: use_hf,<br>        &quot;path&quot;: dataset_path<br>    },<br>    &quot;verbose&quot;: True<br>}<br><br># setting up auto tokenizer<br>tokenizer = AutoTokenizer.from_pretrained(model_name)<br>tokenizer.pad_token = tokenizer.eos_token<br>train_dataset, test_dataset = tokenize_and_split_data(training_config, tokenizer)<br><br># set up a baseline model<br>base_model = AutoModelForCausalLM.from_pretrained(model_name)<br><br># gpu parallization<br>device_count = torch.cuda.device_count()<br>if device_count &gt; 0:<br>    logger.debug(&quot;Select GPU device&quot;)<br>    device = torch.device(&quot;cuda&quot;)<br>else:<br>    logger.debug(&quot;Select CPU device&quot;)<br>    device = torch.device(&quot;cpu&quot;)</pre><h4>Setup training to fine-tune the model</h4><p>Finally, we set up training argument parameters with hyperparameters such as a learning rate, epochs, batch size, output directory, eval steps, save, and warmup steps, evaluation and logging strategy, etc to fine-tune custom training dataset.</p><pre>max_steps = 3<br><br># trained model name<br>trained_model_name = f&quot;lamini_docs_{max_steps}_steps&quot;<br>output_dir = trained_model_name<br><br>training_args = TrainingArguments(<br>  # Learning rate<br>  learning_rate=1.0e-5,<br>  # Number of training epochs<br>  num_train_epochs=1,<br><br>  # Max steps to train for (each step is a batch of data)<br>  # Overrides num_train_epochs, if not -1<br>  max_steps=max_steps,<br><br>  # Batch size for training<br>  per_device_train_batch_size=1,<br><br>  # Directory to save model checkpoints<br>  output_dir=output_dir,<br><br>  # Other arguments<br>  overwrite_output_dir=False, # Overwrite the content of the output directory<br>  disable_tqdm=False, # Disable progress bars<br>  eval_steps=120, # Number of update steps between two evaluations<br>  save_steps=120, # After # steps model is saved<br>  warmup_steps=1, # Number of warmup steps for learning rate scheduler<br>  per_device_eval_batch_size=1, # Batch size for evaluation<br>  evaluation_strategy=&quot;steps&quot;,<br>  logging_strategy=&quot;steps&quot;,<br>  logging_steps=1,<br>  optim=&quot;adafactor&quot;,<br>  gradient_accumulation_steps = 4,<br>  gradient_checkpointing=False,<br><br>  # Parameters for early stopping<br>  load_best_model_at_end=True,<br>  save_total_limit=1,<br>  metric_for_best_model=&quot;eval_loss&quot;,<br>  greater_is_better=False<br>)</pre><p>Once the training arguments are set, the model’s floating-point operations per second (FLOPs) are calculated based on the input size and gradient accumulation steps, providing an insight into the computational load. Memory usage is also assessed, giving an estimate of the model’s footprint in gigabytes. After these calculations, a Trainer is initialised with the base model, FLOPs, total training steps, and the previously prepared datasets for training and evaluation. This setup ensures that the training process is optimised and allows for monitoring resource utilisation, which is crucial for handling large-scale model fine-tuning effectively. At the end of the training model, we get the fine-tuned model which can be deployed on the cloud to serve users as an API.</p><pre># model parameters<br>model_flops = (<br>  base_model.floating_point_ops(<br>    {<br>       &quot;input_ids&quot;: torch.zeros(<br>           (1, training_config[&quot;model&quot;][&quot;max_length&quot;])<br>      )<br>    }<br>  )<br>  * training_args.gradient_accumulation_steps<br>)<br><br>print(base_model)<br>print(&quot;Memory footprint&quot;, base_model.get_memory_footprint() / 1e9, &quot;GB&quot;)<br>print(&quot;Flops&quot;, model_flops / 1e9, &quot;GFLOPs&quot;)<br><br># Set up a trainer<br>trainer = Trainer(<br>    model=base_model,<br>    model_flops=model_flops,<br>    total_steps=max_steps,<br>    args=training_args,<br>    train_dataset=train_dataset,<br>    eval_dataset=test_dataset,<br>)</pre><h3>Conclusion</h3><p>In conclusion, this article covers an in-depth guide to understanding the need for fine-tuning large language models using the Lamini platform. This article gives a comprehensive overview of why we need to fine-tune the model for custom datasets and business use cases as well as the benefits of using Lamini tools. We also saw a step-by-step guide to fine-tune the model using a custom dataset and LLM with tools from Lamini. Let’s summarise key takeaways from the blog.</p><h4>Key takeaways:</h4><ol><li>Learning need of fine-tuning models against prompt engineering and retrieval augmented generation methods.</li><li>Utilisation of platforms like Lamini for easy-to-use hardware setup and deployment techniques for fine-tuned models to serve the user requirements</li><li>We are preparing data for the fine-tuning task and setting up a pipeline to train a base model using a wide range of hyperparameters.</li></ol><h4>FAQs:</h4><p><strong>Q1: How to fine-tune my models?</strong></p><p><strong>A:</strong> The fine-tuning process starts with context-specific requirement understanding, dataset preparation, tokenization, and setting up training setups like hardware requirements, training configs, training arguments and eventually running a training job for model development.</p><p><strong>Q2: What does fine-tuning of LLMs mean?</strong></p><p><strong>A: </strong>Fine-tuning a large language model means training a base model on a specific custom dataset to generate accurate and context-relevant outputs for specific queries as per the use case.</p><p><strong>Q3: What is Lamini in LLM fine-tuning?</strong></p><p><strong>A: </strong>Lamini provides an integrated language model fine-tuning, inference and GPU setup for the seamless development of large language models efficiently and cost-effectively.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dde12f51d82" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>